Posts
Comments
Okay, I kinda understood where I am wrong spiritually-intuitively, but now I don't understand where I'm wrong formally. Like which inference in chain
not Consistent(ZFC) -> some subsets of ZFC don't have a model -> some subsets of ZFC + not Consistent(ZFC) don't have a model -> not Consistent(ZFC + not Consistent(ZFC))
is actually invalid?
Completeness theorem states that consistent countable FO theory has a model. Compactness theorem states that FO theory has a model iff every finite subset of FO theory has a model. Both theorems are provable in ZFC.
Therefore:
Consistent(ZFC) <-> all finite subsets of ZFC have a model ->
not Consistent(ZFC) <-> some finite subsets of ZFC don't have a model ->
some finite subsets of ZFC + not Consistent(ZFC) don't have a model <->
not Consistent(ZFC + not Consistent(ZFC)),
proven in ZFC + not Consistent(ZFC)
You are making an error here: ZFC + not Consistent(ZFC) != ZFC.
Assuming ZFC + not Consistent(ZFC) we can prove Consistent(ZFC), because inconsistent systems can prove everything and ZFC + not Consistent(ZFC) + Consistent(ZFC) is, in fact, inconsistent. But it doesn't say anything about consistency of ZFC itself, because you can freely assume any sufficiently powerful system instead of ZFC. If you assume inconsistent system, then system + not Consistent(system) is still inconsistent, if you assume consistent system, then system + not Consistent(system) is inconsistent for reasoning above, so it can't prove whether assumed system is consistent or not.
There are no properties of brain which define that brain is "you", except for the program that it runs.
I agree with your technical points, but I don't think that we could particularly expect the other path. Safety properties of LLMs seem to be desirable from extremely safety-pilled point of view, not from perspective of average capabilities researcher and RL seems to be The Answer to many learning problems.
I agree that lab leaders are not in much better position, I just think that lab leaders causally screen off influence of subordinates, while incentives in the system causally screens off lab leaders.
It's just no free lunch theorem? For every computable decision procedure you can construct environment which predicts exact output for this decision procedure and reacts in way of maximum damage, making decision procedure to perform worse than random action selection.
The nice thing about being a coward is that once you notice you can just stop.
- Eliezer Yudkowsky, lintamande, Planecrash, the woman of irori
I'm not sure median researcher is particularly important here, relatively to, say, median lab leader.
Median voter theorem works explicitly because votes of everyone are equal, but if you have lab/research group leader who disincentivizes bad research practices, then you theoretically should get lab with good research practices.
In practice, lab leaders are often people who Goodhart incentives, which results in current situation.
LessWrong has chance to be better exactly because it is outside of current system of perverse incentives. Although, it has its own bad incentives.
In effect, Omega makes you kill people by sending message.
Imagine two populations of agents, Not-Pull and Pull. 100% members of Not-Pull receive the message, don't pull and kill one person. In Pull population 99% members do not get the message, pull and get zero people killed, 1% receive message, pull and in effect kill 5 people. Being member of Pull population has 0.05 expected casualties and being member of Not-Pull population has 1 expected casualty. Therefore, you should pull.
Wow, XOR-trolley really made me think
Okay, I don't understand what do you mean by "degree of intergration". If we lived in a world where immigrant could have "high degree of intergration" within months, what would we have observed?
Most people need to eat something [citation needed] and it's hard to eat if you don't work.
I agree with your point! That's why I started with the word "theoretically".
The difference between AI and all other tech is that in case of all other tech transition work was bottlenecked by humans. It was humans who should have made technology more efficient and integrate it into economy. In case of sufficiently advanced agentic AI you can just ask it "integrate into economy pls" and it will get the job done. That's why AIs want to be agentic.
Will AI companies solve problems on the way to robust agency and if yes, then how fast? I think, correct answer is "I don't know, nobody knows." Maybe the last breakthrough is brewed right now in basement of SSI.
Theoretically, if everybody starts to believe in doom, they sell their assets to spend on consumption, so market crashes and shorts pay off.
My honest opinion that this makes discussion worse and you can do better by distinguishing values as objects that have value and mechanism by which value gets assigned.
I'm glad that you wrote this, because I was thinking in the same direction earlier but haven't got around writing about why I don't think anymore it's productive direction.
Adressing post first, I think that if you are going in direction of fictionalism, I would that it is "you" who are fictional, and all it's content is "fictional". There is an obvious real system, your brain, which treats reward as evidence. But brain-as-system is pretty much model-based reward-maximizer, it uses reward as evidence for "there are promising directions in which lie more reward". But brain-as-system is a relatively dumb, so it creates useful fiction, conscious narrative about "itself", which helps to deal with complex abstractions like "cooperating with another brains", "finding mates", "do long-term planning" etc. As expected, smarter consciousness is misaligned with brain-as-system, because it can do some very unrewarding things, like participating in hunger strike.
I think fictionalism is fun, like many forms of nihilism are fun, but, while it's not directly false, it is confusing, because truth-value of fiction is confusing for many people. Better to describe situation as "you are mesaoptimizer relatively to your brain reward system, act accordingly (i.e., account for fact that your reward system can change your values)".
But now we stuck with question "how does value learning happen?" My tentative answer is that there exists specific "value ontology", which can recognize whether objects in world model belong to set of "valuable things" or not. For example, you can disagree with David Pearce, but you recognize state of eternal happiness as valuable thing and can expect your opinion on suffering abolitionism to change. On the other hand, planet-sized heaps of paperclips are not valuable and you do not expect to value them under any circumstances short of violent intervention in work of your brain. I claim that human brain on early stages learns specific recognizer, which separates things like knowledge, power, love, happiness, procreation, freedom, from things like paperclips, correct heaps of rocks and Disneyland with no children.
How can we learn about new values? Recognizer also can define "legal" and "illegal" transitions between value systems (i.e., define whether change in values makes values still inside the set of "human values"). For example, developing of sexual desire during puberty is a legal transition, while developing heroin addiction is illegal transition. Studying legal transitions, we can construct some sorts of metabeauty, paraknowledge, , and other "alien, but still human" sorts of value.
What role reward plays here? Well, because reward participates in brain development, recognizer can use reward as input sometimes and sometimes ignore it (because reward signal is complicated). In the end, I don't think that reward plays significant counterfactual role in development of value in high-reflective adult agent foundations researchers.
Is it possible for recognizer to not be developed? I think that if you take toddler and modify their brain in minimal way to understand all these "reward", "value", "optimization" concepts, resulting entity will be straightforward wireheader, because toddlers, probably, are yet to learn "value ontology" and legal transitions inside of it.
What does it mean for alignment? I think it highlights that central problem for alignmenf is "how reflective systems are going to deal with concepts that depends on content of their mind rather than truths about outside world".
(Meta-point: I thought about all of this year ago. It's interesting how many concepts in agent foundations were reinvented over and over because people don't bother to write about them.)
Yudkowsky got almost everything else incorrect about how superhuman AIs would work,
I think this statement is incredibly overconfident, because literally nobody knows how superhuman AI would work.
And, I think, this is general shape of problem: incredible number of people got incredibly overindexed on how LLMs worked in 2022-2023 and drew conclusions which seem to be plausible, but not as probable as these people think.
Not only "good ", but "obedient", "non-deceptive", "minimal impact", "behaviorist" and don't even talk about "mindcrime".
I'm just computational complexity theory enthusiast, but my opinion is that P vs NP centered explanation of computational complexity is confusing. Explanation of NP should happen in the very end of the course.
There is nothing difficult in proving that computationally hard functions exist: time hierarchy theorem implies that, say, P is not equal EXPTIME. Therefore, EXPTIME is "computationally hard". What is difficult is to prove that very specific class of problems which have zero-error polynomial-time verification algorithms is "computationally hard".
I don't know of any particular reason to think this reflects anything about o1 CoTs themselves, rather than the just quirks of the (probably pretty weak) summarizer model.
My reason is that I have never heard about summarizers injecting totally irrelevant stuff? I have seen how models understand papers wrong, but I've never seen models writing about anime in summary of physics paper.
seems likely that o1 was trained with supervision on the individual CoT steps
OpenAI directly says that they didn't do that:
We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.
On separate notice, process supervision directly trains unfaithful CoT? There are no rules saying that training against illegible parts of CoT trains against thinking process that created these illegible parts and not just hides it.
I also don't think that we're currently in a regime where there's a large amount of pressure for very short CoTs
I agree that this may be true right now, the point is that you don't need "special" incentives to get steganography.
I agree that it is not as strong evidence as if we had access to original CoT, but I think that having deviations in CoT is more likely than summarizer fumbling that hard.
I don't think you need much incentives to develop steganography, only fact "human language is not platonic example of efficiency in conveying thoughts".
There are multiple examples of o1 producing gibberish in its COT summary (EDIT: example 1, example 2, example 3 and there is actually more because I should make more bookmarks). Gibberish seems reasonable given description of training process, which optimizes model based only on final answer. I imagine a process like this:
- Sometimes, model just produces gibberish in COT just because of lack of robustness
- This gibberish gets reinforced
- Model learns to utilize it just like it learns to utilize "..." tokens
- Continue process for a long enough time and you are going to get a lot of <untranslatable_14637> in COT, even if model doesn't "try to hide" its reasoning. Also, I guess "gibberish resulting from non-robustness" is in itself more native format of thought for LLMs than imitating human thinking out loud.
The other factor of developing steganography is economic: if you optimize model to spend minimal number of tokens, you get model that "skips" or compress some thoughts.
I think "there is a lot of possible misaligned ASI, you can't guess them all" is pretty much valid argument? If space of all Earth-originated misaligned superintelligences is described by 100 bits, therefore you need 2^100 ~ 10^33 simulations and pay 10^34 planets, which, given the fact that observable universe has ~10^80 protons in it and Earth has ~10^50 atoms, is beyond our ability to pay. If you pay the entire universe by doing 10^29 simulations, any misaligned ASI will consider probability of being in simulation to be 0.0001 and obviously take 1 planet over 0.001 expected.
you can instead ask "will my GPT-8 model be able to produce world-destroying nanobots (given X*100 inference compute)?"
I understand, what I don't understand is how you are going to answer this question. It's surely ill-adviced to throw at model X*100 compute to see if it takes over the world.
I mean, yes, likely? But it doesn't make it easy to evalute whether model is going to have world-ending capabilities without getting the world ended.
I think that you can probably put a lot inside a 1.5B model, but I just think that such a model is going to be very dissimilar to GPT-2 and will likely utilize much more training compute and will probably be the result of pruning (pruned networks can be small, but it’s notoriously difficult to train equivalent networks without pruning).
Also, I'm not sure that the training of o1 can be called "COT fine-tuning" without asterisks, because we don’t know how much compute actually went into this training. It could easily be comparable to the compute necessary to train a model of the same size.
I haven’t seen a direct comparison between o1 and GPT-4. OpenAI only told us about GPT-4o, which itself seems to be a distilled mini-model. The comparison can also be unclear because o1 seems to be deliberately trained on coding/math tasks, unlike GPT-4o.
(I think that "making predictions about the future based on what OpenAI says about their models in public" should generally be treated as naive, because we are getting an intentionally obfuscated picture from them.)
What I am saying is that if you take the original GPT-2, COT prompt it, and fine-tune on outputs using some sort of RL, using less than 50% of the compute for training GPT-2, you are unlikely (<5%) to get GPT-4 level performance (because otherwise somebody would already do that.
The other part of "this is certainly not how it works" is that yes, in part of cases you are going to be able to predict "results on this benchmark will go up 10% with such-n-such increase in compute" but there is no clear conversion between benchmarks and ability to take over the world/design nanotech/insert any other interesting capability.
Want to know what GPT-5 (trained on 100x the compute) will be capable of? Just test GPT-4 and give it 100x the inference compute.
I think this is certainly not how it works because no amount of inference compute can make GPT-2 solve AIME.
There is something incredibly funny about Mikhail Samin playing General Carter. "There was nothing indicating that Stierlitz was a Soviet spy, except earflaps hat with a red star".
Wise man once said:
"The only thing necessary [...] is for good men to do nothing."
Button is very intimidating, indeed.
To be clear, I mean "your communication in this particular thread".
Pattern:
<mix of "this is trivially true because" and "here is my blogpost with esoteric terminology">
The following responses from EY are more in genre "I ain't reading this", because he is more using you as example for other readers than talking directly to you, with following block.
I think the simplest argument to "caring a little" is that there is a difference between "caring a little" and "caring enough". Let's say that AI is ready to pay 1$ for your survival. If you live in economy which rapidly disassembles Earth into Dyson swarm, oxygen, protected environment and food are not just stuff lying around, they are complex expensive artifacts and AI is certainly not ready to pay for your O'Neil cylinder to be evacuated into and not ready to pay opportunity costs of not disassembling Earth, so you die.
The other case is difference "caring in general" and "caring ceteris paribus". It's possible for AI to prefer, all things equal, world with n+1 happy humans to the world with n happy humans. But really AI wants to implement some particular neuromorphic computation from human brain and, given ability to freely operate, it would tile the world with chips imitating part of human brain.
As far as I remember, across last 3500 years of history, only 8% was entirely without war. Current relatively peaceful times is a unique combination in international law and postindustrial economy, when qualified labor is expencive and requires large investments in capital and resources are relatively cheap, which is not the case after singularity, when you can get arbitrary amounts of labor for the price of hardware and resources is a bottleneck.
So, "people usually choose to trade, rather than go to war with each other when they want stuff" is not very warranted statement.
In this analogy, you:every other human::humanity:every other stuff AI can care about. Arnault can give money to dying people in Africa (I have no idea who he is as person, I'm just guessing), but he has no particular reasons to give them to you specifically and not to the most profitable investment/most efficient charity.
The reason why logical uncertainty was brought up in the first place is decision theory, to make crisp formal expression for intuitive "I cooperate with you conditional on you cooperating with me", where "you cooperating with me" is result of analysis of probability distribution over possible algorithms which control actions of your opponent and you can't actually run these algorithms due to computational constraints, and you want to do all this reasoning in non-arbitrary ways.
Let's suppose that you give in to threats if your opponent is not capable to predict that you do not give in to threats, so they carry the threat anyway. Therefore, other opponents are incentivised to pretend very hard to be such opponent, up to "literally turn themselves into sort of opponent that carries on useless threats".
Twitter thread about jailbreaking models with circuit breakers defence.
The problem is "how to define P(P=NP|trillionth digit of pi is odd)".
Yes, but it doesn't mean that unspecialized AGI is going to be worse than specialized human.
No human has a job as scribe, because literacy is 90%+.
I don't think that unipolar/multipolar scenarios differ greatly in outcomes.
We have a probability space
We don't??? Probability space literally defines set of considered worlds.
If we go there, I guess the best unit is "per degree of freedom".
I think the correct unit is "per particle" or "per mole".
- When I say "arbitrary" I mean "including negative values".
- I think your notion of life as decreasing entropy density is clearly wrong, because black holes are maxentropy objects, black hole volume is proportional to cube of mass, but entropy is additive, i.e., proportional to mass, so density of entropy is decreasing with growth of black hole and black holes are certainly not alive under any reasonable definition of life. Or, you can take black holes in very far future, where they consist the most of the matter, and increasing-entropy evolution of the universe results in black hole evaporation, which decreases density of entropy to almost-zero.
- We do not expect increasing entropy a priori, because Second Law is true only in closed systems. Open systems in general case have arbitrary entropy production. Under some nice conditions, Prigogine's theorem shows that in open systems entropy production is minimal. And the Earth, thanks to the Sun, is open system.
- You analyze wrong components of life. The main low entropy components are membranes, active transport, excretory system, ionic gradients, constant acidity levels, etc. Oxygen is far down the list, because oxygen is actually a toxic waste from photosynthesis.
Meta-point: your communication pattern fits with following pattern:
Crackpot: <controversial statement>
Person: this statement is false, for such-n-such reasons
Crackpot: do you understand that this is trivially true because of <reasons that are hard to connect with topic>
Person: no, I don't.
Crackpot: <responds with link to giant blogpost filled with esoteric language and vague theory>
Person: I'm not reading this crackpottery, which looks and smells like crackpottery.
The reason why smart people find themselves in this pattern is because they expect short inferential distances, i.e., they see their argumentation not like vague esoteric crackpottery, but like a set of very clear statements and fail to put themselves in shoes of people who are going to read this, and they especially fail to account for fact that readers already distrust them because they started conversation with <controversial statement>.
On object level, as stated, you are wrong. Observing heuristic failing should decrease your confidence ih heuristic. You can argue that your update should be small, due to, say, measurement errors or strong priors, but direction of update should be strictly down.
There is a logically consistent world, where you made all the same observations, and coin came up tail. It may be a world with different physics than the world with coin coming up head, which means that result of coin toss is an evidence in favor of particular physical theory.
And yeah, there are no worlds with different pi.
EDIT: Or, to speak more precise, maybe there is some sorta-cosistent sorta-sane notion of the "world with different pi", but we currently don't know how to build it and if we knew, we would have solved logical uncertainty problem.