AGI Ruin: A List of Lethalities

That requires, not the ability to read this document and nod along with it, but the ability to spontaneously write it from scratch without anybody else prompting you; that is what makes somebody a peer of its author. It's guaranteed that some of my analysis is mistaken, though not necessarily in a hopeful direction. The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I published it, which nobody apparently did, despite my having had other things to do than write this up for the last five years or so. Some of that silence may, possibly, optimistically, be due to nobody else in this field having the ability to write things comprehensibly - such that somebody out there had the knowledge to write all of this themselves, if they could only have written it up, but they couldn't write, so didn't try. I'm not particularly hopeful of this turning out to be true in real life, but I suppose it's one possible place for a "positive model violation" (miracle). The fact that, twenty-one years into my entering this death game, seven years into other EAs noticing the death game, and two years into even normies starting to notice the death game, it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that. I knew I did not actually have the physical stamina to be a star researcher, I tried really really hard to replace myself before my health deteriorated further, and yet here I am writing this. That's not what surviving worlds look like.

To say that somebody else should have written up this list before is such a ridiculously unfair criticism. This is an assorted list of some thoughts which are relevant to AI alignment—just by the combinatorics of how many such thoughts there are and how many you chose to include in this list, of course nobody will have written up something like it before. Every time anybody writes up any overview of AI safety, they have to make tradeoffs between what they want to include and what they don't want to include that will inevitably leave some things off and include some things depending on what the author personally believes is most important/relevant to say—ensuring that all such introductions will always inevitably cover somewhat different material. Furthermore, many of these are responses to particular bad alignment plans, of which there are far too many to expect anyone to have previously written up specific responses to.

Nevertheless, I am confident that every core technical idea in this post has been written about before by either me, Paul Christiano, Richard Ngo, or Scott Garrabrant. Certainly, they have been written up in different ways than how Eliezer describes them, but all of the core ideas are there. Let's go through the list:

(1, 2, 4, 15) AGI safety from first principles [? · GW]

(3) This is a common concept, see e.g. Homogeneity vs. heterogeneity in AI takeoff scenarios [AF · GW] (“Homogeneity makes the alignment of the first advanced AI system absolutely critical (in a similar way to fast/discontinuous takeoff without the takeoff actually needing to be fast/discontinuous), since whether the first AI is aligned or not is highly likely tano determine/be highly correlated with whether all future AIs built after that point are aligned as well.”).

(4) This is just answering a particular bad plan.

(5, 6, 7) This is just the concept of competitiveness, see e.g. An overview of 11 proposals for building safe advanced AI [AF · GW].

(8) Risks from Learned Optimization in Advanced Machine Learning Systems: Conditions for Mesa-Optimization [? · GW]

(9) Worst-case guarantees (Revisted)

(10, 13, 14) Risks from Learned Optimization in Advanced Machine Learning Systems: Deceptive Alignment: Distributional shift and deceptive alignment [? · GW]

(11) Another specific bad plan.

(12, 35) Robustness to Scale [AF · GW]

(16, 17, 19) Risks from Learned Optimization in Advanced Machine Learning Systems [? · GW]

(18, 20) ARC's first technical report: Eliciting Latent Knowledge [AF · GW]

(21, 22) 2-D Robustness [AF · GW] for the concept, Risks from Learned Optimization in Advanced Machine Learning Systems [? · GW] for why it occurs.

(23, 24.2) Towards a mechanistic understanding of corrigibility [AF · GW]

(24.1) Risks from Learned Optimization in Advanced Machine Learning Systems: Deceptive Alignment: Internalization or deception after extensive training [? · GW]

(25, 26, 27, 29, 31) Acceptability Verification: A Research Agenda

(28) Relaxed adversarial training for inner alignment [AF · GW]

(30, 33) Chris Olah’s views on AGI safety: What if interpretability breaks down as AI gets more powerful? [AF · GW]

(32) An overview of 11 proposals for building safe advanced AI [AF · GW]

(34) Response to a particular bad plan.

(36) Outer alignment and imitative amplification: The case for imitative amplification [AF · GW]

To spot check the above list, I generated the following three random numbers from 1 - 36 after I wrote the list: 32, 34, 15. Since 34 corresponds to a particular bad plan, I then generated another to replace it: 14. Let's spot check those three—14, 15, 32—more carefully.

(14) Eliezer claims that “Some problems, like 'the AGI has an option that (looks to it like) it could successfully kill and replace the programmers to fully optimize over its environment', seem like their natural order of appearance could be that they first appear only in fully dangerous domains.” In Risks from Learned Optimization in Advanced Machine Learning Systems, we say very directly:

In current AI systems, a small amount of distributional shift between training and deployment need not be problematic: so long as the difference is small enough in the task-relevant areas, the training distribution does not need to perfectly reflect the deployment distribution. However, this may not be the case for a deceptively aligned mesa-optimizer. If a deceptively aligned mesa-optimizer is sufficiently advanced, it may detect very subtle distributional shifts for the purpose of inferring when the threat of modification has ceased.

[...] Some examples of differences that a mesa-optimizer might be able to detect include:

• [...]
• The presence or absence of good opportunities for the mesa-optimizer to defect against its programmers.

(15) Eliezer says “Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously.” Richard says:

If AI development proceeds very quickly, then our ability to react appropriately will be much lower. In particular, we should be interested in how long it will take for AGIs to proceed from human-level intelligence to superintelligence, which we’ll call the takeoff period. The history of systems like AlphaStar, AlphaGo and OpenAI Five provides some evidence that this takeoff period will be short: after a long development period, each of them was able to improve rapidly from top amateur level to superhuman performance. A similar phenomenon occurred during human evolution, where it only took us a few million years to become much more intelligent than chimpanzees. In our case one of the key factors was scaling up our brain hardware - which, as I have already discussed, will be much easier for AGIs than it was for humans.

While the question of what returns we will get from scaling up hardware and training time is an important one, in the long term the most important question is what returns we should expect from scaling up the intelligence of scientific researchers - because eventually AGIs themselves will be doing the vast majority of research in AI and related fields (in a process I’ve been calling recursive improvement). In particular, within the range of intelligence we’re interested in, will a given increase δ in the intelligence of an AGI increase the intelligence of the best successor that AGI can develop by more than or less than δ? If more, then recursive improvement will eventually speed up the rate of progress in AI research dramatically.

Note: for this one, I originally had the link above point to AGI safety from first principles: Superintelligence [AF · GW] specifically, but changed it to point to the whole sequence after I realized during the spot-checking that Richard mostly talks about this in the Control [AF · GW] section.

(32) Eliezer says “This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents, which are only impoverished subsystems of human thoughts; unless that system is powerful enough to contain inner intelligences figuring out the humans, and at that point it is no longer really working as imitative human thought.” In An overview of 11 proposals for building safe advanced AI, I say:

if RL is necessary to do anything powerful and simple language modeling is insufficient, then whether or not language modeling is easier is a moot point. Whether RL is really necessary seems likely to depend on the extent to which it is necessary to explicitly train agents—which is very much an open question. Furthermore, even if agency is required, it could potentially be obtained just by imitating an actor such as a human that already has it rather than training it directly via RL.

and

the training competitiveness of imitative amplification is likely to depend on whether pure imitation can be turned into a rich enough reward signal to facilitate highly sample-efficient learning. In my opinion, it seems likely that human language imitation (where language includes embedded images, videos, etc.) combined with techniques to improve sample efficiency will be competitive at some tasks—namely highly-cognitive tasks such as general-purpose decision-making—but not at others, such as fine motor control. If that’s true, then as long as the primary economic use cases for AGI fall into the highly-cognitive category, imitative amplification should be training competitive. For a more detailed analysis of this question, see “Outer alignment and imitative amplification [AF · GW].”

I'm sorry to hear that your health is poor and you feel that this is all on you. Maybe you're right about the likelihood of doom, and even if I knew you were, I'd be sorry that it troubles you this way.

I think you've done an amazing job of building the AI safety field and now, even when the field has a degree of momentum of its own, it does seem to be less focused on doom than it should be, and I think you continuing to push people to focus on doom is valuable.

I don't think its easy to get people to take weird ideas seriously. I've had many experiences where I've had ideas about how people should change their approach to a project that weren't particularly far out and (in my view) were right for very straightforward reasons, and yet for the most part I was ignored altogether. What you've accomplished in building the AI safety field is amazing because AI doom ideas seemed really crazy when you started talking about them.

Nevertheless, I think some of the things you've said in this post are counterproductive. Most of the post is good, but insulting people who might contribute to solving the problem is not, nor is demanding that people acknowledge that you are smarter than they are. I'm not telling you that people don't deserve to be insulted, nor that you have no right to consider yourself smarter than them - I'm telling you that you shouldn't say it in public.

My concrete suggestion is this: if you are criticising or otherwise passing pessimistic judgement on people or a group of people
 - Give more details about what it is they've done to merit this criticism ("pretend plan that can fool EAs too 'modest' to trust their own judgments" - what are modest EAs actually doing that you think is wrong? Paying not enough attention to AI doom?)
 - Avoid talking about yourself ("So most organizations don't have plans, because I haven't taken the time to personally yell at them")

Many people are proud, including me. If working in AI safety means I have to be regularly reminded that the fact that I didn't go into the field sooner will be held as a mark against me, then that is a reason for me not to do it. Maybe not a decisive reason, but it is a reason. If working in AI safety means that you are going to ask me to publicly acknowledge that you're smarter than me, that's a reason for me not to do it. Maybe not decisive, but it's a reason. I think there might be others who feel similarly.

If you want people to accept what you're saying, it helps let people change their minds without embarrassing them. There are plenty of other things to do - many of which, as I've said, you seem to be much better at doing than me - but this one is important too. I wonder if you might say something like "anyone turned off by these comments can't be of any value to the project". If you think that - I just don't. There are many, many smart people with dumb motivations, and many of them can do valuable work if they can be motivated to do it. This includes thinking deeply about things they were previously motivated not to think about.

You are a key, maybe the key, person in the AI safety field. What you say is attended to people in, around and even disconnected from the field. I don't think you can reasonably claim that you shouldn't be this important to the field. I think you should take this fact seriously, and that means exercising discipline in the things you say.

I say all this because I think that a decent amount of EA/AI safety seems to neglect AI doom an unreasonable amount, and certainly the field of AI in general neglects it. I find statements of the type I pointed out above off-putting, and I suspect I'm not alone.

I'd have more hope - not significant hope, but more hope - in separating the concerns of (a) credibly promising to pay big money retrospectively for good work to anyone who produces it, and (b) venturing prospective payments to somebody who is predicted to maybe produce good work later.

I desperately want to make this ecosystem exist, either as part of Manifold Markets, or separately. Some people call it "impact certificates" or "retroactive public goods funding"; I call it "equity for public goods", or "Manifund" in the specific case.

If anyone is interested in:

a) Being a retroactive funder for good work (aka bounties, prizes)

b) Getting funding through this kind of mechanism (aka income share agreements, angel investment)

c) Working on this project full time (full-stack web dev, ops, community management)

Please get in touch! Reply here, or message austin@manifold.markets~

It's as good as time as any to re-iterate my reasons for disagreeing with what I see as the Yudkowskian view of future AI. What follows isn't intended as a rebuttal of any specific argument in this essay, but merely a pointer that I'm providing for readers, that may help explain why some people might disagree with the conclusion and reasoning contained within.

I'll provide my cruxes point-by-point,

• I think raw intelligence, while important, is not the primary factor that explains why humanity-as-a-species is much more powerful than chimpanzees-as-a-species. Notably, humans were once much less powerful, in our hunter-gatherer days, but over time, through the gradual process of accumulating technology, knowledge, and culture, humans now possess vast productive capacities that far outstrip our ancient powers.
  
  Similarly, our ability to coordinate through language also plays a huge role in explaining our power compared to other animals. But, on a first approximation, other animals can't coordinate at all, making this distinction much less impressive. The first AGIs we construct will be born into a culture already capable of coordinating, and sharing knowledge, making the potential power difference between AGI and humans relatively much smaller than between humans and other animals, at least at first.
  
  Consequently, the first slightly smarter-than-human agent will probably not be able to leverage its raw intelligence to unilaterally take over the world, for pretty much the same reason that an individual human would not be able to unilaterally take over a band of chimps, in the state of nature, despite the intelligence advantage of the human.
• There's a large range of human intelligence, such that it makes sense to talk about AI slowly going from 50th percentile to 99.999th percentile on pretty much any important general intellectual task, rather than AI suddenly jumping to superhuman levels after a single major insight. In cases where progress in performance does happen rapidly, the usual reason is that there wasn't much effort previously being put into getting better at the task.
  
  The case of AlphaGo is instructive here: improving the SOTA on Go bots is not very profitable. We should expect, therefore, that there will be relatively few resources being put into that task, compared to the overall size of the economy. However, if a single rich company, like Google, at some point does decide to invest considerable resources into improving Go performance, then we could easily observe a discontinuity in progress. Yet, this discontinuity in output merely reflects a discontinuity in inputs, not a discontinuity as a response to small changes in those inputs, as is usually a prerequisite for foom in theoretical models.
• Hardware progress and experimentation are much stronger drivers of AI progress than novel theoretical insights. The most impressive insights, like backpropagation and transformers, are probably in our past. And as the field becomes more mature, it will likely become even harder to make important theoretical discoveries.
  
  These points make the primacy of recursive self-improvement, and as a consequence, unipolarity in AI takeoff [LW · GW], less likely in the future development of AI. That's because hardware progress and AI experimentation are, for the most part, society-wide inputs, which can be contributed by a wide variety of actors, don't exhibit strong feedback loops on an individual level, and more-or-less have smooth responses to small changes in their inputs. Absent some way of making AI far better via a small theoretical tweak, it seems that we should expect smooth, gradual progress by default, even if overall economic growth becomes very high after the invention of AGI.
• [Update (June 2023): While I think these considerations are still important, I think the picture I painted in this section was misleading. I wrote about my views of AI services here [LW · GW].] There are strong pressures -- including the principle of comparative advantage, diseconomies of scale, and gains from specialization -- that incentivize making economic services narrow and modular, rather than general and all-encompassing. Illustratively, a large factory where each worker specializes in their particular role will be much more productive than a factory in which each worker is trained to be a generalist, even though no one understands any particular component of the production process very well.
  
  What is true in human economics will apply to AI services as well. This implies we should expect something like Eric Drexler's AI perspective, which emphasizes economic production across many agents who trade and produce narrow services, as opposed to monolithic agents that command and control.
• Having seen undeniable, large economic effects from AI, policymakers will eventually realize that AGI is important, and will launch massive efforts to regulate it. The current lack of concern almost certainly reflects the fact that powerful AI hasn't arrived yet.
  
  There's a long history of people regulating industries after disasters -- like nuclear energy -- and, given the above theses, it seems likely that there will be at least a few "warning shots" which will provide a trigger for companies and governments to crack down and invest heavily into making things go the way they want. 
  
  (Note that this does not imply any sort of optimism about the effects of these regulations, only that they will exist and will have a large effect on the trajectory of AI)
• The effect of the above points is not to provide us uniform optimism about AI safety, and our collective future. It is true that, if we accept the previous theses, then many of the points in Eliezer's list of AI lethalities become far less plausible. But, equally, one could view these theses pessimistically, by thinking that they imply the trajectory of future AI is much harder to intervene on, and do anything about, relative to the Yudkowskian view.

First, some remarks about the meta-level:

The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I published it, which nobody apparently did, despite my having had other things to do than write this up for the last five years or so. Some of that silence may, possibly, optimistically, be due to nobody else in this field having the ability to write things comprehensibly - such that somebody out there had the knowledge to write all of this themselves, if they could only have written it up, but they couldn't write, so didn't try. I'm not particularly hopeful of this turning out to be true in real life, but I suppose it's one possible place for a "positive model violation" (miracle). The fact that, twenty-one years into my entering this death game, seven years into other EAs noticing the death game, and two years into even normies starting to notice the death game, it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that.

Actually, I don't feel like I learned that much reading this list, compared to what I already knew. [EDIT: To be clear, this knowledge owes a lot to prior inputs from Yudkowsky and the surrounding intellectual circle, I am making no claim that I would derive it all independently in a world in which Yudkowsky and MIRI didn't exit.] To be sure, it didn't feel like a waste of time, and I liked some particular framings (e.g. in A.4 separating the difficulty into "unlimited time but 1 try" and "limited time with retries"), but I think I could write something that would be similar (in terms of content; it would be very likely much worse in terms of writing quality).

One reason I didn't write such a list is, I don't have the ability to write things comprehensibly. Empirically, everything of substance that I write is notoriously difficult for readers to understand. Another reason is, at some point I decided to write top-level posts only when I have substantial novel mathematical results, with rare exceptions. This is in part because I feel like the field has too much hand-waving and philosophizing and too little hard math (which rhymes with C.38). In part it is because, even if people can't understand the informal component of my reasoning, they can at least understand there is math here and, given sufficient background, follow the definitions/theorems/proofs (although tbh few people follow).

There's no plan

Actually, I do have a plan. It doesn't have an amazing probability of success (my biggest concerns are (i) not enough remaining time and (ii) even if the theory is ready in time, the implementation can be bungled, in particular for reasons of operational adequacy [LW · GW]), but it is also not practically useless. The last time I tried to communicate it was 4 years ago [LW · GW], since which time it obviously evolved. Maybe it's about time to make another attempt, although I'm wary of spending a lot of effort on something which few people will understand.

Now, some technical remarks:

Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again: the first semi-outer-aligned solutions found, in the search ordering of a real-world bounded optimization process, are not inner-aligned solutions.

This is true, but it is notable that deep learning is not equivalent to evolution, and the differences are important. Consider for example a system that is designed to separately (i) learn a generative model of the environment and (ii) search for plans effective on this model (model-based RL). Then, module ii doesn't inherently have the problem where the solution only optimizes the correct thing in the training environment. Because, this module is not bounded by available training data, but only by compute. The question is then, to 1st approximation, whether module i is able to correctly generalize from the training data (obviously there are theoretical bounds on how good such this generalization can be; but we want this generalization to be at least as good as human ability and without dangerous biases). I do not think current systems do such generalization correctly, although they do seem to have some ingredients right, in particular Occam's razor / simplicity bias. But we can imagine some algorithm that does.

...on the current optimization paradigm there is no general idea of how to get particular inner properties into a system, or verify that they're there, rather than just observable outer ones you can run a loss function over.

Also true, but there is nuance. The key problem is that we don't know why deep learning works, or more specifically w.r.t. which prior does it satisfy good generalization bounds. If we knew what this prior is, then we could predict some inner properties. For example, if you know your algorithm follows Occam's razor, for a reasonable formalization of "Occam's razor", and you trained it on the sun setting every day for a million days, then you can predict that the algorithm will not confidently predict the sun is going to fail to to set on any given future day. Moreover, our not knowing such generalization bounds for deep learning is a fact about our present state of mathematical ignorance, not a fact about the algorithms themselves.

...there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment.

It is true that (AFAIK) nothing like this was accomplished in practice, but the distance to that might not be too great. For example, I can imagine training an ANN to implement a POMDP which simultaneously successfully predicts the environment and complies with some "ontological hypothesis" about how the environment needs to be structured in order for the-things-we-want-to-point-at to be well-defined (technically, this POMDP needs to be a refinement of some infra-POMPD [LW · GW] that represents the ontological hypothesis).

The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our Real Actual Values is far out of reach for our FIRST TRY at AGI. Yes I mean specifically that the dataset, meta-learning algorithm, and what needs to be learned, is far out of reach for our first try. It's not just non-hand-codable, it is unteachable on-the-first-try because the thing you are trying to teach is too weird and complicated.

There is a big chunk of what you're trying to teach which not weird and complicated, namely: "find this other agent, and what their values are". Because, "agents" and "values" are natural concepts, for reasons strongly related to "there's a relatively simple core structure that explains why complicated cognitive machines work". Admittedly, my rough proposal (PreDCA [LW(p) · GW(p)]) does have some "weird and complicated" parts because of the acausal attack problem.

Any pivotal act that is not something we can go do right now, will take advantage of the AGI figuring out things about the world we don't know so that it can make plans we wouldn't be able to make ourselves. It knows, at the least, the fact we didn't previously know, that some action sequence results in the world we want. Then humans will not be competent to use their own knowledge of the world to figure out all the results of that action sequence.

This is inaccurate, because $P\ne NP$. It is possible to imagine an AI that provides us with a plan for which we simultaneously (i) can understand why it works and (ii) wouldn't think of it ourselves without thinking for a very long time that we don't have. At the very least, the AI could suggest a way of building a more powerful aligned AI. Of course, in itself this doesn't save us at all: instead of producing such a helpful plan, the AI can produce a deceitful plan instead. Or a plan that literally makes everyone who reads it go insane in very specific ways. Or the AI could just hack the hardware/software system inside which it's embedded to produce a result which counts for it as a high reward but which for us wouldn't look anything like "producing a plan the overseer rates high". But, this direction might [LW(p) · GW(p)] be not completely unsalvageable^[1].

Human thought partially exposes only a partially scrutable outer surface layer. Words only trace our real thoughts. Words are not an AGI-complete data representation in its native style. The underparts of human thought are not exposed for direct imitation learning and can't be put in any dataset. This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents, which are only impoverished subsystems of human thoughts; unless that system is powerful enough to contain inner intelligences figuring out the humans, and at that point it is no longer really working as imitative human thought.

I agree that the process of inferring human thought from the surface artifacts of human thought require powerful non-human thought which is dangerous in itself. But this doesn't necessarily mean that the idea of imitating human though doesn't help at all. We can combine it with techniques such as counterfactual oracles and confidence thresholds to try to make sure the resulting agent is truly only optimizing for accurate imitation (which still leaves problems like attacks from counterfactuals [LW(p) · GW(p)] and non-Cartesian daemons [LW · GW], and also not knowing which features of the data are important to imitate might be a big capability handicap).

On Twitter, Eric Rogstad wrote:

"the thing where it keeps being literally him doing this stuff is quite a bad sign"

I'm a bit confused by this part. Some thoughts on why it seems odd for him (or others) to express that sentiment...

1. I parse the original as, "a collection of EY's thoughts on why safe AI is hard". It's EY's thoughts, why would someone else (other than @robbensinger) write a collection of EY's thoughts?

(And if we generalize to asking why no-one else would write about why safe AI is hard, then what about Superintelligence, or the AI stuff in cold-takes, or ...?)

2. Was there anything new in this doc? It's prob useful to collect all in one place, but we don't ask, "why did no one else write this" for every bit of useful writing out there, right?

Why was it so overwhelmingly important that someone write this summary at this time, that we're at all scratching our heads about why no one else did it?

Copying over my reply to Eric:

My shoulder Eliezer (who I agree with on alignment, and who speaks more bluntly and with less hedging than I normally would) says:

1. The list is true, to the best of my knowledge, and the details actually matter.
   
   Many civilizations try to make a canonical list like this in 1980 and end up dying where they would have lived just because they left off one item, or under-weighted the importance of the last three sentences of another item, or included ten distracting less-important items.
    
2. There are probably not many civilizations that wait until 2022 to make this list, and yet survive.
    
3. It's true that many of the points in the list have been made before. But it's very doomy that they were made by me.
    
4. Nearly all of the field's active alignment research is predicated on a false assumption that's contradicted by one of the items in sections A or B. If the field had recognized everything in A and B sooner, we could have put our recent years of effort into work that might actually help on the mainline, as opposed to work that just hopes a core difficulty won't manifest and has no Plan B for what to do when reality says "no, we're on the mainline".

So the answer to 'Why would someone else write EY's thoughts?' is 'It has nothing to do with an individual's thoughts; it's about civilizations needing a very solid and detailed understanding of what's true on these fronts, or they die'.

Re "(And if we generalize to asking why no-one else would write about why safe AI is hard, then what about Superintelligence, or the AI stuff in cold-takes, or ...?)": 

The point is not 'humanity needs to write a convincing-sounding essay for the thesis Safe AI Is Hard, so we can convince people'. The point is 'humanity needs to actually have a full and detailed understanding of the problem so we can do the engineering work of solving it'.

If it helps, imagine that humanity invents AGI tomorrow and has to actually go align it now. In that situation, you need to actually be able to do all the requisite work, not just be able to write essays that would make a debate judge go 'ah yes, well argued.'

When you imagine having water cooler arguments about the importance of AI alignment work, then sure, it's no big deal if you got a few of the details wrong.

When you imagine actually trying to build aligned AGI the day after tomorrow, I think it comes much more into relief why it matters to get those details right, when the "details" are as core and general as this.

I think that this is a really good exercise that more people should try. Imagine that you're running a project yourself that's developing AGI first, in real life. Imagine that you are personally responsible for figuring out how to make the thing go well. Yes, maybe you're not the perfect person for the job; that's a sunk cost. Just think about what specific things you would actually do to make things go well, what things you'd want to do to prepare 2 years or 6 years in advance, etc.

Try to think your way into near-mode with regard to AGI development, without thereby assuming (without justification) that it must all be very normal just because it's near. Be able to visualize it near-mode and weird/novel. If it helps, start by trying to adopt a near-mode, pragmatic, gearsy mindset toward the weirdest realistic/plausible hypothesis first, then progress to the less-weird possibilities.

I think there's a tendency for EAs and rationalists to instead fall into one of these two mindsets with regard to AGI development, pivotal acts, etc.:

1. Fun Thought Experiment Mindset.  On this mindset, pivotal acts, alignment, etc. are mentally categorized as a sort of game, a cute intellectual puzzle or a neat thing to chat about.
   
   This is mostly a good mindset, IMO, because it makes it easy to freely explore ideas, attend to the logical structure of arguments, brainstorm, focus on gears, etc.
   
   Its main defect is a lack of rigor and a more general lack of drive: because on some level you're not taking the question seriously, you're easily distracted by fun, cute, or elegant lines of thought, and you won't necessarily push yourself to red-team proposals, spontaneously take into account other pragmatic facts/constraints you're aware of from outside the current conversational locus, etc. The whole exercise sort of floats in a fantasy bubble, rather than being a thing people bring their full knowledge, mental firepower, and lucidity/rationality to bear on.
    
2. Serious Respectable Person Mindset.  Alternatively, when EAs and rationalists do start taking this stuff seriously, I think they tend to sort of turn off the natural flexibility, freeness, and object-levelness of their thinking, and let their mind go to a very fearful or far-mode place. The world's gears become a lot less salient, and "Is it OK to say/think that?" becomes a more dominant driver of thought.
   
   Example: In Fun Thought Experiment Mindset, IME, it's easier to think about governments in a reductionist and unsentimental way, as specific messy groups of people with specific institutional dysfunctions, psychological hang-ups, etc. In Serious Respectable Person Mindset, there's more of a temptation to go far-mode, glom on to happy-sounding narratives and scenarios, or even just resist the push to concretely visualize the future at all -- thinking instead in terms of abstract labels and normal-sounding platitudes.

The entire fact that EA and rationalism mostly managed to avert their gaze from the concept of "pivotal acts" for years, is in my opinion an example of how these two mindsets often fail.

"In the endgame, AGI will probably be pretty competitive, and if a bunch of people deploy AGI then at least one will destroy the world" is a thing I think most LWers and many longtermist EAs would have considered obvious. As a community, however, we mostly managed to just-not-think the obvious next thought, "In order to prevent the world's destruction in this scenario, one of the first AGI groups needs to find some fast way to prevent the proliferation of AGI."

Fun Thought Experiment Mindset, I think, encouraged this mental avoidance because it thought of AGI alignment (to some extent) as a fun game in the genre of "math puzzle" or "science fiction scenario", not as a pragmatic, real-world dilemma we actually have to solve, taking into account all of our real-world knowledge and specific facts on the ground. The 'rules of the game', many people apparently felt, were to think about certain specific parts of the action chain leading up to an awesome future lightcone, rather than taking ownership of the entire problem and trying to figure out what humanity should in-real-life do, start to finish.

(What primarily makes this weird is that many alignment questions crucially hinge on 'what task are we aligning the AGI on?'. These are not remotely orthogonal topics.)

Serious Respectable Person Mindset, I think, encouraged this mental avoidance more actively, because pivotal acts are a weird and scary-sounding idea once you leave 'these are just fun thought experiments' land.

What I'd like to see instead is something like Weirdness-Tolerant Project Leader Mindset, or Thought Experiments Plus Actual Rigor And Pragmatism And Drive Mindset, or something.

I think a lot of the confusion around EY's post comes from the difference between thinking of these posts (on some level) as fun debate fodder or persuasion/outreach tools, versus attending to the fact that humanity has to actually align AGI systems if we're going to make it out of this problem, and this is an attempt by humanity to distill where we're currently at, so we can actually proceed to go solve alignment right now and save the world.

Imagine that this is v0 of a series of documents that need to evolve into humanity's (/ some specific group's) actual business plan for saving the world. The details really, really matter. Understanding the shape of the problem really matters, because we need to engineer a solution, not just 'persuade people to care about AI risk'.

If you disagree with the OP... that's pretty important! Share your thoughts. If you agree, that's important to know too, so we can prioritize some disagreements over others and zero in on critical next actions. There's a mindset here that I think is important, that isn't about "agree with Eliezer on arbitrary topics" or "stop thinking laterally"; it's about approaching the problem seriously, neither falling into despair nor wishful thinking, neither far-mode nor forced normality, neither impracticality nor propriety.

-3.  I'm assuming you are already familiar with some basics, and already know what 'orthogonality' and 'instrumental convergence' are and why they're true.

I think this is actually the part that I most "disagree" with. (I put "disagree" in quotes, because there are forms of these theses that I'm persuaded by. However, I'm not so confident that they'll be relevant for the kinds of AIs we'll actually build.)

1. The smart part is not the agent-y part

It seems to me that what's powerful about modern ML systems is their ability to do data compression / pattern recognition. That's where the real cognitive power (to borrow Eliezer's term) comes from. And I think that this is the same as what makes us smart.

GPT-3 does unsupervised learning on text data. Our brains do predictive processing on sensory inputs. My guess (which I'd love to hear arguments against!) is that there's a true and deep analogy between the two, and that they lead to impressive abilities for fundamentally the same reason.

If so, it seems to me that that's where all the juice is. That's where the intelligence comes from. (In the past, I've called this the core smarts [LW(p) · GW(p)] of our brains.)

On this view, all the agent-y, planful, System 2 stuff that we do is the analogue of prompt programming. It's a set of not-very-deep, not-especially-complex algorithms meant to cajole the actually smart stuff into doing something useful.

When I try to extrapolate what this means for how AI systems will be built, I imagine a bunch of Drexler-style [LW · GW]AI services.

Yes, in some cases people will want to chain services together to form something like an agent, with something like goals. However, the agent part isn't the smart part. It's just some simple algorithms on top of a giant pile of pattern recognition and data compression.

Why is that relevant? Isn't an algorithmically simple superintelligent agent just as scary as (if not moreso than) a complex one? In a sense yes, it would still be very scary. But to me it suggests a different intervention point.

If the agency is not inextricably tied to the intelligence, then maybe a reasonable path forward is to try to wring as much productivity as we can out of the passive, superhuman, quasi-oracular just-dumb-data-predictors. And avoid as much as we can ever creating closed-loop, open-ended, free-rein agents.

Am I just recapitulating the case for Oracle-AI / Tool-AI? Maybe so.

But if agency is not a fundamental part of intelligence, and rather something that can just be added in on top, or not, and if we're at a loss for how to either align a superintelligent agent with CEV or else make it corrigible, then why not try to avoid creating the agent part of superintelligent agent?

I think that might be easier than many think...

2. The AI does not care about your atoms either

The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

https://intelligence.org/files/AIPosNegFactor.pdf

Suppose we have (something like) an agent, with (something like) a utility function. I think it's important to keep in mind the domain of the utility function. (I'll be making basically the same point repeatedly throughout the rest of this comment.)

By default, I don't expect systems that we build, with agent-like behavior (even superintelligently smart systems!), to care about all the atoms in the future light cone.

Humans (and other animals) care about atoms. We care about (our sensory perceptions of) macroscopic events, forward in time, because we evolved to. But that is not the default domain of an agent's utility function.

For example, I claim that while AlphaGo could be said to be agent-y, it does not care about atoms. And I think that we could make it fantastically more superhuman at Go, and it would still not care about atoms. Atoms are just not in the domain of its utility function.

In particular, I don't think it has an incentive to break out into the real world to somehow get itself more compute, so that it can think more about its next move. It's just not modeling the real world at all. It's not even trying to rack up a bunch of wins over time. It's just playing the single platonic game of Go.

Giant caveat (that you may already be shouting into your screen): abstractions are leaky.

The ML system is not actually trained to play the platonic game of Go. It's trained to play the-game-of-Go-as-implemented-on-particular-hardware, or something like minimize-this-loss-function-informed-by-Go-game-results. The difference between the platonic game and the embodied game can lead to clever and unexpected behavior.

However, it seems to me that these kinds of hacks are going to look a lot more like a system short-circuiting than it out-of-nowhere building a model of, and starting to care about, the whole universe.

3. Orthogonality squared

I really liked Eliezer's Arbital article on Epistemic and instrumental efficiency. He writes:

An agent that is "efficient", relative to you, within a domain, is one that never makes a real error that you can systematically predict in advance.

I think this very succinctly captures what would be so scary about being up against a (sufficiently) superintelligent agent with conflicting goals to yours. If you think you see a flaw in its plan, that says more about your seeing than it does about its plan. In other words, you're toast.

But as above, I think it's important to keep in mind what an agent's goals are actually about.

Just as the utility function of an agent is orthogonal from its intelligence, it seems to me that the domain of its utility function is another dimension of potential orthogonality.

If you're playing chess against AlphaZero Chess, you're going to lose. But suppose you're secretly playing [LW(p) · GW(p)] "Who has the most pawns after 10 moves?" I think you've got a chance to win! Even though it cares about pawns!

(Of course if you continue playing out the chess game after the10th move, it'll win at that. But by assumption, that's fine, it's not what you cared about.)

If you and another agent have different goals for the same set of objects, you're going to be in conflict. It's going to be zero sum. But if the stuff you care about is only tangentially related to the stuff it cares about, then the results can be positive sum. You can both win!

In particular, you can both get what you want without either of you turning the other off. (And if you know that, you don't have to preemptively try to turn each other off to prevent being turned off either.)

4. Programs, agents, and real-world agents

Agents are a tiny subset of all programs. And agents whose utility functions are defined over the real world are a tiny subset of all agents.

If we think about all the programs we could potentially write that take in inputs and produce outputs, it will make sense to talk about some of those as agents. These are the programs that seem to be optimizing something. Or seem to have goals and make plans.

But, crucially, all that optimization takes place with respect to some environment. And if the input and output of an agent-y program is hooked up to the wrong environment (or hooked up to the right environment in the wrong way), it'll cease to be agent-y.

For example, if you hook me up to the real world by sticking me in outer space (sans suit), I will cease to be very agent-y. Or, if you hook up the inputs and outputs of AlphaGo to a chess board, it will cease to be formidable (until you retrain it). (In other words, the isAgent() predicate is not a one-place function.)

This suggests to me that we could build agent-y, superintelligent systems that are not a threat to us. (Because they are not agent-y with respect to the real world.)

Yes, we're likely to (drastically) oversample from the subset of agents that are agent-y w.r.t. the real world, because we're going to want to build systems that are useful to us.

But if I'm right about the short-circuiting argument above, even our agent-y systems won't have coherent goals defined over events far outside their original domain (e.g. the arrangement of all the atoms in the future light cone) by default.

So even if our systems are agent-y (w.r.t. some environment), and have some knowledge of and take some actions in the real world, they won't automatically have a utility function defined over the configurations of all atoms.

On the other hand, the more we train them as open-ended agents with wide remit to act in the real world (or a simulation thereof), the more we'll have a (potentially superintelligently lethal) problem on our hands.

To me that suggests that what we need to care about are things like: how open-ended we make our systems, whether we train them via evolution-like competition between agents in a high-def simulation of the real world, and what kind of systems are incentivized to be developed and deployed, society-wide.

5. Conclusion

If I'm right in the above thinking, then orthogonality is more relevant and instrumental convergence is less relevant than it might otherwise appear.

Instrumental convergence would only end up being a concern for agents that care about the same objects / resources / domain that you do. If their utility function is just not about those things, IC will drive them to acquire a totally different set of resources that is not in conflict with your resources (e.g. a positional chess advantage in a go game, or trading for your knight while you try to acquire pawns).

This would mean that we need to be very worried about open-ended real-world agents. But less worried about intelligence in general, or even agents in general.

To be clear, I'm not claiming that it's all roses from here on out. But this reasoning leads me to conclude that the key problems may not be the ones described in the post above.

I agree with pretty much everything here, and I would add into the mix two more claims that I think are especially cruxy and therefore should maybe be called out explicitly to facilitate better discussion:

Claim A: “There’s no defense against an out-of-control omnicidal AGI, not even with the help of an equally-capable (or more-capable) aligned AGI, except via aggressive outside-the-Overton-window acts like preventing the omnicidal AGI from being created in the first place.”

I think this claim is true, on account of gray goo and lots of other things, and I suspect Eliezer does too, and I’m pretty sure other people disagree with this claim.

If someone disagrees with this claim (i.e., if they think that if DeepMind can make an aligned and Overton-window-abiding “helper” AGI, then we don’t have to worry about Meta making a similarly-capable out-of-control omnicidal misaligned AGI the following year, because DeepMind’s AGI will figure out how to protect us), and also believes in extremely slow takeoff, I can see how such a person might be substantially less pessimistic about AGI doom than I am.

Claim B: “Shortly after (i.e., years not decades after) we have dangerous AGI, we will have dangerous AGI requiring amounts of compute that many many many actors have access to.”

Again I think this claim is true [LW · GW], and I suspect Eliezer does too. In fact, my guess is that there are already single GPU chips with enough FLOP/s to run human-level, human-speed, AGI, or at least in that ballpark. All that we need is to figure out the right learning algorithms, which of course is happening as we speak.

If someone disagrees with this claim, I think they could plausibly be less pessimistic than I am about prospects for coordinating not to build AGI, or coordinating in other ways, because it just wouldn’t be that many actors, and maybe they could all be accounted for and reach agreement (e.g. after a headline-grabbing near-miss catastrophe or something).

(I think most people in AI alignment, especially “scaling hypothesis” people, are expecting early AGIs to involve truly mindboggling amounts of compute, followed by some very long period where the required compute very gradually decreases on account of algorithmic advances. That’s not what I expect; instead I expect the discovery of new better learning algorithms with a different scaling curve that zooms to AGI and beyond quite quickly.)

Found this to be an interesting list of challenges, but I disagree with a few points. (Not trying to be comprehensive here, just a few thoughts after the first read-through.)

• Several of the points here are premised on needing to do a pivotal act that is way out of distribution from anything the agent has been trained on. But it's much safer to deploy AI iteratively; increasing the stakes, time horizons, and autonomy a little bit each time. With this iterative approach to deployment, you only need to generalize a little bit out of distribution. Further, you can use Agent N to help you closely supervise Agent N+1 before giving it any power.
• One claim is that Capabilities generalize further than alignment once capabilities start to generalize far. The argument is that an agent's world model and tactics will be automatically fixed by reasoning and data, but its inner objective won't be changed by these things. I agree with the preceding sentence, but I would draw a different (and more optimistic) conclusion from it. That it might be possible to establish an agent's inner objective when training on easy problems, when the agent isn't very capable, such that this objective remains stable as the agent becomes more powerful.
  Also, there's empirical evidence that alignment generalizes surprisingly well: several thousand instruction following examples radically improve the aligned behavior on a wide distribution of language tasks (InstructGPT paper) a prompt with about 20 conversations gives much better behavior on a wide variety of conversational inputs (HHH paper). Making a contemporary language model well-behaved seems to be much easier than teaching it a new cognitive skill.
• Human raters make systematic errors - regular, compactly describable, predictable errors.... This is indeed one of the big problems of outer alignment, but there's lots of ongoing research and promising ideas for fixing it. Namely, using models to help amplify and improve the human feedback signal. Because P!=NP it's easier to verify proofs than to write them. Obviously alignment isn't about writing proofs, but the general principle does apply. You can reduce "behaving well" to "answering questions truthfully" by asking questions like "did the agent follow the instructions in this episode?", and use those to define the reward function. These questions are not formulated in formal language where verification is easy, but there's reason to believe that verification is also easier than proof-generation for informal arguments.

I think until recently, I've been consistently more pessimistic than Eliezer about AI existential safety. Here's a 2004 SL4 post for example where I tried to argue against MIRI (SIAI at the time) trying to build a safe AI (and again in 2011 [LW · GW]). I've made my own list of sources of AI risk [LW · GW] that's somewhat similar to this list. But it seems to me that there are still various "outs" from certain doom, such that my probability of a good outcome is closer to 20% (maybe a range of 10-30% depending on my mood) than 1%.

32. Human thought partially exposes only a partially scrutable outer surface layer. Words only trace our real thoughts. Words are not an AGI-complete data representation in its native style. The underparts of human thought are not exposed for direct imitation learning and can't be put in any dataset. This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents, which are only impoverished subsystems of human thoughts; unless that system is powerful enough to contain inner intelligences figuring out the humans, and at that point it is no longer really working as imitative human thought.

One of the biggest "outs" I see is that it turns out to be not that hard "to train a powerful system entirely on imitation of human words or other human-legible contents", we (e.g., a relatively responsible AI lab) train such a system and then use it to differentially accelerate AI safety research. I definitely think that it's very risky [AF · GW] to rely on such black-box human imitations for existential safety, and that a competent civilization would be pursuing other plans where they can end up with greater certainty of success, but it seems there's something like a 20% chance that it just works out anyway.

To explain my thinking a bit more, human children have to learn how to think human thoughts through "imitation of human words or other human-legible contents". It's possible that they can only do this successfully because their genes contain certain key ingredients that enable human thinking, but it also seems possible that children are just implementations of some generic imitation learning algorithm, so our artificial learning algorithms (once they become advanced/powerful enough) won't be worse at learning to think like humans. I don't know how to rule out the latter possibility with very high confidence. Eliezer, if you do, can you please explain this more?

[This is a nitpick of the form "one of your side-rants went a bit too far IMO;" feel free to ignore]

The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I published it, which nobody apparently did, despite my having had other things to do than write this up for the last five years or so. Some of that silence may, possibly, optimistically, be due to nobody else in this field having the ability to write things comprehensibly - such that somebody out there had the knowledge to write all of this themselves, if they could only have written it up, but they couldn't write, so didn't try. ... The fact that, twenty-one years into my entering this death game, seven years into other EAs noticing the death game, and two years into even normies starting to notice the death game, it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that.

The third option this seems to miss is that there are people who could have written this document, but they also thought they had better things to do than write it. I'm thinking of people like Paul Christiano, Nate Soares, John Wentworth, Ajeya Cotra... there are dozens of people who have thought deeply about this stuff and also talked with you (Yudkowsky) and I bet they could have written something approximately as good as this if they tried. Perhaps, like you, they decided to instead spend their time working directly on the problem.

I do agree with you that they seem to on average be way way too optimistic, but I don't think it's because they are ignorant of the considerations and arguments you've made here.

A big source of optimism for Paul, for example, seems to be his timelines + views about takeoff speeds, which are mostly independent of the claims made in this post. I too would be cautiously optimistic if I thought we had 30 years left and that by the time things really went crazy we'd have decades of experience with just-slightly-dumber systems automating big chunks of the economy & AI alignment would be a big prestigious field with lots of geniuses being mentored by older geniuses etc. (Many of the points you make here would still apply, so it would still be a pretty scary situation...)

Reading this post made me more optimistic about alignment and AI [LW(p) · GW(p)]. My suspension of disbelief snapped; I realized how vague and bad a lot of these "classic" alignment arguments are, and how many of them are secretly vague analogies [LW(p) · GW(p)] and intuitions about evolution.

While I agree with a few points on this list, I think this list is fundamentally misguided. The list is written in a language which assigns short encodings to confused and incorrect ideas [LW · GW]. I think a person who tries to deeply internalize this post's worldview will end up more confused about alignment and AI, and urge new researchers to not spend too much time trying to internalize this post's ideas. (Definitely consider whether I am right in my claims here. Think for yourself. If you don't know how to think for yourself, I wrote about exactly how to do it [LW · GW]! But my guess is that deeply engaging with this post is, at best, a waste of time.^[1])

I think this piece is not "overconfident", because "overconfident" suggests that Lethalities is simply assigning extreme credences to reasonable questions (like "is deceptive alignment the default?"). Rather, I think both its predictions and questions are not reasonable because they are not located by good evidence or arguments. (Example: I think that deceptive alignment is only supported by flimsy arguments [LW(p) · GW(p)].) 

I personally think Eliezer's alignment worldview (as I understand it!) appears to exist in an alternative reality derived from unjustified background assumptions.^[2] Given those assumptions, then sure, Eliezer's reasoning steps are probably locally valid. But I think that in reality, most of this worldview ends up irrelevant and misleading because the background assumptions don't hold. 

I think this kind of worldview (socially attempts to) shield itself from falsification [LW(p) · GW(p)] by e.g. claiming that modern systems "don't count" for various reasons which I consider flimsy [LW(p) · GW(p)]. But I think that deep learning experiments provide plenty of evidence on alignment questions [LW(p) · GW(p)].

But, hey, why not still include this piece in the review? I think it's interesting to know what a particular influential person thought at a given point in time.

1. ^{^}

   Related writing of mine: Some of my disagreements with List of Lethalities [LW · GW], Inner and outer alignment decompose one hard problem into two extremely hard problems [LW · GW]. 

   Recommended further critiques of this worldview: Evolution is a bad analogy for AGI: inner alignment [LW · GW], Evolution provides no evidence for the sharp left turn [LW · GW], My Objections to "We’re All Gonna Die with Eliezer Yudkowsky" [LW · GW].

2. ^{^}

   Since Eliezer claims to have figured out so many ideas in the 2000s, his assumptions presumably were locked in before the advent of deep learning. This constitutes a "bottom line [LW · GW]." 

I would summarize a dimension of the difficulty like this. There are the conditions that give rise to intellectual scenes, intellectual scenes being necessary for novel work in ambiguous domains. There are the conditions that give rise to the sort of orgs that output actions consistent with something like Six Dimensions of Operational Adequacy [LW · GW]. The intersection of these two things is incredibly rare but not unheard of. The Manhattan Project was a Scene that had security mindset. This is why I am not that hopeful. Humans are not the ones building the AGI, egregores are, and spending egregore sums of money. It is very difficult for individuals to support a scene of such magnitude, even if they wanted to. Ultra high net worth individuals seem much poorer relative to the wealth of society than in the past, where scenes and universities (a scene generator) could be funded by individuals or families. I'd guess this is partially because the opportunity cost for smart people is much higher now, and you need to match that (cue title card: Baumol's cost disease kills everyone). In practice I expect some will give objections along various seemingly practical lines, but my experience so far is that these objections are actually generated by an environment that isn't willing to be seen spending gobs of money on low status researchers who mostly produce nothing. i.e. funding the 90%+ percent of a scene that isn't obviously contributing to the emergence of a small cluster that actually does the thing.

Note: I think there's a bunch of additional reasons for doom, surrounding "civilizational adequacy / organizational competence / societal dynamics". Eliezer briefly alluded to these, but AFAICT he's mostly focused on lethality that comes "early", and then didn't address them much. My model of Andrew Critch has a bunch of concerns about doom that show up later, because there's a bunch of additional challenges you have to solve if AI doesn't dramatically win/lose early on (i.e. multi/multi dynamics and how they spiral out of control)

I know a bunch of people whose hope funnels through "We'll be able to carefully iterate on slightly-smarter-than-human-intelligences, build schemes to play them against each other, leverage them to make some progress on alignment that we can use to build slightly-more-advanced-safer-systems". (Let's call this the "Careful Bootstrap plan")

I do actually feel nonzero optimism about that plan, but when I talk to people who are optimistic about that I feel a missing mood about the kind of difficulty that is involved here.

I'll attempt to write up some concrete things here later, but wanted to note this for now.

What concerns me the most is the lack of any coherent effort anywhere, towards solving the biggest problem: identifying a goal (value system, utility function, decision theory, decision architecture...) suitable for an autonomous superhuman AI. 

In these discussions, Coherent Extrapolated Volition (CEV) is the usual concrete formulation of what such a goal might be. But I've now learned [LW(p) · GW(p)] that MIRI's central strategy is not to finish figuring out the theory and practice of CEV - that's considered too hard (see item 24 in this post). Instead, the hope is to use safe AGI to freeze all unsafe AGI development everywhere, for long enough that humanity can properly figure out what to do. Presumably this freeze (the "pivotal act") would be carried out by whichever government or corporation or university crossed the AGI threshold first; ideally there might even become a consensus among many of the contenders that this is the right thing to do. 

I think it's very appropriate that some thought along these lines be carried out. If AGI is a threat to the human race, and it arrives before we know how to safely set it free, then we will need ways to try to neutralize that dangerous potential. But I also think it's vital that we try to solve that biggest problem, e.g. by figuring out how to concretely implement CEV. And if one is concerned that this is just too much for human intellect to figure out, remember that AI capabilities are rising. If humans can't figure out CEV unaided, maybe they can do it with the help of AI. To me, that's the critical pathway that we should be analyzing. 

P.S. I have many more thoughts on what this might involve, but I don't know when I will be able to sort through them all. So for now I will just list a few people whose work is on my shortlist of definitely or potentially relevant (certainly not a complete list): June Ku, Vanessa Kosoy, Jessica Taylor, Steven Byrnes, Stuart Armstrong. 

Thank you, this was very helpful. As a bright-eyed youngster, it's hard to make sense of the bitterness and pessimism I often see in the field. I've read the old debates, but I didn't participate in them, and that probably makes them easier to dismiss. Object level arguments like these help me understand your point of view. 

Alright, now that I've read this post, I'll try to respond to what I think you got wrong, and importantly illustrate some general principles.

To respond to this first:

3.  We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.  This includes, for example: (a) something smart enough to build a nanosystem which has been explicitly authorized to build a nanosystem; or (b) something smart enough to build a nanosystem and also smart enough to gain unauthorized access to the Internet and pay a human to put together the ingredients for a nanosystem; or (c) something smart enough to get unauthorized access to the Internet and build something smarter than itself on the number of machines it can hack; or (d) something smart enough to treat humans as manipulable machinery and which has any authorized or unauthorized two-way causal channel with humans; or (e) something smart enough to improve itself enough to do (b) or (d); etcetera.  We can gather all sorts of information beforehand from less powerful systems that will not kill us if we screw up operating them; but once we are running more powerful systems, we can no longer update on sufficiently catastrophic errors.  This is where practically all of the real lethality comes from, that we have to get things right on the first sufficiently-critical try.  If we had unlimited retries - if every time an AGI destroyed all the galaxies we got to go back in time four years and try again - we would in a hundred years figure out which bright ideas actually worked.  Human beings can figure out pretty difficult things over time, when they get lots of tries; when a failed guess kills literally everyone, that is harder.  That we have to get a bunch of key stuff right on the first try is where most of the lethality really and ultimately comes from; likewise the fact that no authority is here to tell us a list of what exactly is 'key' and will kill us if we get it wrong.  (One remarks that most people are so absolutely and flatly unprepared by their 'scientific' educations to challenge pre-paradigmatic puzzles with no scholarly authoritative supervision, that they do not even realize how much harder that is, or how incredibly lethal it is to demand getting that right on the first critical try.)

I think this is actually wrong, because of synthetic data letting us control what the AI learns and what they value, and in particular we can place honeypots that are practically indistingushiable from the real world, such that if we detected an AI trying to deceive or gain power, the AI almost certainly doesn't know whether we tested it or whether it's in the the real world:

Because of this, given our theoretical complete control over the data input of these models, it will be possible to train them inside what is effectively a simulated world slightly different from our own. This will allow us to maximize their capabilities along axes that are safe and useful to us, while minimizing the risks of deception and take-over and being able to setup traps to detect when these are likely. If the model is not trained on human data, then we can make its understanding of our world subtly different from the reality. If done carefully and cleverly, this will make it very easy to fool and be able to discover e.g. deceptiveness, or attempted takeover in a ‘realistic’ simulated world that the model is put inside during red-teaming. While deceiving an adversary with direct access to your cognition is hard enough, doing so when you have been brought up in a world entirely controlled by that adversary, and which is feeding you subtly incorrect and misleading information about the adversary and their capabilities is likely orders of magnitude more difficult. Jacob Cannell has argued along similar lines [LW · GW] of training potential AGI systems in simulations of fantasy worlds, where takeover can be simulated but is far enough away that skills do not transfer. Fully synthetic datasets are necessary for this and indeed, a fantasy world is likely far from what we need. Instead, we only need to simulate a world very close to our own but with slightly different affordances – maybe specific software vulnerabilities are simulated to exist but are actually patched in our world, maybe certain peptide-making companies accept all orders in the simulated world but have more checks in our world, maybe certain interpretability techniques are claimed not to work in the simulation, but do in fact work in our world – to cause any attempted takeover behaviour or adversarial deception to appear extremely plausible to the AI but in fact be detectable and defendable in our reality. In fact, we should design simulated worlds with ‘honeytraps’, which appear to be loopholes subtly planted in the AI’s input data to encourage them to attempt to pursue their misaligned behaviour, but which we know of ahead of time and can monitor and catch.

It's the same reason for why we can't break out of the simulation IRL, except we don't have to face adversarial cognition, so the AI's task is even harder than our task.

See also this link:

https://www.beren.io/2024-05-11-Alignment-in-the-Age-of-Synthetic-Data/

For this:

6.  We need to align the performance of some large task, a 'pivotal act' that prevents other people from building an unaligned AGI that destroys the world. While the number of actors with AGI is few or one, they must execute some "pivotal act", strong enough to flip the gameboard, using an AGI powerful enough to do that.  It's not enough to be able to align a weak system - we need to align a system that can do some single very large thing.  The example I usually give is "burn all GPUs".  This is not what I think you'd actually want to do with a powerful AGI - the nanomachines would need to operate in an incredibly complicated open environment to hunt down all the GPUs, and that would be needlessly difficult to align.  However, all known pivotal acts are currently outside the Overton Window, and I expect them to stay there.  So I picked an example where if anybody says "how dare you propose burning all GPUs?" I can say "Oh, well, I don't actually advocate doing that; it's just a mild overestimate for the rough power level of what you'd have to do, and the rough level of machine cognition required to do that, in order to prevent somebody else from destroying the world in six months or three years."  (If it wasn't a mild overestimate, then 'burn all GPUs' would actually be the minimal pivotal task and hence correct answer, and I wouldn't be able to give that denial.)  Many clever-sounding proposals for alignment fall apart as soon as you ask "How could you use this to align a system that you could use to shut down all the GPUs in the world?" because it's then clear that the system can't do something that powerful, or, if it can do that, the system wouldn't be easy to align.  A GPU-burner is also a system powerful enough to, and purportedly authorized to, build nanotechnology, so it requires operating in a dangerous domain at a dangerous level of intelligence and capability; and this goes along with any non-fantasy attempt to name a way an AGI could change the world such that a half-dozen other would-be AGI-builders won't destroy the world 6 months later.

I think this is wrong, and a lot of why I disagree with the pivotal act framing is probably due to disagreeing with the assumption that future technology will be radically biased towards to offense, and while I do think biotechnology is probably pretty offense-biased today, I also think it's tractable to reduce bio-risk without trying for pivotal acts.

Also, I think @evhub [LW · GW]'s point about homogeneity of AI takeoff bears on this here, and while I don't agree with all the implications, like there being no warning shot for deceptive alignment (because of synthetic data), I think there's a point in which a lot of AIs are very likely to be very homogenous, and thus break your point here:

https://www.lesswrong.com/posts/mKBfa8v4S9pNKSyKK/homogeneity-vs-heterogeneity-in-ai-takeoff-scenarios [LW · GW]

Running AGIs doing something pivotal are not passively safe, they're the equivalent of nuclear cores that require actively maintained design properties to not go supercritical and melt down.

I think that AGIs are more robust to things going wrong than nuclear cores, and more generally I think there is much better evidence for AI robustness than fragility.

@jdp [LW · GW]'s comment provides more evidence on why this is the case:

Where our understanding begins to diverge is how we think about the robustness of these systems. You think of deep neural networks as being basically fragile in the same way that a Boeing 747 is fragile. If you remove a few parts of that system it will stop functioning, possibly at a deeply inconvenient time like when you're in the air. When I say you are systematically overindexing, I mean that you think of problems like SolidGoldMagikarp [LW · GW] as central examples of neural network failures. This is evidenced by Eliezer Yudkowsky calling investigation of it "one of the more hopeful processes happening on Earth". This is also probably why you focus so much on things like adversarial examples as evidence of un-robustness, even though many critics like Quintin Pope point out that adversarial robustness would make AI systems strictly less corrigible.

By contrast I tend to think of neural net representations as relatively robust. They get this property from being continuous systems with a range of operating parameters, which means instead of just trying to represent the things they see they implicitly try to represent the interobjects between what they've seen through a navigable latent geometry. I think of things like SolidGoldMagikarp as weird edge cases where they suddenly display discontinuous behavior, and that there are probably a finite number of these edge cases. It helps to realize that these glitch tokens were simply never trained, they were holdovers from earlier versions of the dataset that no longer contain the data the tokens were associated with. When you put one of these glitch tokens into the model, it is presumably just a random vector into the GPT-N latent space. That is, this isn't a learned program in the neural net that we've discovered doing glitchy things, but an essentially out of distribution input with privileged access to the network geometry through a programming oversight. In essence, it's a normal software error not a revelation about neural nets. Most such errors don't even produce effects that interesting, the usual thing that happens if you write a bug in your neural net code is the resulting system becomes less performant. Basically every experienced deep learning researcher has had the experience of writing multiple errors that partially cancel each other out to produce a working system during training, only to later realize their mistake.

Moreover the parts of the deep learning literature you think of as an emerging science of artificial minds tend to agree with my understanding. For example it turns out that if you ablate parts of a neural network later parts will correct the errors without retraining. This implies that these networks function as something like an in-context error correcting code, which helps them generalize over the many inputs they are exposed to during training. We even have papers analyzing mechanistic parts of this error correcting code like copy suppression heads. One simple proxy for out of distribution performance is to inject Gaussian noise, since a Gaussian can be thought of like the distribution over distributions. In fact if you inject noise into GPT-N word embeddings the resulting model becomes more performant in general, not just on out of distribution tasks. So the out of distribution performance of these models is highly tied to their in-distribution performance, they wouldn't be able to generalize within the distribution well if they couldn't also generalize out of distribution somewhat. Basically the fact that these models are vulnerable to adversarial examples is not a good fact to generalize about their overall robustness from as representations.

Link here:

https://www.lesswrong.com/posts/JcLhYQQADzTsAEaXd/?commentId=7iBb7aF4ctfjLH6AC [LW · GW]

10.  You can't train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators, assigning a loss, and doing supervised learning.  On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions.  (Some generalization of this seems like it would have to be true even outside that paradigm; you wouldn't be working on a live unaligned superintelligence to align it.)  This alone is a point that is sufficient to kill a lot of naive proposals from people who never did or could concretely sketch out any specific scenario of what training they'd do, in order to align what output - which is why, of course, they never concretely sketch anything like that.  Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn't kill you.  This is where a huge amount of lethality comes from on anything remotely resembling the present paradigm.  Unaligned operation at a dangerous level of intelligence*capability will kill you; so, if you're starting with an unaligned system and labeling outputs in order to get it to learn alignment, the training regime or building regime must be operating at some lower level of intelligence*capability that is passively safe, where its currently-unaligned operation does not pose any threat.  (Note that anything substantially smarter than you poses a threat given any realistic level of capability.  Eg, "being able to produce outputs that humans look at" is probably sufficient for a generally much-smarter-than-human AGI to navigate its way out of the causal systems that are humans, especially in the real world where somebody trained the system on terabytes of Internet text, rather than somehow keeping it ignorant of the latent causes of its source code and training environments.)

I think that there will be generalization of alignment, and more generally I think that alignment generalizes further than capabilities by default, contra you and Nate Soares because of these reasons:

2.) Reward modelling is much simpler with respect to uncertainty, at least if you want to be conservative. If you are uncertain about the reward of something, you can just assume it will be bad and generally you will do fine. This reward conservatism is often not optimal for agents who have to navigate an explore/exploit tradeoff but seems very sensible for alignment of an AGI where we really do not want to ‘explore’ too far in value space. Uncertainty for ‘capabilities’ is significantly more problematic since you have to be able to explore and guard against uncertainty in precisely the right way to actually optimize a stochastic world towards a specific desired point.

3.) There are general theoretical complexity priors to believe that judging is easier than generating. There are many theoretical results of the form that it is significantly asymptotically easier to e.g. verify a proof than generate a new one. This seems to be a fundamental feature of our reality, and this to some extent maps to the distinction between alignment and capabilities. Just intuitively, it also seems true. It is relatively easy to understand if a hypothetical situation would be good or not. It is much much harder to actually find a path to materialize that situation in the real world.

4.) We see a similar situation with humans. Almost all human problems are caused by a.) not knowing what you want and b.) being unable to actually optimize the world towards that state. Very few problems are caused by incorrectly judging or misgeneralizing bad situations as good and vice-versa. For the AI, we aim to solve part a.) as a general part of outer alignment and b.) is the general problem of capabilities. It is much much much easier for people to judge and critique outcomes than actually materialize them in practice, as evidenced by the very large amount of people who do the former compared to the latter.

5.) Similarly, understanding of values and ability to assess situations for value arises much earlier and robustly in human development than ability to actually steer outcomes. Young children are very good at knowing what they want and when things don’t go how they want, even new situations for them, and are significantly worse at actually being able to bring about their desires in the world.

In general, it makes sense that, in some sense, specifying our values and a model to judge latent states is simpler than the ability to optimize the world. Values are relatively computationally simple and are learnt as part of a general unsupervised world model where there is ample data to learn them from (humans love to discuss values!). Values thus fall out mostly’for free’ from general unsupervised learning. As evidenced by the general struggles of AI agents, ability to actually optimize coherently in complex stochastic ‘real-world’ environments over long time horizons is fundamentally more difficult than simply building a detailed linguistic understanding of the world.

See also this link for more, but I think that's the gist for why I expect AI alignment to generalize much further than AI capabilities. I'd further add that I think evolutionary psychology got this very wrong, and predicted much more complex and fragile values in humans than is actually the case:

https://www.beren.io/2024-05-15-Alignment-Likely-Generalizes-Further-Than-Capabilities/

11.  If cognitive machinery doesn't generalize far out of the distribution where you did tons of training, it can't solve problems on the order of 'build nanotechnology' where it would be too expensive to run a million training runs of failing to build nanotechnology.  There is no pivotal act this weak; there's no known case where you can entrain a safe level of ability on a safe environment where you can cheaply do millions of runs, and deploy that capability to save the world and prevent the next AGI project up from destroying the world two years later.  Pivotal weak acts like this aren't known, and not for want of people looking for them.  So, again, you end up needing alignment to generalize way out of the training distribution - not just because the training environment needs to be safe, but because the training environment probably also needs to be cheaper than evaluating some real-world domain in which the AGI needs to do some huge act.  You don't get 1000 failed tries at burning all GPUs - because people will notice, even leaving out the consequences of capabilities success and alignment failure.

This is covered by my points on why alignment generalizes further than capabilities and why we don't need pivotal acts and why we actually have safe testing grounds for deceptive AI.

15.  Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously.  Given otherwise insufficient foresight by the operators, I'd expect a lot of those problems to appear approximately simultaneously after a sharp capability gain.  See, again, the case of human intelligence.  We didn't break alignment with the 'inclusive reproductive fitness' outer loss function, immediately after the introduction of farming - something like 40,000 years into a 50,000 year Cro-Magnon takeoff, as was itself running very quickly relative to the outer optimization loop of natural selection.  Instead, we got a lot of technology more advanced than was in the ancestral environment, including contraception, in one very fast burst relative to the speed of the outer optimization loop, late in the general intelligence game.  We started reflecting on ourselves a lot more, started being programmed a lot more by cultural evolution, and lots and lots of assumptions underlying our alignment in the ancestral training environment broke simultaneously.  (People will perhaps rationalize reasons why this abstract description doesn't carry over to gradient descent; eg, “gradient descent has less of an information bottleneck”.  My model of this variety of reader has an inside view, which they will label an outside view, that assigns great relevance to some other data points that are not observed cases of an outer optimization loop producing an inner general intelligence, and assigns little importance to our one data point actually featuring the phenomenon in question.  When an outer optimization loop actually produced general intelligence, it broke alignment after it turned general, and did so relatively late in the game of that general intelligence accumulating capability and knowledge, almost immediately before it turned 'lethally' dangerous relative to the outer optimization loop of natural selection.  Consider skepticism, if someone is ignoring this one warning, especially if they are not presenting equally lethal and dangerous things that they say will go wrong instead.)

Re the sharp capability gain breaking alignment properties, one very crucial advantage we have over evolution is that our goals are much more densely defined, constraining the AI more than evolution, where very, very sparse reward was the norm, and critically sparse-reward RL does not work for capabilities right now, and there are reasons to think it will be way less tractable than RL where rewards are more densely specified.

Another advantage we have over evolution, and chimpanzees/gorillas/orangutans is far, far more control over their data sources, which strongly influences their goals.

This is also helpful to point towards more explanation of what the differences are between dense and sparse RL rewards:

This also means that minimal-instrumentality training objectives may suffer from reduced capability compared to an optimization process where you had more open, but still correctly specified, bounds. This seems like a necessary tradeoff in a context where we don't know how to correctly specify bounds.

Fortunately, this seems to still apply to capabilities at the moment- the expected result for using RL in a sufficiently unconstrained environment often ranges from "complete failure" to "insane useless crap." It's notable that some of the strongest RL agents are built off of a foundation of noninstrumental world models.

https://www.lesswrong.com/posts/rZ6wam9gFGFQrCWHc/#mT792uAy4ih3qCDfx [LW · GW]

16.  Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments.  Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction.  This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again: the first semi-outer-aligned solutions found, in the search ordering of a real-world bounded optimization process, are not inner-aligned solutions.  This is sufficient on its own, even ignoring many other items on this list, to trash entire categories of naive alignment proposals which assume that if you optimize a bunch on a loss function calculated using some simple concept, you get perfect inner alignment on that concept.

Yeah, I covered this above, but evolution's loss function was neither that simple, compared to human goals, and it was ridiculously inexact compared to our attempts to optimize AIs loss functions, for the reasons I gave above.

17.  More generally, a superproblem of 'outer optimization doesn't produce inner alignment' is that on the current optimization paradigm there is no general idea of how to get particular inner properties into a system, or verify that they're there, rather than just observable outer ones you can run a loss function over.  This is a problem when you're trying to generalize out of the original training distribution, because, eg, the outer behaviors you see could have been produced by an inner-misaligned system that is deliberately producing outer behaviors that will fool you.  We don't know how to get any bits of information into the inner system rather than the outer behaviors, in any systematic or general way, on the current optimization paradigm.

I've answered that concern above in synthetic data for why we have the ability to get particular inner behaviors into a system.

18.  There's no reliable Cartesian-sensory ground truth (reliable loss-function-calculator) about whether an output is 'aligned', because some outputs destroy (or fool) the human operators and produce a different environmental causal chain behind the externally-registered loss function.  That is, if you show an agent a reward signal that's currently being generated by humans, the signal is not in general a reliable perfect ground truth about how aligned an action was, because another way of producing a high reward signal is to deceive, corrupt, or replace the human operators with a different causal system which generates that reward signal.  When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do; even if it ends up perfectly inner-aligned on that reward signal, or learning some concept that exactly corresponds to 'wanting states of the environment which result in a high reward signal being sent', an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).

19.  More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment - to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.  This isn't to say that nothing in the system’s goal (whatever goal accidentally ends up being inner-optimized over) could ever point to anything in the environment by accident.  Humans ended up pointing to their environments at least partially, though we've got lots of internally oriented motivational pointers as well.  But insofar as the current paradigm works at all, the on-paper design properties say that it only works for aligning on known direct functions of sense data and reward functions.  All of these kill you if optimized-over by a sufficiently powerful intelligence, because they imply strategies like 'kill everyone in the world using nanotech to strike before they know they're in a battle, and have control of your reward button forever after'.  It just isn't true that we know a function on webcam input such that every world with that webcam showing the right things is safe for us creatures outside the webcam.  This general problem is a fact about the territory, not the map; it's a fact about the actual environment, not the particular optimizer, that lethal-to-us possibilities exist in some possible environments underlying every given sense input.

The points were covered above, but synthetic data early in training + densely defined reward/utility functions = alignment, because they don't know how to fool humans when they get data corresponding to values yet.

21.  There's something like a single answer, or a single bucket of answers, for questions like 'What's the environment really like?' and 'How do I figure out the environment?' and 'Which of my possible outputs interact with reality in a way that causes reality to have certain properties?', where a simple outer optimization loop will straightforwardly shove optimizees into this bucket.  When you have a wrong belief, reality hits back at your wrong predictions.  When you have a broken belief-updater, reality hits back at your broken predictive mechanism via predictive losses, and a gradient descent update fixes the problem in a simple way that can easily cohere with all the other predictive stuff.  In contrast, when it comes to a choice of utility function, there are unbounded degrees of freedom and multiple reflectively coherent fixpoints.  Reality doesn't 'hit back' against things that are locally aligned with the loss function on a particular range of test cases, but globally misaligned on a wider range of test cases.  This is the very abstract story about why hominids, once they finally started to generalize, generalized their capabilities to Moon landings, but their inner optimization no longer adhered very well to the outer-optimization goal of 'relative inclusive reproductive fitness' - even though they were in their ancestral environment optimized very strictly around this one thing and nothing else.  This abstract dynamic is something you'd expect to be true about outer optimization loops on the order of both 'natural selection' and 'gradient descent'.  The central result:  Capabilities generalize further than alignment once capabilities start to generalize far.

The key is that data on values is what constrains the choice of utility functions, and while values aren't in physics, they are in human books, and I've explained why alignment generalizes further than capabilities.

22.  There's a relatively simple core structure that explains why complicated cognitive machines work; which is why such a thing as general intelligence exists and not just a lot of unrelated special-purpose solutions; which is why capabilities generalize after outer optimization infuses them into something that has been optimized enough to become a powerful inner optimizer.  The fact that this core structure is simple and relates generically to low-entropy high-structure environments is why humans can walk on the Moon.  There is no analogous truth about there being a simple core of alignment, especially not one that is even easier for gradient descent to find than it would have been for natural selection to just find 'want inclusive reproductive fitness' as a well-generalizing solution within ancestral humans.  Therefore, capabilities generalize further out-of-distribution than alignment, once they start to generalize at all.

I think that there is actually a simple core of alignment to human values, and a lot of the reasons for why I believe this is because I believe about 80-90%, if not more of our values is broadly shaped by the data, and not the prior, and that the same algorithms that power our capabilities is also used to influence our values, though the data matters much more than the algorithm for what values you have.

More generally, I've become convinced that evopsych was mostly wrong about how humans form values, and how they get their capabilities in ways that are very alignment relevant.

I also disbelieve the claim that humans had a special algorithm that other species don't have, and broadly think human success was due to more compute, data and cultural evolution.

23.  Corrigibility is anti-natural to consequentialist reasoning; "you can't bring the coffee if you're dead" for almost every kind of coffee.  We (MIRI) tried and failed [LW · GW] to find a coherent formula for an agent that would let itself be shut down (without that agent actively trying to get shut down).  Furthermore, many anti-corrigible lines of reasoning like this may only first appear at high levels of intelligence.

Alright, while I think your formalizations of corrigibility failed to get any results, I do think there's a property close to corrigibility that is likely to be compatible with consequentialist reasoning, and that's instruction following, and there are reasons to think that instruction following and consequentialist reasoning go together:

https://www.lesswrong.com/posts/7NvKrqoQgJkZJmcuD/instruction-following-agi-is-easier-and-more-likely-than [LW · GW]

https://www.lesswrong.com/posts/ZdBmKvxBKJH2PBg9W/corrigibility-or-dwim-is-an-attractive-primary-goal-for-agi [LW · GW]

https://www.lesswrong.com/posts/k48vB92mjE9Z28C3s/implied-utilities-of-simulators-are-broad-dense-and-shallow [LW · GW]

https://www.lesswrong.com/posts/EBKJq2gkhvdMg5nTQ/instrumentality-makes-agents-agenty [LW · GW]

https://www.lesswrong.com/posts/vs49tuFuaMEd4iskA/one-path-to-coherence-conditionalization [LW · GW]

24.  There are two fundamentally different approaches you can potentially take to alignment, which are unsolvable for two different sets of reasons; therefore, by becoming confused and ambiguating between the two approaches, you can confuse yourself about whether alignment is necessarily difficult.  The first approach is to build a CEV-style Sovereign which wants exactly what we extrapolated-want and is therefore safe to let optimize all the future galaxies without it accepting any human input trying to stop it.  The second course is to build corrigible AGI which doesn't want exactly what we want, and yet somehow fails to kill us and take over the galaxies despite that being a convergent incentive there.

1. The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our Real Actual Values is far out of reach for our FIRST TRY at AGI.  Yes I mean specifically that the dataset, meta-learning algorithm, and what needs to be learned, is far out of reach for our first try.  It's not just non-hand-codable, it is unteachable on-the-first-try because the thing you are trying to teach is too weird and complicated.
2. The second thing looks unworkable (less so than CEV, but still lethally unworkable) because corrigibility runs actively counter to instrumentally convergent behaviors within a core of general intelligence (the capability that generalizes far out of its original distribution).  You're not trying to make it have an opinion on something the core was previously neutral on.  You're trying to take a system implicitly trained on lots of arithmetic problems until its machinery started to reflect the common coherent core of arithmetic, and get it to say that as a special case 222 + 222 = 555.  You can maybe train something to do this in a particular training distribution, but it's incredibly likely to break when you present it with new math problems far outside that training distribution, on a system which successfully generalizes capabilities that far at all

I'm very skeptical that a CEV exists for the reasons @Steven Byrnes [LW · GW] addresses in the Valence sequence here:

https://www.lesswrong.com/posts/SqgRtCwueovvwxpDQ/valence-series-2-valence-and-normativity#2_7_Moral_reasoning [LW · GW]

But it is also unnecessary for value learning, because of the data on human values and alignment generalizing farther than capabilities.

I addressed why we don't need a first try above.

For the point on corrigibility, I disagree that it's like training it to say that as a special case 222 + 222 = 555, for 2 reasons:

1. I think instrumental convergence pressures are quite a lot weaker than you do.
2. Instruction following can be pretty easily done with synthetic data, and more importantly I think that you can have optimizers who's goals point to another's goals.

25.  We've got no idea what's actually going on inside the giant inscrutable matrices and tensors of floating-point numbers.  Drawing interesting graphs of where a transformer layer is focusing attention doesn't help if the question that needs answering is "So was it planning how to kill us or not?"

I disagree with this, but I do think that mechanistic interpretability does have lots of work to do.

28.  The AGI is smarter than us in whatever domain we're trying to operate it inside, so we cannot mentally check all the possibilities it examines, and we cannot see all the consequences of its outputs using our own mental talent.  A powerful AI searches parts of the option space we don't, and we can't foresee all its options.

29.  The outputs of an AGI go through a huge, not-fully-known-to-us domain (the real world) before they have their real consequences.  Human beings cannot inspect an AGI's output to determine whether the consequences will be good.

The key disagreement is I believe we don't need to check all the possibilities, and that even for smarter AIs, we can almost certainly still verify their work, and generally believe verification is way, way easier than generation.

32.  Human thought partially exposes only a partially scrutable outer surface layer.  Words only trace our real thoughts.  Words are not an AGI-complete data representation in its native style.  The underparts of human thought are not exposed for direct imitation learning and can't be put in any dataset.  This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents, which are only impoverished subsystems of human thoughts; unless that system is powerful enough to contain inner intelligences figuring out the humans, and at that point it is no longer really working as imitative human thought.

I basically disagree with this, both in the assumption that language is very weak, and importantly I believe no AGI-complete problems are left, for the following reasons quoted from Near-mode thinking on AI:

"But for the more important insight: The history of AI is littered with the skulls of people who claimed that some task is AI-complete, when in retrospect this has been obviously false. And while I would have definitely denied that getting IMO gold would be AI-complete, I was surprised by the narrowness of the system DeepMind used."

"I think I was too much in the far-mode headspace of one needing Real Intelligence - namely, a foundation model stronger than current ones - to do well on the IMO, rather than thinking near-mode "okay, imagine DeepMind took a stab at the IMO; what kind of methods would they use, and how well would those work?"

"I also updated away from a "some tasks are AI-complete" type of view, towards "often the first system to do X will not be the first systems to do Y".

I've come to realize that being "superhuman" at something is often much more mundane than I've thought. (Maybe focusing on full superintelligence - something better than humanity on practically any task of interest - has thrown me off.)"

Like:

"In chess, you can just look a bit more ahead, be a bit better at weighting factors, make a bit sharper tradeoffs, make just a bit fewer errors. If I showed you a video of a robot that was superhuman at juggling, it probably wouldn't look all that impressive to you (or me, despite being a juggler). It would just be a robot juggling a couple balls more than a human can, throwing a bit higher, moving a bit faster, with just a bit more accuracy. The first language models to be superhuman at persuasion won't rely on any wildly incomprehensible pathways that break the human user (c.f. List of Lethalities, items 18 and 20). They just choose their words a bit more carefully, leverage a bit more information about the user in a bit more useful way, have a bit more persuasive writing style, being a bit more subtle in their ways. (Indeed, already GPT-4 is better than your average study participant in persuasiveness.) You don't need any fundamental breakthroughs in AI to reach superhuman programming skills. Language models just know a lot more stuff, are a lot faster and cheaper, are a lot more consistent, make fewer simple bugs, can keep track of more information at once. (Indeed, current best models are already useful for programming.) (Maybe these systems are subhuman or merely human-level in some aspects, but they can compensate for that by being a lot better on other dimensions.)"

"As a consequence, I now think that the first transformatively useful AIs could look behaviorally quite mundane."

https://www.lesswrong.com/posts/ASLHfy92vCwduvBRZ/near-mode-thinking-on-ai [LW · GW]

To address an epistemic point:

39.  I figured this stuff out using the null string as input, and frankly, I have a hard time myself feeling hopeful about getting real alignment work out of somebody who previously sat around waiting for somebody else to input a persuasive argument into them.  This ability to "notice lethal difficulties without Eliezer Yudkowsky arguing you into noticing them" currently is an opaque piece of cognitive machinery to me, I do not know how to train it into others.  It probably relates to 'security mindset', and a mental motion where you refuse to play out scripts, and being able to operate in a field that's in a state of chaos.

You cannot actually do this and hope to get any quality of reasoning, for the same reason that you can't update on nothing/no evidence.

The data matters way more than you think, and there's no algorithm that can figure out stuff with 0 data, and Eric Drexler didn't figure out nanotechnology using the null string as input.

This should have been a much larger red flag for problems, but people somehow didn't realize how wrong this claim was.

And that's the end of my very long comment on the problems with this post.

Mod note: I activated two-axis voting on this post, since it seemed like it would make the conversation go better.

If someone could find a way to rewrite this post, except in language comprehensible to policymakers, tech executives, or ML researchers, then it would probably achieve a lot.

Eliezer, thanks for sharing these ideas so that more people can be on the lookout for failures.  Personally, I think something like 15% of AGI dev teams (weighted by success probability) would destroy the world more-or-less immediately, and I think it's not crazy to think the fraction is more like 90% or higher (which I judge to be your view).

FWIW, I do not agree with the following stance, because I think it exposes the world to more x-risk:

So far as I'm concerned, if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty percent change of killing more than one billion people, I'll take it. 

Specifically, I think a considerable fraction of the remaining AI x-risk facing humanity stems from people pulling desperate (unsafe) moves with AGI to head off other AGI projects.  So, in that regard, I think that particular comment of yours is probably increasing x-risk a bit.  If I were a 90%-er like you, it's possible I'd endorse it, but even then it might make things worse by encouraging more desperate unilateral actions.

That said, overall I think this post is a big help, because it helps to put responsibility in the hands of more people to not do the crazy/stupid/reckless things you're describing here... and while I might disagree on the fraction/probability, I agree that some groups would destroy humanity more or less immediately if they developed AGI.  And, while I might disagree on some of the details of how human extinction eventually plays out, I do think human extinction remains the default outcome of humanity's path toward replacing itself with automation, probably within our lifetimes unfortunately.

Thanks Eliezer for writing up this list, it's great to have these arguments in one place! Here are my quick takes (which mostly agree with Paul's response). 

Section A (strategic challenges?):

Agree with #1-2 and #8. Agree with #3 in the sense that we can't iterate in dangerous domains (by definition) but not in the sense that we can't learn from experiments on easier domains (see Paul's Disagreement #1). 

Mostly disagree with #4 - I think that coordination not to build AGI (at least between Western AI labs) is difficult but feasible, especially after a warning shot. A single AGI lab that decides not to build AGI can produce compelling demos of misbehavior that can help convince other actors. A number of powerful actors coordinating not to build AGI could buy a lot of time, e.g. through regulation of potential AGI projects (auditing any projects that use a certain level of compute, etc) and stigmatizing deployment of potential AGI systems (e.g. if it is viewed similarly to deploying nuclear weapons). 

Mostly disagree with the pivotal act arguments and framing (#6, 7, 9). I agree it is necessary to end the acute risk period [LW(p) · GW(p)], but I find it unhelpful when this is framed as "a pivotal act", which assumes it's a single action taken unilaterally [LW · GW] by a small number of people or an AGI system. I think that human coordination (possibly assisted by narrow AI tools, e.g. auditing techniques) can be sufficient to prevent unaligned AGI from being deployed. While it's true that a pivotal act requires power and an AGI wielding this power would pose an existential risk, a group of humans + narrow AI wielding this power would not. This may require more advanced narrow AI than we currently have, so opportunities for pivotal acts could arise as we get closer to AGI that are not currently available. 

Mostly disagree with section B.1 (distributional leap):

Agree with #10 - the distributional shift is large by default. However, I think there is a decent chance that we can monitor the increase in system capabilities and learn from experiments on less advanced systems, which would allow us to iterate alignment approaches to deal with the distributional shift. 

Disagree with #11 - I think we can learn from experiments on less dangerous domains (see Paul's Disagreement #15).  

Uncertain on #13-14. I agree that many problems would most naturally first occur at higher levels of intelligence / in dangerous domains. However, we can discover these problems through thought experiments and then look for examples in less advanced systems that we would not have found otherwise (e.g. this worked for goal misgeneralization and reward tampering). 

Mostly agree with B.2 (central difficulties):

Agree with #17 that there is currently no way to instill and verify specific inner properties in a system, though it seems possible in principle with more advanced interpretability techniques.  

Agree with #21 that capabilities generalize further than alignment by default. Addressing this would require methods for modeling and monitoring system capabilities, which would allow us to stop training the system before capabilities start generalizing very quickly. 

I mostly agree with #23 (corrigibility is anti-natural), though I think there are ways to make corrigibility more of an attractor, e.g. through utility uncertainty or detecting and penalizing incorrigible reasoning. Paul's argument on corrigibility being a crisp property [LW(p) · GW(p)] assuming good enough human feedback also seems compelling. 

I agree with #24 that it's important to be clear whether an approach is aiming for a sovereign or corrigible AI, though I haven't seen people conflating these in practice.

Mostly disagree with B.3 (interpretability):

I think Eliezer is generally overly pessimistic about interpretability.

Agree with #26 that interpretability alone isn't enough to build a system that doesn't want to kill us. However, it would help to select against such systems, and would allow us to produce compelling demos of misalignment that help humans coordinate to not build AGI. 

Agree with #27 that training with interpretability tools could also select for undetectable deception, but it's unclear how much this is a problem in practice. It's plausibly quite difficult to learn to perform undetectable deception without first doing a bunch of detectable deception that would then be penalized and selected against, producing a system that generally avoids deception. 

Disagree with #30 - the argument that verification is much easier than generation is pretty compelling (see Paul's Disagreement #19). 

Disagree with #33 that an AGI system will have completely alien concepts / world model. I think this relies on the natural abstraction hypothesis being false, which seems unlikely. 

Section B.4 (miscellaneous unworkable schemes) and Section C (civilizational inadequacy?)

Uncertain on these arguments, but they don't seem load-bearing to me. 

Since Divia said, and Eliezer retweeted, that good things might happen if people give their honest, detailed reactions:

My honest, non-detailed reaction is AAAAAAH. In more detail -

1. Yup, this seems right.
2. This is technobabble to me, since I don't actually understand nanomachines, but it makes me rather more optimistic about my death being painless than my most likely theory, which is that a superhuman AI takes over first and has better uses for our atoms later.
3. (If we had unlimited retries - if every time an AGI destroyed all the galaxies we got to go back in time four years and try again - we would in a hundred years figure out which bright ideas actually worked.) My brain immediately starts looking for ways to set up some kind of fast testing for ways to do this in a closed, limited world without letting it know ours exists... which is already answered below, under 10. Yup, doomed.
4. And then we all died.
5. Yup.
6. I imagine it would be theoretically - but not practically - possible to fire off a spaceship accelerating fast enough (that is, with enough lead time) that it could outrun the AI and so escape an Earth about to be eaten by an AI (a pivotal act well short of melting all CPUs that would save at least a part of humanity), but that given that the AI could probably take over the ship just by flashing lights at it, that seems unlikely to actually work in practice.
7. I think the closest thing I get to a "pivotal weak act" would be persuading everyone to halt all AI research with a GPT-5 that can be superhumanly persuasive at writing arguments to persuade humans, but doesn't yet have a model of the world-as-real-and-affecting-it that it could use to realize that it could achieve its goals by taking over the world, but I don't actually expect this would work - that would be a very narrow belt of competence and I'm skeptical it could be achieved.
8. Not qualified to comment.
9. Seems right.
10. Yeah, we're doomed.
11. Doomed.
12. Seems right to me. If the AI never tries a plan because it correctly knows it won't work, this doesn't tell you anything about the AI not trying a plan when it would work.
13. "It's not that we can't roll one twenty, it's that we'll roll a one eventually." I don't think humanity has successfully overcome this genre of problem, and we encounter it a lot. (In practice, our solutions are fail-safe systems, requiring multiple humans to concur to do anything, and removing these problems from people's environments, none of which really work in context.)
14. Doomed.
15. Yup, doomed.
16. I'd also add a lot of "we have lots of experience with bosses trying to make their underlings serve them instead of serving themselves and none of them really work", as more very weak evidence in the same direction.
17. Doomed.
18. Doomed.
19. Not qualified to discuss this.
20. We are really very doomed, aren't we.
21. This seems very logical and probably correct, both about the high-level points Eliezer makes and the history of human alignment with other humans.
22. Seems valid.
23. Not qualified to comment.
24. You know, I'd take something that was imperfectly aligned with my Real Actual Values as long as it gave me enough Space Heroin, if the alternative was death. I'd rather the thing aligned with my Real Actual Values, but if we can't manage that, Space Heroin seems better than nothing. (Also, yup, doomed.)
25. Not qualified to comment.
26. This seems valid but I don't know enough about current AI to comment.
27. Good point!
28. Yup.
29. Yup.
30. Yup.
31. Yup.
32. Good point.
33. Doomed.
34. We do seem doomed, yup.
35. Doomed.
36. Indeed, humans already work this way!
37. This is a good point about social dynamics but does not immediately make me go 'we're all doomed', I think because social dynamics seem potentially contingent.
38. You're the expert and I'm not; I don't know the field well enough to comment.
39. No comment.
40. No comment; this seems plausible but I don't know enough to say.
41. No comment.
42. No comment.
43. No comment.

As a bystander who can understand this, and find the arguments and conclusions sound, I must say I feel very hopeless and "kinda" scared at this point. I'm living in at least an environment, if not a world, where even explaining something comparatively simple like how life extension is a net good is a struggle. Explaining or discussing this is definitely impossible - I've tried with the cleverer, more transhumanistic/rationalistic minded people I know, and it just doesn't click for them, to the contrary, I find people like to push in the other direction, as if it were a game.

And at the same time, I realize it is unlikely I can contribute anything remotely significant to a solution myself. So I can only spectate. This is literally maddening, especially so when most everyone seems to underreact.

That requires, not the ability to read this document and nod along with it, but the ability to spontaneously write it from scratch without anybody else prompting you; that is what makes somebody a peer of its author.  It's guaranteed that some of my analysis is mistaken, though not necessarily in a hopeful direction.  The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I published it, which nobody apparently did, despite my having had other things to do than write this up for the last five years or so.  Some of that silence may, possibly, optimistically, be due to nobody else in this field having the ability to write things comprehensibly - such that somebody out there had the knowledge to write all of this themselves, if they could only have written it up, but they couldn't write, so didn't try.  I'm not particularly hopeful of this turning out to be true in real life, but I suppose it's one possible place for a "positive model violation" (miracle).  The fact that, twenty-one years into my entering this death game, seven years into other EAs noticing the death game, and two years into even normies starting to notice the death game, it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that.  I knew I did not actually have the physical stamina to be a star researcher, I tried really really hard to replace myself before my health deteriorated further, and yet here I am writing this.  That's not what surviving worlds look like.

Something bugged me about this paragraph, until I realized: If you actually wanted to know whether or not this was true, you could have just asked Nate Soares, Paul Christiano, or anybody else you respected to write this post first, then removed all doubt by making a private comparison. If you had enough confidence in the community you could have even made it into a sequence; gather up all of the big alignment researchers' intuitions on where the Filters are and then let us make our own opinion up on which was most salient. 

Instead, now we're in a situation where, I expect, if anybody writes something basically similar you will just posit that they can't really do alignment research because they couldn't have written it "from the null string" like you did. Doing this would literally have saved you work on expectation, and it seems obvious enough for me to be suspicious as to why you didn't think of it.

Could I put in a request to see a brain dump from Eliezer of ways to gain dignity points?

Lots I disagree with here, so let's go through the list.

There are no pivotal weak acts.

Strong disagree.  

EY and I don't seem to agree that "nuke every semiconductor fab" is a weakly pivotal act (since I think AI is hardware-limited and he thinks it is awaiting a clever algorithm).  But I think even "build nanobots that melt every GPU" could be built using an AI that is aligned in the "less than 50% chance of murdering us all" sense.  For example, we could simulate [LW · GW]a bunch of human-level scientists trying to build nanobots and also checking each-other's work.

On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions.

Nope.  I think that you could build a useful AI (e.g. the hive of scientists) without doing any out-of-distribution stuff.  

there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment

I am significantly more optimistic about explainable AI than EY.

There is no analogous truth about there being a simple core of alignment

I do not consider this at all obvious [LW · GW].  

Corrigibility is anti-natural to consequentialist reasoning

Roll to disbelief.  Cooperation is a natural equilibrium in many games.

you can't rely on behavioral inspection to determine facts about an AI which that AI might want to deceive you about

Sure you can.  Just train an AI that "wants" to be honest.  This probably means training an AI with the objective function "accurately predict reality" and then using it to do other things (like make paperclips) rather than training it with an objective function "make paperclips".

Coordination schemes between superintelligences are not things that humans can participate in

I don't think this is as relevant as EY does.  Even if it's true that unaugmented humans are basically irrelevant to an economy of superintelligent AIs, that doesn't mean we can't have a future where augmented or tool-AI assisted humans can have meaningful influence.

Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you're playing them against each other

I believe there is an intermediate level of AI between "utterly useless" and "immediately solves the acausal trading problem and begins coordinating perfectly against humans".  This window may be rather wide.

What makes an air conditioner 'magic' from the perspective of say the thirteenth century, is that even if you correctly show them the design of the air conditioner in advance, they won't be able to understand from seeing that design why the air comes out cold

I'm virtually certain I could explain to Aristotle or DaVinci how an air-conditioner works.

There's a pattern that's played out quite often, over all the times the Earth has spun around the Sun, in which some bright-eyed young scientist, young engineer, young entrepreneur, proceeds in full bright-eyed optimism to challenge some problem that turns out to be really quite difficult.  Very often the cynical old veterans of the field try to warn them about this, and the bright-eyed youngsters don't listen, because, like, who wants to hear about all that stuff, they want to go solve the problem!

There's also a pattern where the venerable scientist is proven wrong by the young scientist too foolish to know what they are doing is impossible.

There's no plan.

There is at least one plan [LW · GW].

This situation you see when you look around you is not what a surviving world looks like

Currently Metaculus estimates 55% chance for "Will there be a positive transition to a world with radically smarter-than-human artificial intelligence?".  Admitted I would like this to be higher, but at the minimum this is what a world that "might survive" looks like.  I have no particular reason [LW · GW] to trust EY vs Metaculus.

I suspect EY and I both agree that if you take existing Reinforcement Learning Architectures, write down the best utility function humans can think of, and then turn the dial up to 11, bad things will happen.  EY seems to believe this is a huge problem because of his belief that "there is no weak pivotal act".  I think this should be taken as a strong warning to not do that.  Rather than scaling architectures that are inherently dangerous, we should focus on making use of architectures that are naturally safe.  For example, EY and I both agree that GPT-N is likely to be safe.  EY simply disagrees with the claim that it might be useful.

EY and I probably also agree that Facebook/Baidu do not have the world's best interest at heart (and are not taking alignment seriously enough or at all).  Hence it is important that people who care about Alignment gain a decisive lead over these efforts.  To me, this logically means that people interested in Alignment should be doing more capabilities research [LW · GW]. To EY, this means that alignment focused institutions need to be using more secrecy.  I'm not utterly opposed to keeping pure-capabilities advancements secret, but if there is a significant overlap between capabilities and alignment, then we need  to be publishing the alignment-relevant bits so that we can cooperate (and hopefully so that Facebook can incorporate them too).

And for completeness, here's a bunch of specific claims by EY I agree with

AGI will not be upper-bounded by human ability or human learning speed.  Things much smarter than human would be able to learn from less evidence than humans require

I think the people who thought this stopped thinking this after move 37.  I hope.

A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure

Strongly agree.  

Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second"

Strongly agree.

We need to get alignment right on the 'first critical try'

Strongly agree.

We can't just "decide not to build AGI"

Strongly agree

Running AGIs doing something pivotal are not passively safe

Agree.  But I don't think this means they are totally unworkable either.

Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn't kill you

Agree.  But I think the lesson here is "don't use powerful AIs until you are sure they are aligned".

Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level

Agree somewhat.  But I don't rule out that "cooperative" or "interesting" is a natural attractor.

Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously

Agree, conditional on our definition of fast.  I think that within a year of training our first "smart human" AI, we can simulate "100 smart humans" using a similar compute budget.  I don't think Foom takes us from "human level AI" to "smarter than all humans AI" in a few minutes simply be rewriting code.

outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction

Agree.  This is why I am skeptical [LW · GW]of utility-functions in general as a method for aligning AI.

Human raters make systematic errors - regular, compactly describable, predictable errors

Duh.

The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our Real Actual Values is far out of reach for our FIRST TRY at AGI.

I am really not very optimistic about CEV.

A powerful AI searches parts of the option space we don't, and we can't foresee all its options.

Yes.

This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents

Agree.  But I don't think you need to make an AI that imitates humans in order to make an AI that is useful.  For example, Codex allows me to write code significantly (2-5x) faster, despite frequently making dumb mistakes.

The AI does not think like you do

Yes.

AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.

Mostly agree.  I think there exist architectures of AI that can be boxed.

You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them.

I think the best approach to funding AI safety is something like Fast Grants where we focus more on quantity than on "quality" since it is nearly impossible to identify who will succeed in advance.

If there was one thing that I could change in this essay, it would be to clearly outline that the existence of nanotechnology advanced enough to do things like melt GPUs isn't necessary even if it is sufficient for achieving singleton status and taking humanity off the field as a meaningful player.

Whenever I see people fixate on critiquing that particular point, I need to step in and point out that merely existing tools and weapons (is there a distinction?) suffice for a Superintelligence to be able to kill the vast majority of humans and reduce our threat to it to negligible levels. Be that wresting control of nuclear arsenals to initiate MAD or simply extrapolating on gain-of-function research to produce extremely virulent yet lethal pathogens that can't be defeated before the majority of humans are infected, such options leave a small minority of humans alive to cower in the wreckage until the biosphere is later dismantled.

That's orthogonal to the issue of whether such nanotechnology is achievable for a Superintelligent AGI, it merely reduces the inferential distance the message has to be conveyed as it doesn't demand familiarity with Drexler.

(Advanced biotechnology already is nanotechnology, but the point is that no stunning capabilities need to be unlocked for an unboxed AI to become immediately lethal)

While I share a large degree of pessimism for similar reasons, I am somewhat more optimistic overall.  

Most of this comes from generic uncertainty and epistemic humility; I'm a big fan of the inside view, but it's worth noting that this can (roughly) be read as a set of 42 statements that need to be true for us to in fact be doomed, and statistically speaking it seems unlikely that all of these statements are true.

However, there are some more specific points I can point to where I think you are overconfident, or at least not providing good reasons for such a high level of confidence (and to my knowledge nobody has).  I'll focus on two disagreements which I think are closest to my true disagreements.

1) I think safe pivotal "weak" acts likely do exist.  It seems likely that we can access vastly superhuman capabilities without inducing huge x-risk using a variety of capability control methods.  If we could build something that was only N<<infinity times smarter than us, then intuitively it seems unlikely that it would be able to reverse engineer details of the outside world or other AI systems source code (cf 35) necessary to break out of the box or start cooperating with its AI overseers.  If I am right, then the reason nobody has come up with one is because they aren't smart enough (in some -- possibly quite narrow -- sense of smart); that's why we need the superhuman AI!  Of course, it could also be that someone has such an idea, but isn't sharing it publicly / with Eliezer.

2) I am not convinced that any superhuman AGI we are likely to have the technical means to build in the near future is going to be highly consequentialist (although this does seem likely).  I think that humans aren't actually that consequentialist, current AI systems even less so, and it seems entirely plausible that you don't just automatically get super consequentialist things no matter what you are doing or how you are training them... if you train something to follow commands in a bounded way using something like supervised learning, maybe you actually end up with something that does something reasonably close to that.  My main reason for expecting consequentialist systems at superhuman-but-not-superintelligent-level AGI is that people will build them that way because of competitive pressures, not because systems that people are trying to make non-consequentialist end up being consequentialist.  

These two points are related: If we think consequentialism is unavoidable (RE 2), then we should be more skeptical that we can safely harness the power of superhuman capabilities at all (RE 1), although we could still hope to use capability control and incentive schemes to harness a superhuman-but-not-superintelligent consequentialist AGI to devise and help execute "weak" pivotal acts.

3) Maybe one more point worth mentioning is the "alien concepts" bit: I also suspect AIs will have alien concepts and thus generalize in weird ways.  Adversarial examples and other robustness issues are evidence in favor of this, but we are also seeing that scaling makes models more robust, so it seems plausible that AGI will actually end up using similar concepts to humans, thus making generalizing in the ways we intend/expect natural for AGI systems.


---------------------------------------------------------------------
The rest of my post is sort of just picking particular places where I think the argumentation is weak, in order to illustrate why I currently think you are, on net, overconfident.

 7.  The reason why nobody in this community has successfully named a 'pivotal weak act' where you do something weak enough with an AGI to be passively safe, but powerful enough to prevent any other AGI from destroying the world a year later - and yet also we can't just go do that right now and need to wait on AI - is that nothing like that exists. 

This contains a dubious implicit assumption, namely: we cannot build safe super-human intelligence, even if it is only slightly superhuman, or superhuman in various narrow-but-strategically-relevant areas.

19.  More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment - to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.  

This basically what CIRL aims to do.  We can train for this sort of thing and study such methods of training empirically in synthetic settings.

23.  Corrigibility is anti-natural to consequentialist reasoning 

Maybe I missed it, but I didn't see any argument for why we end up with consequentialist reasoning.

30. [...] There is no pivotal output of an AGI that is humanly checkable and can be used to safely save the world but only after checking it; this is another form of pivotal weak act which does not exist.

It seems like such things are likely to exist by analogy with complexity theory (checking is easier than proposing).

36. AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.

I figured it was worth noting that this part doesn't explicitly say that relatively weak AGIs can't perform pivotal acts.  

Thanks for writing this, I agree that people have underinvested in writing documents like this. I agree with many of your points, and disagree with others. For the purposes of this comment, I'll focus on a few key disagreeements.

My model of this variety of reader has an inside view, which they will label an outside view, that assigns great relevance to some other data points that are not observed cases of an outer optimization loop producing an inner general intelligence, and assigns little importance to our one data point actually featuring the phenomenon in question. Consider skepticism, if someone is ignoring this one warning, especially if they are not presenting equally lethal and dangerous things that they say will go wrong instead.

There are some ways in which AGI will be analogous to human evolution. There are some ways in which it will be disanalogous. Any solution to alignment will exploit at least one of the ways in which it's disanalogous. Pointing to the example of humans without analysing the analogies and disanalogies more deeply doesn't help distinguish between alignment proposals which usefully exploit disanalogies, and proposals which don't.

Alpha Zero blew past all accumulated human knowledge about Go after a day or so of self-play, with no reliance on human playbooks or sample games.

It seems useful to distinguish between how fast any given model advances during training, and how fast the frontier of our best models advances.  AlphaZero seems like a good example of why we should expect the former to be fast; but for automated oversight techniques, the latter is more relevant.

if a textbook from one hundred years in the future fell into our hands, containing all of the simple ideas that actually work robustly in practice, we could probably build an aligned superintelligence in six months.

Maybe one way to pin down a disagreement here: imagine the minimum-intelligence AGI that could write this textbook (including describing the experiments required to verify all the claims it made) in a year if it tried. How many Yudkowsky-years does it take to safely evaluate whether following a textbook which that AGI spent a year writing will kill you?

This situation you see when you look around you is not what a surviving world looks like.  The worlds of humanity that survive have plans.

It would be great to have a well-justified plan at this point. But I think you're also overestimating the value of planning, in a way that's related to you using the phrase "miracle" to mean "positive model violation". Nobody throughout human history has ever had a model of the future accurate enough to justify equivocating those two terms. Every big scientific breakthrough is a model violation to a bunch of geniuses who have been looking at the problem really hard, but not quite at the right angle. This is why I pushed you, during our debates, to produce predictions rather than postdictions, so that I could distinguish you from all the other geniuses who ran into big model violations.

Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction.

Humans haven't been optimized to pursue inclusive genetic fitness for very long, because humans haven't been around for very long. Instead they inherited the crude heuristics pointing towards inclusive genetic fitness from their cognitively much less sophisticated predecessors. And those still kinda work!

If we are still around in a couple of million years I wouldn't be surprised if there was inner alignment in the sense that almost all humans in almost all practically encountered environments end up consciously optimising inclusive genetic fitness. 

More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment - to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.

Generally, I think that people draw the wrong conclusions from mesa-optimisers and the examples of human evolutionary alignment. 

Saying that we would like to solve alignment by specifying exactly what we want and then let the AI learn exactly what we want, is like saying that we would like to solve transportation by inventing teleportation. Yeah, would be nice but unfortunately it seems like you will have to move through space instead. 

The conclusion we should take from the concept of mesa-optimisation isn't "oh no alignment is impossible", that's equivalent to "oh no learning is impossible". But learning is possible. So the correct conclusion is "alignment has to work via mesa-optimisation". 

Because alignment in the human examples (i.e. human alignment to evolution's objective and humans alignment to human values) works by bootstrapping from incredibly crude heuristics. Think three dark patches for a face. 

Humans are mesa-optimized to adhere to human values. If we were actually inner aligned to the crude heuristics that evolution installed in us for bootstrapping the entire process, we would be totally disfunctional weirdoes. 

I mean even more so ...

To me the human examples suggest that there has to be a possibility to get from gesturing at what we want to getting what we want. And I think we can gesture a lot better than evolution! Well, at least using much more information than 3.2 billion base pairs. 

If alignment has to be a bootstrapped open ended learning process there is also the possibility that it will work better with more intelligent systems or really only start working with fairly intelligent systems. 

Maybe bootstrapping with cake, kittens and cuddles will still get us paperclipped, I don't know. It certainly seems awfully easy to just run straight off a cliff. But I think looking at the only known examples of alignment of intelligences does allow us more optimistic takes than are prevalent on this page. 

↑ comment by Rob Bensinger (RobbBB) · 2022-06-08T05:42:57.015Z · LW(p) · GW(p)

The conclusion we should take from the concept of mesa-optimisation isn't "oh no alignment is impossible", that's equivalent to "oh no learning is impossible".

The OP isn't claiming that alignment is impossible.

If we were actually inner aligned to the crude heuristics that evolution installed in us for bootstrapping the entire process, we would be totally disfunctional weirdoes. 

I don't understand the point you're making here.

Replies from: p.b.

↑ comment by p.b. · 2022-06-08T07:33:21.400Z · LW(p) · GW(p)

The point I'm making is that the human example tells us that: 

If first we realize that we can't code up our values, therefore alignment is hard. Then, when we realize that mesa-optimisation is a thing. we shouldn't update towards "alignment is even harder". We should update in the opposite direction. 

Because the human example tells us that a mesa-optimiser can reliably point to a complex thing even if the optimiser points to only a few crude things. 

But I only ever see these three points, human example, inability to code up values, mesa-optimisation to separately argue for "alignment is even harder than previously thought". But taken together that is just not the picture. 

Replies from: Eliezer_Yudkowsky, quintin-pope

↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2022-06-08T07:38:25.929Z · LW(p) · GW(p)

Humans point to some complicated things, but not via a process that suggests an analogous way to use natural selection or gradient descent to make a mesa-optimizer point to particular externally specifiable complicated things.

Replies from: TurnTrout, david-johnston

↑ comment by TurnTrout · 2022-06-08T17:34:53.454Z · LW(p) · GW(p)

Why do you think that? Why is the process by which humans come to reliably care about the real world, not a process we could leverage analogously to make AIs care about the real world? 

Likewise, when you wrote,

This isn't to say that nothing in the system’s goal (whatever goal accidentally ends up being inner-optimized over) could ever point to anything in the environment by accident. 

Where is the accident? Did evolution accidentally find a way to reliably orient terminal human values towards the real world? Do people each, individually, accidentally learn to terminally care about the real world? Because the former implies the existence of a better alignment paradigm (that which occurs within the human brain, to take an empty-slate human and grow them into an intelligence which terminally cares about objects in reality), and the latter is extremely unlikely. Let me know if you meant something else.

EDIT: Updated a few confusing words.

Replies from: RobbBB, Vaniver, lc

↑ comment by Rob Bensinger (RobbBB) · 2022-06-08T19:41:31.814Z · LW(p) · GW(p)

Why is the process by which humans come to reliably care about the real world, not a process we could leverage analogously to make AIs care about the real world? 

Maybe I'm not understanding your proposal, but on the face of it this seems like a change of topic. I don't see Eliezer claiming 'there's no way to make the AGI care about the real world vs. caring about (say) internal experiences in its own head'. Maybe he does think that, but mostly I'd guess he doesn't care, because the important thing is whether you can point the AGI at very, very specific real-world tasks.

Where is the accident? Did evolution accidentally find a way to reliably orient people towards the real world? Do people each, individually, accidentally learn to care about the real world?

Same objection/confusion here, except now I'm also a bit confused about what you mean by "orient people towards the real world". Your previous language made it sound like you were talking about causing the optimizer's goals to point at things in the real world, but now your language makes it sound like you're talking about causing the optimizer to model the real world or causing the optimizer to instrumentally care about the state of the real world....? Those all seem very different to me.

Or, in summary, I'm not seeing the connection between:

• "Terminally valuing anything physical at all" vs. "terminally valuing very specific physical things".
• "Terminally valuing anything physical at all" vs. "instrumentally valuing anything physical at all".
• "Terminally valuing very specific physical things" vs. "instrumentally valuing very specific physical things".
• Any of the above vs. "modeling / thinking about physical things at all", or "modeling / thinking about very specific physical things".

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-08T20:10:36.672Z · LW(p) · GW(p)

Hm, I'll give this another stab. I understand the first part of your comment as "sure, it's possible for minds to care about reality, but we don't know how to target value formation so that the mind cares about a particular part of reality." Is this a good summary? 

I don't see Eliezer claiming 'there's no way to make the AGI care about the real world vs. caring about (say) internal experiences in its own head'.

Let me distinguish three alignment feats:

1. Producing a mind which terminally values sensory entities. 
2. Producing a mind which reliably terminally values some kind of non-sensory entity in the world, like dogs or bananas. 
   1. AFAIK we have no idea how to ensure this happens reliably -- to produce an AGI which terminally values some element of {diamonds, dogs, cats, tree branches, other real-world objects}, such that there's a low probability that the AGI actually just cares about high-reward sensory observations. 
   2. In other words: Design a mind which cares about anything at all in reality which isn't a shallow sensory phenomenon which is directly observable by the agent. Like, maybe I have a mind-training procedure, where I don't know what the final trained mind will value (dogs, diamonds, trees having particular kinds of cross-sections at year 5 of their growth), but I'm damn sure the AI will care about something besides its own sensory signals. 
   3. I was, first, pointing out that this problem has to be solvable, since the human genome solves it millions of times every day! 
3. Producing a mind which reliably terminally values a specific non-sensory entity, like diamonds.
   1. Design a mind which cares about a particular kind of object. We could target the mind-training process to care about diamonds, or about dogs, or about trees, but to solve this problem, we have to ensure the trained mind significantly cares about one kind of real-world entity in particular. Therefore, feat #3 is strictly harder than feat #2.
   2. This is what you point out as a potential crux.

(EDIT: Added a few sub-points to clarify list.)

From my shard theory document: 

We (alignment researchers) have had no idea how to actually build a mind which intrinsically (not instrumentally!) values a latent, non-sensory object in the real world. Witness the confusion on this point in Arbital’s ontology identification article. 

To my knowledge, we still haven't solved this problem. We have no reward function to give AIXI which makes AIXI maximize real-world diamonds. A deep learning agent might learn to care about the real world, yes, but it might learn sensory preferences instead. Ignorance about the outcome is not a mechanistic account of why the agent convergently will care about specific real-world objects instead of its sensory feedback signals. 

Under this account, caring about the real world is just one particular outcome among many. Hence, the "classic paradigms" imply that real-world caring is (relatively) improbable.

While we have stories about entities which value paperclips, I do not think we have known how to design them. Nor have we had any mechanistic explanation for why people care about the real world in particular.

As you point out, we obviously need to figure problem 3 out in order to usefully align an AGI. I will now argue that the genome solves problem 3, albeit not in the sense of aligning humans with inclusive genetic fitness (you can forget about human/evolution alignment, I won't be discussing that in this comment). 

The genome solves problem #3 in the sense of: if a child grows up with a dog, then that child will (with high probability) terminally value that dog. 

Isn't that an amazing alignment feat!?

Therefore, there has to be a reliable method of initializing a mind from scratch, training it, and having the resultant intelligence care about dogs. Not only does it exist in principle, it succeeds in practice, and we can think about what that method might be. I think this method isn't some uber-complicated alignment solution. The shard theory explanation for dog-value formation is quite simple.

now your language makes it sound like you're talking about causing the optimizer to model the real world or causing the optimizer to instrumentally care about the state of the real world....? Those all seem very different to me.

Nope, wasn't meaning any of these! I was talking about "causing the optimizer's goals to point at things in the real world" the whole time.

Replies from: RobbBB

↑ comment by Rob Bensinger (RobbBB) · 2022-06-08T20:58:36.371Z · LW(p) · GW(p)

I understand the first part of your comment as "sure, it's possible for minds to care about reality, but we don't know how to target value formation so that the mind cares about a particular part of reality." Is this a good summary? 

Yes!

I was, first, pointing out that this problem has to be solvable, since the human genome solves it millions of times every day! 

True! Though everyone already agreed (e.g., EY asserted this in the OP) that it's possible in principle. The updatey thing would be if the case of the human genome / brain development suggests it's more tractable than we otherwise would have thought (in AI).

Seems to me like it's at least a small update about tractability, though I'm not sure it's a big one? Would be interesting to think about the level of agreement between different individual humans with regard to 'how much particular external-world things matter'. Especially interesting would be cases where humans consistently, robustly care about a particular external-world thingie even though it doesn't have a simple sensory correlate.

(E.g., humans developing to care about sex is less promising insofar as it depends on sensory-level reinforcement such as orgasms. Humans developing to care about 'not being in the Matrix / not being in an experience machine' is possibly more promising, because it seems like a pretty common preference that doesn't get directly shaped by sensory rewards.)

3. Producing a mind which reliably terminally values a specific non-sensory entity, like diamonds

Is the distinction between 2 and 3 that "dog" is an imprecise concept, while "diamond" is precise? FWIW, 2 and 3 currently sound very similar to me, if 2 is 'maximize the number of dogs' and 3 is 'maximize the number of diamonds'.

If you could reliably build a dog maximizer, I think that would also be a massive win and would maybe mean that the alignment problem is mostly-solved. (Indeed, I'm inclined to think that's a harder feat than building a diamond maximizer, and I think being able to build a diamond maximizer would also suggest the strawberry-grade alignment problem is mostly solved.)

But maybe I'm misunderstanding 2.

Nope, wasn't meaning any of these! I was talking about "causing the optimizer's goals to point at things in the real world" the whole time.

Cool!

I'll look more at your shards document and think about your arguments here. :)

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-09T01:36:57.140Z · LW(p) · GW(p)

Is the distinction between 2 and 3 that "dog" is an imprecise concept, while "diamond" is precise? FWIW, 2 and 3 currently sound very similar to me, if 2 is 'maximize the number of dogs' and 3 is 'maximize the number of diamonds'.

Feat #2 is: Design a mind which cares about anything at all in reality which isn't a shallow sensory phenomenon which is directly observable by the agent. Like, maybe I have a mind-training procedure, where I don't know what the final trained mind will value (dogs, diamonds, trees having particular kinds of cross-sections at year 5 of their growth), but I'm damn sure the AI will care about something besides its own sensory signals. Such a procedure would accomplish feat #2, but not #3.

Feat #3 is: Design a mind which cares about a particular kind of object. We could target the mind-training process to care about diamonds, or about dogs, or about trees, but to solve this problem, we have to ensure the trained mind significantly cares about one kind of real-world entity in particular. Therefore, feat #3 is strictly harder than feat #2.

If you could reliably build a dog maximizer, I think that would also be a massive win and would maybe mean that the alignment problem is mostly-solved. (Indeed, I'm inclined to think that's a harder feat than building a diamond maximizer

I actually think that the dog- and diamond-maximization problems are about equally hard, and, to be totally honest, neither seems that bad^[1] in the shard theory paradigm. 

Surprisingly, I weakly suspect the harder part is getting the agent to maximize real-world dogs in expectation, not getting the agent to maximize real-world dogs in expectation. I think "figure out how to build a mind which cares about the number of real-world dogs, such that the mind intelligently selects plans which lead to a lot of dogs" is significantly easier than building a dog-maximizer.

1. ^{^}

   I appreciate that this claim is hard to swallow. In any case, I want to focus on inferentially-closer questions first, like how human values form.

↑ comment by Vaniver · 2022-06-08T18:36:29.538Z · LW(p) · GW(p)

Why is the process by which humans come to reliably care about the real world

IMO this process seems pretty unreliable and fragile, to me. Drugs are popular; video games are popular; people-in-aggregate put more effort into obtaining imaginary afterlives than life extension or cryonics.

But also humans have a much harder time 'optimizing against themselves' than AIs will, I think. I don't have a great mechanistic sense of what it will look like for an AI to reliably care about the real world.

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-08T19:01:59.590Z · LW(p) · GW(p)

One of the problems with English is that it doesn't natively support orders of magnitude for "unreliable." Do you mean "unreliable" as in "between 1% and 50% of people end up with part of their values not related to objects-in-reality", or as in "there is no a priori reason why anyone would ever care about anything not directly sensorially observable, except as a fluke of their training process"? Because the latter is what current alignment paradigms mispredict, and the former might be a reasonable claim about what really happens for human beings.

EDIT:  My reader-model is flagging this whole comment as pedagogically inadequate, so I'll point to the second half of section 5 in my shard theory document.

↑ comment by lc · 2022-06-08T17:43:55.557Z · LW(p) · GW(p)

Why do you think that? Why is the process by which humans come to reliably care about the real world, not a process we could leverage analogously to make AIs care about the real world? 

Humans came to their goals while being trained by evolution on genetic inclusive fitness, but they don't explicitly optimize for that. They "optimize" for something pretty random, that looks like genetic inclusive fitness in the training environment but then in this weird modern out-of-sample environment looks completely different. We can definitely train an AI to care about the real world, but his point is that, by doing something analogous to what happened with humans, we will end up with some completely different inner goal than the goal we're training for, as happened with humans.

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-08T17:49:39.239Z · LW(p) · GW(p)

I'm not talking about running evolution again, that is not what I meant by "the process by which humans come to reliably care about the real world." The human genome must specify machinery which reliably grows a mind which cares about reality. I'm asking why we can't use the alignment paradigm leveraged by that machinery, which is empirically successful at pointing people's values to certain kinds of real-world objects.

Replies from: lc

↑ comment by lc · 2022-06-08T18:01:34.601Z · LW(p) · GW(p)

Ah, I misunderstood. 

Well, for starters, because if the history of ML is anything to go by, we're gonna be designing the thing analogous to evolution, and not the brain. We don't pick the actual weights in these transformers, we just design the architecture and then run stochastic gradient descent or some other meta-learning algorithm. That meta-learning algorithm is going to be what decides to go in the DNA, so in order to get the DNA right, we will need to get the meta-learning algorithm correct. Evolution doesn't have much to teach us about that except as a negative example.

But (I think) the answer is similar to this:

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-08T18:14:07.682Z · LW(p) · GW(p)

we're gonna be designing the thing analogous to evolution, and not the brain. We don't pick the actual weights in these transformers, we just design the architecture and then run stochastic gradient descent or some other meta-learning algorithm.

But, ah, the genome also doesn't "pick the actual weights" for the human brain which it later grows. So whatever the brain does to align people to care about latent real-world objects, I strongly believe that that process must be compatible with blank-slate initialization and then learning.

That meta-learning algorithm is going to be what decides to go in the DNA, not some human architect.

In the evolution/mainstream-ML analogy, we humans are specifying the DNA, not the search process over DNA specifications. We specify the learning architecture, and then the learning process fills in the rest. 

 

I confess that I already have a somewhat sharp picture [LW(p) · GW(p)] of the alignment paradigm used by the brain, that I already have concrete reasons to believe it's miles better than anything we have dreamed so far. I was originally querying what Eliezer thinks about the "genome->human alignment properties" situation, rather than expressing innocent ignorance of how any of this works.

Replies from: lc

↑ comment by lc · 2022-06-08T18:22:09.665Z · LW(p) · GW(p)

I think I disagree with you, but I don't really understand what you're saying or how these analogies are being used to point to the real world anymore. It seems to me like you might be taking something that makes the problem of "learning from evolution" even more complicated (evolution -> protein -> something -> brain vs. evolution -> protein -> brain) and using that to argue the issues are solved, in the same vein as the "just don't use a value function" people. But I haven't read shard theory, so, GL.

In the evolution/mainstream-ML analogy, we humans are specifying the DNA, not the search process over DNA specifications.

You mean, we are specifying the ATCG strands, or we are specifying the "architecture" behind how DNA influences the development of the human body? It seems to me like we are definitely also choosing how the search for the correct ATCG strands and how they're identified, in this analogy. The DNA doesn't "align" new babies out of the womb, it's just a specification of how to copy the existing, already """aligned""" code.

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-08T18:55:24.508Z · LW(p) · GW(p)

"learning from evolution" even more complicated (evolution -> protein -> something -> brain vs. evolution -> protein -> brain)

ah, no, this isn't what I'm saying. Hm. Let me try again.

The following is not a handwavy analogy, it is something which actually happened: 

1. Evolution found the human genome. 
2. The human genome specifies the human brain.
3. The human brain learns most of its values and knowledge over time.
4. Human brains reliably learn to care about certain classes of real-world objects like dogs.  

Therefore, somewhere in the "genome -> brain -> (learning) -> values" process, there must be a process which reliably produces values over real-world objects. Shard theory aims to explain this process. The shard-theoretic explanation is actually pretty simple.

Furthermore, we don't have to rerun evolution to access this alignment process. For the sake of engaging with my points, please forget completely about running evolution. I will never suggest rerunning evolution, because it's unwise and irrelevant to my present points. I also currently don't see why the genome's alignment process requires more than crude hard-coded reward circuitry, reinforcement learning, and self-supervised predictive learning. 

Replies from: FireStormOOO

↑ comment by FireStormOOO · 2022-06-09T06:07:04.218Z · LW(p) · GW(p)

That does seem worth looking at and there's probably ideas worth stealing from biology.  I'm not sure you can call that a robustly aligned system that's getting bootstrapped though.  Existing in a society of (roughly) peers and the lack of a huge power disparity between any given person and the rest of humans is anologous to the AGI that can't take over the world yet.  Humans that aquire significant power do not seem aligned wrt what a typical person would profess to and outwardly seem to care about.

I think your point still mostly follows despite that; even when humans can be deceptive and power seeking, there's an astounding amount of regularity in what we end up caring about.

Replies from: TurnTrout

↑ comment by TurnTrout · 2022-06-09T16:29:57.761Z · LW(p) · GW(p)

there's an astounding amount of regularity in what we end up caring about.

Yes, this is my claim. Not that eg >95% of people form values which we would want to form within an AGI.

↑ comment by David Johnston (david-johnston) · 2022-06-08T09:46:31.795Z · LW(p) · GW(p)

Humans can, to some extent, be pointed to complicated external things. This suggests that using natural selection on biology can get you mesa-optimizers that can be pointed to particular externally specifiable complicated things. Doesn't prove it (or, doesn't prove you can do it again), but you only asked for a suggestion.

Replies from: Eliezer_Yudkowsky, RobbBB

↑ comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2022-06-08T22:34:15.628Z · LW(p) · GW(p)

Humans can be pointed at complicated external things by other humans on their own cognitive level, not by their lower maker of natural selection.

Replies from: TurnTrout, david-johnston

↑ comment by TurnTrout · 2022-06-09T00:57:07.761Z · LW(p) · GW(p)

I don't think I understand what, exactly, is being discussed. Are "dogs" or "flowers" or "people you meet face-to-face" examples of "complicated external things"? 

↑ comment by David Johnston (david-johnston) · 2022-06-08T22:50:54.229Z · LW(p) · GW(p)

Right, but the goal is to make AGI you can point at things, not to make AGI you can point at things using some particular technique.

(Tangentially, I also think the jury is still out on whether humans are bad fitness maximizers, and if we're ultimately particularly good at it - e.g. let's say, barring AGI disaster, we'd eventually colonise the galaxy - that probably means AGI alignment is harder, not easier)

↑ comment by Rob Bensinger (RobbBB) · 2022-06-08T19:34:40.780Z · LW(p) · GW(p)

To my eye, this seems like it mostly establishes 'it's not impossible in principle for an optimizer to have a goal that relates to the physical world'. But we had no reason to doubt this in the first place, and it doesn't give us a way to reliably pick in advance which physical things the optimizer cares about. "It's not impossible" is a given for basically everything in AI, in principle, if you have arbitrary amounts of time and arbitrarily deep understanding.

Replies from: david-johnston

↑ comment by David Johnston (david-johnston) · 2022-06-09T02:09:04.963Z · LW(p) · GW(p)

As I said (a few times!) in the discussion about orthogonality, indifference about the measure of "agents" that have particular properties seems crazy to me. Having an example of "agents" that behave in a particular way is a enormously different to having an unproven claim that such agents might be mathematically possible.

↑ comment by Quintin Pope (quintin-pope) · 2022-06-08T07:46:46.675Z · LW(p) · GW(p)

I think this is correct. Shard theory is intended as an account of how inner misalignment produces human values. I also think [LW · GW] that human values aren't as complex or weird as they introspectively appear. 

I read an early draft of this awhile and am glad to have it publicly available.  And I do think the updates in structure/introduction were worth the wait. Thanks!

>There is no pivotal output of an AGI that is humanly checkable and can be used to safely save the world but only after checking it

This is a sort of surprising claim. From an abstract point of view, assuming NP >> P, checking can be way easier than inventing. To stick with your example, it kind of seems, at an intuitive guess, like a plan to use nanobots to melt all GPUs should be very complicated but not way superhumanly complicated? (Superhuman to invent, though.) Like, you show me the plans for the bootstrap nanofactory, the workhorse nanofactory, the standard nanobots, the software for coordinating the nanobots, the software for low-power autonomous behavior, the transportation around the world, the homing in on GPUs, and the melting process. That's really complicated, way more complicated than anything humans have done before, but not by 1000x? Maybe like 100x? Maybe only 10x if you count whole operating systems or scientific fields. Does this seem quantitatively in the right ballpark, and you're saying, that quantitatively large but not crazy amount of checking is infeasible?

To point 4 and related ones, OpenAI has this on their charter page:

We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”

What about the possibility of persuading the top several biggest actors (DeepMind, FAIR, etc.) to agree to something like that?  (Note that they define AGI on the page to mean "highly autonomous systems that outperform humans at most economically valuable work".)  It's not very fleshed out, either the conditions that trigger the pledge or how the transition goes, but it's a start.  The hope would be that someone would make something "sufficiently impressive to trigger the pledge" that doesn't quite kill us, and then ideally (a) the top actors stopping would buy us some time and (b) the top actors devoting their people to helping out (I figure they could write test suites at minimum) could accelerate the alignment work.

I see possible problems with this, but is this at least in the realm of "things worth trying"?

+9. This is a powerful set of arguments pointing out how humanity will literally go extinct soon due to AI development (or have something similarly bad happen to us). A lot of thought and research went into an understanding of the problem that can produce this level of understanding of the problems we face, and I'm extremely glad it was written up.

Is there a plausible pivotal act that doesn't amount to some variant of "cripple human civilization so that it can't make or use computers until it recovers"?

Wow, 510 karma and counting. This post currently has the 14th most karma all time and most for this year. Makes me think back to this excerpt from Explainers Shoot High. Aim Low! [LW · GW].

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual. He had thought it would be useful to academics outside the field, or even reporters. This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

The lesson was not that his fellow scientists were stupid, but that we tend to enormously underestimate the effort required to properly explain things.

I'm confused about A6, from which I get "Yudkowsky is aiming for a pivotal act to prevent the formation of unaligned AGI that's outside the Overton Window and on the order of burning all GPUs". This seems counter to the notion in Q4 of Death with Dignity [LW · GW] where Yudkowsky says

It's relatively safe to be around an Eliezer Yudkowsky while the world is ending, because he's not going to do anything extreme and unethical unless it would really actually save the world in real life, and there are no extreme unethical actions that would really actually save the world the way these things play out in real life, and he knows that.  He knows that the next stupid sacrifice-of-ethics proposed won't work to save the world either, actually in real life. 

I would estimate that burning all AGI-capable compute would disrupt every factor of the global economy for years and cause tens of millions of deaths^[1], and that's what Yudkowsky considers the more mentionable example. Do the other options outside the Overton Window somehow not qualify as unsafe/extreme unethical actions (by the standards of the audience of Death with Dignity)? Has Yudkowsky changed his mind on what options would actually save the world? Does Yudkowsky think that the chances of finding a pivotal act that would significantly delay unsafe AGI are so slim that he's safe to be around despite him being unsafe in the hypothetical that such a pivotal act is achievable? I'm confused.

Also, I'm not sure how much overlap there is between people who do Bayesian updates and people for who whatever Yudkowsky is thinking of is outside the Overton Window, but in general, if someone says that what they actually want is outside your Overton Window, I see only two directions to update in: either shift your Overton Window to include their intent, or shift your opinion of them to outside your Overton Window. If the first option isn't going to happen, as Yudkowsky says (for public discussion on lesswrong at least), that leaves the second.

1. ^{^}

   Compare modern estimates of the damage that would be caused by a solar flare equivalent to the Carrington Event. Factories, food supply, long-distance communication, digital currency - many critical services nowadays are dependent on compute, and that portion will only increase by the time you would actually pull the trigger.

Having read the original post and may of the comments made so far, I'll add an epistemological observation that I have not seen others make yet quite so forcefully. From the original post:

Here, from my perspective, are some different true things that could be said, to contradict various false things that various different people seem to believe, about why AGI would be survivable [...]

I want to highlight that many of the different 'true things' on the long numbered list in the OP are in fact purely speculative claims about the probable nature of future AGI technology, a technology nobody has seen yet.

The claimed truth of several of these 'true things' is often backed up by nothing more than Eliezer's best-guess informed-gut-feeling predictions about what future AGI must necessarily be like. These predictions often directly contradict the best-guess informed-gut-feeling predictions of others, as is admirably demonstrated in the 2021 MIRI conversations.

Some of Eliezer's best guesses also directly contradict my own best-guess informed-gut-feeling predictions. I rank the credibility of my own informed guesses far above those of Eliezer.

So overall, based on my own best guesses here, I am much more optimistic about avoiding AGI ruin than Eliezer is. I am also much less dissatisfied about how much progress has been made so far.

We need to align the performance of some large task, a 'pivotal act' that prevents other people from building an unaligned AGI that destroys the world. 

What is the argument for why it's not worth pursuing a pivotal act without our own AGI? I certainly would not say it was likely that current human actors could pull it off, but if we are in a "dying with more dignity" context anyway, it doesn't seem like the odds are zero.

My idea, which I'll include more as a demonstration of what I mean than a real proposal, would be to develop a "cause area" for influencing military/political institutions as quickly as possible. Yes, I know this sounds too slow and too hard and a mismatch with the community's skills, but consider:

1. Militaries/governments are "where the money is": they probably do have the coercive power necessary to perform a pivotal act, or at least buy a lot of time. If the PRC is able to completely lock down its giant sophisticated cities, it could probably halt domestic AI research. The West hasn't really tried to do extreme control in a while, for various good reasons, but (just e.g.) the WW2 war economy was awfully tightly managed. We are good at slowing stuff down with crazy red tape. Also there are a lot of nukes
   • Yes, there are lots of reasons this is hard, but remember we're looking for hail marys [LW · GW].
2. "The other guy might develop massive offensive capability soon" is an extremely compelling narrative to normal people, and the culture definitely possesses the meme of "mad scientists have a crazy new weapon". Convincing some generals that we need to shut down TSMC or else China will build terminators might be easier than convincing ML researchers they are doing evil.
   • Sure, if this narrative became super salient, it could possibly lead to a quicker technological arms-race dynamic, but there are other possible dynamics it might lead to, such as (just e.g.) urgency on non-proliferation, or urgency for preemptive military victory using current (non-AGI) tools.
   • I know attempts to get normal people to agree with EA-type thinking have been pretty dispiriting, but I'm not sure how much real energy has gone into making a truly adequate effort, and I think the "military threat" angle might be a lot catchier to the right folks. The "they'll take our jobs" narrative also has a lot of appeal.
   • Importantly, even if convincing people is impossible now, we could prepare for a future regime where we've gotten lucky and some giant smoke alarm event has happened without killing us. You can even imagine both white-hat and black-hat ways of making such an alarm more likely, which might be very high value.
   • Again, remember we're looking for hail marys [LW · GW]. When all you have is an out-of-the-money call option, more volatility is good.
3. The rationalist community's libertarian bent might create a blind spot here. Yes governments and militaries are incredibly dumb, but they do occasionally muddle their way into giant intentional actions.
4. Also with respect to biases, it smells a little bit like we are looking for an "AI-shaped key to unlock an AI-shaped lock", so we should make sure we are putting enough effort into non-AI pivotal actions even if my proposal here is wrong.

Thanks for writing this. I agree with all of these except for #30, since it seems like checking the output of the AI for correctness/safety should be possible even if the AI is smarter than us, just like checking a mathematical proof can be much easier than coming up with the proof in the first place. It would take a lot of competence, and a dedicated team of computer security / program correctness geniuses, but definitely seems within human abilities. (Obviously the AI would have to be below the level of capability where it can just write down an argument that convinces the proof checkers to let it out of the box. This is a sense in which having the AI produce uncommented machine code may actually be safer than letting it write English at us.)

So what should I do with this information, like what other option than "nod along and go on living their lives" is there for me?