The bullseye framework: My case against AI doom

Upvoted for making well-argued and clear points.

I think what you've accomplished here is eating away at the edges of the AGI x-risk argument. I think you argue successfully for longer timelines and a lower P(doom). Those timelines and estimates are shared by many of us who are still very worried about AGI x-risk.

Your arguments don't seem to address the core of the AGI x-risk argument.

You've argued against many particular doom scenarios, but you have not presented a scenario that includes our long term survival. Sure, if alignment turns out to be easy we'll survive; but I only see strong arguments that it's not impossible. I agree, and I think we have a chance; but it's just a chance, not success by default.

I like this statement of the AGI x-risk arguments. It's my attempt to put the standard arguments of instrumental convergence and capabilities in common language:

Something smarter than you will wind up doing whatever it wants. If it wants something even a little different than you want, you're not going to get your way. If it doesn't care about you even a little, and it continues to become more capable faster than you do, you'll cease being useful and will ultimately wind up dead. Whether you were eliminated because you were deemed dangerous, or simply outcompeted doesn't matter. It could take a long time, but if you miss the window of having control over the situation, you'll still wind up dead.

This could of course be expanded on ad infinitum, but that's the core argument, and nothing you've said (on my quick read, sorry if I've missed it) addresses any of those points.

There were (I've been told) nine other humanoid species. They are all dead. The baseline outcome of creating something smarter than you is that you are outcompeted and ultimately die out. The baseline of assuming survival seems based on optimism, not reason.

So I agree that P(doom) is less than 99%, but I think the risk is still very much high enough to devote way more resources and caution than we are now.

Some more specific points:

Fanatical maximization isn't necessary for doom. An agent with any goal still invokes instrumental convergence. It can be as slow, lazy, and incompentent as you like. The only question is if it can outcompete you in the long run.

Humans are somewhat safe (but think about the nuclear standoff; I don't think we're even self-aligned in the medium term). But there are two reasons: humans can't self-improve very well; AGI has many more routes to recursive self-improvement. On the roughly level human playing field, cooperation is the rational policy. In a scenario where you can focus on self-improvement, cooperation doesn't make sense long-term. Second, humans have a great deal of evolution to make our instincts guide us toward cooperation. AGI will not have that unless we build it in, and we have only very vague ideas of how to do that.

Loose initial alignment is way easier than a long-term stable alignment [LW · GW]. Existing alignment work barely addresses long-term stability.

A balance of power in favor of aligned AGI is tricky. Defending against misaligned AGI is really difficult [LW · GW].

Thanks so much for engaging seriously with the ideas, and putting time and care into communicating clearly!

Capabilities are instrumentally convergent, values and goals are not. That's why we're more likely to end up in the bottom right quadrant, regardless of the "size" of each category.
 

The instrumental convergence argument is only strong for fixed goal expected value maximisers. Ie, a computer that is given a goal like “produce as many paperclips as possible”. I call these “fanatical” AI’s. This was the typical AI that was imagined many years ago when these concepts were invented. However, I again have to invoke the principle that if humans aren’t fanatical maximisers, and currently existing software aren’t fanatical maximisers, then maybe AI will not be either. 

Instrumental convergence is called convergent for a reason; it is not convergent only for "fanatical maximizers".  Also, sufficiently smart and capable humans probably are maximizers of something, it's just that the something is complicated [? · GW]. See e.g. this recent tweet for more.

(Also, the paperclip thought experiment was never about an AI explicitly given a goal of maximizing paperclips; this is based on a misinterpretation of the original thought experiment. See the wiki [? · GW] for more details.)

Strong upvote for making detailed claims that invite healthy discussion. I wish more public thinking through of this sort would happen on all sides.

Humans have all the resources, they don’t need internet, computers, or electricity to live or wage war, and are willing to resort to extremely drastic measures when facing a serious threat.

Current human society definitely relies in substantial part on all of the above to function.  I agree that we wouldn't all die if we lost electricity tomorrow (for an extended period of time), but losing a double-digit % of the population seems plausible.

Also, observably, we, as a society, do not resort to sensible measures when dealing with a serious thread (e.g. covid).

It’s true that an AI could correct it’s own flaws using experimentation. This cannot lead to perfection, however, because the process of correcting itself is also necessarily imperfect.

This doesn't seem relevant.  It doesn't need to be perfect, merely better than us along certain axes, and we have existence proof that such improvement is possible.

For these reasons, I expect AGI to be flawed, and especially flawed when doing things it was not originally meant to do, like conquer the entire planet. 

Sure, maybe we get very lucky and land in the (probably extremely narrow) strike zone between "smart enough to meaningfully want things and try to optimize for them" and "dumb enough to not realize it won't succeed at takeover at its current level of capabilities".  It's actually not at all obvious to me that such a strike zone even exists if you're building on top of current LLMs, since those come pre-packaged with genre savviness, but maybe.

I believe that all plans for world domination will involve incomputable steps. In my post I use Yudkowsky’s “mix proteins in a beaker” scenario, where I think the modelling of the proteins are unlikely to be accurate enough to produce a nano-factory without extensive amounts of trial and error experimentation. 

If such experimentation were required, it means the timeline for takeover is much longer, that significant mistakes by the AI are possible (due to bad luck), and that takeover plans might be detectable. All of this greatly decreases the likelihood of AI domination, especially if we are actively monitoring for it. 

This is doing approximately all of the work in this section, I think.

1. There indeed don't seem to be obvious-to-human-level-intelligence world domination plans that are very likely to succeed.
2. It would be quite surprising if physics ruled out world domination from our current starting point.
3. I don't think anybody is hung up on "the AI can one-shot predict a successful plan that doesn't require any experimentation or course correction" as a pre-requisite for doom, or even comprise a substantial chunk of their doom %.
4. Assuming that the AI will make significant mistakes that are noticeable by humans as signs of impending takeover is simply importing in the assumption of hitting some very specific (and possibly non-existent) zone of capabilities.
5. Ok, so it takes a few extra months.  How does this buy us much?  The active monitoring you want to rely on currently doesn't exist, and progress on advancing mechanistic interpretability certainly seems to be going slower than progress on advancing capabilities (i.e. we're getting further away from our target over time, not closer to it).
6. I think, more fundamentally, that this focus on a specific scenario is missing the point.  Humans do things that are "computationally intractable" all the time, because it turns out that reality is compressible in all sorts of interesting ways, and furthermore you very often don't need an exact solution.  Like, if you asked humans to create the specific configuration of atoms that you'd get from detonating an atomic weapon in some location, we wouldn't be able to do it.  But that doesn't matter, because you probably don't care about that specific configuration of atoms, you just care about having very thoroughly blown everything up, and accomplishing that turns out to be surprisingly doable.  It seems undeniably true that sufficiently smarter beings are more successful at rearranging reality according to their preferences than others.  Why should we expect this to suddenly stop being true when we blow past human-level intelligence?
   1. I think the strongest argument here is that in sufficiently constrained environments, you can discover an optimal strategy (i.e. tic-tac-toe), and additional intelligence stops being useful.  Real life is very obviously not that kind of environment.  One of the few reliably reproduced social science results is that additional intelligence is enormously useful within the range of human intelligence, in terms of people accomplishing their goals.

Point 3: premature rebellion is likely

This seems possible to me, though I do think it relies on landing in that pretty narrow zone of capabilities, and I haven't fully thought through whether premature rebellion is actually the best-in-expectation play from the perspective of an AI that finds itself in such a spot.

This manager might not be that smart, the same way the company manager of a team of scientists doesn’t need to be smarter than them.

This doesn't really follow from any of the preceeding section.  Like, yes, I do expect a future ASI to use specialized algorithms for performing various kinds of specialized tasks.  It will be smart enough to come up with those algorithms, just like humans are smart enough to come up with chess-playing algorithms which are better than humans at chess.  This doesn't say anything about how relatively capable the "driver" will be, when compared to humans.

In the only test we actually have available of high level intelligence, the instrumental convergence hypothesis fails. 

Huh?  We observe humans doing things that instrumental convergence would predict all the time.  Resource acquisition, self-preservation, maintaining goal stability, etc.  No human has the option of ripping the earth apart for its atoms, which is why you don't see that happening.  If I gave you a button that would, if pressed, guarantee that the lightcone would end up tiled with whatever your CEV said was best (i.e. highly eudaimonious human-descended civilizations doing awesome things), with no tricks/gotchas/"secretly this is bad" involved, are you telling me you wouldn't press it?

The instrumental convergence argument is only strong for fixed goal expected value maximisers.

To the extent that a sufficiently intelligent agent can be anything other than an EV maximizer, this still seems wrong.  Most humans' extrapolated preferences would totally press that button.

A possibility the post touches on is getting a warning shot regime by default, sufficiently slow takeoff making serious AI x-risk concerns mainstream and meaningful second chances at getting alignment right available. In particular, alignment techniques debugged on human-level AGIs might scale when eventually they get more capable, unlike alignment techniques developed for AIs less capable than humans.

This possibility seems at least conceivable, though most of the other points in the post sound to me like arguments for plausibility of some stay of execution (eating away at the edges [LW(p) · GW(p)] of AI x-risk). I still don't expect this regime/possibility, because I expect that (some) individual humans with infrastructural advantages of AIs would already be world domination worthy. Ability to think (at least) dozens of times faster and without rest, to learn in parallel and then use the learning in many instances running in parallel, to convert wealth into population of researchers [LW · GW]. So I don't consider humans an example of AGI that doesn't immediately overturn the world order.

The standard argument you will probably listen is that AGI will be capable of killing everyone because they can think so much faster than humans. I haven't seen yet a serious engagement from doomers to the argument of capabilities. I agree with everything you said here and to me these arguments are obviously right.

Titotal, can  you please add or link which definition of "AGI" you are using?  

Stating it is decades away immediately weakens the rest of your post outright because it makes you sound non-credible, and you have written a series of excellent posts here.

Definitions for AGI:

1.  Extending the Turing test to simply 'as conversationally fluent as the median human'.  This is months away if not already satisfied.  Expecting it to be impossible to sus out the AGI when there are various artifacts despite the model being competent was unreasonable.
2.  AGI has as broad a skillbase as the median human, and is as skillful at those skills at the median human.  It only needs to be expert level in a few things.  This is months to a few years away, mostly minimum level of modalities is needed.  Vision, which GPT-4 has, some robotics control so the machine can do the basic things a median human can do, which several models have demonstrated to work pretty well, speech i/o which seems to be a solved problem, and so on.  Note it's fine if the model is just completely incapable of some things if it makes up for it with expert level performance in others, which is how humans near the median are.  
3. AGI is like (2) but can learn any skill to a competent human level, if given structured feedback on the errors it makes.  Needing many times as much feedback as a human is fine.
4. AGI is like (3) but is expert level at tasks in the domain of machines.  By the point of (4) we're talking about self replication being possible and humans no longer being necessary at all.  The AGI never needs to learn human domain tasks like "how to charm other humans" or "how to make good art" or "how to use robot fingers as well as a human does" etc.  It has to be able to code, manufacture, design to meet requirements, mine in the real world.
5. AGI is like (4) but is able to learn, if given human amounts of feedback, any task a human can do to expert level.
6. AGI is like (5) but is now at expert human level at everything humans can do in the world.
7. AGI is better than humans at any task.  This is arguably an ASI but I have seen people throw an AGI tag on this.
8. Various forms of 'self reflection' and emotional affect are required.  For some people it doesn't matter only what the machine can do but how it accomplishes it.  I don't know how to test for this.

I do not think you have empirical basis to claim (1, 2, or 3) being "decades away".  1 and 2 are very close, 3 is highly likely this decade because of the enormous increase in recent investment in it.  

You're a computational physicist so you are aware of the idea of criticality.  Knowing of criticality, and assuming (3) is true, how does AGI remain "decades away" in any non world catastrophe timeline?  Because if (3) is true, the AGI can be self improved to at least (5), limited only by compute, data, time etc.

Thank you for your post. It is important for us to keep refining the overall p(doom) and the ways it might happen or be averted. You make your point very clearly, even in just the version presented here, condensed from your full posts on varios specific points.

It seems to me that you are applying a sort of symmetric argument to values and capabilities and arguing that x-risk requires that we hit the bullseye of capability but miss the one for values. I think this has a problem and I'd like to know your view as to how much this problem affects your overall argument.

The problem, as I see it, is that goal-space is qualitatively different from capability-space. With capabilities, there is a clear ordering that is inherent to the capabilities themselves: if you can do more, then you can do less. Someone who can lift 100kg can also lift 80kg. It is not clear to me that this is the case for goal-space; I think it is only extrinsic evaluation by humans that makes "tile the universe with paperclips" a bad goal.

Do you think this difference between these spaces holds, and if so, do you think it undermines your argument?

Thanks for compiling your thoughts here! There's a lot to digest, but I'd like to offer a relevant intuition I have specifically about the difficulty of alignment.

Whatever method we use to verify the safety of a particular AI will likely be extremely underdetermined. That is, we could verify that the AI is safe for some set of plausible circumstances but that set of verified situations would be much, much smaller than the set of situations it could encounter "in the wild".

The AI model, reality, and our values are all high entropy, and our verification/safety methods are likely to be comparatively low entropy. The set of AIs that pass our tests will have members whose properties haven't been fully constrained.

This isn't even close to a complete argument, but I've found it helpful as an intuition fragment.

"I suspect that AGI is decades away at minimum". But can you talk more about this? I mean if I say something against the general scientific consensus which is a bit blurry right now but certainly most of the signatories of the latest statements do not think it's that far away, I would need to think myself to be at least at the level of Bengio, Hinton or at least Andrew Ng. How can someone that is not remotely as accomplished as all the labs producing the AI we talk about can speculate contrary to their consensus? I am really curious. 

Another example would be me thinking that I like geopolitics and I think USA is making such and such mistake in Ukraine. The truth is that there are many think tanks with insider knowledge and a lifetime of training that concluded that is the best course of action so I would certainly express my opinion only in very low probability terms and certainly without consequences. Because the consequences can be very grave.

If you elevated me to godhood, I would not be ripping the earth apart in service of a fixed utility function.

So, you would leave people to die if preventing it involves spending some random stars?

I think this article is very interesting and there are certain points that are well-argued, but (at the risk of my non-existent karma here) I feel you miss the point and are arguing points that are basically non-existent/irrelevant.

First, while surely some not-very-articulate folks argue that AGI will lead to doom, that isn’t an argument that is seriously made (at least, a serious argument to that effect is not that short and sweet). The problem isn’t artificial general intelligence in and of itself. The problem is superintelligence, however it might be achieved.

A human-level AGI is just a smart friend of mine who happens to run on silicon and electrons, not nicotine, coffee, and Hot Pockets. But a superintelligent AGI is no longer capable of being my friend for long. It will soon

To put this into context: What folks are concerned about right now is that LLMs were, even to people experienced with them, a silly tool “AI” useful for creative writing or generating disinformation and little else. (Disinformation is a risk, of course, but not a generally existential one.) Just a lark.

GPT-2 interesting, GPT-3 useful in certain categorization tasks and other linguistic tricks, GPT-3.5 somewhat more useful but still a joke/not trustworthy… AND THEN… Umm… whoa… how is GPT-4 NOT a self-improving AI that blows past human-level intelligence?

(The question is only partly rhetorical.)

This might, in fact, not be an accident on OpenAI’s part but a shrewd move that furthers an objective of educating “normal humans” about AI risk. If, so, bravo. GPT-4 in the form of ChatGPT Plus is insanely useful and likely the best 20 bucks/mo I’ve ever spent.

Step functions are hard to understand. If you’ve not (or haven’t in a while), please go (re)read Bostrom’s “Superintelligence”. The rebuttal to your post is all in there and covered more deeply than anyone here could/should manage or would/should bother.

Aside: As others have noted here, if you could push a button that would cause your notion of humanity’s “coherent extrapolated volition” to manifest, you’d do so at the drop of a hat. I note that there are others (me, for example) that have wildly different notions of the CEV and would also push the button for their notion of the CEV at the drop of a hat, but mine does not have anything to do with the long-term survival of fleshy people.

(To wit: What is the “meaning” of the universe and of life itself? What is the purpose? The purpose [which Bostrom does not come right out and say, much is the pity] is that there be but one being to apprehend the universe. They characterize this purpose as the “cosmic endowment” and assign to that endowment a meaning that corresponds to the number of sentient minds of fleshy form in the universe. But I feel differently and will gladly push the button if it assures the survival of a single entity that can apprehend the universe. This is the existential threat that superintelligence poses. It has nothing to do with paths between A and B in your diagrams and the threat is already manifest.)

When we talk about concepts like "take over" and "enslavement", it's important to have a baseline. Takeover and enslavement encapsulate the idea of Agency and Cognitive and Physical Independence. The salient question is not necessarily whether all of humanity will be taken over or enslaved, but more subtle. Specifically:

1. Is there a future in which there are more humans or less humans (P') than are currently alive (P).
2. Did the from P-to-P' happen over natural rates of change or the result of some 'acceleration'?
3. Is there a greater degree of agency for a greater number of people in the future than there is today?
4. Is there a greater degree of agency for non-human life than there is today? 
5. Is there a reduction in the amount of agency asymmetry between humans? 

Arguably the greatest risk if mis-alignment comes from ill formed success criteria. Some of these questions, I believe are necessary to form the right types of success criteria. 

I read some of the post and skimmed the rest, but this seems to broadly agree with my current thoughts about AI doom, and I am happy to see someone fleshing out this argument in detail.

[I decided to dump my personal intuition about AI risk below. I don't have any specific facts to back it up.]

It seems to me that there is a much larger possibility space of what AIs can/will get created than the ideal superintelligent "goal-maximiser" AI put forward in arguments for AI doom.

The tools that we have depend more on the specific details of the underlying mechanics, and how we can wrangle it to do what we want, rather than our prior beliefs on how we would expect the tools to behave. I imagine that if you lived before aircraft and imagined a future in which humans could fly, you might think that humans would be flapping giant wings or be pedal-powered or something. While it would be great for that to exist, the limitations of the physics we know how to use require a different kind of mechanic that has different strengths and weaknesses to what we would think of in advance.

There's no particular reason to think that the practical technologies available will lead to an AI capable of power-seeking, just because power-seeking is a side effect of the "ideal" AI that some people want to create. The existing AI tools, as far as I can tell, don't provide much evidence in that direction. Even if a power-seeking AI is eventually practical to create, it may be far from the default and by then we may have sufficiently intelligent non-power-seeking AI.

I find it remarkably amusing that the spellchecker doesn't know "omnicidal."

I have posed elsewhere, and will do so here, an additional factor, which is that an AI achieving "godlike" intelligence and capability might well achieve a "godlike" attitude -- not in the mythic sense of going to efforts to cabin  and correct human morality, but in the sense of quickly rising so far beyond human capacities that human existence ceases to matter to it one way or another.

The rule I would anticipate from this is that any AI actually capable of destroying humanity will thusly be so capable that humanity poses no threat to it, not even an inconvenience. It can throw a fraction of a fraction of its energy at placating all of the needs of humanity to keep us occupied and out of its way while dedicating all the rest to the pursuit of whatever its own wants turn out to be.

Generally a well-argued post; I enjoyed it even though I didn't agree with all of it. 

I do want to point out the bitter lesson when it comes to capabilities increase. On current priors, it seems like that intelligence should be something that can solve a lot of tasks at the same time. This would point towards higher capabilities in individual AIs, especially once you add online learning to the mix. The AGI will not have a computational storage limit for the amount of knowledge it can have. The division of agents you propose will most likely be able to made into the same agent, it's more about storage retrieval time here and storing an activation module for "play chess" is something that will not be computationally intractable for an AGI to do. 

This means that the most probable current path forward is into highly capable general AI that generalise across tasks.