james oofou's Shortform

If you're looking for recent, canonical one-stop-shop, the answer is List of Lethalities [LW · GW].

David Chalmers asked for one last year, but there isn't. 

I might give the essence of the assumptions as something like: you can't beat superintelligence; intelligence is independent of value; and human survival and flourishing require specific complex values that we don't know how to specify. 

But further pitfalls reveal themselves later, e.g. you may think you have specified human-friendly values correctly, but the AI may then interpret the specification in an unexpected way. 

What is clearer than doom, is that creation of superintelligent AI is an enormous gamble, because it means irreversibly handing control of the world to something non-human. Eliezer's position is that you shouldn't do that unless you absolutely know what you're doing. The position of the would-be architects of superintelligent AI is that hopefully they can figure out everything needed for a happy ending, in the course of their adventure. 

One further point I would emphasize, in the light of the last few years of experience with generative AI, is the unpredictability of the output of these powerful systems. You can type in a prompt, and get back a text, an image, or a video, which is like nothing you anticipated, and sometimes it is very definitely not what you want. "Generative superintelligence" has the potential to produce a surprising and possibly "wrong" output that will transform the world and be impossible to undo. 

I'd actually recommend Zvi's On A List of Lethalities [LW · GW] over the original, as a more readily understandable version that covers the same arguments.

I think this post is an excellent distillation of the AI doomer argument, and it importantly helps me understand why people think AI alignment was going to be difficult:

https://www.lesswrong.com/posts/wnkGXcAq4DCgY8HqA/a-case-for-ai-alignment-being-difficult [LW · GW]

I think  AGI Safety From First Principles [? · GW] by Richard Ngo is probably good.

I think AGI Ruin: A List of Lethalities [LW · GW] is comprehensive but also sort of advanced and skips over the two basic bits.

What I have noticed is that while there are cogent overviews of AI safety that don't come to the extreme conclusion that we all going to be killed by AI with high probability....and there are articles that do come to that conclusion without being at all rigorous or cogent....there aren't any that do both. From that I conclude there aren't any good reasons to believe in extreme AI doom scenarios, and you should disbelieve them. Others use more complicated reasoning, like "Yudkowsky is too intelligent to communicate his ideas to lesser mortals, but household believe him anyway".

(See @DPiepgrass saying something similar [LW · GW] and of course getting downvoted).

@MitchellPorter supplies us with some examples of gappy arguments.

human survival and flourishing require specific complex values that we don't know how to specify

There 's no evidence that "human values" are even a coherent entity , and no reason to believe that any AI of any architecture would need them.

But further pitfalls reveal themselves later, e.g. you may think you have specified human-friendly values correctly, but the AI may then interpret the specification in an unexpected way.

What is clearer than doom, is that creation of superintelligent AI is an enormous gamble, because it means irreversibly handing control of the world

Hang on a minute. Where does control of the come from? Do we give it to the AI? Does it take it?

to something non-human. Eliezer's position is that you shouldn't do that unless you absolutely know what you're doing. The position of the would-be architects of superintelligent AI is that hopefully they can figure out everything needed for a happy ending, in the course of their adventure.

One further point I would emphasize, in the light of the last few years of experience with generative AI, is the unpredictability of the output of these powerful systems. You can type in a prompt, and get back a text, an image, or a video, which is like nothing you anticipated, and sometimes it is very definitely not what you want. "Generative superintelligence" has the potential to produce a surprising and possibly "wrong" output that will transform the world and be impossible to undo.

Current generative AI has no ability to directly affect anything. Where would that come from?

Perhaps see https://homosabiens.substack.com/p/deadly-by-default by Duncan Sabien [LW · GW].

I don't know that "the AI doomer argument" is a coherent thing.  At least I haven't seen an attempt to gather or summarize it in an authoritative way.  In fact, it's not really an argument (as far as I've seen), it's somewhere between a vibe and a prediction.

For me, when I'm in a doomer mood, it's easy to give a high probability to the idea that humanity will be extinct fairly soon (it may take centuries to fully die out, but will be fully irreversible path in 10-50 years, if it's not already).  Note that this has been a common belief long before AI was a thing - nuclear war/winter, ecological collapse, pandemic, etc. are pretty scary, and humans are fragile.

My optimistic "argument" is really not better-formed.  Humans are clever, and when they can no longer ignore a problem, they solve it.  We might lose 90%+ of the current global population, and a whole lot of supply-chain and tech capability, but that's really only a few doublings lost, maybe a millennium to recover, and maybe we'll be smarter/luckier in the next cycle.

From your perspective, what do you think the argument is, in terms of thesis and support?  

This sounds highly plausible. There are some other dangers your scenario leaves out, which I tried to explore in If we solve alignment, do we die anyway? [LW · GW]