Most arguments for AI Doom are either bad or weak

Here's my serious claim after giving this an awful lot of thought and study: you are correct about the arguments for doom being either incomplete or bad.

But the arguments for survival are equally incomplete and bad.

It's like arguing about whether humans would survive driving a speedboat at ninety miles an hour before anyone's invented the internal combustion engine. There are some decent arguments that they wouldn't, and some decent argument that their engineering would be up to the challenge. People could've debated endlessly.

The correct answer is that humans often survive driving speedboats at ninety miles and hour and sometimes die doing it. It depends on the quality of engineering and the wisdom of the pilot (and how their motivations are shaped by competition;).

So, to make progress on actual good arguments, you have to talk about specifics: what type of AGI we'll create, who will be in charge of creating it and perhaps directing it, and exactly what strategies we'll use to ensure it does what we want and that humanity survives. (That is what my work focuses on.)

In the absence of doing detailed reasoning about specific scenarios, you're basically taking a guess. It's no wonder people have wildly varying p(doom) estimates from that method.

If it's a guess, the base rate is key. You've said you refuse to change it, but to understand the issue we have to address it. That's the primary difference between your p(doom) and mine. I've spent a ton of time trying to address specific scenarios, but my base rate is still the dominant factor in my total, because we don't have enough information yet and havent' modeled specific scenarios well enough yet.

I think we'll probably survive a decade after the advent of AGI on something like your odds, but I can't see farther than that, so my long-range p(doom) goes to around .5 with my base rate. (I'm counting permanent dystopias as doom). (incidentally, Paul Christiano's actual p(doom) is similar - modest in the near term after AGI but rising to 40% as things progress from there).

It's tough to arrive at a base rate for something that's never happened before. There is no reference class, which is why debates about the proper reference class are endless. No species has ever designed and created a new entity smarter than itself. So I put it around .5 for sheer lack of evidence either way. I think people using lower base rates are succumbing to believing what feels easy and good- motivated reasoning. There simply are no relevant observations.

A strong good argument has the following properties:

• it is logically simple (can be stated in a sentence or two)
  • This is important, because the longer your argument, the more details that have to be true, and the more likely that you have made a mistake.  Outside the realm of pure-mathematics, it is rare for an argument that chains together multiple "therefore"s to not get swamped by the fact that

No, this is obviously wrong.

1. Argument length is substantially a function of shared premises.  I would need many more sentences to convey a novel argument about AI x-risk to someone who had never thought about the subject before, than to someone who has spent a lot of time in the field, because in all likelihood I would first need to communicate and justify many of the foundational concepts that we take for granted.
   1. Note that even here, on LessWrong, this kind of detailed argumentation is necessary to ward off misunderstandings.  
2. Argument strength is not an inverse function with respect to argument length, because not every additional "piece" of an argument is a logical conjunction which, if false, renders the entire argument false.  Many details in any specific argument are narrowing down which argument the speaker is making, but are not themselves load-bearing (& conjunctive) claims that all have to be true for the argument to be valid.  (These are often necessary; see #1.)

Anyways, the trivial argument that AI doom is likely (given that you already believe we're likely to develop ASI in the next few decades, and that it will be capable of scientific R&D that sounds like sci-fi today) is that it's not going to have values that are friendly to humans, because we don't know how to build AI systems in the current paradigm with any particular set of values at all, and the people pushing frontier AI capabilities mostly don't think this is a real problem that needs to be figured out^[1].  This is self-evidently true, but you (and many others) disagree.  What now?

1. ^{^}

   A moderately uncharitable compression of a more detailed disagreement, which wouldn't fit into one sentence.

To be clear, if you put doom at 2-20%, you're still quite worried then? Like, wishing humanity was dedicating more resources towards ensuring AI goes well, trying to make the world better positioned to handle this situation, and saddened by the fact that most people don't see it as an issue?

Over what timeframe?  2-20% seems a reasonable range to me, and I would not call it "very low".  I'm not sure there is a true consensus, even around the LW frequent posters, but maybe I'm wrong and it is very low in some circles, though it's not in the group I watch most.   It seems plenty high to motivate behaviors or actions you see as influencing it.

Doom aside, do you expect AI to be smarter than humans? If so, do you nonetheless expect humans to still control the world? 

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-12T22:45:05.154Z · LW(p) · GW(p)

so, do you nonetheless expect humans to still control the world? 

 

I personally don't control the world now.  I (on average) expect to be treated about as well by our new AGI overlords as I am treated by the current batch of rulers.

Replies from: tailcalled

↑ comment by tailcalled · 2024-10-13T07:51:27.635Z · LW(p) · GW(p)

Why do you expect that? Our current batch of rulers need to treat humans reasonably well in order for their societies to be healthy. Is there a similar principle that makes AI overlords need to treat us well?

Replies from: logan-zoellner

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T10:40:28.142Z · LW(p) · GW(p)

50% of the humans currently on Earth want kill me because of my political/religious beliefs.  My survival depends on the existence of a nice game-theory equilibrium, not because of the benevolence of other humans.  I agree (note the 1 bit) that the new game-theory equilibrium after AGI could be different.  However, historically, increasing the level of technology/economic growth has led to less genocide/war/etc, not more.

Replies from: tailcalled, MinusGix, avturchin

↑ comment by tailcalled · 2024-10-13T10:53:23.554Z · LW(p) · GW(p)

Has it? I'm under the impression technology has lead to much more genocide and war. WWI and WWII were dependent on automatic weapons, the Holocaust was additionally dependent on trains etc., the Rwandan genocide was dependent on radio.

Technology mainly has the ability to be net good despite this because:

1. Technology also leads to more growth, better/faster recovery after war, etc..
2. War leads to fear of war, so with NATO, nuclear disarmament, etc., people are reducing the dangers of war

But it's not clear that point 2 is going to be relevant until after AI has been applied in war, and the question is whether that will be too late. Basically we could factor P(doom) into P(doom|AI gets used in war)P(AI gets used in war). Though of course that's only one of multiple dangers.

50% of the humans currently on Earth want kill me because of my political/religious beliefs.

Which political/religious beliefs?

Replies from: logan-zoellner

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T11:01:14.752Z · LW(p) · GW(p)

Has it? I'm under the impression technology has lead to much more genocide and war.

 

You're impression is wrong.  Technology is (on average) a civilizing force.

Which political/religious beliefs?

 

I'm not going into details about which people want to murder me and why for the obvious reason.  You can probably easily imagine any number of groups whose existence is tolerated in America but not elsewhere.

Replies from: tailcalled

↑ comment by tailcalled · 2024-10-13T11:15:16.661Z · LW(p) · GW(p)

You're impression is wrong.

You link this chart:

... but it just shows the percentage of years with wars without taking the severity of the wars into account.

Your link with genocides includes genocides linked with colonialism, but colonialism seems driven by technological progress to me.

Technology is (on average) a civilizing force.

This stuff is long-tailed, so past average is no indicator of future averages. A single event could entirely overwhelm the average.

Replies from: habryka4, logan-zoellner

↑ comment by habryka (habryka4) · 2024-10-14T05:34:23.608Z · LW(p) · GW(p)

See also this classical blogpost: https://blog.givewell.org/2015/07/08/has-violence-declined-when-large-scale-atrocities-are-systematically-included/ 

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T11:34:40.858Z · LW(p) · GW(p)

but it just shows the percentage of years with wars without taking the severity of the wars into account.

 

If you look at the probability of dying by violence, it shows a similar trend

This stuff is long-tailed, so past average is no indicator of future averages.

I agree that tail risks are important.  What I disagree with is that only tail risks from AGI are important.  If you wish to convince me that tail-risks from AGI are somehow worse than (nuclear war, killer drone swarms, biological weapons, global warming, etc) you will need evidence.  Otherwise, you have simply recreated the weak argument (which I already agree with) "AGI will be different, therefore it could be bad".

Replies from: tailcalled

↑ comment by tailcalled · 2024-10-13T11:50:04.513Z · LW(p) · GW(p)

If you look at the probability of dying by violence, it shows a similar trend

Probability normalizes by population though.

I agree that tail risks are important. What I disagree with is that only tail risks from AGI are important.

My claim is not that the tail risks of AGI are important, my claim is that AGI is a tail risk of technology. Like the correct way to handle tail risks of a broad domain like technology is to perform root cause analysis into narrower factors like "AGI", "nuclear weapons" vs "speed boats" etc., so you can specifically address the risks of severe stuff like AGI without getting caught up in basic stuff like speed boats.

Replies from: logan-zoellner

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T11:58:49.994Z · LW(p) · GW(p)

My claim is not that the tail risks of AGI are important, my claim is that AGI is a tail risk of technology. 

 

Okay, I'm not really sure why we're talking about this, then.

Consider this post a call to action of the form "please provide reasons why I should update away from the expert-consensus that AGI is probably going to turn out okay"

I agree talking about how we could handle technological changes as a broader framework is a meaningful and useful thing to do.  I'm just don't think it's related to this post.

Replies from: tailcalled

↑ comment by tailcalled · 2024-10-13T12:03:42.745Z · LW(p) · GW(p)

My previous comment was in opposition to "handling technological changes as a broader framework". Like I was saying, you shouldn't use "technology" broadly as a reference at all, you should consider narrower categories like AGI which individually have high probabilities of being destructive.

Replies from: logan-zoellner

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T12:05:46.688Z · LW(p) · GW(p)

narrower categories like AGI which individually have high probabilities of being destructive.

 

If AGI has a "high probably of being destructive", show me the evidence. What amazingly compelling argument has led you to have beliefs that are wildly different from the expert-consensus?

Replies from: tailcalled

↑ comment by tailcalled · 2024-10-13T12:14:04.992Z · LW(p) · GW(p)

I've already posted my argument here [LW(p) · GW(p)], I don't know why you have dodged responding to it.

Replies from: logan-zoellner

↑ comment by Logan Zoellner (logan-zoellner) · 2024-10-13T12:17:53.804Z · LW(p) · GW(p)

my apologizes.  that is in a totally different thread, which I will respond to.

↑ comment by MinusGix · 2024-10-14T18:30:23.315Z · LW(p) · GW(p)

It has also led to many shifts in power between groups based on how well they exploit reality. From hunter-gatherers to agriculture, to grand armies spreading an empire, to ideologies changing the fates of entire countries, and to economic & nuclear super-powers making complex treaties.

↑ comment by avturchin · 2024-10-13T17:34:16.768Z · LW(p) · GW(p)

Soon anyone will be able to build a drone which will fly around a globe and will kill exact person they hate. 

Right now, every powerful intelligence (e.g. nation-states) is built out of humans, so the only way for such organizations to thrive is to make sure the constituent humans thrive, for instance by ensuring food, clean air and access to accurate information.

AI is going to loosen up this default pull. If we are limited to reflex-based tool AIs like current LLMs, probably we'll make it through just fine, but if we start doing wild adversarial searches that combine tons of the tool-like activities into something very powerful and autonomous, these can determine ~everything about the world. Unless all winners of such searches actively promote human thriving in their search instead of just getting rid of humanity or exploiting us for raw resources, we're doomed.

There's lots of places where we'd expect adversarial searches to be incentivized, most notably:

• War/national security
• Crime and law enforcement
• Propaganda to sway elections
• Market share acquisition for companies (not just in advertising but also in undermining competitors)

The current situation for war/national security is already super precarious due to nukes, and I tend to reason by an assumption that if a nuke is used again then that's going to be the end of society. I don't know whether that assumption is true, but insofar as it is reasonable, it becomes natural to think of AI weapons as Nuke 2.0.

The situation with nukes suggests that maybe sometimes we can have an indefinite holdoff on using certain methods, but again it already seems quite precarious here, and it's unclear how to generalize this to other case. For instance, outlawing propaganda would seem to interfere with free speech, and enforcing such laws without tyrannizing people who are just confused seems subtle.

So a plausible model seems to me to be, people are gradually developing ways of integrating computers with the physical world, by giving them deeper knowledge of how the world works and more effective routines for handling small tasks. Right now, this interface is very brittle and so breaks down when even slight pressure is applied to it, but as it gets more and more robust and well-understood, it becomes more and more feasible to run searches over it to find more powerful activities. In non-adversarial circumstances, such searches don't have to ensure robustness or completeness and thus can just "do the thing" you're asking them to, but in adversarial circumstances, the adversaries will exploit your weakness and so you actually have to do it similar to a dangerous utility-maximizer.

I start with a very low prior of AGI doom ... I give 1 bit of evidence collectively for the weak arguments

Imagine that you have to argue with someone who believes in 50% doom^[1] on priors. Then you'd need to articulate the reasons for adopting your priors. Priors are not arguments, it's hard to put them into arguments. Sometimes priors are formed by exposure to arguments, but even then they are not necessarily captured by those arguments. Articulation of your priors might then look similarly tangential and ultimately unconvincing to those who don't share them, if it doesn't form a larger narrative that's no longer a specific argument.

a very low pDoom (2-20%) compared to most people on LessWrong

Were there surveys that put most people on LW higher than that? My impression is that lower bounds of 10-20% are not too low for most, at least for doom in the sense of extinction.

I think the cumulative argument works:

1. There are dozens of independent ways in which AI can cause a mass extinction event at different stages of its existence.
2. While each may have around a 10 percent chance a priori, cumulatively there is more than a 99 percent chance that at least one bad thing will happen.

This doesn't mean that all humans will go extinct for sure, everywhere and forever. Some may survive the mass extinction event.

As an aside, most arguments for almost anything are bad or weak, whether the conclusion is true/real or not. Science, politics, economics, really any field where there's room for uncertainty and a lot of people interested in the answer. As such, this is not strong evidence in and of itself. One sufficiently strong argument can outweigh all the bad ones. At least in terms of logical evidence. There are many, many, many people who understand your points about nuclear power, for example, but they have been unable to sway political processes for the past few decades and the bad arguments to the contrary are still bandied about constantly. I do think the theoretical minimum price of solar is lower than nuclear for a given level of safety and reliability, though.

That said: I think you'll find there's a lot more people with similar estimates of p(doom) here than you are expecting. I'm one of them. I also think the appropriate reaction to "I believe there's a 2-20% chance that anyone born today will see human extinction before they turn 25" is not to say that number is low, but to say, "We need to mobilize the entire planet as though for total war, if that would help, in order to make sure that doesn't happen."

I do think completely ignoring long lists is a mistake, especially when it's something like the list of lethalities [LW · GW]. When you're dealing with extinction-level events, we don't get to be wrong even once. We don't get to overlook even one plausible case. We must be prepared to to avoid extinction in every instance that might lead to it.

Have you read the sequences? My response depends on whether you have or haven't.

Here's an entirely separate weak argument, improving on your straw man:

AGI will be powerful. Powerful agentic things do whatever they want. People will try to make AGI do what they want. They might succeed or fail. Nobody has tried doing this before, so we have to guess what the odds of their succeeding are. We should notice that they won't get many second chances because agents want to keep doing what they want to do. And notice that humans have screwed up big projects in surprisingly dumb (in retrospect) ways.

If some people do succeed at making AGI do what they want, they might or might not want things the rest of humanity wants. So we have to estimate the odds that the types of people who will wind up in charge of AGI (not the ones that start out in charge) are "good people". Do they want things that would be better described as doom or flourishing for humanity? This matters, because they now have AGI which we agree is powerful. It may be powerful enough that a subset of power-hungry people now control the future - quite possibly all of it.

If you look at people who've held power historically, the appear to often be pretty damned selfish, and all too often downright sadistic toward their perceived enemies and sometimes toward their own subjects.

I basically only believe the standard "weak argument" you point at here, and that puts my probability of doom given strong AI at 10-90% ("radical uncertainty" might be more appropriate).

It would indeed seem to me that either I) you are using the wrong base-rate or 2) you are making unreasonably weak updates given the observation that people are currently building AI, and it turns out it's not that hard.

I'm personally also radically uncertain about correct base rates (given that we're now building AI) so I don't have a strong argument for why yours is wrong. But my guess is your argument for why yours is right doesn't hold up.

I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).

You link to a prediction market (Manifold's "Will AI wipe out humanity before the year 2100", curretly at 13%).

Problems I see with using it for this question, in random order:

1. It ends in 2100 so the incentive is effectively about what people will believe a few years from now, not about the question. It is a Keynesian beauty contest. (Better than nothing.)
2. Even with the stated question, you win only if it resolves NO, so it is strategically correct to bet NO.
3. It is dynamically inconsistent, if you think that humans have power over the outcome and that such markets influence what humans do about it. Illustrative story: "The market says P(doom)=1%, ok I can relax and not work on AI safety" => everyone says that => the market says P(doom)=99% because no AI safety work => "AAAAH SOMEONE DO SOMETHING" => marker P(doom)=1% => ...

Note that some of the best arguments are of the shape "AI will cause doom because it's not that hard to build the following..." followed by insights about how to build an AI that causes doom. Those arguments are best rederived privately rather than shared publicly, and by asking publicly you're filtering the strength of arguments you might get exposed to.

Get a dozen AI risk skeptics together, and I suspect you'll get majority support from the group for each and every point that the AI risk case depends on. You, in particular, seem to be extremely aligned with the "doom" arguments.

The "guy-on-the-street" skeptic thinks that AGI is science fiction, and it's silly to worry about it. Judging by your other answers, it seems like you disagree, and fully believe that AGI is coming. Go deep into the weeds, and you'll find Sutton and Page and the radical e/accs who believe that AI will wipe out humanity, and that's a good thing, and that wanting to preserve humanity and human control is just another form of racism. A little further out, plenty of AI engineers believe that AGI would normally wipe out humanity, but they're going to solve the alignment problem in time so no need to worry. Some contrarians like to argue that intelligence has nothing to do with power, and that superintelligence will permanently live under humanity's thumb because we have better access to physical force. And then, some optimists believe that AI will inevitably be benevolent, so no need to worry.

If I'm understanding your comments correctly, your position is something like "ASI can and will take over the world, but we'll be fine", a position so unusual I didn't even think to include it detail in my lengthy taxonomy [LW · GW] of "everything turns out okay" arguments. I am unable to make even a basic guess as to how you arrived at the position (though I would be interested in learning).

Please notice that your position is extremely non-intuitive to basically everyone. If you start with expert consensus regarding the basis of your own position in particular, you don't get 87% chance that you're right, you get a look of incredulity and an arbitrarily small number. If you instead want to examine the broader case for AI risk, most of the "good arguments" are going to look more like "no really, AI keeps getting smarter, look at this graph" and things like Yudkowsky's "The Power of Intelligence", both of which (if I understand correctly) you already think are obviously correct.

If you want to find good arguments for "humanity is good, actually", don't ask AI risk people, ask random "normal" people.

My apologies if I've completely misunderstood your position.

(PS: Extinction markets do not work, since they can't pay out after extinction.)