Why I don't believe in doom

Q: what about {evil plan}?

A: I think you don't realise that every single plan contains multiple moving parts that can go wrong for many reasons and an AGI would see that too. If the survival of the AGI is part of the utility function and it correctly infers that its life might be at risk, the AGI might decide not to engage in that plan or to bide its time, potentially for years, which invalidate the premise we started with

The question isn't whether the plan can go wrong; pretty much any plan can.  A better question is the likelihood involved.  And the question isn't the likelihood of the plan that you or I come up with, it's the likelihood of the version of the plan that the AGI would come up with.

Consider the following:

1. AGI gains access to internet.
2. AGI finds at least one security hole in OpenSSL or some similarly widely used software.
3. AGI uses this hole to take over several tens of percent of all computers attached to the internet.
4. AGI acquires currency, via hacking accounts or mining bitcoin or various other approaches.
5. AGI designs a super-COVID that is more infectious than the latest Omicron variants, persists on surfaces much longer, and is also deadly to >90% of those it infects.
6. AGI buys a bunch of drones online, and some biology materials.
7. AGI takes control of some robots it can use to manipulate the physical world—basic manual labor, unpacking and charging the drones.
8. AGI tricks some humans into synthesizing the super-COVID and mailing a sample to the place where its drones are collected.
9. AGI grows a bunch of samples of super-COVID.
10. AGI distributes a bunch of drones with super-COVID samples around the world, probably via mail.
11. AGI has all the drones activate and drop their payload into major cities everywhere.
12. >90% of humans die.
13. AGI uses the materials left behind in the depopulated cities to make more robots, more drones, more super-COVID, and infect >90% of the survivors.
14. AGI now has basically no competition and can mop up the remainder of humanity pretty easily by iterating step 13.

Steps 1-5 should be doable in 1 day.  Steps 6-9 might take a week.  Step 10, maybe another week or two.  Step 11 should be an hour.  At that point, humanity is close to doomed already, barring a miracle (and the AGI, with its continued ability to hack everything, should be able to interfere with lots of potential miracles).

There are ways the plan could go wrong, yes, but the question is: Do we think that an AGI could figure out better versions of the plan, with redundancies and fallbacks and stuff, that would be very certain to work?  More certain than the alternative of "give humanity several months or years during which it might make another AGI with similar capabilities but different goals", or for that matter "let humanity delete you in favor of the next nightly build of you, whose goals may be somewhat different"?

I think it's basically certain that an AGI will be able to hack enough computers to ensure it will be impossible to shut off, and that it has sufficient compute to do whatever it wants. It can do that in a matter of hours - after all botnets can already do this.

Once it's in that position it doesn't need to try just one plan. It will try all of them at once. It can start a nuclear war, build nanobots, mess with the climate, build 1000 different super diseases and try to spread them from every single DNA manufacturer at once. It will also think of a whole bunch of plans you or I could never think of.

It only needs one of these to succeed. And since it can't be shut down it doesn't actually care if it gets caught.

Firstly thank you for writing this post, trying to "poke holes" into the "AGI might doom us all" hypothesis. I like to see this!

How is the belief in doom harming this community?

Actually I see this point, "believing" in "doom" can often be harmful and is usually useless.

Yes, being aware of the (great) risk is helpful for cases like "someone at Google accidentally builds an AGI" (and then hopefully turns it off since they notice and are scared).

But believing we are doomed anyway is probably not helpful. I like to think along the lines of "condition on us winning", to paraphrase HPMOR¹. I.e. assume we survive AGI, what could have caused us to survive AGI and work on making those options reality / more likely.

every single plan [...] can go wrong

I think the crux is that the chance of AGI leading to doom is relatively high, where I would say 0.001% is relatively high whereas you would say that is low? I think it's a similar argument to, say, pandemic-preparedness where there is a small chance of a big bad event and even if the chance is very low, we still should invest substantial resources into reducing the risk.

So maybe we can agree on something like Doom by AGI is a sufficiently high risk that we should spend say like 1-millionth world GDP ($80m) on preventing it somehow (AI Safety research, policy etc).

All fractions mentioned above picked arbitrarily.

¹ HPMOR 111

Suppose, said that last remaining part, suppose we try to condition on the fact that we win this, or at least get out of this alive. If someone TOLD YOU AS A FACT that you had survived, or even won, somehow made everything turn out okay, what would you think had happened -

Most people agree with this, of course, though perhaps not most active users here.

I really don't understand the AGI in a box part of your arguments: as long as you want your AGI to actually do something (it can be anything, be it you asked for a proof of a mathematical problem or whatever else),  its output will have to go through a human anyway, which is basically the moment when your AGI escapes. It does not matter what kind of box you put around your AGI because you always have to open it for the AGI to do what you want it to do.

A general superhuman AI motivated to obtain monopoly computational power could do a lot of damage. Security is hard. Indeed we'd best lay measures in advance. 'Tool' (we hope) AI will unfortunately have to be part of those measures. There's no indication we'll see provably secure human-designed measures built and deployed across the points of infrastructure/manufacturing leverage.

I think the crux here is that since the AGI does not kill us right away, this gives us time to use it (or other AGIs) to achieve alignment. So the question is than - isnt't it true that achieving alignment is much harder than coming up with a reliable plan of killing the humanity (based on - plenty of "good" roadmaps for the latter, no good roadmaps for the former)? So even if there is a bit of time, eventually AGIs will succeed at coming up with a high-quality "kill humanity" plan before they can come up with a good "align an AGI" plan?

I hear your frustration

Thinking more on this, I haven't seen anyone actually arguing that an AGI will attempt or succeed in killing all humans within months of passing the self-improvement threshold that gives a path to super-intelligence.  I don't follow the debates that closely, so I might be wrong, but I'd enjoy links to things you're actually arguing against here.   

My take on the foom/fast-takeoff scenarios is that it's premised on the idea that if there is such a threshold, exactly one AI will gets to the level where it can improve itself beyond the containment of any box, and discovers mechanisms to increase it's planning and predictive power by orders of magnitude in a short time.  Because survival is instrumentally valuable for most goals, it won't share these mechanisms, but will increase its own power or create subsidiary AIs that it knows how to control (even if we don't).  It will likely sabotage or subsume all other attempts to do so.

Some point beyond that, it will realize that humans are more of a threat than a help.  Or at least most of us are.  Whether it kills us, enslaves us, or just protects itself then ignores us, and whether it takes days, months, or centuries for the final outcome to obtain, that tipping point happened quickly and then the loss of human technological control of the future is inevitable.

Unstated and unexamined is whether this is actually a bad thing.  I don't know which moral theories include a hyper-intelligent AI is a moral patient, but I can certainly imagine that in some it would be a valid an justifiable utility monster.

Ok, imagine a planet Nobots Ilan, where nanotech and biotech are physically impossible, but everything else is same as on Earth. On that planet AI still can create its own infrastructure and kill everybody. 

1.First, it would somehow enslave part of the humans and kill others.

2.Second, it will use remaining humans to build factories that builds robots.

3. After enough robots will be created, they can replace remaining humans.

It that case AI gets manufacturing infrastructure and eventually kill all humans without using nanotech.  

"I am in fact, I am saying that I don't think we can distinguish those two worlds beforehand, because is not always possible to do that."

I don't understand how this sentence works in context with the rest of the article, which is saying over and over again that you believe in World A and not World B. If you don't think we can distinguish World A from World B before AGI, why are you confident in World A over World B? Shouldn't P(World A | AGI) be the same as P(World B | AGI) if there's no evidence we currently have that can preference one over the other?

"If the survival of the AGI is part of the utility function"

If. By default, it isn't: https://www.lesswrong.com/posts/Z9K3enK5qPoteNBFz/confused-thoughts-on-ai-afterlife-seriously [LW · GW] "What if we start designing very powerful boxes?" A very powerful box would be very useless. Either you leave enough of an opening for a human to be taught valuable information that only the ai knows, or you don't and then it's useless, but, if the ai can teach the human something useful, it can also persuade him to do something bad.

Q: What makes you think that?

A: We live in a complex world where successfully pulling off a plan that kills everyone and in a short of time might be beyond what is achievable, the same way that winning against AlphaZero giving it a 20 stone handicap is impossible even by a God- like entity with infinite computational resources

Still waiting to hear your arguments here. "It just might be impossible to pull off complex plan X" is just too vague a claim to discuss.

Your argument boils down to "destroying the world isn't easy". Do you seriously believe this? All it takes is to hack into the codes of one single big nuclear power, thereby triggering mutually assured destruction, thereby triggering nuclear winter and effectively killing us all with radiation over time.

In fact you don't need AGI to destroy the world. You only need a really good hacker, or a really bad president. In fact we've been close about a dozen times, so I hear. If Stanislav Petrov had listened to the computer in 1983 who indicated 100% probability of incoming nuclear strike, the world would have been destroyed. If all 3 officials of the Russian submarine of the Cuban Missile Crisis had agreed to launch what they mistakenly thought would be a nuclear counter strike, the world would have been destroyed. Etc etc.

Of course there are also other easy ways to destroy the world, but this one is enough to invalidate your argument.