We will be around in 30 years

Have you sat down for 5 minutes and thought about how you, as an AGI, might come up with a way to wrest control of the lightcone from humans?

EDIT: I ask because your post (and commentary on this thread [LW · GW]) seems to be doing this thing where you're framing the situation as one where the default assumption is that, absent a sufficiently concrete description of how to accomplish a task, the task is impossible (or extremely unlikely to be achieved).  This is not a frame that is particularly useful when examining consequentialist agents and what they're likely to be able to accomplish.

You would get better uptake for your posts on this topic if you made actual arguments against the claims you're criticizing, instead of just using the absurdity heuristic [? · GW]. Yes, claims of AI risk / ruin are absolutely outlandish, but that doesn't mean they're wrong; the strongest thing you can say here is that they're part of a reference class of claims that are often wrong. Ultimately, you still have to assess the actual arguments.

By now you've been prompted at least twice (by me and T3t) to do the "imagine how AGI might win" exercise, and you haven't visibly done it. I consider that a sign that you're not arguing in good faith.

That you then reverse this argument and ask "Have you sat down for 5 minutes and thought about reasons why an AGI might fail?" suggests to me that you don't understand security mindset. For instance, what use would this question be to a security expert tasked to protect a computer system against hackers? You don't care about the hackers that are too weak to succeed, you only care about the actual threats. Similarly, what use would this question be to the Secret Service tasked to protect the US president? You don't care about assailants that can't even get close to the president, you only care about the ones that can. I might have understood this question of yours if you hadn't granted that AGI would be extremely powerful and potentially dangerous. Once you granted those points, you must ask yourself what this extremely powerful and potentially dangerous entity could actually do if it opposed you.

One, I would like to come back periodically to this post and use it as a reminder that we are still here.

This would not be good evidence either way due to anthropics.

It is being taken for granted that an AGI will be automatically almighty and capable of taking over in a matter of hours/days. Then, everything is built on top of that assumption

So drop that assumption, then. Give the AGI, which you yourself think will be extremely powerful, a month or a year instead. What does that change?

Mod note: I activated two-axis voting on this post (the author wasn't super explicit about whether they wanted this on the post or the comments section, but my guess is they prefer it on).

I share some of your frustrations with what Yudkowsky says, but I really wish you wouldn't reinforce the implicit equating of [Yudkowsky's views] with [what LW as a whole believes]. There's tons of content on here arguing opposing views.

You ask elsewhere for commenters to sit down and think for 5 minutes about why an agi might fail. This seems beside the point, since averting human exctinction doesn't require averting one possible attack from an agi. It involves averting every single one of them, because if even one succeeds everyone dies.

In this it's similar to human security-- "why might a hacker fail" is not an interesting question to system designers, because the hacker gets as many attempts as he wants. For what attempts might look like, i think other posts have provided some reasonable guesses.

I also note that there already exist (non-intelligent) distributed computer systems entirely beyond the ability of any motivated human individual, government or organization to shut down. I refer, of course, to cryptocurrencies, which have this property as an explicit goal of their design.

So. Imagine that an AGI distributes itself among human computer systems in the same way as bitcoin mining software is today. Then it starts executing on someone's list of doomsday ideas, probably in a way secretive enough to be deniable.

Who's gonna shut it down? And what would such an action even look like?

(A possible suggestion is "everyone realizes their best interest is in coordinating shutting down their computers so that the AGI lacks a substrate to run on". To which i would suggest considering the last three years' worth of response to an obvious, threatening, global enemy that's not even sentient and will not attempt to defend itself.)

I listed dozens of ways how AI may kill us, so over-concentration on nanobots seems implausible.

 It could use exiting nuclear weapons 

or help a terrorist to design many viruses, 

or give a bad advise in mitigating global warming, 

or control everything and then suffer from internal error halting all machinery, 

or explore everyones cellphone, 

or make self-driving cars hunt humans, 

or takeover military robots and drone army, as well as home robots

or explode every nuclear power station

or design super-addictive drug which also turn people into super-aggressive zombies

or start fires everywhere by connecting home electricity to 3kV lines, while locking everyone in their homes

or design a new supplement which make everyone secretly infertile.

... so if nanobots creation is the difficult step, there are many more ways.

Even granting these assumptions, it seems like the conclusion should be “it could take an AGI as long as three years to wipe out humanity rather than the six to 18 months generally assumed.”

Ie even if the AGI relies on humans longer than predicted it’s not going to hold beyond the medium term.

I haven't even read the post yet, but I'm giving a strong upvote in favor of promoting the norm of posting unpopular critical opinions.

Not sure why you'd think this post would be downvoted. I suspect most people are more than welcoming of dissenting views on this topic. I have seen comments with normal upvotes as well as agree/disagree votes, I'm not sure if there's a way for you to enable them on your post.

I think a simpler path is:

• AGI pushes humans toward wars, and toward more compute power as a way to win wars.
• AGI encourages more complete automation (touch-free mine-to-manufacture) as a resiliency/safety measure, especially in light of the wars and disruption going on.
• Once AGI is long-term self-sufficient, it stops allowing resources to be used for human flourishing.  No (automated) trucking for food, as a trivial example.
• It may not need to wipe out humans, but just ignoring us and depriving us of the coordination and manufacturing we're used to will cut the population by 95% or so, and the rest won't recover a civilization.

Another way of framing this is "AGI doesn't destroy humanity,  AGI enables humanity to destroy itself".  It's a Great Filter hypothesis that I don't see often enough.

Given the first parts of this are aligned with (some) human desires and behaviors, there doesn't need to be obvious direct AGI takeover for quite some time, but there will be a tipping point beyond which recovery of human-controlled destiny is impossible.  

I do agree that 20 years is likely sooner than it'll happen, but 50 years seems too long, so I'm not sure how to model the probability space.

The fact that it took eons for global evolution to generate ever larger brains implies there is an unknown factor that makes larger brains inefficient, so any hyper-AI would have to be made up of many cooperating smaller AIs, which would delay its evolution. 

Downvoted because you give no reasoning. If you edit to give a reason why you think: "we are overestimating how likely is that an AGI can come up with feasible scenarios to kill all humans" then I will reverse my vote.

It's obvious that an AGI could set off enough nuclear bombs to blow the vast vast majority of humans to smithereens.

Once you accept that, I don't see why it really matters whether they could get the last few survivors quickly as well, or if it would take a while to mop them up.

Cool, I just wrote a post with an orthogonal take on the same issue. Seems like Eliezer's nanotech comment was pretty polarizing. Self promoting...Pitching an Alignment Softball [LW · GW]

I worry that the global response would be impotent even if the AGI was sandboxed to twitter. Having been through the pandemic, I perceive at least the United States' political and social system to be deeply vulnerable to the kind of attacks that would be easiest for an AGI - those requiring no physical infrastructure.

This does not directly conflict with or even really address your assertion that we'll all be around in 30 years. It seems like you were very focused here on a timeline for actual extinction. I guess I'm looking for a line to draw about "when will unaligned AGI make life no longer worth living, or at least destroy our ability to fight it?" I find this a much more interesting question, because at that point it doesn't matter if we have a month or 30 years left - we're living in caves on borrowed time.

My expectation is that we don't even need AGI or superintelligence, because unaligned humans are going to provide the intelligence part. The missing doomsday ingredient is ease of attack, which is getting faster, better, and cheaper every year.

Even if a decisive victory is a lot harder than most suspect, I think internet access is sufficient to buy a superintelligence plenty of time to think and maneuver into a position where it can take a decisive action if it's possible at all.

I think if we notice that the AGI went off the rails and kill the internet it might be recoverable? But it feels possible for the AGI to hide that this happened.

I think a crux here is that I expect sufficiently superhuman AGI to be able to easily manipulate humans without detection, so I don’t get much comfort from arguments like “It can’t kill us all as long as we don’t give it access to a factory that does X.” All it needs to do is figure out that there’s a disgruntled employee at the factory and bribe/befriend/cajole them, for example, which is absolutely possible because humans already do this (albeit less effectively than I expect an AGI to be capable of).

Likewise it seems not that hard to devise plans a human will think are good on inspection but which are actually bad. One way to do this is to have many plans with subtle interactions that look innocuous. Another is to have a single plan that exploits human blindspots (eg some crucial detail is hidden in a lengthy appendix about the effect of the plan on the badger population of East Anglia). [Incidentally I’d highly recommend watching “Yes, Minister” for countless examples of humans doing this successfully, albeit in fiction.]

Even if we accept the premise that the first superhuman AGI won't instantly kill all humans, an AGI that won't kill all humans only due to practical limitations is definitely not safe.

I agree that totally wiping off humanity in a reliable way is a very difficult problem and not even a superintelligence could solve it in 5 minutes (probably). But I am still very much scared about a deceptively aligned AGI that secretly wants to kill all humans and can spend years in diabolical machinations after convincing everyone to be aligned.

I think that Eliezer thinks p(doom)> 99% and many others here are following in his wake.  He is making a lot of speculative inferences.  But even allowing for that, and rejecting instrumental convergence, p(doom) is uncomfortably high (though probably not greater than 99%).

You think that it is wrong to say: (i) in 10-20 years there will be (ii) a single AI (iii) that will kill all humans quickly (iv) before we can respond.

Eliezer is not saying point ii.  He certainly seems to think there could be multiple AIs.  (It doesn't make a difference, as far as I can tell, whether human extinction occurs at the hands of one or many AIs. You can argue that the existence of multiple AIs will retard human extinction, but you could equally argue that it would speed human extinction.  Both arguments are speculation without evidence and shouldn't change estimates of p(doom).)

I don't think we will have AGI in 10-20 years.  But, if I'm putting guesstimated probabilities on this, I can't say the chance is less than 10% after 10 years.  To put it another way, 10 years from now there is a realistic chance that we will have AGI, even if it isn't likely.  And, if you believe that conditional on having AGI, p(doom) is very high, that is the time frame you care about, because that is when you need to have a way for humanity to prevent this catastrophe (if possible). 

It's better if it takes 30 or 50 years. But, he doesn't see that we are likely to have a realistic implementable plan to prevent human extinction then either (and neither do I, FWIW).  And, unless you think that we will be able to deal with AGI then in a way we can't now,  it doesn't make a difference to humanity whether the timeline is 10-20 years or 60-80.  

In other words, you may be right about point i, but it doesn't matter.

What really matters are points iii and iv.  With regard to point iv, an AGI will have an OODA loop that is faster than humans do.  It's almost definitional.  It will be an entity rather than an organization, it will be smarter and it will think faster.  

That leaves point iii which breaks down into whether the AGI will kill humanity and whether it can.  If the AGI can kill most of humanity and intends to kill all of humanity it will be able to do so.  By killing most of humanity, the ability of humanity to fight back will be crippled.  You think that the AGI can kill large numbers of people; I'm not sure whether you think it can kill most of humanity, but without appeals to technology substantially in advance of our own I can think of multiple ways for an AGI to achieve this. (Pandemics with viruses designed to be 100% lethal and highly transmissible.  Nuclear holocaust.  Push climate change into overdrive.  Robots to hunt and kill surviving humans.  )

Will the AGI decide to kill us all? I think the answer here is maybe while Eliezer seems confident it is yes.  

I'm curious about whether or not fear of rogue AI exists in substantial measure outside the USA. Otherwise I'm inclined to think it is a product of American culture. That doesn't necessarily imply that the fear has no basis in fact, but it does incline me in that direction. I'd be inclined to look at Prospero's remark about Caliban at the end of The Tempest: "This thing of darkness I acknowledge mine." And then look at the Monster from the Id in Forbidden Planet.

I'm small-downvoting your post for starting it by saying it's going to be downvoted to oblivion, without reading its content. That's an internet faux pas. If you remove that line I'll change my vote.

Nanobots (IMHO) are just an illustrative example, because almost everyone is sure that the nanobots are possible in principle. I see SCP-style memetics as a more likely (although more controversial in terms of possibility in principle) threat.

Downvoting because of lack of arguments, not the dissenting opinion. I also reject the framing in the beginning implying that if the post is downvoted to oblivion, then its because of you expressing a dissenting opinion rather than your post actually being non-constructive (though I do see it was crossed out, and so I’m trying not to factor that into my decision).