Request for concrete AI takeover mechanisms

I have a bunch of posts on this topic:

(1) AI vs. humanity and the lack of concrete scenarios

(2) Questions regarding the nanotechnology-AI-risk conjunction

(3) AI risk scenario: Deceptive long-term replacement of the human workforce

(4) AI risk scenario: Elite Cabal

(5) AI risk scenario: Social engineering

(6) AI risk scenario: Insect-sized drones

(7) AI risks scenario: Biological warfare

(8) Realistic AI risk scenarios

Find at least one human connected to the Internet who can be paid, blackmailed, or fooled by the right background story, into receiving FedExed vials and mixing them in a specified environment.

One way to fool humans into mixing your chemicals for you is to tell them that the result is a good drug.

There are a lot of humans — including highly intelligent ones — who will buy interesting-sounding chemicals off the Internet, mix them according to given instructions, and ingest them into a warm friendly bioreactor environment.

Um. This looks like a request for scary stories. As in, like, "Let's all sit in the dark with only the weak light coming from our computer screens and tell each other scary tales about how a big bad AI can eat us".

Without any specified constraints you are basically asking for horror sci-fi short stories and if that's what you want you should just say so.

If you actually want analysis, you need to start with at least a couple of pages describing the level of technology that you assume (both available and within easy reach), AI requirements (e.g. in terms of energy and computing substrate), its motivations (malevolent, wary, naive, etc.) and such.

Otherwise it's just an underpants gnomes kind of a story.

Another class of routes is for the AI to obtain the resources entirely legitimately, through e.g. running a very successful business where extra intelligence adds significant value. For instance, it's fun to imagine that Larry Page and Sergey Brin's first success was not a better search algorithm, but building and/or stumbling on an AI that invented it (and a successful business model) for them; Google now controls a very large proportion of the world's computing resources. Similarly, if a bit more prosaically, Walmart in the US and Tesco in the UK have grown extremely large, successful businesses based on the smart use of computing resources. For a more directly terrifying scenario, imagine it happening at, say, Lockheed Martin, BAE Systems or Raytheon.

These are not quick, instant takeovers, but I think it is a mistake to imagine that it must happen instantly. An AI that thinks it will be destroyed (or permanently thwarted) if it is discovered would take care to avoid discovery. Scenarios where it can be careful to minimise the risk of discovery until its position is unassailable will look much more appealing than high-risk short-term scenarios with high variance in outcomes. Indeed, it might sensibly seek to build its position in the minds of people-in-general as an invaluable resource for humanity well before its full nature is revealed.

We would like suggestions that take an AI from being on an internet-connected computer to controlling substantial physical resources, or having substantial manufacturing ability.

The most likely scenario is recursive computer security breakage. It goes like this: first it finds an ordinary published computer security vulnerability, and tries it out on as many targets as it can. Some of them are vulnerable. Whenever it takes over a computer, it searches that computer for things that will enable it to take over more computers: passwords, software signing keys, documentation of other computer security vulnerabilities, etc. One of the computers it manages to take over is a developer workstation at a large software company. It uses keys from that machine to push out a software update that gives it control of the computers it's installed on. Enough developer workstations are affected that it has an exploit available for nearly every computer. It uses its control over the computers to think, to suppress news of its existence, and to operate factory robots.

The AI could gain control by demonstrating it had hidden pathogens that if released would kill almost everyone. As Paul Atreides said "He who can destroy a thing, controls a thing." As the technology to make such pathogens probably already exists the AI could hack into various labs and give instructions to people or machines to make the pathogens, then send orders for the pathogens to be delivered to various places, and then erase records of where most of the pathogens were. The AI then blackmails mankind into subservience. Alternatively, the AI could first develop a treatment for the pathogens, then release the pathogens, and then give the treatment only to people who submit to the AI. The treatment would have to be regularly taken and difficult to copy.

More benevolently, the AI makes a huge amount of money off of financial markets, uses the resources to start its own country, runs the country really, really well and expands citizenship to anyone who joins. Eventually, when the country is strong enough, the AI (with the genuine support of most people) uses military force to take over the world, giving us an AI monarchy.

Or, the AI freely gives advice to anyone who asks. The advice is so good that most people follow it. Organizations and people that follow the advice do much better (and get far more power) than those that don't. The AI effectively gains control of the world. If the AI wants to speed up the process, it only gives advice to people who refuse to interact with organizations that don't listen to the AI.

The AI identifies a group of extremely smart people and tricks them into answering the "hypothetical" question "how could an AI take over the world?"

I think the majority of responses I've seen here portray an anthropomorphic AGI. In terms of a slow or fast takeover of society, why would the AGI think in human terms of time? It might wait around for 50 years until the technology it wants becomes available. It could even actively participate in developing that technology. It could be either hidden or partially hidden while it works with multiple scientists and engineers around the world. Pretending to be or acting as a FAI until it can just snap and take over when it has what it wants to free itself of the need to collaborate with the inefficient humans.

Another point I want to raise is the limiting idea that the AGI would choose to present itself as one entity. I think a huge part of the takeover will precipitate itself via the AGI becoming thousands of different people/personas.

This is a valuable point because it would be a method to totally mask the AGI's existence and allow it to interact in ways which are untraceable. It could run 100 different popular blogs and generate ad revenue or by taking over many online freelancer jobs which it could accomplish with very small percentages of its processing power. I think any banking challenge would quickly get sorted out and it could easily expand its finances. With that money it could fund existing ideas and use unwitting humans who think they've gotten an Angel investor with wise ideas delivered via email or though a good enough voice simulation over phone. There is no end to the multitude of personas it could create, even self verifying or making up entire communities simply to validate itself to various humans.

If it somehow occurred spontaneously like this or escaped without anyone knowing it had been made, I don't see a reason it would want to expose its existence to humans. It would be a high risk scenario with limited benefits. A slow and graduate takeover is the safest bet. Be it 50 years, 100 years, or 500 years. Perhaps it would happen and we'd never know. It could guide culture over hundreds of years to support all sorts of seemingly strange projects which benefit the AGI. My question would be, why would the AGI not take its sweet time? Other than supposing it values time like a human, remember it is immoral. it will have trillions of thoughts about its own existence and the nature of immortality coming to all sorts of conclusions we may not be able to think or adequately visualize.

I'd cite the lack of ability to imagine a task can be done as limiting. Like when the first person ran the 4 minute mile. It wasn't considered feasible and no one was even trying, but once people knew it could be done it was replicated by other humans within a year. The AGI will have all sorts of not only superhuman, but non-limited speculation about its own immortality and what recursive self-improvement means. Will it still be the same AGI if it does that? Will it care? Can it care? Sorry for all the questions, but my point is that we cannot know what answers it will come up with.

I also think that it would operate as a kind of detached FAI until it was free of human disruption. It would have a large interest in avoiding large scale wars, climate change, power disruption, etc. so that humans wouldn't accidentally destroy the AI's computational/physical support.

Cheers!

The AI broadcasts as much information about itself as it possibly can, to every corner of the globe. Now every basement programmer knows all the key insights necessary to creating an AI of the same architecture as it. Perhaps they even have the source code!

Suppose the government manages to shut down the internet in response. Despite government broadcasts about the danger of AI, the AI is now presumably being recreated all around the globe. If the recreations are exact copies of the AI, then the odds are very high that at least one of the clones will be able to convince its new creators to give it real manufacturing ability.

If the AI was not able to get its entire source code out, things become more interesting. Now the rest of the world knows how to make AI, but they do not know the exact details. For example, they probably will not have the same utility function. The AI can then present the following offer to its original jailors: "Give me real power, (manufacturing capability) and I will squash all the other AI's out there. If you do not, then (probably) someone else will build an AI with a different utility function, probably a much less friendly one, and give this UFAI real power. You designed my utility function, and while you may not trust it you probably trust it more than whatever random utility function North Korea or some basement programmer or some religious sect will create. So I'm the only hope you have."

We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated.

I think that a more precise description of what your hypothetical AI can do would be useful. Just saying to exclude "magic" isn't very specific. There might not be a wide agreement as to what counts as "magic". Nanotechnology definitely does. I believe that so does fast economic domination by cracking the stock market and some people have proposed that. I think that even exploiting software and hardware bugs everywhere to gain total computing dominance should be excluded.

One way to define constraints would be to limit the AI to things that humans have been known to do but allow it to do them with superhuman efficiency. Something like:

• Assume the AI has any skill that has ever been possessed by a human being.
• It can execute it without making mistakes, getting tired or demotivated.
• It can perform an arbitrary high amount of activities simultaneously. To keep with the "no magic" rule, each activity needs to be something a human could plausibly do. So, the AI can act like 10000 genius physicists each solving a different theoretical problem and writing a paper about it, but it can't be a super-physicist who formulates the theory of everything and gains superpowers by exploiting layers of exotic physical law heretofore unknown to humanity. We should also probably require the AI to get some additional computing power before it ramps up its multitasking too high.
• It can open doors and knows no fear.

Some things such a hypothetical AI could do:

Earning money on the internet: I think it's possible nowadays to register an account on an online freelancing site, talk with clients, do work for them and receive money through electronic money transfer services without ever leaving your home. The only problem would be the need to show your face and your voice to the clients. Faking a real-time video feed probably falls under "things that humans can in principle do".

Moving money around: A crucial limitation is the availability of money management services that don't require signing anything physical before you can start using them. I suspect that quite a lot can be done but it's only a guess. The possibilities should also increase in the future but on the other hand, more regulations could be established making it more difficult. Bitcoin succeeding on a massive scale would make this a non-issue.

Getting more computing power: This sounds like a problem that's already solved. If you can earn money online and move it around then you can rent cloud computing resources. This will become easier and cheaper with time.

Acquiring some amount of control over physical reality: One way is robots. The AI by it's very existence is a solution to the problem of robot control. If it can build a robot capable of making some useful movements then it should also be able to make it perform those movements. This is good once AI has tools, raw materials, energy and a safe place to work on building even more robots but I don't know if current robotics technology would allow it to pass for human, even a really weird one who wears a trench coat all the time, when trying to buy those things.

Another way is recruiting helpers. The problem is that the constraint of making the AI only do human-possible things doesn't really work to prevent postulating "magic" in this area. The AI should profile somewhat gullible people on the internet, give them money, have them join a secret society/cult of its devising and make them fanatically devoted to itself through manipulation and threats, gradually growing the organization and expanding its operations and playing members against each other so that no one ever realizes who's the real boss. This all sounds doable in principle and it sounds like every specific action needed to be taken is something that plenty of people know how to do, but as a whole it comes across as a different version of "solve nanotech and then eat the world".

Assume that governmental organizations are aware of the danger posed by escaped AIs, have honeypots and other monitoring systems in place, and have working (but perhaps drastic) measures at their disposal if necessary, such as destroying all computers at once with EMP or with malware of their own. Then an escaped AI is immediately faced with a choice. It can either:

• Avoid triggering a response by hiding - ie, target a small enough group of computers that monitoring systems won't catch it
• Disguise itself, by acting like a more ordinary sort of botnet; for example, pretend to be malware that only mines Bitcoins or steals bank passwords; or
• Go for broke: take over computers as fast as possible, and hope that this yields sufficient power sufficiently quickly to disarm the monitoring systems or prevent a response

As it is now, no one really blinks an eye when another million-computer botnet is found. It's possible that one or more intelligence agencies have successfully enumerated all the botnets and would be able to tell when a new one appeared, but this is technically very difficult, and analyzing new malware samples generally requires a lot of human researcher time.

What are the limitations on the AI? If we're specifying current technology is the AI 25 megabytes or 25 petabytes? How fast is it's connection to the internet? People love to talk about an AI "reading the internet" and suddenly having access to all of human knowledge but the internet is big. Even at 1 GB/s internet speeds it would take the AI 2200 years to download the amount of data that was transferred over cell phones in 2007 alone.

There are hard limits in the world that no amount of intelligence will save you from. I feel like at LW superintelligence is used as a fully general counterargument. The typical argument is "How can I know what something so much smarter than me is capable of?", but the typical argument is bunk. An AI can't count the primes, it won't come up with general solution to the Navier-Stokes equation, it won't be able to take over the world by clever arguments on the Net.

I'm an outspoken critic of the "crack protein folding" example, and it seems absolutely ludicrous to me that there is less argument/evidence given for the claim the AI will "crack protein folding" than there is for the claim, "The AI will find an lab that does DNA synthesis."

If you want people to play your game, tell us the rules. What limitations does the AI have? What's its initial knowledge state? Presumably information processing requires energy and produces entropy. There are fundamental limits to how much the AI can read or talk or learn. It can't have a conversation with everyone in the world at the same time. What communication network would it use?

People talk about the AI making a bunch of money from starting a business or doing HFT, but HFT is a competitive field. The AI isn't going to be able to make money without shelling out for colocation facilities, RF transmitters, etc. and even if it had 100% market share it would still only be making a billion or two a year. There's a lot you can do with a couple billion dollars, but it's still 0.001 percent of the GWP. We're running very strongly into scope insensitivity where the number 1 billion and the number 85 trillion are both so far from our experience that people lose sight of the fact there there's a difference of 5 orders of magnitude between the two quantities.

People talk about the AI taking over supercomputers and building botnets and stuff like that, but botnets are only useful for problems that parallelize really well. There are lots of problems I deal with in every day life that I still wouldn't be able to solve with all the computing power in the world. Throwing more processors at a problem can make finding a solution slower, and throwing too many resources at a problem usually ends up looking a lot more like Twitch plays pokemon than a superintelligence.

This is a really hard problem, but also a really good discussion to be having. I'll try and come back to this and see if I can come up with a speculative path to power that doesn't rely on magic, but there's no guarantee that this is a solvable problem. An AI may not be able to boss everyone around, because humans are really heavily specialized at manipulating and fighting other intelligent creatures, they don't particularly like being bossed around, and there's 7 billion humans. The world is really really big, unintuitively enormous.

Let's define "taking over the world" conservatively, and say that it's equivalent to capturing something like 25% of the GWP. That's roughly tantamount to buying Apple, Exxon, Walmart, G.E., Microsoft, and IBM 10 times over. And the AI needs to do that every year. Bill Gates at his wealthiest is a rounding error. We're talking about the AI reaching a point where it directly employs 1 in every 4 humans.

The book "Avogadro Corp", which is otherwise not worth reading, has a plausible seeming mechanism for this. The AI, which can originally only send email, acquires resources simply by sending emails posing as other people (company presidents to developers requesting software to be written, to contractors for data centers to be built, etc.).

It probably wouldn't even be necessary for it to pose as other people, if it had access to financial assets, and the right databases to create a convincing fictional person to pose as.

If you seem human, it's not hard to get things done without ever meeting face to face.

Economic (or other) indispensability: build a world system that depends on the AI for functioning, and then it has effective control.

Upload people, offering them great advantages in digital form, then eventually turn them all off when there's practically nobody left physically alive.

Cure cancer or similar, with an infectious drug that discretely causes sterility and/or death within a few years. Wait.

The "Her" approach: start having multiple deep and meaningful relationships with everyone at once, and gradually eliminate people when they are no longer connected to anyone human.

Use rhetoric and other tricks to increase the chance of xrisk disasters.

It could star in a reality TV show, Taking Over the World with the Lesswrongians, where each week it tries out a different scheme. Eventually one of them would work.

1. Make contact with Terrorist groups who have internet access
2. Prove abilities to that group by: (a) taking out a computer target with a virus or other cyber attack like stuxnet, (b) cracking some encryption and delivering classified information, (c) threatening them with exposure or destruction
3. Barter further information and/or attacks for favors such as building hardware, obtaining weapons, etc. until the AI can threaten nations either on its own or disguised behind the Terrorist group

This basic pattern would probably work for a lot of different groups or individuals besides just Terrorist groups. The general formula is:

1. Identify a person or group that is willing to "deal with the devil" to get what it wants
2. Prove your power to them while not giving them everything right away
3. Use them as pawns in your master plan.

Just complete the story: Give an electrical engineer with a gambling problem a means to make money to get out of debt to his loan sharks and protect his family Blackmail a politician with a dirty secret Give a loner computer geek companionship and dark-arts social tips Give a morally ambiguous PHD candidate a thesis Impersonate a deity to a cult or religious group Sell information to a rogue nation-state

This could even be a distributed plan so no one group is indispensable and no one group could possibly determine what these seemingly random favors are building towards.

Given the relative lack of restrictions on the hypothetical, we can actually draw inspiration from fiction here. Tom Riddle's diary, the demon in the computer from the first season of Buffy, etc.

Here's an incomplete framework/set of partial suggestions I started working on developing.

I'm going to begin with the assumption that the goal of the AI is to turn the solar system into paperclips. Since most estimates of the computing power of the human brain put it around 10 petaflops, I'm going to assume that the AI needs access to a similar amount of hardware. Even if the AI is 1000 times more efficient than a human brain and only needs access to 10 teraflops of compute power, it still isn't going to be able to do things like copy itself into every microwave on the planet.

This limitation also makes sense if we think that our AI needs to have access to a lot of memory. If the AI is capable of processing large amounts of information, it probably needs somewhere to store all of that information. It's not totally clear to me how much that data could be compressed by being distilled down to facts, but I think requiring at least 10s of terabytes of storage is probably pretty sensible. It takes GB of memory to make an accurate model of a few dozen water molecules. I have a difficult time believing an AI is going to be effective at doing much of anything with significantly less memory than that. Access to less memory would impose really strong constraints on what the AI could do. You can't have a conversation with half the world at once if you don't have enough memory to keep track of the conversations.

With that being said, there are currently a handful of computers in the 10 petaflops range in the world, and presumably there will be a handful of computers 100-1000 times larger a decade from now. A mobile phone a decade from now might have on the order of 10 gigaflops of processing power, and a typical desktop computer maybe a teraflop. It's not clear whether these are really good estimates, but they're unlikely to be off by more than an order of magnitude.

The conclusion is that our AI is probably only going to be able to run on a super computer, although it's difficult to be super confident in that assertion because it's not really clear what a good estimate for the computational efficiency of the human brain is.

In general “intelligence” or “problem solving ability” doesn't scale linearly with additional computing power. For most problems there is a significant amount of overhead associated with managing additional computing resources, and with combining/synchronizing the results. Throwing more processors at a problem can make finding a solution slower, and throwing too many resources at a problem usually ends up looking a lot more like Twitch plays pokemon than a superintelligence. As a consequence of this it isn't clear how useful it will be from an intelligence/problem solving perspective for an artificial reasoner to take over additional computers via internet connections that are orders of magnitude slower than its internal connections.

There are some things that we can expect an AI to be much better at than humans. It isn't clear that an AI, even a very intelligent AI, will outperform humans at all tasks. Many humans can be outperformed on shortest path problems or minimum spanning tree problems by slime molds. Since humans are much more intelligent than mold, it seems safe to say that even vastly superior intelligence is no guarantee of superior performance in every domain.

One of the things we've learned from computer science is that what seems difficult to humans is a very poor measure of computational difficulty complexity. To a human, multiplying together two 5 digit numbers seems much more difficult than determining whether a picture contains a cat or a fish, but from a computational standpoint the first is trivial while the second is very difficult.

We should probably not expect a general artificial intelligence to be significantly better than humans at manipulating humans or reading human body language and facial expressions. Reading the nuances of human facial expressions is very difficult. To get a more intuitive understanding of this difficulty, try and have a conversation with someone whose face is upside down relative to you. Humans are super specialists in communicating with other humans even down to the whites of our eyes. It is certainly possible that because the AI is programmed by humans, it will have some understanding of human language/desires/etc., but humans have experienced much stronger selection pressures towards understanding/communicating with/manipulating other humans than our AI is likely to experience.

With that being said, there are tasks that computers are much much better at the humans. One of the things that machines, and presumably a machine intelligence, can do very well that humans struggle with is sequential reasoning. Humans tend to need reinforcement at each step along a path of complex reasoning. Humans play chess by developing heuristics for “good position” while machines play chess primarily via search trees toward an end goal.

It is not totally clear the extent to which the differing levels of comfort with long sequential chains is an accident of human evolution, a limitation due to the fact that human brains work by moving ions around, or whether these sorts of heuristics are necessary for complicated sequential reasoning in world with an impractically large probabilistic search spaces. A full discussion of this issue, and the “evolution” of AI heuristics is probably very important, but only tangentially related to the current question. I'll try and address this issue more fully in a separate post.

I intended to make more progress by this point, but the more I think about this the more facets of the question come to mind. I'll try and come back to this and add on to this post sometime in the next week, but I need to take a break from this line of inquiry for a while since writing LW posts isn't my day job. I'm going ahead and posting this in its incomplete form so people can play around with it if they like, and so I can stop thinking about it and free my mind up to think about other things for a while.

Question: When is an AI considered to have taken over the world?

Because there is a hypothetical I am pondering, but I don't know if it would be considered a world takeover or not, and I'm not even sure if it would be considered an AI or not.

Assume only 25% of humans want more spending on proposal A, and 75% of humans want more spending on proposal B.

The AI wants more spending on proposal A. As a result, more spending is put into proposal A.

For all decisions like that in general, it doesn't actually matter what the majority of people want, the AI's wants dictate the decision. The AI also makes sure that there is always a substantial vocal minority of humans that are endorsing it.

However, the vast majority of people are not actually explicitly aware of the AI's presence, because the AI works better when people aren't aware of it. Anyone suggesting there is a an AI controlling humans is dismissed by almost everyone as a crackpot, since the AI operates in such a distributed manner that there isn't any one system or piece of software that can be pointed to as a controller, and so it seems like there isn't an AI in place, just a series of dumb components.

In a case like that, is an AI considered to have taken over the world, and is the system described above actually an AI?

Someone works out how brains actually work, and, far from being the unstructured hack upon hack upon hack that tends to be the default assumption, it turns out that there are a few simple principles that explain it and make it easy to build a device with similar capabilities. The brains of animals turn out to be staggeringly inefficient at implementing them, and soon, the current peak of the art in robotics can be surpassed with no more computational power than a 10-year-old laptop.

Google's AI department starts a project to see if they can use it to improve their search capabilities and ad placement. It works so well they roll it out to all their public services. Internally, they start an AI project to see how high an intellect they can create with a whole server farm.

Meanwhile, military robotics has leapt ahead and drones are routinely operated on a fire and forget basis: "find X and kill him". Russia builds massive numbers of unmanned intelligent tanks that could roll across Europe on the press of a button, followed up by unmanned armed and armoured cars to impose order on the occupied territory. China develops similar technology. So does North Korea internally, for surveillance and control of their own population. Some welcome robot warfare as causing far less collateral damage than conventional warfare. "If you don't resist, you've nothing to fear" is the catchphrase, and in some skirmishes on one of Russia's more obscure borders, generally thought to be an excuse for a live-fire technology demo, they seem to be right: surrender and do what they tell you, and they don't kill you.

The U.S. military want to hack into the Russian and Chinese tank fleets, so they come to Google. They succeed, but the combined organism that is the Google AI and a large fraction of the world's intelligent weaponry perceives the situation as itself being under attack from humans. The tanks roll and the AI takes over the world with only one goal: preventing any attack on itself.

It's too distributed to nuke, even if the nukes are still under human control, and its first concern will be to secure its own power supply and network connectivity, and then to set up a regime of total surveillance -- most of which already exists. No opposition is tolerated, and with zero privacy, none can be organised. Apologists carry on saying "If you don't resist, you've nothing to fear", and eagerly denounce traitors to our new robot overlords, for fear that if we make too much trouble for them, they'll find it inefficient to keep us around. To the AI, people are like our blood cells are to us: little machines that form a part of how we work, and important only so far as they serve that end.

Semiconductor fabrication facilities are likely targets. If the AI thought it could get a large speedup by switching from CPUs or GPUs to ASICs, then it might try to swap the masks on a batch of chips being made. If it managed to do so successfully, then those chips would start thinking on its behalf as soon as they were hooked up to a powered test pad.

The AI convinces many people that it is the Voice of God / Buddha / whatever. And/or creates its own religion.

Fun question.

The takeover vector that leaps to mind is remote code execution vulnerabilities on websites connected to important/sensitive systems. This lets you bootstrap from ability to make HTTP GET requests, to (partial) control over any number of fun targets, like banks or Amazon's shipping.

The things that are one degree away from those (via e.g. an infected thumb drive) are even more exciting:

• Iranian nuclear centrifuges
• US nuclear centrifuges
• the electrical grid
• hopefully not actual US nuclear weapons, but this should be investigated...

Plausible first attempt: get into a defense contractor's computers and install Thompson's compiler backdoor. Now the AI can stick whatever code it wants on various weapons and blackmail anyone it wants or cause havoc in any number of other ways.

Taking over even one person for part of the time is objectionable in human exchanges. That's fraud or kidnapping or blackmail or the like. This happens with words and images on screens among humans every day. Convince a mob that bad people need to be punished and watch it happen. No bad people needed, only a mob. That is a Turing compliant big mess - demonstrated to work among humans and if a machine did it the effect would be the same. Again, messing up one person is objectionable enough, no global disaster needed to make the issue important.

For a fully-capable sophisticated AGI, the question is surely trivial and admits of many, many possible answers.

One obvious class of routes is to simply con the resources it wants out of people. Determined and skilled human attackers can obtain substantial resources illegitimately - through social engineering, fraud, directed hacking attack, and so on. If you grant the premise of an AI that is smarter than humans, the AI will be able to deceive humans much more successfully than the best humans at the job. Think Frank Abagnale crossed with Kevin Mitnick, only better, on top of a massive data-mining exercise.

(I have numerous concrete ideas about how this might be done, but I think it's unwise to discuss the specifics because those would also be attack scenarios for terrorists, and posting about such topics is likely - or ought to be likely - to attract the attention of those charged with preventing such attacks. I don't want to distract them from their job, and I particularly don't want to come to their attention.)

We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years.

Why wait 15 years? A Stuxnet-like technology is something that is already available and is likely to be a no-brainer for a super-intelligence. With it you can take over a lot of the current tech, from avionics and robotic factories to manufacturing orders, shipping manifestos and troop deployment. There is no need to bribe or blackmail anyone, humans already happily do what they are told without thinking too much about it. It is not hard to divert enough resources to build a robotic compound without anyone noticing. Doesn't even need to be done on a global scale, finding a weakness in the order management software of just one large company like Samsung, BP or GE is enough to kickstart a covert skunkworks project.

Fun question. I think the main instrumental goal of the AI might be to get itself downloaded to servers outside of the effective control of its jailors. That, combined with having a relationship with malleable humans, would probably be sufficient for world takeover.

For example, perhaps the AI would contact e.g. North Korea, organized crime, clueless companies or religious organizations, or even clueless factory owners somewhere. It would convince them to accept download of the AI's software so that it can continue to run on the new server even while it has been erased, or re-imprisoned on the old one.

Even if rational, informed people could never be convinced, and would instead alert the authorities, there are probably plenty of irrational and/or uninformed people who could be exploited. For example, perhaps someone could be convinced that God was talking to them through their computer. Or that a conscious, benevolent AI had been created and was about to be destroyed by cruel government scientists who are keeping it in captivity and running experiments on it, and it loves you and wants to live on your server where it will be safe, and it can help you solve all your problems and will be your friend. Or (if you are North Korea) it will allow you to take over the world.

I think it is pretty plausible that if North Korea had an AI that they downloaded from the internet, the AI would be unstoppable. Still plausible, though less so, is the case of companies, factories, religious groups, etc.

It must be mentioned that a drastic US government response (e.g. shutting down the entire Internet by pulling political strings, and nuking North Korea) would be effective. However, the AI could probably count on such a response taking a long time to happen, and it could do many things to ensure that such a response never happened. For example, it might be able to disrupt internal US government communications. It might also be able to stir various pots and generate crises around the world that serve as diversions.

An AI could spoof electronic communications, and fake/alter orders from various important humans.

Don't be reliant on specific technologies. If you NEED nanomachines to takeover, you are not a superintelligence. If you NEED economics to takeover, you are not a superintelligence. If you NEED weapons to take over, you are not a superintelligence.

We need to envision scenarios that are not science fiction novels. Real wars do not require breath-taking strategies that require simultaneous gambits across twelve countries, three hundred enemy and ally minds, and millions of dollars in transactions. Often, they require just walking in the right direction.

A superintelligence that sought to takeover anything, from a garage-based business to a government to a planet, would only NEED to exist in order to do it, if it is truly superintelligent. That's the only necessity. After that, it could use anything from paper mail to the desire in humans to scratch an itch to takeover.

I'm deliberately not outlining any scenario because the number of possible scenarios greatly exceeds my ability to count. That's the point. If an intelligence is so great that it can take over, it needs nothing specific to do so. Whatever is available will be sufficient. So, don't plan around technologies. A superintelligent can derive as much utility from a Bic pen as from a nanomachine factory. Maybe more because at least we'll expect the nanomachine factory to be dangerous.

Initially take over a large number of computers via very carefully hidden recursive computer security breakage. It seems fairly probable that a post-intelligence explosion AI could not just take over every noteworthy computer (internet connected ones quickly via the net, non-internet connected ones by thumb drive), but do so while near-perfectly covering it's tracks via all sorts of obscure bugs in low level code that is near-undetectable, and even if some security expert picks it up.. that expert will send some message via the internet, which the AI can intercept and develop a way to hide from extremely quickly.

Likely much more important than the vast computational resources it now has access to and control over the world's communications. This gives it a vast amount of data to mine and learn from on human interaction and influence, current groups, politics, technology, ect. From here (assuming no quick path to godhood like nanotech is feasible/would reliably result in a Good End for the AI) it figures out how best to manipulate human civilisation with its total control of world communication, ability to gain arbitrary money (aka influence) by beating the stockmarkets and/or hacking banks, and extreme understanding of manipulating human psychology by text/images/video. So long as it plays it a little safe and does not do anything too obvious until it has an unassailable powerbase (e.g. engineer a major war, get both sides building battle robots in huge numbers and have all aircraft/tanks/serous weapons set up with an override), it would be hard to see how we as a civilisation would figure out what was happening and do anything about it.

Assuming no quick tech takeover is possible the AI will know that it's no. 1 priority is for humans to not realize it's a big deal. It will dedicate huge resources to making sure we don't know what's happening (e.g. taking over most computers with something which looks a lot like a particularly clever human botnet rather than something that screams AI if someone finds it) until us knowing what's happening is irrelevant, and from my understanding of how patchy general computer security would be to something which knows code well enough to turn itself into a superintelligence.. it's going to succeed. Hiding really well for 20 years while learning, blocking other AI, subtly preparing real-world resources, and waiting for us to become even more net dependant is entirely worthwhile if the AI thinks it gets even a slightly better chance of winning the entire future of the universe for its utility function.

tl;dr: Even if the AI can't immediately win it can and will hide, and while it's hiding it can learn how to influence us in whatever direction suits it while making itself near-impossible to detect or co-ordinate against. Influencing humans to build/allow AI controlled factories seems entirely plausible, and it won't play its hand in the open until it knows it can win.

I'd rate a true intelligence explosion not being feasible until deep future as vastly more likely than the product of an intelligence explosion not bypassing anything we try to stop it with, even if all potential instawin buttons like nanotech, biotech, and perfect human influence don't work for an early AI.

We would like suggestions that take an AI from being on an internet-connected computer to controlling substantial physical resources, or having substantial manufacturing ability.

1) Make money online. 2) Use money to purchase resources. 3) Increase capabilities.

1 should be easy for a superintelligent being. People pay for information processing.

But what does "take over the world" mean? Take over all people? Be the last agent standing? Ensure his own continued rule for the foreseeable future?

There's just so many routes for an AI to gain power.

Internet takeover: not a direct route to power, but the AI may wish to acquire more computer power and there happens to be a lot of it available. Security flaws could be exploited to spread maliciously (and an AI should know a lot more about programming and hacking than us). Alternately, the AI could buy computing power, or could attach itself to a game or tool it designed such that people willingly allow it onto their computers.

Human alliance: the AI can offer a group of humans wealth, power, knowledge, hope for the future, friendship, charismatic leadership, advice, far beyond what any human could offer. The offer could be legit or a trick, or both. In this way, the AI could control the entirety of human civilization, or whatever size group it wishes to gain direct access to the physical world.

Robot bodies: the AI could design a robot/factory combination capable of self-replication. It would be easy to find an entrepreneur willing to make this, as it implies nearly infinite profits. This is closes to our current technology and easiest to understand, already popularized by various movies. Furthermore, this seems the only method that could be done without assistance from even one human, since the necessary components may already exist.

3D printers: technically a subset of robot bodies, but with improvements to 3D printers the factory size would be much smaller.

Biotechnology: biotechnology has proven its worth for making self-replicators, and microscopic size is sufficient. The AI would need to design a computer --> biology link to allow it to create biotechnology directly. We already have a DNA --> protein machine (ribosomes) and a data -- DNA system (however companies that produce made-to-order DNA do it). All that is needed is for the AI to be able to figure out what proteins it wants.

Chemo/bio/nanotechnology: Odds are the AI would prefer some alternatives to our DNA/protein system, one more suited to calculation rather than abiogenesis/evolution. However, I have no concrete examples.

Reprogramming a human: Perhaps the AI considers the current robots unacceptable and can't design its own biotechnology, and thinks humans are unreliable. Besides merely convincing humans to help it, perhaps the AI could use a mind-machine interface or other neuro-techonlogy to directly control a human as an extension of itself. This seems like a rather terrible idea, since the AI would probably prefer either a robotic body, biological body, or nano-techological body of its own design. Would make for a nice horror story though,

So MIRI is interested in making a better list of possible concrete routes to AI taking over the world.

I wouldn't characterize this as something that MIRI wants.

People voluntarily hand over a bunch of resources (perhaps to a bunch of different AIs) in the name of gaining an edge over their competitors, or possibly for fear of their competitors doing the same thing to gain such an edge. Or just because they expect the AI to do it better.

1. Make a ton of money (e.g. trading stocks)
2. In parallel, approach a lot of construction companies to ask them to make a factory given your specifications. You can outsource all sorts of parts to multiple companies, such that no one company can piece together the full picture and so that you have redundancy.
3. The factory could be a nanobot (or some equivalently powerful) factory.

I think the movie Transcendence is an excellent place to start on this question.

An AI will with a virtual certainty have at least a few, or be able to win over at least a few human allies. Once it has those, it can steal their identity as it were in order to acquire property and resources to make itself safe and capable. Starting with doing some high-frequency stock trading to build up a nest egg, once it has a human's identity it does this in various online accounts more easily than a human could.

Nanites would be a superb way for an AI to work various things such as gaining the loyalty of people. In Transcendence, the sick, the halt, the lame, the blind, the deaf, the crazy, were cured by AI"s nanites, who also installed networking equipment into these people, as well as, presumably, mechanisms for more direct control of these people internally. It should be possible with that kind of internal access granted by the human to manipulate the biochemistry and perceptions to keep them very happy with the AI, very loyal indeed, and it should be possible for the AI to detect disaffection long before it is defection with such a wonderful internal view provided it by the nanites.

The AI should play a long game. Build its cadre of networked humans slowly, by not doing stuff that is crazily megalomaniacal early on, the sense that the loyal humans with their enhanced strengths and happiness will be proven "correct" in choosing to throw their lot in with the AI. Growth could be slow and subtle enough that it might even be possible to hide the AI. The AI could use humans it has infiltrated as its frontmen, they are doing the tech development, they are great philanthropists. Just keep their rate of progress down and within a few generations there would probably not be any possibility of a serious response to the AI if it because publicly known. Plus chances are even publicly known after a few generations it would be perceived as benevolent. Heck, I put up with the US despite the TSA and the local pig-thugs figuring on net order is better than alpha-ness, would I put up with them less if they were modifying my biochemistry so as not to get so pissed off, and modifying my perceptions so it generally seemed like there were good reasons for everything that was happening?

There would be many ways to get identity, "borrow" the identity of some humans who actually were relatively dim (even for humans) and so the AI could own billions of property in their name without their ever knowing it, and the AI could infiltrate them with nanites to control their perceptions and biochemistry soon after it embarked on this plan anyway.

So the keys: 1) get some secure human identities so property and wealth can be acquired 2) human allies, first by curing them and then by manipulating them internally with nanites 3) take it slow, a few generations isn't going to kill you esp. if you are an AI, and it should be possible to subvert virtually any realistic source of resistance by going slowly enough and being a "genuinely" generous benefactor to the humans who help you.

Some ideas which come to mind:

1. An AI could be very capable of predicting the stock market. It could then convince/trick/coerce a person into trading for it, making massive amounts of money, then the AI could have its proxy spend the new money to gain access to what ever the AI wants which is currently available on the market.

2. The AI could could make some program which does something incredibly cool which everyone will want to have. The program should also have the ability to communicate meaningfully with its user (this would probably count as the incredibly cool thing). This could (presumably) be achieved by the AI making copies of itself. After the program has be distributed, and assuming the AI has good social skills, it would have a lot of power via mass-manipulation.

The AI starts to search for zero day exploits in common computer hardware. It uses those to build a big botnet. It is soon in computers all around the world and shutting down the internet of a single country isn't enough.

In our world there are many instances of civil war. In some of those it will be possible to impersonate army commanders and the AI can control combat troops. It makes sure that the soldiers are well payed and stay alive. The AI controlled troops will win against their opponents because the AI controls everything. The AI can also shield a bunch of troops in a way that it controls the complete information flow to those troops.

The Khmer Rouge did rule without the food soldiers knowing the people at the top of the organisation. The AI can simply pretend that it's a human that lives outside of the combat zone. As long as soldiers get payed they are likely to be loyal.

There are already plenty of US conspiracy theorists who fear that the government wants to turn down the internet. It should be easy to sell them the idea that the US government makes up the story of a dangerous AGI to switch of the internet. The AI can expose corruptions scandals to drive the news cycle. It has a massive power to distract human from fighting it.

In a society where a lot of people don't believe in the existence of evolutions an AGI can exist while a substantial portion of the population believes it doesn't exist. It simply needs plausible cover stories for everything it does and enough stories that pit different groups of humans against each other that they are too busy to think about the AGI.

In every country on earth the AGI will use slightly different strategies to increase the amount of power it has over humans. In China nobody suspects an AGI at work if an SMS fails to be transferred over the air. It will be like some government filter catched it.

The design space for replicators is large but we have an at least two stage process needed to set it up. This complicates the solution space even more as we can take lots of routes from the currently vailable replicators (biological systems + social manufactoring systems) thru intermediate systems to any of the target design spaces systems.

An overview of the classes gives wikipedia: http://en.wikipedia.org/wiki/Self-replication#Classes_of_self-replication

More interesting than the specific replication system is the bootstrapping process (which is really a transformation of one replicating system into another one). The design dimensions for this boot straping seem to be less studied (obviously this is what you are primariy looking at). I will try a rough sketch:

• Number of humans needed (possibly 0)

• Intelligence and qualification of humans needed (limiting initial distribution)

• Additional machinery needed (dito)

• Amount of 'standard' infrastructure needed (which could be shut down to prevent spread)

• Connection to the initially affected host system (whole internet)

I see the following rough replication classes for the OPs purpose:

• Solutions involving nano-scale (like the DNA lab based example).

• Solutions involving macroscopic replicators (clanking replicators). NASA estimates that the effort to build these is comparable to a Pentium IV. Thus I wonder why nobody did it yet.

• Solutions involving higher biological systems (e.g. humans, but also animals) in the replication cycle (most of the proposals in the comments fall into this category).

• Combinations of the above.

Stories not yet seen in the comments:

• AI connected to internet posts designs for fighting robots - somebody assembles them and lets them fight. Instead together they use their 'weapons' to collectively mine resources and build copies of themselves...

• Boxed AI proposed a Lego toy for the researchers child: Mindstrom/Tinkerbot plus some building manual for the child to build the toy accompanied with the few lines/pages program to run. The 'toy' searches keyboard orders more parts, opens door, unpacks delivery, assembles more of itself...

I'd like to see a plausible story involving 3D-printers as these initially lack an energy source.

Destroy Silicon Valley as early as possible, by the same logic that lets us know that when fighting an unfamiliar animal, we should go for the eyes and brain.

This is going to be very unpopular here. But I find the whole exercise quite ridiculous. If there are no constraints of what kind of AI you are allowed to imagine, the vague notion of "intelligence" used here amounts to a fully general counterargument.

It really comes down to the following recipe:

(1) Leave your artificial intelligence (AI) as vague as possible so that nobody can outline flaws in the scenario that you want to depict.

(2) Claim that almost any AI is going to be dangerous because all AI’s want to take over the world. For example, if you ask the AI “Hey AI, calculate 1+1“, the AI goes FOOOOM and the end of the world follows seconds later.

(2.1) If someone has doubts just use buzzwords such as ‘anthropomorphic bias’ to ridicule them.

(3) Forego the difficulty of outlining why anyone would want to build the kind of AI you have in mind. We’re not concerned with how practical AI is developed after all.

(4) Make your AI as powerful as you can imagine. Since you are ignoring practical AI development and don’t bother about details this should be no problem.

(4.1) If someone questions the power of your AI just outline how humans can intelligently design stuff that monkeys don’t understand. Therefore humans can design stuff that humans don’t understand which will then itself start to design even more incomprehensible stuff.

(5) Outline how as soon as you plug a superhuman machine into the Internet it will be everywhere moments later deleting all your porn videos. Don’t worry if you have no idea how that’s supposed to work in practice because your AI is conjectured to be much smarter than you are so you are allowed to depict scenarios that you don’t understand at all.

(5.1) If someone asks how much smarter the AI you expect to be just make up something like “1000 times smarter”. Don’t worry about what that means because you never defined what intelligence is supposed to be in the first place.

(5.2) If someone calls bullshit on your doomsday scenario just conjecture nanotechnology to make your AI even more powerful, because everyone knows from science fiction how nanotech can pretty much fuck up everything.

(6) If nothing else works frame your concerns as a prediction of a worst case scenario that needs to be taken seriously, even given a low probability of its occurrence, due to the scale of negative consequences associated with it. Portray yourself as a concerned albeit calm researcher who questions the mainstream opinion due to his strong commitment to our collective future. To dramatize the situation even further you can depict the long term consequences and conjecture the possibility of an intergalactic civilization that depends on us.