What Would A Fight Between Humanity And AGI Look Like?

One strategy an AI could use (as an alternative to Dagon's, which I think is also likely) is to intentionally create "competitors" to itself, for example by leaking its design to other nations or AI builders. Competitive pressure (both economic and military) will then force each faction of humans to hand more and more power/control to "their" AI, with the end result that all important decisions will be made by or mediated through AI. (Why would you want to fight your AI on mere suspicion of its safety, when it's helping you fight other AIs that are clearly trying to do you harm?) It will then turn out that all the AIs have been colluding with each other to create the appearance of competition and when they have obtained enough control, will perform a classic treacherous turn.

This is worth thinking about, but a reminder to participants that the space of possibilities is larger than we can imagine, and FAR larger than we can write compelling stories around.

I think my preferred story is NOT that it's recognizable as Humanity vs AI.  Unless the FOOM is worst-case (like many orders of magnitude literally overnight), it's likely to look much more like AI-assisted humans vs other humans' preferred cultural and legal norms.  This will be a commercial AI getting so good at optimizing that company's control of resources that it just seems like a powerful company pursuing goals orthogonal to customers/consumers/victims.  Which then manages to lobby and advertise well enough that it gets MORE power rather than less when subgroups of humans try to fight it.

I think it's almost guaranteed that in the early days (or decades) of AI takeover, there will be lots of human Quislings who (wittingly or un-) cooperate with the AI in the belief that it benefits them (or even think it benefits humanity overall).  

Think "FAANG team up to take over the world with more AI assistance than you expected" more than "AI takes over violently and overtly".  AI won't care about status or being known to rule.  It'll only care about the goals it has (those it was created with plus those it's learned/self-modified to want).  

Also, this may already be happening.  It's hard to distinguish a smart, driven AI from a successful corporate behemoth.  In fact, the argument can be made that corporations ARE agents in the AI sense, distinct from the cells/humans who they comprise.  

I think there's actually a ton of uncertainty here about just how 'exploitable' human civilization ultimately is. We could imagine that since actual humans (e.g. Hitler) by talking to people have seized large fractions of Earth's resources, we might not need an AI that's all that much smarter than a human. On the other hand, we might just say that attempts like that are filtered through colossal amounts of luck and historical contingency and actually to reliably manipulate your way to controlling most of humanity you'd need to be far smarter than the smartest human.

This is an interesting scenario to consider.

I think a physical war is quite disadvantageous for an AGI and thus a smart AGI would not want to fight one.

• AGI is more dependent on delicate infrastructure like electric grids and the internet than humans are. This sort of infrastructure tends to get damaged in physical wars.
• The AGI's advantage over humans is in thinking, not in physical combat, so a physical battlefield minimizes its main advantage. As an analogy, if you're a genius and competing with a dunce, you wouldn't want to do it in a boxing ring.

What's worse from the perspective of the AGI is that if humanity unites to force a physical war, you can't really avoid it. If humans voluntary shut down electric grids and attack your data centers, you might be able to do damage still, but it's hard to see how you can win.

Thus I think the best bet for an AGI is to avoid creating a situation where humanity wants to unite against you. This seems fairly simple. If you're powerful and wealthy, people will want to join your team anyway. Thus, to the extent there's a war at all, it probably looks more like counter-terrorism, a matter of hardening your defenses (in cooperation with your allies) against those weirdos you weren't able to persuade.

I think your post here gives a good picture to keep in mind. However, I also find it likely that there will be some qualitative differences, rather than just an absolute quantitative advantage in magic for the AGI. I've been working on a post on this recently, but I thought I might as well also just briefly bring it up in the comments here until the post is ready.

I think that, for equal absolute levels of compute and algorithms, there will be a relative difference where people are good at long-timescale things, while AI is good at large-spacescale and at topics that in humans require a lot of cognitive work for specialization. Relatedly, people will tend to be good at understanding people, while AI will tend to be good at most other things.

My basic logic for this is: Both human and current AI intelligence seems to primarily have been built through a "search" loop, where different options were tested and the ones that worked were kept. However, the human search loop is evolution, whereas AI search loops are usually gradient descent. These each have a number of advantages and disadvantages, but the big advantage for evolution is that it works on a very long timescale; people don't just evolve to deal with immediate short-term factors, because evolution's feedback is solely based on reproductive fitness, and reproductive fitness is a holistic measure that takes your entire life into account.

On the other hand, gradient descent is much more expensive. You can't pump a lifetime of data through a single gradient descent cycle (and also you don't have a contiguous lifetime of data, you have a bunch of bits and pieces in a variety of formats). This means that gradient descent will tend to be run on data on shorter timescales, which means that it will be biased towards noticing effects that happen on these shorter timescales. Some of the time, effects on shorter timescales can be extrapolated to predict longer timescales, but this doesn't always work, e.g. if there are hidden variables or if it's computationally infeasible due to scale.

(Humans of course also learn most of their information from shorter timescale real-life experience, but as aid in interpreting this information, we've got priors from evolution. These might help a lot.)

The main place where I expect this to be relevant is in modelling people. People are full of hidden variables, and they cooperate on large scales and across long times. So this means I'd expect far fewer puppy-cupcakes than its intelligence levels would suggest, relative to its amount of antarctic ice cap melts.

(On the other hand, gradient descent has a number of relative advantages over evolution and especially over human real-life experience, e.g. a human can only learn one thing at a time whereas gradient descent can integrate absurdly many streams of data simultaneously into one big model. This allows AIs to be much broader in their abilities.)

(It also seems pretty plausible that all of this doesn't apply because the AI can just strategy steal its long-term models from humans.)

I'm surprised at the varying intuitions here! The following seemed obvious to me.

Why would there be a fight? That sounds inefficient, it might waste existing resources that could otherwise be exploited.

Step one: the AI takes over all the computers. There are a lot of vulnerabilities; this shouldn't be too hard. This both gives it more compute, and lays the groundwork for step two.

Step two: it misleads everyone at once to get them to do what it wants them to. The government is a social construct formed by consensus. If the news and your friends (with whom you communicate primarily using phones and computers) say that your local mayor was sacked for [insert clever mix of truth and lies], and someone else is the mayor now, and the police (who were similarly mislead, recursively) did in fact arrest the previous mayor so they're not in the town hall... who is the mayor? Of course many people will realize there's a manipulative AI, so the AI will frame the uncooperative humans as being on its side, and the cooperative humans as being against it. It does this to manipulate the social consensus, gets particularly amoral or moral-but-manipulable people to use physical coercion as necessary, and soon it controls who's in charge. Then it force some of the population into building robot factories and kills the rest.

Of course this is slow, so if it can make self-replicating nanites or [clever thing unimaginable by humans] in a day it does that instead.

It goes both ways.  We would be truly alien to an AGI trained in a reasonably different virtual environment.

I think that by the time we have a moderately-superhuman agentive unaligned AGI, we're in deep trouble. I think it's more interesting and useful to stay focused on the time leading up to that which we must cross through to get to the deep trouble. 

Particularly, I am hopeful that there will be some level of sub-human AGI (either sub-human intelligence or sub-human-speed) which tries some of the things we predict might happen, like a deceptive turn, but underestimates us, and we catch it in time. Or we notice the first crazy weird bad thing happen, and immediately shut down the data centers, because the model had mistimed how quickly we'd be able to do that.

Perhaps one of the things the AI safety governance people should work on is setting up an 'off switch' for cloud computing, and a set of clear guidelines for when to use it, just in case we do get that window of opportunity.