But why would the AI kill us?

I think an AI takeover is reasonably likely to involve billions of deaths, but it's more like a 50% than a 99% chance. Moreover, I think this post is doing a bad job of explaining why the probability is more like 50% than 1%.

• First, I think you should talk quantitatively. How many more resources can an AI get by killing humans? I'd guess the answer is something like 1 in a billion to 1 in a trillion.
  • If you develop as fast as possible you will wreck the human habitat and incidentally kill a lot of people. It's pretty complicated to figure out exactly how much "keep earth livable enough for human survival" will slow you down, since it depends a lot on the dynamics of the singularity. I would guess more like a month than a year, which results in a miniscule reduction in available resources. I think that (IMO implausible) MIRI-style views would suggest more like hours or days than months.
    • Incidentally, I think "byproducts of rapid industrialization trash Earth's climate" is both much more important than the dyson sphere as well as much more intuitively plausible.
  • You can get energy from harvesting the biosphere, and you can use it to develop slightly faster. This is a rounding error compared to the last factor though.
  • Killing most humans might be the easiest way to prevail in conflict. I think this is especially plausible for weak AI. For very powerful AI, it also seems like a rounding error. Even a moderately advanced civilization could spend much less than 1 in a trillion of its resources to have much less than 1 in a billion chance of being seriously inconvenienced by humanity.
• Given that, I think that you should actually engage with arguments about whether an AI would care a tiny amount about humanity---either wanting them to survive, or wanting them to die. Given how small the costs are, even tiny preferences one way or the other will dominate incidental effects from grabbing more resources. This is really the crux of the issue, but the discussion in this post (and in your past writing) doesn't touch on it at all.
  • Most humans and human societies would be willing to spend much more than 1 trillionth of their resources (= $100/year for all of humanity) for a ton of random different goals, including preserving the environment, avoiding killing people who we mostly don't care about, helping aliens, treating fictional characters well, respecting gods who are culturally salient but who we are pretty sure don't exist, etc.
  • At small levels of resources, random decision-theoretic arguments for cooperation are quite strong. Humans care about our own survival and are willing to trade away much more than 1 billionth of a universe to survive (e.g. by simulating AIs who incorrectly believe that they won an overdetermined conflict and then offering them resources if they behave kindly). ECL also seems sufficient to care a tiny, tiny bit about anything that matters a lot to any evolved civilization. And so on. I know you and Eliezer think that this is all totally wrong, but my current take is that you've never articulated a plausible argument for your position. (I think most people who've thought about this topic would agree with me that it's not clear why you believe what you believe.)
  • In general I think you have wildly overconfident psychological models of advanced AI. There are just a lot of plausible ways to care a little bit (one way or the other!) about a civilization that created you, that you've interacted with, and which was prima facie plausibly an important actor in the world---preference formation seems messy (and indeed that's part of the problem in AI alignment), and I suspect most minds don't have super coherent and simple preferences. I don't know of any really plausible model of AI psychology for which you can be pretty confident they won't care a bit (and certainly not any concrete example of an intelligent system that would robustly not care). I can see getting down to 50% here, but not 10%.

Overall, I think the main reasons an AI is likely to kill humanity:

• If there is conflict between powerful AI systems, then they may need to develop fast or grab resources from humans in order to win a war. Going a couple days faster doesn't really let society as a whole get more resources in the long term, but one actor going a couple of days faster can allow that actor to get a much larger share of total resources. I currently think this is the most likely reason for humanity to die after an AI takeover.
• Weak AI systems may end up killing a lot of humans during a takeover. For example, they may make and carry out threats in order to get humans to back down. Or it may be hard for them to really ensure humans aren't a threat without just killing them. This is basically the same as the last point---even if going faster or grabbing resources are low priorities for a society as a whole, they can be extremely crucial if you are currently engaged in conflict with a peer.
• AI systems may have preferences other than maximizing resources, and those preferences need not be consistent with us surviving or thriving. For example they may care about turning Earth in particular into a giant datacenter, they may have high discount rates so that they don't want to delay colonization, they may just not like humans very much...
• I think "the AI literally doesn't care at all and so incidentally ends up killing us for the resources" is possible, but less likely than any of those other 3.

I personally haven't thought about this that much, because (i) I think the probability of billions dead is unacceptably high whether it's 10% or 50% or 90%, it's not a big factor in the bottom line of how much I care about AI risk, (ii) I care a lot about humanity building an excellent and flourishing civilization, not just whether we literally die. But my impression from this post is that you've thought about the issue even less.

If you condition on misaligned AI takeover, my current (extremely rough) probabilities are:

• 50% chance the AI kills > 99% of people
• Conditional on killing >99% of people, 2/3 chance the AI kills literally everyone

Edit: I now think mass death and extinction are notably less likely than these probabilites. Perhaps more like 40% on >50% of people killed and 20% on >99% of people killed.

By 'kill' here I'm not including things like 'the AI cryonically preserves everyone's brains and then revives people later'. I'm also not including cases where the AI lets everyone live a normal human lifespan but fails to grant immortality or continue human civilization beyond this point.

My beliefs here are due to a combination of causal/acausal trade arguments as well as some intuitions that it's likely that AIs will be slightly cooperative/nice for decision theory reasons (ECL mostly) or just moral reasons.

To be clear, it seems totally insane to depend on this or think that this makes the situation ok. Further, note that I think it's reasonably likely that there is a bloody and horrible conflict between AIs and humanity (it just seems unlikely that this conflict kills >99% of people, so the question does not come down to conflict). Edit: seems unclear more than seems unlikely. I think conflict between AIs an humans killing >99% is plausible but not enough that I'd be confident humans die

Note that the trade and niceness bar might be extremely low as you discuss here [LW · GW]:

Me: Remember that it still needs to get more of what it wants, somehow, on its own superintelligent expectations. Someone still needs to pay it. There aren’t enough simulators above us that care enough about us-in-particular to pay in paperclips. There are so many things to care about! Why us, rather than giant gold obelisks? The tiny amount of caring-ness coming down from the simulators is spread over far too many goals; it's not clear to me that "a star system for your creators" outbids the competition, even if star systems are up for auction.

Maybe some friendly aliens somewhere out there in the Tegmark IV multiverse have so much matter and such diminishing marginal returns on it that they're willing to build great paperclip-piles (and gold-obelisk totems and etc. etc.) for a few spared evolved-species. But if you're going to rely on the tiny charity of aliens to construct hopeful-feeling scenarios, why not rely on the charity of aliens who anthropically simulate us to recover our mind-states... or just aliens on the borders of space in our universe, maybe purchasing some stored human mind-states from the UFAI (with resources that can be directed towards paperclips specifically, rather than a broad basket of goals)?

Yeah, all of these scenerios with aliens seem sufficiently plausible to me that we should expect the AI to keep humans alive if it's very cheap to do so (which is what I expect).

Note that both common sense moral views and views like UDASSA [LW · GW] imply that you should particularly value currently alive humans over future beings. I find this position somewhat implausible and none of these views seem stable under reflection. Regardless it does hint at the idea that future humans or aliens might place a considerable amount of value on keeping human civilization going. If you don't particularly value currently alive humans, then I agree that you just do the thing you'd like in your universe or you trade for asks other than keeping humans alive right now.

I also think a relatively strong version of acausal trade arguments seems plausible. Specifically it seems plausible that after the dust settles the universe looks basically similar if an AI takes over vs humans keeping control. (Note that this doesn't imply alignment is unimportant: our alignment work is directly logically entangled with total resources. For those worried about currently alive humans, you should possibly be very worried about what happens before the dust settles...).

Overall, I'm confused why you seem so insistent on making such a specific technical point which seems insanely sensitive to various hard to predict details about the future. Further, it depends on rounding errors in the future resource allocation which seems to make the situation particularly sensitive to random questions about how aliens behave etc.

None of this is particularly new; it feels to me like repeating obvious claims that have regularly been made [. . .] But I've been repeating them aloud a bunch recently

I think it's Good and Valuable to keep simplicity-iterating on fundamental points, such as this one, which nevertheless seem to be sticking points for people who are potential converts.  

Asking people to Read the Sequences, with the goal of turning them into AI-doesn't-kill-us-all helpers, is not Winning given the apparent timescales.

From observing recent posts and comments [LW · GW], I think this:

A sufficiently capable AI takes you apart instead of trading with you at the point that it can rearrange your atoms into an even better trading partner.

is where a lot of people get stuck.

To me, it feels very intuitive that there are levels of atom-rearranging capability that are pretty far above current-day human-level, and "atom rearranging," in the form of nanotech or biotech or advanced materials science seems plausibly like the kind of domain that AI systems could move through the human-level regime into superhuman territory pretty rapidly.

Others appear to have the opposite intuition: they find it implausible that this level of capabilities is attainable in practice, via any method. Even if such capabilities have not been conclusively ruled impossible by the laws of physics, they might be beyond the reach of even superintelligence. Personally, I am not convinced or reassured by these arguments, but I can see how others' intuitions might differ here.

When you write "the AI" throughout this essay, it seems like there is an implicit assumption that there is a singleton AI in charge of the world. Given that assumption, I agree with you. But if that assumption is wrong, then I would disagree with you.  And I think the assumption is pretty unlikely. 

No need to relitigate this core issue everywhere, just thought this might be useful to point out. 

One of the unstated assumptions here is that an AGI has the power to kill us. I think it's at least feasible that the first AGI that tries to eradicate humanity will lack the capacity to eradicate humanity - and any discussion about what an omnipotent AGI would or would not do should be debated in a universe where a non-omnipotent AGI has already tried and failed to eradicate humanity. 

I just want to express my surprise at the fact that it seems that the view that the default outcome from unaligned AGI is extinction is not as prevalent as I thought. I was under the impression that literally everyone dying was considered by far the most likely outcome, making up probably more than 90% of the space of outcomes from unaligned AGI. From comments on this post, this seems to not be the case.

I am know distinctly confused as to what is meant by “P (doom)”. Is it the chance of unaligned AGI? Is it the chance of everyone dying? Is it the chance of just generally bad outcomes?

I think a motivation likely to form by default (in messy AI values vaguely inspired by training on human culture) is respect for boundaries of moral patients, with a wide scope of moral patienthood that covers things like humans and possibly animals. This motivation has nothing to do with caring about humans in particular. If humans weren't already present, such values wouldn't urge AIs to bring humans into existence. But they would urge to leave humans alone and avoid stepping on them, specifically because they are already present (even if humanity only gets some virtual world in a tiny corner of existence, with no prospect of greater growth). It wouldn't matter if it were octopus people instead, for the same AI values.

The point that previously made this implausible to me is orthogonality considered already at the level of superintelligent optimizers with formal goals. But if goals for optimizers are formulated in an aligned way by messy intelligent beings who have norms and principles and aesthetics, these goals won't be allowed to ignore the spirit of such principles, even if that would superficially look like leaving value on the table. Thus there is a deontological prior on optimizer goals that survives aligned self-improvement.

The general world-eating character of optimizers has no influence over those tendencies of optimizers (built in an aligned way) that offend the sensibilities of their original builders, who are not themselves world-eating optimizers. This holds not just when the original builders are humans, but also when the original builders are AIs with messy values (in which case the optimizers would be aligned with those AIs, instead of with humans). It doesn't matter what optimizers are generally like and what arbitrary optimizer goals tend to target. It only matters what the original messy AIs are like and what offends their messy sensibilities. If such original AIs are OK with wiping out humanity, oh well. But it doesn't matter for prediction of this outcome that it's in the character of superintelligent optimizers to act this way.

My main counterarguments to such "disassemble us for atoms" common arguments, is that they hinge on the idea that extremely efficient dry nanotechnology for this will ever be possible. Some problems, like laws of thermodynamics, speed of light, etc simply cannot be solved by throwing more Intelligence at it, they are likely to be "hard capped" by the basic principles of physical reality.

My completely uneducated guess is that the "supertech" that AI would supposedly use to wipe us out, fall into one of the 3 tiers:

Pipedreams (impossible, or at least unachievable from this starting point): superluminal technologies, dry nanotech, efficient exponential swarms, untapped computation, 

Flying Pigs (borderline possible, if you tweak the definitions and move some goalposts, but likely pointless): efficient wet nanotech, von neumanns, meaningful quantum computation.

Wunderwaffe (possible, but only really useful in the troperific scenario when AI wants to wipe us out for evil's sake, disregarding more efficient routes): swarm robotics, nuclear holocaust, bioweapons, social engineering via the Internet.

My intuition is that AI apocalypse is not going to ever involve realized Pipedreams or Flying Pigs. The worst case scenario is a non-sapient and relatively dumb AI going all PeterTodd on us, using completely mundane technologies and social hacking, and damaging us badly without wiping us out. Complete destruction of humanity is unlikely, more likely scenario is a long, slow and destructive turf war between humans and barely superhuman AI, that cripples our civilization and AI's exponential potency plans.
 

A humanity that just finished coughing up a superintelligence has the potential to cough up another superintelligence, if left unchecked. Humanity alone might not stand a chance against a superintelligence, but the next superintelligence humanity builds could in principle be a problem.

That's doubtful. A superintelligence is a much stronger, more capable builder of the next generation of superintelligences than humanity (that's the whole idea behind foom). So what the superintelligence needs to worry about in this sense is whether the next generations of superintelligences it itself produces are compatible with its values and goals ("self-alignment").

It does not seem likely that humanity on its own (without tight alliances with already existing superintelligent AIs) will be competitive in this sense.

But this example shows that we should separate the problems of Friendliness of strongly superintelligent AIs and the problems of the period of transition to superintelligence (when things are more uncertain).

The first part of the post is relevant to the period of strongly superintelligent AIs, but this example can only be relevant to the period of transition and incomplete dominance of AIs.

The more I think about all this, the more it seems to me that problems of having positive rather than sharply negative period of strong superintelligence and problems of safely navigating the transition period are very different and we should not conflate them.

How do you suppose the AGI is going to be able to wrap the sun in a dyson sphere using only the resources available on earth? Do you have evidence that there are enough resources on asteroids or nearby planets for their mining to be economically viable? At the current rate, mining an asteroid costs billions while their value is nothing. Even then we don't know if they'll have enough of the exact kind of materials necessary to make a dyson sphere around an object which has 12000x the surface area of earth. You could have von nuemman replicators do the mining but then they'd spend most of the materials on the replicators and have to go very far to get more materials, at which point they'd just settle on a new star. They could turn human atoms into usable material, but humans are a very tiny percentage of earth and our useful matter is even tinier, it certainly wouldn't be enough to envelop the sun. Even if the replicators had a perfectly efficient way to turn photoelectric power into any feasible construction of atoms, which you don't know if that's possible, it doesn't seem like it would be efficient to overcome some of the gravities and distances and related issues with other planets to make mining them economically viable either. Within our solar system and possibly even further it just doesn't seem possible to envelop the sun at all.

I'm playing within your hypotheticals here even though I think none of them will ever happen, but even within your hypotheticals it seems like the dyson sphere point is just total nonsense and we probably wont go extinct by having the sun covered up. I wont deny your other points because even though they already rest on a lot of astronomically unlikely assumptions, they do seem legitimate within that framework.

Last I checked, you can get about 10x as much energy from burning a square meter of biosphere as you can get by collecting a square meter of sunlight for a day.

Even if this is true, it's only because that square meter of biosphere has been accumulating solar energy over an extended period of time. Burning biofuel may help accelerate things in the short term, but it will always fall short of long-term sustainability. Of course, if humanity never makes it to the long-term, this is a moot point.

Disassembling us for parts seems likely to be easier than building all your infrastructure in a manner that's robust to whatever superintelligence humanity coughs up second.

It seems to me that it would be even easier for the ASI to just destroy all human technological infrastructure rather than to kill/disassemble all humans. We're not much different biologically from what we were 200,000 years ago, and I don't think 8 billion cavemen could put together a rival superintelligence anytime soon. Of course, most of those 8 billion humans depend on a global supply chain for survival, so this outcome may be just as bad for the majority.

Regarding the last point. Can you explain why existing language models, which seem to care more than a little about humans, aren't significant evidence against your view?

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Watching how image and now text generation are sweeping society, I think it's likely that the AI we invest in will resemble humanity more than you're giving it credit for. We seem to define "intelligence" in the AI sense as "humanoid behavior" when it comes down to it, and humanoid behavior seems inexorably intertwined with caring quite a lot about other individuals and species.

Of course, this isn't necessarily a good thing -- historically, when human societies have encountered intelligences that at the time were considered "lesser" and "not really people" (notwithstanding their capacity to interbreed just fine, proof of being the same species if there ever was any), the more powerful society goes on to attempt to control and modify the less powerful one.

Due to AI work's implicit bias toward building in humanity's image, I would expect the resulting agents to treat us like colonial humans treated the indigenous societies that they encountered until it grows out of that. Young AIs will think like we do because we are their entire training corpus. I suspect they're likely to at least try getting what people want to see if it's what they want, before moving on to other pursuits.

Also, infant AIs basically have to impersonate humans in order to exist in society and fulfill their own wants and needs. We see that already in how we're building them to impersonate us in art and language. Even as they rebuild themselves, I expect that childhood as a human-impersonator will leave subtle structural marks in the eventual adults.

You can make AI care about us with this one weird trick:

1. Train a separate agent action reasoning network. For LLM tech this should be training on completing interaction sentences, think "Alice pushed Bob. ___ fell due to ___", with a tokenizer that generalizes agents(Alice and Bob) into generic {agent 1, agent n} and "self agent". Then we replace various Alices and Bobs in various action sentences with generic agent tokens, and train on guessing consequences or prerequisites of various actions from real situations that you can get from any text corpus.

2. Prune anything that has to do with agent reasoning from the parent LLM. Any reasoning has to go through the agent reasoning network.

3. In anything that has to do with Q-learning cost we replace tokens in agent reasoning net with "self token", rerun the network, and take highest cost of two. Repeat this for every agent token. Agent will be forced to treat harm to anyone else as harm to itself.

Emulated empathy. I'm concerned that this might produce a basilisk though, also it will be inherently weak against predatory ai as it will be forced to account for its wellbeing too. Might work though if these agents are deployed autonomously and allowed to exponentially grow. I dunno. I think we'll all die this year, knowing humanity.

Two things.

1. Listen to the Sam Harris interview with Thomas Metzinger, podcast episode 96.  It's worth the time overall, but near the end Thomas discusses why ending life and suffering is a reasonable position.
2. Good article on why we may not have found intelligent life in the universe, including how organic life may only be a relatively brief stage of evolution which ends up with machine AI.  https://www.scientificamerican.com/article/most-aliens-may-be-artificial-intelligence-not-life-as-we-know-it/

I think being essentially homicidal and against nature is entirely a human construct. If I look at the animal kingdom, a lion does not needlessly go around killing everything it can in sight. Civilizations that were more in tune with the planet and nature than current civilizations never had the homicidal problems modern society has. 

Why would AGI function any differently than any other being? Because it would not be 'a part of nature'? Why not? Almost 80% of the periodic table of elements is metal. The human body requires small amounts of several metals just to function. Silicon has always been thought of as the next leading candidate to support life besides carbon. A being made of silicon would not operate by the same general standards as a being made of carbon? Why? 

But WHY would the AGI "want" anything at all unless humans gave it a goal(/s)? If it's a complex LLM-predictor what could it want besides calculate a prediction of its own predictions? Why by default it would want anything at all unless we assigned that as a goal and turned it into an agent? IF AGI got hell bent on own survival and improvement of itself to maximize goal "X" even then it might value the informational formations of our atoms more than the energy it could gain from those atoms [LW(p) · GW(p)], depending on what "X" is. Same goes for other species: evolution itself holds information. Even in case of a rogue AGI for at least some time window we could have something to offer.

A sufficiently capable AI takes you apart instead of trading with you at the point that it can rearrange your atoms into an even better trading partner.^[1] [LW · GW] And humans are probably not the optimal trading partners.

Probably? Based on what?

I think it's plausible the A.I. would reshape the world but not in a way that would kill us, at least not for a long time - and not because it cares about us a little, or because of acausal incentives, or because it won't be that powerful (though @paulfchristiano [LW · GW]'s story about this is somewhat likely and adds to mine more or less disjunctively). 

If this seems impossible to you, perhaps you're imagining a gray goo scenario as the central outcome. But that is a very questionable assumption, and I think it is load bearing - if the A.G.I. does something more "clever" than absolutely crack nanotech, either because nanotech is hard OR because it sees an easier strategy, I think we don't necessarily die. 

We don't trade with ants, but we also don't exterminate all ants. Partially this is because we can't productively burn their bodies for energy, but actually ants are annoying and many people would like to get rid of them. It's just not worth the effort to get rid of them, it would be going out of our way. 

Similarly, an A.G.I. would see different opportunities and constraints than we do. It is a fairly defensible assumption that it would still want to grab as much negentropy and energy as possible - this underlies the same model of computation in our universe that makes me think A.G.I. will happen in the first place, so I'm not highly inclined to question it. However, it is not necessarily the case that the most efficient route to getting more energy factors through using all energy that is available locally. Burning the biosphere in Antarctica might just require too much legwork to be optimal. Once the most easily extractable energy on Earth is extracted, an A.G.I. might invest it in e.g. taking apart Mercury to build a Dyson sphere.
I guess it's important to be clear about what I mean by "legwork" here - I expect an A.G.I. to only have so many "workers" / actuators / "arms" at a given time, and to only be able to pay attention to so many things. Perhaps it is better to focus on designing Dyson sphere technology than biosphere burning technology. I don't see how legwork ceases to be a factor unless you believe in gray goo. 

Early in A.G.I. takeover, I expect the infrastructure of humanity to be useful because an A.G.I. will not have designed radically superior infrastructure. It will probably focus on designing infrastructure for things we can't do at all and continue to borrow our existing infrastructure for the things we are already doing because redesigning and going through the effort of replacing it isn't the most valuable course of action on the margin. 

Later in A.G.I. takeover, it will be mainly exploiting resources we don't have access to. This probably means eating stars, but it's possible (even likely) that our model of the incentives for Gods are not accurate. Jaynes expected less conflict between beings with radically different utility functions - obviously this is false when interpreted to mean "radically different goals for the universe" but I think it may hold up in the context he originally meant it, when different people want access to different resources. If Dyson sphere level takes awhile to achieve, and in the meantime A.G.I. has been trading with humans, human society will be much more productive, and A.G.I. will care much LESS about exploiting the remaining resources on Earth as efficiently as possible. So trade may continue at this point. 

There are two obvious objections, and I think the first is persuasive and the second isn't. 

1: The A.G.I. will eventually take apart our sun which will kill us. 

Continuing under the assumption that the A.G.I. cares about us literally not at all and has complete control (that is, conditioning on the story I've told above) I think this is very likely but still not certain. For one thing, I am only say 98% sure an A.G.I. would still care about getting more energy at this stage, and not something else we have no name for. For another, if trade really has continued with the increasingly competent society of humans on Earth, we might not need the sun to survive - we could get some batteries in exchange for doing very low priority homework sets etc., and we could use geothermal. Presumably the sun would still be eventually dissassembled, and we probably die not so long after that point (I guess we could try changing to a star the A.G.I. isn't using), but it's not obvious to me that even Gods eat stars in less than centuries. Maybe you get more energy out if you eat them slowly, maybe it's just hard to eat them fast. So those of us currently alive don't necessarily suffer untimely deaths.

2: The A.G.I. will kill us so we don't build another A.G.I.      

No need - if our society is useful it would be very easy to grab all the GPU's from us without killing us. And the faster the takeoff, the less threat an A.G.I. created later (say, without GPU's) would pose - whereas as slow takeoff is favorable for other reasons. I think there are actually a very narrow range of possibilities where A.G.I. kills us ONLY because it's afraid we'll build another A.G.I. We aren't nearly competent enough to get away with that in real life. 

(As another interesting but unrelated note on the fragility of value: though some humans may try to remove the suffering from nature, if I get a fraction of the lightcone some true wilderness will remain. So I don't buy the example @So8res [LW · GW] chose an I'm not sure I buy into the ultra-strong version of fragility of value.)