In defense of technological unemployment as the main AI concern

If humanity was indefinitely competitive in most tasks, the AIs would want to trade with us or enslave us instead of murdering us or letting us starve to death.

This sentence (in the context of the broader post) seems to assume that "being competetive in most tasks" and "technological unemployment" are the same. However, they very importantly are not. In-general, because of comparative advantage dynamics (i.e. situations where one party might totally dominate on productivity of all tasks you still have opportunity for trade), I don't think there is a pure economic case that technological unemployment would be correlated with lack of competitiveness compared to AI. 

And so I don't really think that existential risk is caused by "unemployment". People are indeed confused about the nature of comparative advantage, and mistakenly assume that lack of competetiveness will lead to loss of jobs, which will then be bad for them.

But the actual risk comes from adversarial dynamics where we don't want to hand over the future of the universe to the AIs, but they AIs sure would like to have it. And if humanity could coordinate better, it would be able to just wait a few decades and seize the future, but it probably won't do that.

It's not like I can't imagine at all calling what is going on here an "unemployment" issue, but I feel like in that case I would need to also call violent revolutions or wars "unemployment" issues, since like, if I could just engage in economic trade with my enemies, we wouldn't need to fight, but clearly my enemies want to take the stuff I already have, and want to use the land that I "own" for their own stuff, and AIs will face the same choice, and that really seems quite different from "unemployment".

I agree with this post, and IMO, technological unemployment is the main issue with AI in my model of the benefit-cost accounting from AI.

One point I do disagree with you on is this:

This loss is what causes existential risk.

I disagree with that, primarily because of 2 things:

1. I still expect humans to be able to control AI, and more importantly I think it's pretty likely that something like corrigibility/personal intent alignment happens by default.

2. I think that existential risk gets overused, because the original definition was a risk that eliminated humanity's potential, and I don't think technological employment actually does this, but I do think that something like a global catastrophe could happen if tech unemployment is handled badly.

That's because I expect at least a few humans to be able to create Dyson Swarms using their superintelligent AIs, and this means that existential risk is avoided in that scenario, and we wouldn't call the bottleneck event that happened 70,000 years ago which reduced human populations to 1,000 people an existential risk.

That said, I agree that something like decoupling professional humans from value production, or the rich from any constraints on how they are able to shape the world can definitely be a big problem.

At this point, I wouldn't agree that that is the main concern. I do think it's a solid third place though. 

My top two are:

1. Deliberately harmful use

   Some human decides they want to kill some other humans, and an AI enables them to acquire more power to do so. This could scale really fast, and actually give a single human the capability to wipe out nearly all of humanity. I would not like to bet my loved ones lives on there not being some person in the world crazy enough to try this.

2. Rogue AGI

   I think it's possible that some crazy human might decide that AI becoming an independent species is a good thing actually, and deliberately creating and releasing an AI system which is sufficiently capable and general that it manages to take off. There's also the possibility an AI will independently develop Omohundro Drives and escape of its own volition. That seems less likely to me, but the two probabilities sum together, so... pick your nightmare, I guess.

Both of these threats route through the possibility of AI enabling offense-dominant self-replicating weaponry (bioweapons and/or nanotech). Self-replicating weaponry is a particularly dangerous category, because a small actor can cheaply initiate huge irrevocable effects thereby.

For more details, see this other comment I wrote today. [LW(p) · GW(p)]

I agree that the negative outcomes from technological unemployment do not get enough attention but my model of how the world will implement Transformative AI is quite different to yours.

Our current society doesn't say "humans should thrive", it says "professional humans should thrive"

Let us define workers to be the set of humans whose primary source of wealth comes from selling their labour. This is a very broad group that includes people colloquially called working class (manual labourers, baristas, office workers, teachers etc) but we are also including many people who are well remunerated for their time, such as surgeons, senior developers or lawyers. 

Ceteris paribus, there is a trend that those who can perform more valuable, difficult and specialised work can sell their labour at a higher value. Among workers, those who earn more are usually "professionals". I believe this is essentially the same point you were making.

However, this is not a complete description of who society allows to "thrive". It neglects a small group of people with very high wealth. This is the group of people who have moved beyond needing to sell their labour and instead are rewarded for owning capital. It is this group who society says should thrive and one of the strongest predictors of whether you will be a member is the amount of wealth your parents give you.

The issue is that this small group is owns a disproportionate proportion of shares in frontier AI companies. 

Assuming we develop techniques to reliably align AGIs to arbitrary goals, there is little reason to expect private entities to intentionally give up power (doing so would be acting contrary to the interests of their shareholders).

Workers unable to compete with artificial agents will find themselves relying on the charity and goodwill of a small group of elites. (And of course, as technology progresses, this group will eventually include all workers.)

Those lucky enough to own substantial equity in AI companies will thrive as the majority of wealth generated by AI workers flows to them.

In itself, this scenario isn't an existential threat. But I suspect many humans would consider their descendants being trapped into serfdom is a very bad outcome.

I worry a focus on preventing the complete extinction of the human race means that we are moving towards AI Safety solutions which lead to rather bleak futures in the majority of timelines.^[1]
 

1. ^{^}

   My personal utility function considers permanent techno-feudalism forever removing the agency of the majority of humans is only slightly better than everyone dying.

   I suspect that some fraction of humans currently alive also consider a permanent loss of freedom to be only marginally better (or even worse) than death.

I thought job loss was a short-term but not a long-term risk and only a secondary cause of x-risks. Which is why we don't worry about it around here much. FWIW being enslaved permanently doesn't sound better to me than being extinct, so I wouldn't frame that distinction as a problem.

I thought the standard plan was: we figure out how to ensure that an aligned AGI/ASI takes over; it helps humans because it wants to. It's way better than us at making stuff, so keeping humanity in good conditions is actually super easy, barely an inconvenience.

That could be either because we got value alignment right, or we put personal intent aligned AGI into the hands of someone with the common decency to help all of humanity with it. And it/they wisely decided to prevent the creation of other AGI.

The ASI makes sub-sentient safe minds to do the grunt work; humanity's biggest challenge is deciding what to do with unlimited time and vast resources.

Anyone who's understood a Burning Man event knows that humans have a blast working on collaborative projects, even when they don't need to be done at all, so I have no concerns that we'll all get sad without necessary or "meaningful" work to do.

So job loss seems like a problem only if it makes us freak out and do something stupider. 

Perhaps you're thinking of the scenario where AGI takes over the economy but doesn't otherwise directly care about humans. I think job loss is a weird way to frame humans being utterly obsoleted and having their oxygen eventually "purchased" for better uses. AGI that doesn't care about humans but does care about property rights would be strange. FWIW, I think people sometimes see AGI proliferation as a potential good outcome, but that's exactly wrong; the multipolar scenario with personal intent aligned AGI serving different competing masters [LW · GW] seems likely to result in doom, too.

rationalists think that it is a bigger risk that the AI might murder us all, and that automation gives humans more wealth and free time and is therefore good.

I don't think this is really accurate. It's not about "bigger." For humanity, surviving and thriving depend on us overcoming or avoiding all the x-risks. Rather, many rationalists expect everyone-not-dying to be the first ASI safety gate we have to get through, temporally and ontologically (if we're worrying about unemployment, we're alive to do so). 

Thanks for writing this, this is something I have thought about before trying to convince people who are more worried about "short-term" issues to take the "long-term" risks seriously. Essentially, one can think of two major "short-term" AI risk scenarios (or, at least "medium-term" ones that "short-term"ists might take seriously), essentially corresponding to the prospects of automating the two factors of production:

1. Mass technological unemployment causing large swathes of workers to become superfluous and then starved out by the now AI-enabled corporations (what you worry about in this post)
2. AI increasingly replacing "fallible" human decision-makers in corporations if not in government, pushed by the necessity to maximize profits to be unfettered by any moral and legal norm (even more so than human executives are already incentivized to be; what Scott worries about here)

But if 1 and 2 happens at the same time, you've got your more traditional scenario: AI taking over the world and killing all humans as they have become superfluous. This doesn't provide a full-blown case for the more Orthodox AI-go-FOOM scenario (you would need ), but at least serve as a case that one should believe Reform AI Alignment is a pressing issue, and those who are convinced about that will ultimately be more likely to take the AI-go-FOOM scenario seriously, or at least operationalize one's differences with believers in only object-level disagreements about intelligence explosion macroeconomics, how powerful is intelligence as a "cognitive superpower", etc. as opposed to the tribalized meta-level disagreements that define the current "AI ethics" v. "AI alignment" discourse.