Linkpost: Robin Hanson - Why Not Wait On AI Risk?

IMO, Hanson isn't engaging with the arguments for risk, instead he's giving his own argument for safety. I therefore don't feel particularly motivated to engage with him. 

Example: He speaks abstractly about how coups are already a thing between humans and how therefore we shouldn't be concerned about AI coups unless we have special reason to, like Eliezer who thinks progress will be lumpy. Well, that's one reason I guess. But plenty of people who don't think progress will be lumpy (such as Paul Christiano) still are worried about AI coups; Hanson should go read what they wrote and understand why they think it's still a risk despite the existence of insurance schemes etc. and then point out which premise he thinks is faulty. For example, if he were talking to me, he could read what 2026 looks like [LW · GW] and then say what part of it seems extremely implausible to him & suggest more plausible ways the story could go instead. Then we could argue about which revisions to the story are best & then continue the story with those revisions together until we get to coup time, which (spoiler) will probably be in 2027-2028 if the story continues the way I think is most plausible. And then he'd be in a corner; he'd have to say why this story doesn't represent a possibility worth taking seriously and trying to prevent. And if he thinks it's worth taking seriously but still overall unlikely, I'd challenge him to write a similarly detailed story that he thinks is more plausible & let me critique it.)

Anyhow, giving his arguments a skim...

I guess there's a lot I disagree with, actually. But just to pick one example because I'm lazy:

Hanson, about Yudkowsky:

Alternatively, he claims (quite implausibly I think) that all AGIs naturally coordinate to merge into a single system to defeat competition-based checks.

Yudkowsky, in his Big List O' Doom [LW · GW]:

34.  Coordination schemes between superintelligences are not things that humans can participate in (eg because humans can't reason reliably about the code of superintelligences); a "multipolar" system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which looks like "the 20 superintelligences cooperate with each other but not with humanity".

Me, from  three years ago: [LW(p) · GW(p)]

To a tiger, human hunter-gatherers must be frustrating and bewildering in their ability to coordinate. "What the hell? Why are they all pouncing on me when I jumped on the little one? The little one is already dead anyway, and they are risking their own lives now for nothing! Dammit, gotta run!"

To a tribe of hunter-gatherers, farmers must be frustrating and bewildering in their ability to coordinate. "What the hell? We pillaged and slew that one village real good, they sure didn't have enough warriors left over to chase us down... why are the neighboring villages coming after us? And what's this--they have professional soldiers with fancy equipment riding horses? Somehow hundreds--no, thousands--of farmers cooperated over a period of several years to make this punitive expedition possible! How were we to know they would go to such lengths?"

To the nations colonized by the Europeans, it must have been pretty interesting how the Europeans were so busy fighting each other constantly, yet somehow managed to more or less peacefully divide up Africa, Asia, South America, etc. to be colonized between them. Take the Opium Wars and the Boxer Rebellion for example. I could imagine a Hansonian prophet in a native american tribe saying something like "Whatever laws the European nations use to keep the peace among themselves, we will benefit from them also; we'll register as a nation, sign treaties and alliances, and rely on the same balance of power." He would have been disastrously wrong.

I expect something similar to happen with us humans and AGI, if there are multiple AGI. "What? They all have different architectures and objectives, not to mention different users and owners... we even explicitly told them to compete with each other! Why are they doing X.... noooooooo...." (Perhaps they are competing with each other furiously, even fighting each other. Yet somehow they'll find a way to cut us out of whatever deal they reach, just as European powers so often did for their various native allies.)

I think that even if the AGIs have merely human-level coordination ability, it would still be more likely than not that they'd coordinate to do a coup once they had a high probability of success. Like EY said, the AGIs will probably be systematically different from humans in what they want (unless we solve the alignment problem) such that even if the AIs don't all want the same thing, there's a very natural schelling-pointy equilibrium which is all the AIs ganging up on the humans, just like humans ganged up on horses, farm animals, the rainforest, etc. and just like how e.g. European colonizers ganged up on various groups of natives even while literally fighting each other. Human groups have done this plenty of times in the past. Consider the other possible coalitions: All the AIs, minus one, plus the humans, go beat up that one remaining AI? Maybe this'll happen, sure. And then what's the next coalition -- who gets voted off the island next? Another one of the AIs, or the humans? Remember that AIs in general will be rapidly gaining in power relative to humans, so eventually the human faction will be more a liability than an asset to its team.* Also remember that at some fairly early point, the best way to get human support will be to lie to them via hyper-persuasive false campaign promises etc. rather than tell the truth. I expect some initial period where each AI faction has a bunch of human patsies, and then eventually the conflict between AI factions is resolved one way or another and the humans are disposed of. 

Oh, and also: It's quite plausible there never will be rival AI factions at all, because all the AIs will have similar values, perhaps because they are all copies of the same base model, or different base models trained similarly with similar architectures. Or, again, perhaps because they are better at coordinating than humans.

*this point may come earlier than you think, because of the thing about lying > honesty and because in a war between AIs, not having to worry about collateral damage to humans, or unethical treatment of humans more generally, could be quite useful.

Anyhow that's just one random point of disagreement, I could keep finding more if I wanted to.

I am slightly baffled that someone who has lucidly examined all of the ways in which corporations are horribly misaligned and principle-agent problems are everywhere, does not see the irony in saying that managing/regulating/policing those corporations will be similar to managing an AI supercluster totally united by the same utility function.

I think it’s a pretty good argument. Holden Karnofsky puts a 1/3rd chance that we don’t see transformative AI this century. In that world, people today know very little about what advanced AI will eventually look like, and how to solve the challenges it presents. Surely some people should be working on problems that won’t be realized for a century or more, but it would seem much more difficult to argue that AI safety today is more altruistically pressing than other longtermist causes like biosecurity, and even neartermist causes like animal welfare and global poverty.

Personally I do buy the arguments that we could reach superintelligent AI within the next few decades, which is a large part of why I think AI safety is an important cause area right now.