Thinking soberly about the context and consequences of Friendly AI

On these three issues - I'll call them post-singularity concentration of power, size and duration of post-singularity civilization, and length of post-singularity lifespan - my preference is to make the low estimate the default, but to regard the high estimates as defensible.

In each case, there are arguments (increasing returns on self-enhancement, no sign of alien civilizations, fun theory / basic physics) in favor of the high estimate (one AI rules the world, the winner on Earth gets this Hubble volume as prize, no psychological or physical limits on lifespan shorter than that of the cosmos). And it's not just an either-or choice between low estimate and high estimate; the low estimates could be off by just a few orders of magnitude, and that would still mean a genuinely transcendent future.

However, the real situation is that we just don't know how much power could accrue to a developing AI, how big a patch of space and time humanity's successors might get to claim as their own, and how much lifespan is achievable or bearable. Science and technology have undermined the intellectual basis for traditional ideas of what the limits are, but they haven't ratified the transcendent scenarios either. So the "sober" thing to do is to say that AIs can become at least as smart as human beings, that we could live throughout the solar system, and that death by old age is probably unnecessary; and to allow that even more than that may be possible, with the extremes being represented by what I've just called the transcendent scenarios.

I think FAI is still completely relevant and necessary, even in the minimal scenario. UFAI can still wipe out the human race in that scenario; it would be just another genocide, just another extinction, in which one group of sapients displaces another, and history is full of such events. Similarly, the usual concerns of FAI - identifying the correct value system, stability under self-enhancement - remain completely relevant even if we're just talking about AIs of roughly humanesque capabilities, as do concepts like CEV and reflective decision theory.

This has all become clear to me when I think what it would take for the concept of Friendly AI (whether under that name, or some other) to really gain traction, culturally and politically. I can make the case for the minimal future scenario; I can make the case that something beyond the minimal scenario is possible, and that we don't know how much beyond. I can live with the existence of people who dream about ruling their own galaxy one day. But it would be very unwise to tie serious FAI advocacy to that sort of dreaming.

Maybe one day there will be mass support, even for that sort of transhumanism; but that will be the extremist wing of the movement, a bit like the role of fundamentalists in conservative American politics. They are part of the conservative coalition, so they matter politically, but when they actually take over, you get craziness.

The assumptions that I criticize may be providing a lot of the motivation, but you can think that e.g. solving the problem of ethical stability under self-modification is important, without believing that stuff; and meanwhile, a lot of people must be encountering the concept of Friendly AI as part of a package which includes the futurist maximalism and the peculiar metaphysics. I suppose I'm saying that Friendly AI needs to be saved from the subculture that most vigorously supports it, because it contains ideas that really are important.

Whether the recursive self-improvement is possible or not, is the property of the universe. Also other details, like how much additional time and energy is necessary for another increase in intelligence.

The answer to this question can make some outcomes more likely. For example, if recursive self-improvement is possible, and at some level you can get a huge increase in intelligence very quickly and relatively cheaply, one of the centers of power could easily overpower the other ones. Perhaps even in situations where every super-agent would read and analyze the source code of all other super-agents all the time; the increased intelligence could allow one of them to make changes that will seem harmless to the other ones.

On the other hand the multiple centers of power scenario is more likely if humankind spreads to many planets, and there is some natural limit how high an intelligence can become before it somehow collapses or starts needing insane amounts of energy; so no super-agent could be smart enough to conquer the rest of the world.

Whether a single AI takes over the world, or whether there are always multiple centers of power is something we choose.

The idea of technological determinism suggests this is hubris.

Tech also seems quite vulnerable to monocultures. Think of file formats, for example. In the early days there are often several formats, but after a while most of them go extinct and the survivors end up being universally used. Image display formats, for example, fall largely into two categories - formats that every computer knows how to display, and formats that hardly anybody uses at all. (Image editing formats are different, I know.) How many word processors have you used recently that can't support .doc format ?

The most likely scenario is that there will be only one center of intelligence, and that although the intelligence isn't really there yet, the center is. You're using it now.

I don't see a difference of purpose between "Friendly AI" and "machine ethics". Friendly AI is basically a school of thought about machine ethics - by far the best that I've seen, when it comes to depth of analysis, methodology, and ingenuity of concepts - that sprang up under unusual circumstances.

In terms of LW concepts, FAI tends to get grouped with "futurism and speculation", and then that topic cluster is separated from "pure and applied rationality". The perspective I want to foster sees more continuity between FAI and the study of human decision-making, and places a little more distance between FAI and the futurism/speculation cluster.

The concept of a hard takeoff plays a central role in how people think about FAI - perhaps this is the "specific sort of problem" you mean - but it can be motivated as a worst case that needs to be considered for safety's sake, rather than as a prediction or dogma about what will actually happen.

"Pet", "vermin", and "wild animal" (as well as "livestock" and "working animal") are all concepts that humans have come up with for our species' relationships with other species that we've been living with since forever, and have developed both instincts and cultural practices to relate to. Why would you expect them to apply to an AI's relationship to humans? Isn't that a bit, well, anthropomorphizing?

A single AI will take over the world [...]

I don't see how this can be avoided if the damn thing is so much smarter [...]

Companies and governments are much smarter than humans. So far, none has taken over the world. Companies compete with other companies. Governments compete with other governments . Like that.

That's possible, but I find it more likely that we have a silent majority of people who are on the periphery of the community and are uncomfortable with a lot of the singularitarian memes. I think that a lot of internet communities with voting have a lot of people who vote but don't comment or comment only occasionally, so you'd expect that voting would reflect more peripheral members more strongly than the discussion would.