The Fermi paradox as evidence against the likelyhood of unfriendly AI

This argues equally against FAI as UFAI. Both are equally capable of expanding at near-lightspeed.

The Fermi paradox provides some evidence against long-lived civilization of any kind, hostile or non-hostile. Entangling the Fermi paradox with questions about the character of future civilization (such as AI risk) doesn't seem very helpful.

Obviously, an intelligence looking only to grow itself (and maximize paperclips or whatever) can do this much more easily than one restrained by its biological-or-similar parents.

I disagree. See this post, and Armstrong and Sandberg's analysis.

• Aliens won't produce a FAI, their successful AI project would have alien values, not ours (complexity of value). It would probably eat us. I suspect even our own humane FAI would eat us, at the very least get rid of the ridiculously resource-hungry robot-body substrate. The opportunity cost of just leaving dumb matter around seems too enormous to compete with whatever arguments there might be for not touching things, under most preferences except those specifically contrived to do so.
• UFAI and FAI are probably about the same kind of thing for the purposes of powerful optimization (after initial steps towards reflective equilibrium normalize away flaws of the initial design, especially for "scruffy" AGI). FAI is just an AGI that happens to be designed to hold our values in particular. UFAI is not characterized by having "simple" values (if that characterization makes any sense in this context, it's not clear in what way should optimization care about the absolute difficulty of the problem, as compared to the relative merits of alternative plans). It might even turn out to be likely for a poorly-designed AGI to have arbitrarily complicated "random noise" values. (It might also turn out to be relatively simple to make an AI with values so opaque that it would need to turn the whole universe into an only instrumentally valuable computer in order to obtain a tiny chance of figuring out where to move a single atom, the only action it ever does for terminal reasons. Make it solve a puzzle of high computational complexity or something.)
• There doesn't appear to be a reason to expect values to influence the speed of expansion to any significant extent, it's astronomical waste (giving an instrumental drive) for almost all values, important to start optimizing the matter as soon as possible.

A problem with Unfriendly and Friendly are that they are quite relative terms. An AI which is Friendly to the baby eaters will be unfriendly to us. This is a subset of the more general truth that an I (natural or artificial) which is friendly to the baby eaters will be unfriendly to us. Indeed as an I, humans have been unfriendly to Chimps, Bonobos, Dodos, and Neanderthals to name some of the more interesting examples.

Yes the Fermi paradox is evidence against UAI. It is also evidence against FAI. In fact, as the pardox in its original form is stated, it is evidence against I other than human, no matter what letters you decorate the I with.

http://meteuphoric.wordpress.com/2010/11/05/light-cone-eating-ai-explosions-are-not-filters/

Maybe interstellar travel is really, really hard--no matter what your level of technology.

Maybe 99% of the habitable planets in the galaxy have been sterilized by unfriendly AI and we owe our current existence to the anthropomorphic principle.

Maybe highly rational entities decide large-scale interstellar travel is suboptimal.

Probably a bunch more possibilities here...

Why are you assuming that we would be more likely to notice an unfriendly SI than a friendly SI? If anything, it seems that an intelligence we would consider friendly is more likely to cause us to observe life than one maximizing something completely orthogonal to our values.

(I don't buy the argument that an unfriendly SI would propagate throughout the universe to a greater degree than a friendly SI. Fully maximizing happiness/consciousness/etc also requires colonizing the galaxy.)

Therefore, it very much seems like no self-replicating unfriendly artificial intelligence has arisen anywhere in the galaxy in the - very roughly - 10 billion years since intelligent life could have arisen somewhere in the galaxy.

I think a more correct formulation would be:

It very much seems that nothing that both (a) desires limitless expansion and (b) is capable of limitless expansion through interstellar distances has arisen anywhere in our galaxy.

I am not quite sure that the application of terminology like AI (and, in particular, FAI or UFAI) to alien entities is meaningful.

"They're made out of meat."

Maybe there are better ways to expand than through spacetime, better ways to make yourself into this sort of maximizing agent, and we are just completely unaware of them because we are comparatively dull next to the sort of AGI that has a brain the size of a planet? Some way to beat out entropy, perhaps. That'd make it inconsistent to see any sort of sky with UFAI or FAI in it.

I can somewhat imagine what these sorts of ways would be, but I have no idea if those things are likely or even feasible, since I am not a world-devouring AGI and can only do wild mass speculation at what's beyond our current understanding of physics.

A simpler explanation could be that AGIs use stealth in pursuing their goals: the ability to camouflage oneself has always been of evolutionary import, and AGIs may find it useful to create a sky which looks like "nothing to see here" to other AGIs. (As they will likely be unfriendly toward each other) Camoflage, if good enough, would allow one to hide from predators (bigger AGI) and sneak up on prey (smaller AGI) Since we would likely be orders of magnitude worse at detecting an AGI's camoflage, we see a sky that looks like there is nothing wrong. This doesn't explain why we haven't been devoured, of course, which is the weakness of the argument.

Or maybe something like acausal trade limits the expansion of AGI. If AGIs realize that fighting over resources is likely to hinder their goals more than help them in the long wrong, they might limit their expansion on the theory that there are other AGIs out there. If I think I am 1 out of a population of a billion, and I don't want to be a target for a billion enemies at once, I might decide that taking over the entire galaxy/universe/beyond isn't worth it. In fact, if these sorts of stand-offs become more common as the scale becomes grander, it might be motivation not to pursue such scales. The problem with this being that you would expect earlier AGIs to be more likely to just take advantage before future ones can really get to the point of being near-equals and defect on this particular dilemma. (A billion planet-eating AGI are probably not a match for one galaxy-eating AGI. So if you see a way to become the galaxy-eater before enough planet-eaters can come to the party, you go for it.)

I don't find any of these satisfying, as one seems to require a sub-set of possibilities for unknown physics and the others seem to lean pretty heavily on the anthropic principle to explain why we, personally, are not dead yet. I see possibilities here, but none of them jump out at me as being exceptionally likely.

All that we know is that we haven't already encountered an UFAI.

Probably.

Humans evolved to having sex to encourage them to have as many children as possible. Yet humans don't have as many children as they can. Utility functions of humans are complicated.

The same goes for AGI. It's likely to do things that are more complicated than paperclip maximisation, which are hard to understand.

My take on this is that civilizations overwhelmingly terminate, overwhelmingly by other means than some independent willed super AI. Which is what I'd expect anyway, because this specific AI doomsday followed by AI expansion scenario is just one of many very speculative ways how a civilization could come to an end.

With regards to "utilities" and "utility functions", one needs to carefully distinguish between a mathematical function (with some fairly abstract input domain) that may plausibly and uncontroversially describe a goal that may be implemented into an AI, and a hypothetical, speculative function which much more closely follows the everyday meaning of the word "function" (i.e. purpose) and has actual reality as it's input domain.

What if the UFAI is just hiding in darkspace for a set period of time?

The assumptions that intelligence is substrate-independent, as well as that intelligent systems will always attempt to become more intelligent lead to the conclusion that, in the words of Paul Davies, "if we ever encounter extraterrestrial intelligence, it is overwhelmingly likely to be post-biological".

It seems like a daft thing to say. Biology is the study of life. Aliens would be alive - and thus plainly biological - for anyone using sensible definitions of these terms.