[draft] Concepts are Difficult, and Unfriendliness is the Default: A Scary Idea Summary

Reading this with my theoretical computer scientist hat on, the whole thing feels rather fuzzy. It feels basically like taking the informal throwing ideas around level of discussion, and putting numbers on the paragraphs to make it seem deep and formal. The hat-influenced persona is mostly upset about how the concept of AGI is such a magical anthropomorphic creature here, when Numbers Being Used sets up the expectation for Crisp Reductive Formalization to make an appearance.

There might be a chapter in the beginning, stating, "We define an AGI to be a system with such and such properties", that grounds and sets up the ensuing discussion.

The other bit of troublesome magical hand-waving is the part where the problems of going from human concepts, desires and intentions into machine formalism come up. The theoretical computer scientist hat is quite familiar with problems of formal specification, but being a piece of headgear with sinister psychic powers, it does not have first-hand familiarity with human concepts, desires and intentions, and would like a formal presentation for them.

The problem here of course is that we don't have a good formal presentation for stuff in human brain states, and things start to get rather tricky. This is basically the Alan Perlis epigram, "One can't proceed from the informal to the formal by formal means", except that now it looks like we need to. A direct attack won't work, but the approach where the massive mess of going from the informal human level in general to the formal AGI implementation level is basically just danced around in the discussion with rather specific problems going from some certain informal human concepts into particularly unfriendliness-engendering AGI designs feels like it leaves the discussion of the particulars rather groundless.

I don't have a good answer to the problems. The basic questions of defining the AGI and the translation from generalized human intentions to AGI design is basically a big chunk of the whole friendly AI endeavor, and can't be handled in the paper. But I still have the sense that there should be some extra crispness there to convince a computer science literate reader that this is something worth paying attention to, instead of just make-work prognostication around an ill-defined concept.

The thing with most engineering-style literature, computer science included, is that while the work is both concerned with mechanical exactitude with respect to the subject matter and inexorably tied to human creativity and judgment in coming up with the subject matter to begin with, it is generally quite silent on trying to apply the standards of mechanical exactitude to the human creativity and judgment half of the process. Some kind of precursor article that focuses on this bit and how an AGI project will need to break the Perlis epigram, which for most people is just an invisible background assumption not even worth stating, might be useful to make people less predisposed to seeing the AGI stuff itself as just strange nonsense.

For what it's worth. Here are some possible objections that certain people might raise.

(Note: I am doing this to help you refine a document that was probably meant to convince critics that they are wrong. It is not an attempt to troll. Everything below this line is written in critique mode.)

The most basic drive of any highly efficient AGI is, in my opinion, the drive to act correctly. You seem to assume that AGI will likely be designed to judge any action with regard to a strict utility-function. You are assuming a very special kind of AGI design with a rigid utility-function that the AGI then cares to satisfy the way it was initially hardcoded. You assume that the AGI won't be able to, respectively does not want to, figure out what its true goals might be.

What makes you think that AGI's will be designed according to those criteria?

If an AGI acts according to a rigid utility-functions, then what makes you think that it won't try to interpret any vagueness in a way that most closely reflects the most probable way it was meant to be interpreted?

If the AGI's utility-function solely consisted of the English language sentence "Make people happy.", then what makes you think that it wouldn't be able to conclude what we actually meant by it and act accordingly? Why would it care to act in a way that does not reflect our true intentions?

My problem is that there seems to be a discontinuity between the superior intelligence of a possible AGI and its inability to discern irrelevant information from relevant information with respect to the correct interpretation of its utility-function.

[Anna Salamon] gave the familiar SIAI argument that, if one picks a mind at random from “mind space”, the odds that it will be Friendly to humans are effectively zero.

This is an incredibly weak argument by intuition. The mind picked at random from "mind space" can be self destructive, for instance, or can be incapable of self improvement. As intuition pump, if you pick a computer program at random from computer program space - run random code - it crashes right off almost all of the time. If you eliminate the crashes you get very simple infinite loops. If you eliminate those, you get very simple loops that count or the like, with many pieces of random code corresponding to exact same behaviour after running it for any significant number of cpu cycles (as most of the code ends up non-functional). You get Kolmogorov's complexity prior even if you just try to run uniformly random x86 code.

The problem with the argument is that you appeal to the random mind space, while discussing the AIs that foom'd from being man made and running at manmade hardware, and which do not self destruct, and thus are anything but random.

One could make equally plausible argument that random mind from the space of the minds that are not self destructive, yet capable of self improvement (which implies considerably broad definition of self) is almost certainly friendly as it would implement the simplest goal system which permits self improvements and forbids self harm, implying likely rather broad and not very specific definition of self harm that would likely include harm to all life. It is not a very friendly AI - it will kill the entire crew of a whaling ship if it has to - but not very destructive. edit: Of course, that's subject to how it tries to maximize value of the life; the diversity and complexity preservation seems natural for the anti-self-harm mechanism. Note: the life is immensely closer to the AI than dead parts of the universe. Note2: Less specific discriminators typically have lower complexity. Note3: I think the safest assumption to make is that the AI doesn't start off as a self aware super genius that will figure out instrumental self preservation from first principles even if the goal is not self preserving.

I'll call this a "Greenpeace by default" argument. It is coming from a software developer (me) with some understanding of what random design spaces tend to look like, so it got to have higher prior than the "Unfriendly by default" which ignores the fact that most of the design space corresponds to unworkable designs and that simpler designs have larger number of working implementations.

Ultimately, this is all fairly baseless speculation and rationalization of culturally, socially, and politically motivated opinions and fears. One does not start with an intuition of the random mind design space - it is obvious that such intuition is likely garbage unless one actually dealt with random design spaces before. One starts with fear and invents that argument. One can start with pro-AI attitude and invent converse, but equally (if not more) plausible argument, by appeal to intuitions of this kind. Bottom line is, all of those are severely privileged hypotheses. The scary idea, the Greenpeace idea of mine, they're baseless speculations - though I do have very strong urge to just promote this Greenpeace idea with same zeal, just to counter the harm done by promoting other privileged hypotheses.

Kaj, this is an excellent article focusing on why an AGI will have a hard time adopting an model of the world similar to the ones that humans have.

However, I think that Ben's main hangup about the scary idea is that he doesn't believe in the complexity and fragility of moral values. In this article he gives "Growth, Choice, and Joy" as a sufficient value system for friendliness. He knows that these terms "concept a vast mass of ambiguity, subtlety and human history," but still, I think this is where Goertzel and SI differ.

The term "formalization" doesn't seem to fit (it's something more like formulation of an argument), and I'm not sure propagating Goetzel's term "Scary Idea" is a good idea.

Is it worth thinking about not just a single AGI system, but technological development in general? Here is an outline for an argument - the actual argument requires a lot of filling in.

Definitions:

• A "singleton" is a system with enough global control to prevent potential competitor AGIs from being developed. I'm assuming that this would imply the ability to obtain control in many other spheres as well, even if the singleton chooses not to use that power
• "disruptive" is a vaguely-defined term which is strictly weaker than "singleton". Disruptive doesn't necessarily imply bad.
• The argument should work for any reasonable definition of "AGI".

Someone creates an AGI. Then one of the following is true:

1. The AGI becomes a singleton. This isn't a job that we would trust to any current human, so for it to be safe the AGI would need not just human-level ethics but really amazing ethics. This is where arguments about the fragility of value and Kaj_Sotala's document come in.

2. The AGI doesn't become a singleton but it creates another AGI that does. This can be rolled into 1 (if we don't distinguish between "creates" and "becomes") or it can be rolled into 3 (if we don't make the distinction between humans and AGIs acting as programmers).

3. The AGI doesn't become a singleton and doesn't create one either. Then we just wait for someone to develop the next AGI.

Notes on point 3:

• New AGIs are likely to arrive thick and fast due to arms races, shared/leaked source code and rising technological waterline. This is the differential technology argument - even if your AGI won't be disruptive, working on it might accelerate the development of more disruptive AGI.
• A possible loophole: we might cycle round here for a long time if singleton creation is an unlikely event. Worth digging up arguments for/against this one.
• Probably worth mentioning that AGIs could cause a lot of disruption, up to and including human extinction, even without becoming a singleton (e.g. out-competing humans economically), but this is only incidental to the argument

AGI will only be Friendly if its goals are the kinds of goals that we would want it to have

At the risk of losing my precious karma, I'll play the devil's advocate and say I disagree.

First some definitions: "Friendly" (AI), according to Wikipedia, is one that is beneficial to humanity (not a human buddy or pet). "General" in AGI means not problem-specific (narrow AI).

My counterexample is an AI system that lacks any motivations, goals or actuators. Think of an AIXI system (or, realistically, a system that approximates it), and subtract any reward mechanisms. It just models its world (looking for short programs that describe its input). You could use it to make (super-intelligent) predictions about the future. This seems clearly beneficial to humanity (until it falls into malicious human hands, but that's besides the argument you are making)