What a practical plan for Friendly AI looks like

Not to be too harsh, but this seems like several steps backwards in designing a Friendly AI. To extend your metaphor, it's as if NASA has started calculating the potential trajectory of a rocket and you're running in shouting "Guys, guys, all this math is going to make the project way too confusing! We can see the moon - let's just aim our rocket at that!".

I think part of the problem is determining the goal of your post. Are you trying to come up with a simple, non-weird sounding explanation of FAI that LWers can explain to people without sounding low-status? If so, good job. This sounds reassuring and intuitive, and I'm fairly confident I could explain it to anyone without inferential-gap problems. Or are you trying to create a blueprint for how FAI "might be pursued in the real world"? If this is your goal, I'm concerned.

Having managers, bureaucrats, and lawyers create a set of laws for an AI to follow just won't work. Human deliberation and consensus can't solve simple, human created problems - why on earth do we think it'll solve AI problems? Constraining an AI with today's value system stifles all moral progress - possibly forever.

Then you would get your programmers and your cognitive scientists to implement that goal condition in a way such that the symbols have the meanings that they are supposed to have.

That's the whole problem. We don't really know what we mean by "right", and we don't even know what "happiness" and "freedom" and similar concepts mean to us. There's no reason to believe doing this would be any easier than CEV, and some evidence to suggest it's impossible.

I have seen too many discussions of Friendly AI, here and elsewhere (e.g. in comments at Michael Anissimov's blog), detached from any concrete idea of how to do it....

At present, it is discussed in conjunction with a whole cornucopia of science fiction notions such as: immortality, conquering the galaxy, omnipresent wish-fulfilling super-AIs, good and bad Jupiter-brains, mind uploads in heaven and hell, and so on. Similarly, we have all these thought-experiments: guessing games with omniscient aliens, decision problems in a branching multiverse, "torture versus dust specks". Whatever the ultimate relevance of such ideas, it is clearly possible to divorce the notion of Friendly AI from all of them....

SIAI, in discussing the quest for the right goal system, emphasizes the difficulties of this process and the unreliability of human judgment. Their idea of a solution is to use artificial intelligence to neuroscientifically deduce the actual algorithmic structure of human decision-making, and to then employ a presently nonexistent branch of decision theory to construct a goal system embodying ideals implicit in the unknown human cognitive algorithms.

In short, there is a dangerous and almost universal tendency to think about FAI (and AGI generally) primarily in far mode. Yes!

However, I'm less enamored with the rest of your post. The reason is that building AGI is simply an altogether higher-risk activity than traveling to the moon. Using "build a chemical powered rocket" as your starting point for getting to the moon is reasonable in part because the worst that could plausibly happen is that the rocket will blow up and kill a lot of volunteers who knew what they were getting into. In the case of FAI, Eliezer Yudkowsky has taken great pains to show that the slightest, subtlest mistake, one which could easily pass through any number of rounds of committee decision making, coding, and code checking, could lead to existence failure for humanity. He has also taken pains to show that approaches to the problem which entire committees have in the past thought were a really good idea, would also lead to such a disaster. As far as I can tell, the LessWrong consensus agrees with him on the level of risk here, at least implicitly.

There is another approach. My own research pertains to automated theorem proving, and its biggest application, software verification. We would still need to produce a formal account of the invariants we'd want the AGI to preserve, i.e., a formal account of what it means to respect human values. When I say "formal", I mean it: a set of sentences in a suitable formal symbolic logic, carefully chosen to suit the task at hand. Then we would produce a mathematical proof that our code preserves the invariants, or, more likely we would use techniques for producing the code and the proof at the same time. So we'd more or less have a mathematical proof that the AI is Friendly. I don't know how the SIAI is trying to think about the problem now, exactly, but I don't think Eliezer would be satisfied by anything less certain than this sort of approach.

Not that this outline, at this point, is satisfactory. The formalization of human value is a massive problem, and arguably where most of the trouble lies anyway. I don't think anyone's ever solved anything even close to this. But I'd argue that this outline does clarify matters a bit, because we have a better idea what a solution to this problem would look like. And it makes it clear how dangerous the loose approach recommended here is: virtually all software has bugs, and a non-verified recursively self-improving AI could magnify a bug in its value system until it no better approximates human values than does paperclip-maximizing. Moreover, the formal proof doesn't do anyone a bit of good if the invariants were not designed correctly.

Good post.

Let me describe to you how it felt to read one line of the post. While reading it, I felt like I was a prominent supporting character in a war movie, one who has to die so that suspension of disbelief is not broken, because otherwise the good guys would mow down hundreds of enemies with their small, rag-tag band not taking any casualties and that would just be silly.

It's a rare thing to have a good guy die, and since he has to be dead and out of the plot, the screenwriter might as well squeeze all the drama he or she can out of that moment. Have him survive a few wounds - not that he first few would be enough to kill a good guy anyway. The dying guy (that's me here) should shudder and convulse dramatically from the force of each hit.

You would get a team of professional ethicists,

BANG (lessdazed is hit by sniper fire.)

lessdazed: "Sir, I...I've been hit...I can't feel my legs..."

Commander: "MEDIC! MEDIC! Don't worry son, just hang in there we'll get you home."

some worldly people like managers,

BOOM (lessdazed is hit by a mortar.)

lessdazed: "...my left side...it's gone...I'll never make it...go on without me..."

Commander: "Stay with me dammit! I told your mother I'd get you home and by gawd I will!"

some legal experts

KABLOOEY (lessdazed has been obliterated by an artillery shell.)

lessdazed: "..." (much of lessdazed's former mass dies in the arms of the commander)

Commander: "Nooooooooooooo! I'll get you, you bastards!" (Commander goes on rampage, does not even stop to reload until 20 minutes and hundreds of dead bad guys later.)

In short, I am not sure that the team you described (but did not endorse, I note, I'm not saying you did) is the best to produce something that behaves ethically.

Whatever the ultimate relevance of such ideas, it is clearly possible to divorce the notion of Friendly AI from all of them.

Take for example Pascal's mugging, if you can't solve it then you need to implement a hack that is largely based on human intuition. Therefore in order to estimate the possibility of solving friendly AI, or to distinguish it from the implementation of fail-safe mechanisms, one needs to account for the difficulty in solving all those sub-problems you mentioned.

As multifoliaterose wrote, we don't even know "how one would start to research the problem of getting a hypothetical AGI to recognize humans as distinguished beings."

...it is simply an exercise in creating artificial intelligence that does the right thing.

Yes, but solving metaethics, to figure out what we mean when we use the word "right", already seems to be ridiculously difficult.

What you need to show is that there is a possibility to solve friendly AI before someone stumbles upon AGI. A possibility that would outweigh the (in my opinion) vastly less effective but easier possibility of creating fail-safe mechanisms that might prevent a full-scale extinction scenario or help us to employ an AGI to solve friendly AI.

Simplicity has its value, but one shouldn't pursue plans any simpler than the simplest plan that might actually work. We don't know whether it's even in principle possible for us to knowingly construct an explicit goal system that matches our values and doesn't offer any room for a super-human AI to cleverly "optimize" in unexpected ways without referencing our values themselves. But I would be extremely skeptical of any attempt inspired by contract law. In your analogy this plan sounds more like "balloon that goes really high" to me. Perhaps the Singinst is pursuing Verneian cannons and no one has thought of rockets yet.

Instead, it only discusses hopelessly impractical models of cognition like AIXI and exact Bayesian inference, that are mostly of theoretical interest.

That sounds backwards. We study these models because it's easier. If we find friendliness very hard to capture even after making strong simplifying assumptions, that means a team of lawyers trying to solve the messier real-life problem is guaranteed to fail.

While fully cognizant of the fact that value is complex, as eloquently attested by Eliezer in many speeches, the practical FAI project would nonetheless choose the AI's goal system in the old-fashioned way, by human deliberation and consensus. You would get a team of professional ethicists, some worldly people like managers, some legal experts in the formulation of contracts, and together you would hammer out a mission statement for the AI. Then you would get your programmers and your cognitive scientists to implement that goal condition in a way such that the symbols have the meanings that they are supposed to have. End of story.

We believe that this won't result in a FAI.

the practical FAI project would nonetheless choose the AI's goal system in the old-fashioned way, by human deliberation and consensus

[...]

Our practical FAI project has "solved" FAI by simply coming to an agreement on what to wish for, and by studying with legalistic care how to avoid pitfalls and loopholes in the finer details of the wish

If we are making an AGI, then humans think too slowly, in comparison, to be able to completely consider every single possible aspect of a "wish", so I don't think legalistic is strong enough, given the large negative utility of a mistake. A mathematical proof of Friendliness should be required, and that is what the formalisations of "hopelessly impractical models of cognition" (e.g. TDT) are a step towards.

If your goal is AGI, then you want a cognitive architecture that will exhibit these behaviors.

FTFY. If you are designing something to have behaviour x then you want behaviour x to definitely occur, possibly being built out of other behaviours, but not just "emerging" out of other behaviours.

Upvoted for having at least some acquaintance with reality. To use the analogy in the post, I would put this into the category of "we should use chemical fuel, maybe in the form of a very large cannon," by contrast with proposals on the level of psychic levitation or breeding kangaroos for the ability to jump really high.

I want to use Steve Omohundro's paper "The Basic AI Drives" in a slightly unusual way. The paper lists a number of behaviors that should be exhibited by a sufficiently sophisticated AI: it will try to model its own operation, clarify its goals, protect them from modification, protect itself from destruction, acquire resources and use them efficiently... The twist I propose is that Omohundro's list of drives should be used as a design specification. If your goal is AGI, then you want a cognitive architecture that will exhibit these emergent behaviors.

IMHO, that doesn't help too much. We mostly know what we want - what we don't know is how to get there.

Incidentally, what most people don't want is just a bunch of universal instrumental values.

All that is coming, says the practical FAI project, and we are here to design these new beings so they will be good citizens

That would be a lie.