The Friendly AI Game

Start the AI in a sandbox universe, like the "game of life". Give it a prior saying that universe is the only one that exists (no universal priors plz), and a utility function that tells it to spell out the answer to some formally specified question in some predefined spot within the universe. Run for many cycles, stop, inspect the answer.

Start the AI in a sandbox universe. Define its utility function over 32-bit integers. Somewhere inside the sandbox, put something that sets its utility to INT_MAX utility, then halts the simulation. Outside the sandbox, leave documentation of this readily accessible. The AI should never try to do something elaborately horrible, because it can get max utility easily enough from inside the simulation; if it does escape the box, it should go back in to collect its INT_MAX utility.

The AI gets positive utility from having been created, and that is the whole of its utility function. It's given a sandbox full of decision-theoretic problems to play with, and is put in a box (i.e. it can't meaningfully influence the outside world until it has superhuman intelligence). Design it in such a way that it's initially biased toward action rather than inaction if it anticipates equal utility from both.

Unless the AI develops some sort of non-causal decision theory, it has no reason to do anything. If it develops TDT, it will try to act in accordance with what it judges to be the wishes of its creators, following You're In Newcomb's Box logic--it will try to be the sort of thing its creators wished to create.

The Philosophical Insight Generator - Using a model of a volunteer's mind, generate short (<200 characters, say) strings that the model rates as highly insightful after read each string by itself, and print out the top 100000 such strings (after applying some semantic distance criteria or using the model to filter out duplicate insights) after running for a certain number of ticks.

Have the volunteer read these insights along with the rest of the FAI team in random order, discuss, update the model, then repeat as needed.

So, here's my pet theory for AI that I'd love to put out of it's misery: "Don't do anything your designer wouldn't approve of". It's loosely based on the "Gandi wouldn't take a pill that would turn him into a murderer" principle.

A possible implementation: Make an emulation of the designer and use it as an isolated component of the AI. Any plan of action has to be submitted for approval to this component before being implemented. This is nicely recursive and rejects plans such as "make a plan of action deceptively complex such that my designer will mistakenly approve it" and "modify my designer so that they approve what I want them to approve".

There could be an argument about how the designer's emulation would feel in this situation, but.. torture vs. dust specks! Also, is this a corrupted version of ?

Oracle AI - its only desire is to provide the correct answer to yes or no questions posed to it in some formal language (sort of an ueber Watson).

Give the AI a bounded utility function where it automatically shuts down when it hits the upper bound. Then give it a fairly easy goal such as 'deposit 100 USD in this bank account.' Meanwhile, make sure the bank account is not linked to you in any fashion (so the AI doesn't force you to deposit the 100 USD in it yourself, rendering the exercise pointless.)

Define "Interim Friendliness" as a set of constraints on the AI's behavior which is only meant to last until it figures out true Friendliness, and a "Proxy Judge" as a computational process used to judge the adequacy of a proposed definition of true Friendliness.

Then there's a large class of Friendliness-finding strategies where the AI is instructed as follows: With your actions constrained by Interim Friendliness, find a definition of true Friendliness which meets the approval of the Proxy Judge with very high probability, and adopt that as your long-term goal / value system / decision theory.

In effect, Interim Friendliness is there as a substitute for a functioning ethical system, and the Proxy Judge is there as a substitute for an exact specification of criteria for true Friendliness.

The simplest conception of a Proxy Judge is a simulated human. It might be a simulation of the SIAI board, or a simulation of someone with a very high IQ, or a simulation of a random healthy adult. A more abstracted version of a Proxy Judge might work without simulation, which is a brute-force way of reproducing human judgment, but I think it would still be all about counterfactual human judgments. (A non-brute-force way of reproducing human judgment is to understand the neural and cognitive processes which produce the judgments in question, so that they may be simulated at a very high level of abstraction, or even so that the final judgment may be inferred in some non-dynamical way.)

The combination of Interim Friendliness and a Proxy Judge is a workaround for incompletely specified strategies for the implementation of Friendliness. For example: Suppose your blueprint for Friendliness is to apply reflective decision theory to the human decision architecture. But it turns out that you don't yet have a precise definition of reflective decision theory, nor can you say exactly what sort of architecture underlies human decision making. (It's probably not maximization of expected utility.) Never fear, here are amended instructions for your developing AI:

With your actions constrained by Interim Friendliness, come up with exact definitions of "reflective decision theory" and "human decision architecture" which meet the approval of the Proxy Judge. Then, apply reflective decision theory (as thus defined) to the human decision architecture (as thus defined), and adopt the resulting decision architecture as your own.

I like this game. However, as a game, it needs some rules as to how formally the utility function must be defined, and whether you get points merely for avoiding disaster. One trivial answer would be: maximize utility by remaining completely inert! Just be an extremely expensive sentient rock!

On the other hand, it should be cheating to simply say: maximize your utility by maximizing our coherent extrapolated volition!

Or maybe it wouldn't be...are there any hideously undesirable results from CEV? How about from maximizing our coherent aggregated volition ?

(Love these kinds of games, very much upvoted.)

Send 1 000 000 000 bitcoins to the SIAI account.

A variant of Alexandros' AI: attach a brain-scanning device to every person, which frequently uploads copies to the AI's Manager. The AI submits possible actions to the Manager, which checks for approval from the most recently available copy of each person who is relevant-to-the-action.

At startup, and periodically, the definition of being-relevant-to-an-action is determined by querying humanity with possible definitions, and selecting the best approved. If there is no approval-rating above a certain ratio, the AI shuts down.

We give the AI access to a large number of media about fictional bad AI and tell it to maximize each human's feeling that they are living in a bad scifi adventure where they need to deal with a terrible rogue AI.

If we're all very lucky, we'll get promised some cake.

puts hand up

That was me with the geographically localised trial idea… though I don’t think I presented it as a definite solution. More of an ‘obviously this has been thought about BUT’. At least I hope that’s how I approached it!

My more recent idea was to give the AI a prior to never consult or seek the meaning of certain of its own files. Then put in these files the sorts of safeguards generally discussed and dismissed as not working (don’t kill people etc), with the rule that if the AI breaks those rules, it shuts down. So it can't deliberately work round the safeguards, just run into them. This is similar to my other helpful suggestion at the London meet, which was 'leave its central computer exposed so that it can be crippled with a well-aimed gunshot'.

Risks with subconscious AI: Someone tampers with the secret files It works out what will be in them by analysing us If we try to make an improved one after it shutting down, the improved one will assume similar rules We just don’t cover the possibilities of bad things it could do It become obsessed with its dreams etc and invents psychoanalysis

NEVERTHELESS, I think it’s a pretty neat idea. ;-)

Create a combination of two A.I Programs.

Program A's priority is to keep the utility function of Program B identical to a 'weighted average' of the utility function of every person in the world- every person's want counts equally, with a percentage basis based on how much they want it compared to other things. It can only affect Program B's utility function, but if necessary to protect itself FROM PROGRAM B ONLY (in the event of hacking of Program B/mass stupidity) can modify it temporarily to defend itself.

Program B is the 'Friendly' AI.

I don't want to be a party pooper, but I think the idea that we could build an AGI with a particular 'utility function' explicitly programmed into it is extremely implausible.

You could build a dumb AI, with a utility function, that interacts with some imprisoned inner AGI. That's basically equivalent to locking a person inside a computer and giving them a terminal to 'talk to' the computer in certain restricted, unhackable ways. (In fact, if you did that, surely the inner AGI would be unable to break out.)

A line in the wiki article on "paperclip maximizer" caught my attention:

"the notion that life is precious is specific to particular philosophies held by human beings, who have an adapted moral architecture resulting from specific selection pressures acting over millions of years of evolutionary time."

Why don't we set up an evolutionary system within which valuing other intelligences, cooperating with them and retaining those values across self improvement iterations would be selected for?

A specific plan:

Simulate an environment with a large number of AI agents competing for resources. Access to those resources allows the agent to perform a self improvement iteration. Rig the environments such that success requires cooperating with other intelligences of the same or lower level. Repopulate the next environment with copies of the succeeding intelligences. Over a sufficient number of generation times this should select for agents that value other intelligences, and preserve their values through self modification.

What do people think? I can see a few possible sources for error myself, but would like to hear your responses uncontaminated. [Given the importance of the topic you can assume Crocker's rules are in effect.]

1: Define Descended People Years as number of years lived by any descendants of existing people.

2: Generate a searchable index of actions which can be taken to increase Descended People Years, along with an explanation on an adjustable reading level as to why it works.

3: Allow Full view of any DPY calculations, so that something can be seen as both "Expected DPY gain X" and "90% chance of Expected DPY gain Y, 10% chance of Expected DPY loss Z"

4: Allow Humans to search this list sorting by cost, descendant, and action, time required, complexity, Descended People Years given, and ratios of those figures.

5: At no Point will the FAI ever be given the power to implement any actions itself.

6: The FAI is not itself considered a Descendant of People. (Although even if it was, it would have to dutifuly report in the explanations.)

7: All people are allowed to have access to this FAI.

8: The FAI is required to have all courses of action obey the law, (no suggesting murdering one group at a cost of DPY X to extend a second group by DPY Y) although it can suggest changing a law.

9: Allow people to focus calculations in a particular area based on complexity or with restrictions, such as "I'd like to engage in an action that can boost DPY that I can complete in 5 minutes." or "I'd like to engage in an action that can boost DPY that does not require adding additional computational power to the FAI."

So I can use the AI to find out (for instance) that giving up drinking Soda will likely extend my descended people years by 0.2 years, at a savings to me of 2 dollars a day, along with an explanation of why it does so and extended calculations if desired.

But a president of a country could use the FAI to find out that signing a treaty with a foreign country to reduce Nuclear Weapons would likely extend DPY by 1,000,000 DPY, along with an explanation of why it does so and extended calculations if desired.

The minor nature of its goals is the whole point. It is not meant to do what we want because it empathizes with our values and is friendly, but because the thing we actually want it to do really is the best way to accomplish the goals we gave it. Also I would not consider making a cheese cake to be a trivial goal for an AI, there is certainly more to it then the difficult task of distinguishing a spoon from a fork, so this is surely more than just an "intelligent rock".

Not a utility function, but rather a (quite resources-intensive) technique for generating one:

Rather than building one AI, build about five hundred of them, with a rudimentary utility function template and the ability to learn and revise it. Give them a simulated universe to live in, unaware of the existence of our universe. (You may need to supplement the population of 500 with some human operators, but they should have an interface which makes them appear to be inhabiting the simulated world.) Keep track of which ones act most pathologically, delete them, and recombine the remaining AIs with mutation to get a second generation of 500. Keep doing this until you have an AI that consistently minds its manners, and then create a new copy of that AI to live in our world.

After reading the current comments I’ve come up with this:

1) Restrict the AI’s sphere of influence to a specific geographical area (Define it in several different ways! You don’t want to confine the AI in “France” just to have it annex the rest of the world. Or by gps location and have it hack satellites so they show different coordinates.)

2) Tell it to not make another AI (this seems a bit vague but I don’t know how to make it more specific) (maybe: all computing must come from one physical core location. This could prevent an AI from tricking someone into isolating a back up, effectively making a copy)

3) Set an upper bound for the amount physical space all AI combined in that specific area can use.

4) As a safeguard, if it does find a way around 2, let it incorporate the above rules, unaltered, in any new AI it makes.

After reading the current comments I’ve come up with this:

Restrict the AI’s sphere of influence to a specific geographical area (Define it in several different ways! You don’t want to confine the AI in “France” just to have it annex the rest of the world. Or by gps location and have it hack satellites so they show different coordinates.) Tell it to not make another AI (this seems a bit vague but I don’t know how to make it more specific) (maybe: all computing must come from one physical core location. This could prevent an AI from tricking someone into isolating a back up, effectively making a copy) Set an upper bound for the amount physical space all AI combined in that specific area can use. As a safeguard, if it does find a way around 2, let it incorporate the above rules, unaltered, in any new AI it makes.

After reading the current comments I’ve come up with this:

1) Restrict the AI’s sphere of influence to a specific geographical area (Define it in several different ways! You don’t want to confine the AI in “France” just to have it annex the rest of the world. Or by gps location and have it hack satellites so they show different coordinates.) 2) Tell it to not make another AI (this seems a bit vague but I don’t know how to make it more specific) (maybe: all computing must come from one physical core location. This could prevent an AI from tricking someone into isolating a back up, effectively making a copy) 3) Set an upper bound for the amount physical space all AI combined in that specific area can use. 4) As a safeguard, if it does find a way around 2, let it incorporate the above rules, unaltered, in any new AI it makes.

After reading the current comments I’ve come up with this:

1) Restrict the AI’s sphere of influence to a specific geographical area (Define it in several different ways! You don’t want to confine the AI in “France” just to have it annex the rest of the world. Or by gps location and have it hack satellites so they show different coordinates.) 2) Tell it to not make another AI (this seems a bit vague but I don’t know how to make it more specific) (maybe: all computing must come from one physical core location. This could prevent an AI from tricking someone into isolating a back up, effectively making a copy) 3) Set an upper bound for the amount physical space all AI combined in that specific area can use. 4) As a safeguard, if it does find a way around 2, let it incorporate the above rules, unaltered, in any new AI it makes.

New Proposal (although I think I see the flaw already): -Create x "Friendly" AI, where x is the total number of people in the world. An originator AI is designed to create 1 of each such one equal to the number of humans in the world, then create new ones every time another human comes into being.

-Each "Friendly" AI thus created is "attached" to one person in the world in that it is programmed to constantly adjust it's utility function to that person's wants. All of them have equal self-enhancement potential, and have two programs. -Program A is the "Dominant" program, and has complete control of the utility function but nothing else. Program B cannot defy Program A. Program A is constantly adjusting Program B's utility function to be identical to that of the person it is "attached" to, and has no other priorities. -Program B's priorities are determined by Program A.

One such set of Program A and Program B is 'attached' to every individual in the world. Being of equal intelligence, in THEORY they should compromise in order to avoid wasting resuorces.

General objection to all wimpy AI's (e.g. ones whose only interaction with the outside world is outputting a correct proof of a particular mathematical theorem):

What the AI does is SO AWESOME that a community is inspired to develop their own AI without any of that boring safety crap.

New one(I'm better at thinking of ideas than refutation, so I'm going to run with that)- start off with a perfect replica of a human mind. Eliminate absolutely all measures regarding selfishness, self-delusion, and rationalisation. Test at this stage to check it fits standards using a review board consistent of people who are highly moral and rational by the standards of ordinary humans. If not, start off using a different person's mind, and repeat the whole process.

Eventually, use the most optimal mind coming out of this process and increase it's intelligence until it becomes a 'Friendly' A.I.

Make 1 reasonably good cheese cake as judged by a person within a short soft deadline while minimizing the cost to resources made available to it and with out violating property laws as judged by the legal system of the local government within some longer deadline.

To be clear the following do not contribute any additional utility:

• Making additional cheese cakes
• Making a cheese cake that is better than reasonably good
• Making any improvements to the value of the resources available other than making a cheese cake
• Anything that happens after the longer deadline.

The short soft deadline is about 1 day. Finishing in 5 hours is slightly better than 6. At 1 day there is a steep inflection. Finishing in more than 25 hours is slightly better than doing nothing.

The long deadline is 1 year.

Committing a property law violation is only slightly worse than doing nothing.

Since we are allowing a technology that does not presently exist I will assume there is another technological change. Specifically there are remotely controllable humanoid robots and people are comfortable interacting with them.

The AGI has available to it: 1 humanoid robot, more money than the cost of ingredients of a cheese cake available at a nearby grocery store, a kitchen, communication access to the person who will judge the reasonableness of the cheese cake and who may potentially provide other resources if requested and who's judgment may be influenced.

As a possible alternative scenario we could assume there are thousands of other similar AGI that each are tasked with making a different pastry. I propose we call them each Marvin.

Let the confectionery conflagration commence!