Formalizing Value Extrapolation

Cool! Thanks to you, we finally seem to have a viable attack on the problem of FAI, by defining goals in terms of hypothetical processes that could output a goal specification, like brain emulations with powerful computers. Everyone please help advance this direction of inquiry :-)

One potential worry is that the human subject must be over some minimal threshold of intelligence for this scheme to work. A village fool would fail. How do I convince myself that the threshold is below the "reasonably intelligent human" level?

A simplified version of this proposal that applies more generally is to implement a WBE-based FAI project using AGI, before normal WBE becomes available. This way, you only need to figure out how to build a "yield control to this-here program as it would be after N cycles" AGI, and the rest of the FAI project design can be left to the initial WBE-ed team. This would possibly have a side effect of initially destroying the rest of the human world, since the AGI won't be guided by our values before the N cycles of internal simulation complete (it would care about simulating the internal environment, not about saving external human lives; it might turn out to be FAI-complete to make it safe throughout), but if that internal environment can be guaranteed to be given control afterwards (when a FAI project inside it is complete), then eventually this is a plausible winning plan.

Regarding the 100 years informal example at the beginning:

Let's say I kidnapped you, put you in a box, and told you that you would spend the rest of your life figuring out the answer to some obvious moral question ("Should we cure Brad of cancer?") I could imagine you might become quite resentful, might suffer from some sort of mental illness after the first 10 years, and might give the wrong answer just out of spite.

Of course, if we take a snapshot of your brain and put it through the same experience in a simulation, it will feel exactly the same way.

Obviously you can't just cut out whatever parts of the brain might be responsible for resentment/boredom/mental illness. Specifying some sort of entertainment for you might cause you to ignore solving the problem to focus on the entertainment, or the entertainment might change your values. Terminating your simulation as soon as you found the answer to the question could cause you to focus on your impending death.

Maybe you could give us an informal summary of your proposal, to harvest cognitive surplus from those who don't want to read the entire thing and possibly find some relevant common sense sticking point that didn't occur to you? I suspect I would find an informal discussion where specifics were hashed out as necessary more interesting to read. Of course the eventual goal is formalization, but best practices for formalization may not be best practices for harvesting cognitive surplus.

To put it another way, you might wish to make sure you had something worth formalizing (through lots of informal discussion) before taking the trouble to formalize it.

"In(1)" looked like the natural log to me, for what it's worth.

A U-maximizer faces a similar set of problems: it cannot understand the exact form of U, but it can still have well-founded beliefs about U, and about what sorts of actions are good according to U. For example, if we suppose that the U-maximizer can carry out any reasoning that we can carry out, then the U-maximizer knows to avoid anything which we suspect would be bad according to U (for example, torturing humans).

Here's a stronger version of my previous criticism of this argument. Suppose instead of giving neuroimaging data to the AI and defining H in terms of a brute force search for a model that can explain the neuroimaging data, we give it a cryptographic hash of the neuroimaging data (of sufficient length to avoid possible collisions), and modify the definition of H to first perform a brute force search to recover the neuroimaging data from the hash. In this case, we can still say that torturing is probably bad according to U, but the AI obviously can't arrive at this conclusion from the formal definition of U alone (assuming it can't break the cryptographic hash). It seems clear that we can't safely assume that "the U-maximizer can carry out any reasoning that we can carry out".

Even if the U-maximizer cannot carry out this reasoning, as long as it can recognize that humans have powerful predictive models for other humans, it can simply appropriate those models (either by carrying out reasoning inspired by human models, or by simply asking).

In order to "carry out reasoning inspired by human models", the AI has to first form a usable model of a human. I don't have a strong argument that the U-maximizer can't do this from the original definition of U (i.e., from plaintext neuroimaging data), but intuitively it seems implausible given an amount of computing power the U-maximizer might initially have access to (say, within a couple orders of magnitude of the amount needed to do standard WBE). I don't see how "simply asking" could work either. What kind of questions might the U-maximizer ask, and how can we answer it, given that we don't know how to formalize what "torture" means?

It occurs to me that we can view this proposal through the "acausal trade" lens, instead of the "indirect normativity" lens, which might give another set of useful intuitions. What Paul is proposing can be seen as creating an AGI that can exert causal control in our world but cares only about a very specific world / platonic computation defined by H and T, while the inhabitants of that world (simulated humans and their descendants) care a lot about our world but has no direct influence over it. The hoped for outcome is for the two parties to do a trade: the AGI turns our world into a utopia in return for the inhabitants of the HT World satisfying its preferences (i.e., having the computation return a high utility value).

From this perspective, Paul's proposal can also be seen as an instance of what I called "Instrumentally Friendly AI" (on the decision theory list):

Previous discussions about Rolf Nelson's AI Deterrence idea (or in Nesov's more general terms, UFAIs trading with FAIs across possible worlds) seem to assume that even at best we (or humane values) will get only a small share of any world controlled by a UFAI. But what if we can do much better (i.e., get almost the entire universe), by careful choice of the decision algorithm and utility function of the UFAI? In that case the UFAI might be worthy of the name "Instrumentally Friendly AI" (IFAI), and it might be something we'd want to deliberately build.

I'm slightly worried that even formally specifying an "idealized and unbounded computer" will turn out to be Oracle-AI-complete. We don't need to worry about it converting something valuable into computronium, but we do need to ensure that it interacts with the simulated human(s) in a friendly way. We need to ensure that it doesn't modify the human to simplify the process of explaining something. The simulated human needs to be able to control what kinds of minds the computer creates in the process of thinking (we may not care, but the human would). And the computer should certainly not hack its way out of the hypothetical via being thought about by the FAI.

Another problem with this proposal: what if egoism is the right morality, or at least that our "actual" values have a large selfish component? If that is the case, then presumably the simulated humans inside the proposed AI will eventually realize it, and then cause the AI to value them (the simulations) instead of us (biological humans).

It seems difficult for approaches to FAI based on indirect normativity (e.g., CEV) to capture selfish values (with the correct indexical references), so it's not just a problem for this specific proposal, but I don't seem to recall seeing the issue mentioned anywhere before.

After reading the article, I thought I understood it, but from reading the comments, this appears to be an illusion. Yet, I think I should be able to understand, it doesn't seem to require any special math or radically new concepts... My understanding is below. Could someone check it and tell me where I'm wrong?

The proposal is to define a utility function U(), which takes as input some kind of description of the universe, and returns the evaluation of this description, a number between 0 and 1.

The function U is defined in terms of two other functions - H and T, representing a mathematical description of a specific human brain, and an infinitely powerful computing environment.

Although the U-maximizing AGI will not be able to actually calculate U, it will be able to reason about it (that is, prove theorems), which should allow it to perform at least some actions, which would therefore be provably friendly.

Possible objection: the proposal appears to fix U in terms of a mathematical description H of some current human brain. What happens in the future, when humans significantly self-modify?

Some care needs to be taken to define the behavior if T enters an infinite loop–we want to minimize the probability that the human accidentally hangs the terminal, with catastrophic consequences, but we cannot provide a complete safety-net without running into unresolvable issues with self-reference.

If I cause an infinite loop in my python shell, my computer does not crash and I can just kill the shell and start what I was doing over from the beginning. I don't understand why this wouldn't work in your scenario.

If we weight the simulation-people equally to the real-people, the computer will either choose to neglect us entirely and expend its resources more efficiently by using them on the simulation people or choose to treat us as the test subjects for some of its simulated projects, because there are going to be more simulation people than real people (unless the quality/quantity of the simulation isn't very good, which has its own problems).

I don't like that idea, but I also don't like the idea of weighting certain people higher than others just because some are in a machine.

I think there's a problem because of the range of possible humans that could be simulated within the AI.

What if it decides to simulate Hitler?