Discussion: Which futures are good enough?

is Eliezer right in thinking that if we get just one piece wrong the whole endeavor is worthless?

To clarify, the linked post by Eliezer actually says the following:

Value isn't just complicated, it's fragile. There is more than one dimension of human value, where if just that one thing is lost, the Future becomes null. A single blow and all value shatters. Not every single blow will shatter all value - but more than one possible "single blow" will do so.

(Unless you mind being simulated, in which case at least you'll never know.)

If I paid you to extend the lives of cute puppies, and instead you bought video games with that money but still sent me very convincing pictures of cute puppies that I had "saved", then you have still screwed me over. I wasn't paying for the experience of feeling that I had saved cute puppies -- I was paying for an increase in the probability of a world-state in which the cute puppies actually lived longer.

Tricking me into thinking that the utility of a world state that I inhabit is higher than it actually is isn't Friendly at all.

Related, for everyone who hasn't seen it yet: http://www.fimfiction.net/story/62074/Friendship-is-Optimal

Simulating the people you interact with in each simulation to a strong enough approximation of reality means you're creating tons of suffering people for each one who has an awesome life, even if a copy of each of those people is living a happy life in their own sim. I don't think I would want a bunch of copies of me being unhappy even if I know one copy of me is in heaven.

I fail to see why the shards have to be perfectly isolate, in this scenario. It would seem plausible that the AI could automatically import all the changes made by my best friend in his simulation into mine, and vice versa; and more extensively, include bits and pieces of other "real actions" into my ongoing narrative. Ultimately, everyone in my universe could be "intermittently real" in proportion to which of their owned actions contributed to my utopia, and the rest of their screen time can be done by an AI stand-in that acted consistently with the way I like them to act. (For example, everyone on Twitter could be a real person in another simulation; me following them would start to leak their reality into mine).

This is sounding oddly familiar, but I can't put my finger on why.

It seems like most other commenters so far don't share my opinion, but I view the above scenario as basically equivalent to wireheading, and consequently see it as only very slightly better than the destruction of all earth-originating intelligence (assuming the AI doesn't do anything else interesting). "Affecting-the-real-world" is actually the one value I would not want to trade off (well, obviously, I'd still trade it off, but only at a prohibitively expensive rate).

I'm much more open to trading off other things, however. For example, if we could get Failed Utopia #4-2 much more easily than the successful utopia, I'd say we should go for it. What specific values are the best to throw away in the pursuit of getting something workable isn't really clear, though. While I don't agree that if we lose one, we lose them all, I'm also not sure that anything in particular can be meaningfully isolated.

Perhaps the best (meta-)value that we could trade off is "optimality" - we should consider that if we see a way to design something stable that's clearly not the best we can do, we should nonetheless go with it if it's considerably easier than better options. For example, if you see a way to specify a particular pretty good future and have the AI build that without going into some failure mode, it might be better to just use that future instead of trying to have the AI design the best possible future.

For those who are interested in acquiring scientific knowledge and learning the nature of the universe, the simulation is accurate enough that everything they learn and discover is true of the real world.

Doesn't this include the information that there's now a simulation doing all of this in the real world?

Would you want to live in such a utopia?

In terms of directions to pursue, it seems like the first thing you want to do is make sure the AI is essentially transparent and that we don't have much of an inferential gap with it. Otherwise when we attempt to have it give a values and tradeoffs solution, we may not get anywhere near what we want.

In essence if the AI should be able to look at all the problems facing earth and say something like "I'm 97% sure our top priority is to build asteroid deflectors, based on these papers, calculations, and projections. The proposed plan of earthquake stabilizers is only 2% likely to be the best course of action based on these other papers, calculations, and projection" If it doesn't have that kind of approach, there seem to be many ways that things can go horribly wrong.

Examples:

A: If the AI can build Robotic Earthquake stabilizers at essentially no cost, and prevent children from being killed in earthquakes, or, it can simulate everyone and have our simulations have that experience at essentially no cost, the AI should probably be aware of the fact that these are different things so we don't say "Yes, build those earthquake stabilizers." and then it uploads everyone, and we say "That isn't what I meant!"

B: And the AI should definitely provide some kind of information about proposed plans/alternatives. If we say "Earthquake stabilizers save the most children, build those!" and the AI is aware "Actually, Asteroid deflectors save ten times more children." it shouldn't just go "Oh well, they SAID earthquake stabilizers, I'm not even going to mention the deflectors."

C: Or maybe: "I thought killing all children was the best way to stop children from suffering, and that this was trivially obvious so of course you wanted me to make a childkiller plague and I did so and released it without telling you when you said "Reduce children's suffering.""

D: Or it could simulate everyone and say "Well, they never said to keep the simulation running after I simulated everyone, so time to shutdown all simulations and save power for their next request."

Once you've got that settled, you can attempt to have the AI do other things, like assess Anti-Earthquake/Asteroid Deflection/Uploading, because you'll actually be able to ask it "Which of these are the right things to do and why based on these values and these value tradeoffs?" and get an answer which makes sense. You may not like or expect the answer, but at least you should be able to understand it given time.

For instance, going back to the sample problem, I don't mind that simulation that much, but I don't mind it because I am assuming it works as advertised. If it has a problem like D and I just didn't realize that and the AI didn't think it noteworthy, that's a problem. Also, for all I know, there is an even better proposed life, that the AI was aware of, and didn't think to even suggest as in B.

Given a sufficiently clear AI, I'd imagine that it could explain things to me sufficiently well that there wouldn't even be a question of which values to trade off, because the solution would be clear, but for all I know, it might come up with "Well, about half of you want to live in a simulated utopia, and about half of you want to live in a real utopia, and this is unresolvable to me because of these factors unless you solve this value tradeoff problem."

It would still however, have collected all the reasons together that explained WHY it couldn't solve that value tradeoff problem, which would still be a handy thing to have anyway, since I don't have that right now.

Edit: Eek, I did not realize the "#" sign bolded things, extra bolds removed.

1. Are there other values that, if we traded them off, might make MFAI much easier?

I don't understand this question. Is it somehow not trivially obvious that the more values you remove from the equation (starting with "complexity"), the easier things become?

1,2. Is there something which is the worst possible error, such that every future that does not contain that error is preferable to every future which does (for example, is the destruction of all earth-originating intelligence worse than anything which does not include it?)

I'm sure the answer is yes, but I'm not sure it's trivial to prove it.