The Empty White Room: Surreal Utilities

The difference between the dust specks and the white room is that in the case of the dust specks, each experience is happening to a different person. The arbitrarily big effect comes from your consideration of arbitrarily many people - if you wish to reject the arbitrarily big effect, you must reject the independence of how you care about people.

In the case of the white room, everything's happening to you. The arbitrarily big effect comes from your consideration of obtaining arbitrarily many material goods. If you wish to reject the arbitrarily big effect, you must reject the independence of how you care about each additional Mona Lisa. But in this case, unlike in dust specks, there's no special reason to have that independence in the first place.

Now, if the room were sufficiently uncomfortable, maybe I'd off Frank - as long as I was sure the situation wasn't symmetrical. But I don't think we need surreal numbers to describe why, if I get three square meals a day in the white room, I won't kill Frank just to get an infinite amount of food.

To show that my utility for Frank is infinite you have to establish that I wouldn't trade an arbitrarily small probability of his death for the nanofab. I would make the trade at sufficiently small probabilities.

Also, the surreal numbers are almost always unnecessarily large. Try the hyperreals first.

I'd kill Frank.

ETA: Even if I'd be the only sentient being in the entire nanofabbed universe, it's still better than 2 people trapped in a boring white room, either forever or until we both die of dehydration.

Would you trade a Mona Lisa picture for a 1/3^^^3 chance of saving Frank's life?

Are you using the same kind of decision-making in your real life?

Problem: People in real life choose the equivalent of the fabricator over Frank all the time, assuming "choosing not to intervene to prevent a death" is equivalent to choosing the fabricator...

Also, people accept risks to their own life all the time.

This seems to me obviously very wrong. Here's why. (Manfred already said something kinda similar, but I want to be more explicit and more detailed.)

My utility function (in so far as I actually have one) operates on states of the world, not on particular things within the world.

It ought to be largely additive for mostly-independent changes to the states of different bits of the world, which is why arguably TORTURE beats DUST SPECKS in Eliezer's scenario. (I won't go further than "arguably"; as I said way back when Eliezer first posted that, I don't trust any bit of my moral machinery in cases so far removed from ones I and my ancestors have actually encountered; neither the bit that says "obviously different people's utility changes can just be added up, at least roughly" nor the bit that says "obviously no number of dust specks can be as important as one instance of TORTURE".

But there's no reason whatever why I should value 100 comfy cushions any more at all than 10 comfy cushions. There's just me and Frank; what is either of us going to do with a hundred cushions that we can't do with 10?

Maybe that's a bit of an exaggeration; perhaps with 100 cushions we could build them into a fort and play soldiers or something. (Not really my thing, but Frank might like it, and it seems like anything that relieves the monotony of this drab white room would be good. And of course the offer actually available says that Frank dies if I get the cushions.) But I'm pretty sure there's literally no benefit to be had from a million cushions beyond what I'd get from ten thousand.

And the same goes even if we consider things other than cushions. There's just only so much benefit any single human being can get from a device like this, and there's no obvious reason why -- even without incommensurable values or anything like them -- that should exceed the value of another human life in tolerable conditions.

In particular, any FAI that successfully avoids disasters like tiling the universe with inert smiley humanoid faces seems likely to come to the same conclusion; so I don't agree that in the Seelie scenario we should expect it to accept Omega's offer unless it has incommensurable values.

There are a few ways that that might be wrong, which I'll list; it seems to me that each of them breaks one of the constraints that make this an argument for incommensurable values.

Possible exception 1: maybe the cushions wear out and I'm immortal in this scenario. But then I guess Frank's immortal too, in which case the possible value of that life we're trading away just went way up (in pretty much exactly the way the value of the cushion-source did).

Possible exception 2: Alternatively, perhaps I'm immortal and Frank isn't. Or perhaps the machine, although it can't make a mind, can make me immortal when I wasn't before. In that case: separate stretches of my immortal life -- say, a million years long each -- might reasonably be treated as largely independent, so then, yes, you can make the same sort of argument for preferring CUSHIONS AND DEATH over STATUS QUO as for preferring TORTURE over DUST SPECKS, and I don't see that one preference is so much more obviously right than the other as to let you conclude that you want incommensurable values after all.

I guess I'm thinking about this wrong. I want to either vaporize Frank or have Frank vaporize me for the same deal. I prefer futures with fewer, happier minds. IOW, I guess I accept Nozick's utility monster.

You need to specify what happens if you decline the offer. Right now it looks as if you and Frank both die of dehydration after couple of days. Or you go insane and one kills the other (and maybe eats him). And then dies anyway. In order for this to be a dilemma, the baseline outcome needs to be more... wholesome.

Also, the temptation isn't very tempting. An ornate chandelier? I could get some value from the novelty from seeing it and maybe staring at it for several hours if it's really ornate. It's status as a super-luxury good would be worthless in the absence of a social hierarchy. I couldn't trade or give away gazillions of them so multiplying wouldn't add anything.

I suppose the nanofab can manufacture novelty (though it isn't quite clear from your description). But it won't make minds. This is a problem. Humans are quite big on belonging to a society. I can't imagine what being an immortal god of solipsistic matrix would feel like but I suspect it could be horrible.

The prohibition against creating minds isn't very clear as we don't have a clear idea on what constitutes a mind. Maybe I could ask Omega to generate the best possible RPG game with an entire simulated world and super-realistic NPCs? Would that be allowed? I don't know if a sufficiently high-fidelity simulation of a person isn't an actual person. And there would be at least one mind -- me. Could I self-modify, grow my sense of empathy to epic proportions and start imagining people into being? And then, to fix my past sins, I'd order a book "Everything You Could Ever Ask About Frank" or something.

Note: I think that the fact that there are only two lives/minds mentally posited in the problem, "You" and "Frank" may significantly modify the perceived value of lives/minds.

After all, consider these problems:

1: The white room contains you, and 999 other people. The cost of the Nanofab is 1 life.

2: The white room contains you, and 999 other people. The cost of the Nanofab is 2 lives.

3: The white room contains you, and 999 other people. The cost of the Nanofab is 500 lives.

4: The white room contains you, and 999 other people. The cost of the Nanofab is 900 lives.

5: The white room contains you, and 1 other person. The cost of the Nanofab is 1 life.

If lives are sacred in general, you should be equally reluctant to buy the Nanofab in all cases. That seems unlikely to be the case for most people.

On the other hand if the sacred value is "When you and someone else are alone, don't sacrifice one of you" Someone might be willing to buy the Nanofab in cases 1-4 and not 5.

(Of course, seeing all options at the same time likely also influences behavior)

I liked this post. The white room doesn't really seem to work so well as an intuition pump, but it's good that someone has brought up the idea of using surreal utilities.

Since they lead to these tears, within which tradeoff happens normally, but across which you don't trade, it would be interesting to see if we actually find that. We might want to trade n lives for n+1 lives, but what other sacred values do humans have, and how do they behave?

The problem with your "white room" scenario is that one human can't actually have Large amounts of utility. The value of the 3^^^3th seat cushion is actually, truly zero.

an eternity of non-boredom

I don't think I could possibly get that in a room containing no other minds.

Decision theory with ordinals is actually well-studied and commonly used, specifically in language and grammar systems. See papers on Optimality Theory.

The resolution to this "tier" problems is assigning every "constraint" (thing that you value) an abstract variable, generating a polynomial algebra in some ungodly number of variables, and then assigning a weight function to that algebra, which is essentially assigning every variable an ordinal number, as you've been doing.

Just as perspective on the abstract problem there are two confounders that I don't see addressed

One is that every time you assign a value to something you should actually be assigning a distribution of possible values to it. It's certainly possible to tighten these distributions in theory but I don't think that human value systems actually do tighten them enough to reduce this to a mathematically tractable problem; and if they DO constrain things that much I'm certain we don't know it. Which is just saying that this problem is going to end up with people reaching different intuitive conclusions.

Two is that it tends to be the case that these systems are wildly underspecified. If you do the appropriate statistics to figure out how people rank constraints, you don't get an answer, you get some statistics about an answer, and the probability distributions on people's preferences are WIIIIIIDE. In order to solve this problem in linguistics people use subject- and problem-specific methods to throw together ad hoc conclusions. So I guess these are really the same complaint; you shouldn't be using single-value assignments and when you stop doing that you lose the computational precision that makes talking about ordinal numbers really interesting.

(for reference my OT knowledge comes entirely from casual conversations with people who do it professionally; I'm fairly confident in these statements but I'd be open to contradiction from a linguist)

The tiered values approach appears to run into continuity troubles, even with surreal numbers.

Seelie 2.0 double checks with its mental copy of your values, finds that you would rather have Frank's life than infinite Fun, and assigns it a tier somewhere in between - for simplicity, let's say that it puts it in the tier. And having done so, it correctly refuses Omega's offer.

How does it compare punching/severely injuring/torturing Frank with your pile of cushions or with infinite fun? What if there is a .0001%/1%/99% probability that Frank will die?

I would say the most obvious flaw with surreal utilities (or, generally, pretty much anything other than real utilities) is simply that you can't sensibly do infinite sums or limits or integration, which is after all what expected value is, which is the entire point of a utility function. If there are only finitely many possibilities you're fine, but if there are infinitely many possibilities you are stuck.

Anyone new to this page: I'm basically talking about Hausner utilities, except with surreal numbers needlessly slapped on.

Could utilities be multi-dimensional? Real vector spaces are much nicer to work with than to surreal numbers.

For example, the utility for frank being alive would be (1,0), while the utility for a seat cushion is (0,1). Using lexicographic ordering, (1,0) > (0,3^^^3).

Vector valued utility functions violate the VNM axiom of continuity, but who cares.

I give seat cushions zero value. I give the comfort they bring me zero value. The only valuable thing about them is the happiness they bring from the comfort. Unless the nanofab can make me as happy as my current happiness plus Frank's combined, nothing it makes will be worth it. It probably could, but that's not the point.

As for the idea of surreal utilities, there's nothing wrong with it in principle. The axiom they violate isn't anything particularly bad to violate. The problem is that, realistically speaking, you might as well just round infinitesimal utility down to zero. If you consider a cushion to be worth infinitesimally many lives, then if you're given a choice that gives you an extra cushion and has zero expected change in the number of lives, you'd take it. But you won't get that choice. You'll get choices where the expected change in number of lives is very small, but the expected value from lives will always be infinitely larger than the expected value from cushions.

I would like to point out that Fun was listed as a separate tier, and that whether or not to put it on the same tier as a human life is entirely up to you. Surreal utilities aren't much of a decision theory, they're just a way to formalize tiered values; the actual decision you make depends entirely on the values you assign by some other method.

To me there is very big difference between 0 probability and an exact infinidesimal probability and I disagree that it is obvious they suffer from the same problems.

For example if I have a unit line and choose some particular point the probability of picking some exact point is epsilon. If I where to pick a point from a unit square the probability would be yet epsilon times smaller, for a total of epsilon*epsilon. If I where to pick a point from a line of lenght 2 the probability would only be half for a total of epsilon/2.

When usage of infinidesimal probabilites often fail is not spesifying which one and treating them all as the same one. It is not so that If you can't multiply an amount finite times and end up with a finite amount then all such amounts must be equal. If I multiply epsilon by the first order infinite I get a finite 1. If I multiply epsilon*epsilon by the first order infinite I get a non-finite positive amount (exactly epsilon).

What impact infinite or infinidesimal probabilities make can largely be adopted by using rules to the same effect. An Example could be distinguishing between "pure" 0 and "almost never" and "pure" 1 and "almost always". For what practical effect they might make consider darts. There are various probabilities conserning in which sector the dart lands and for example whether it it lands on a line dividing areas. However the numbers being passed around conserning lines will live a life largely separated by the math done for the areas. Now I can either take the separatedness as known fact outside the analysis or have the analysis show the separtedness of them. And there will be multiple types of zero probabilities. For example given that the board was hit, the probability for the dart not hitting any spesific area, line separating areas or a connection between lines is zero. However if I throw a dart I know I should not expect to hit that exact spot during the evening, the probability of it's recurrence is an "impure" zero. The dart can still land there and it won't magically avoid that spot. And no matter how many darts I throw the probability of hitting an old spot increases but I am not expecting to actually hit one. However if I notice that my probability of hitting an area divider or a line intersection is vanishing, in practise I know to focus on the area ratios, but I won't accuse of someone of lying if they report a single such occurrence during the time I know such a man. However if they report 2 such occurrences I have reason to be suspicious.

While you might not kill Frank to get the machine, there has to be some small epsilon, such that you would take the machine in exchange for an additional probability epsilon of Frank dying today. Wouldn't Frank agree to that trade?Wouldn't you agree to a small additional probability of dying yourself in exchange for the machine?

Otherwise, living in a big white room is going to be a bit - ahem - dull for both of you.

I agree there is a difficulty here for any utility function. The machine can make unlimited quantities of secular goods, so if 3^^^3 beautiful paintings are worth probability epsilon of Frank dying, why aren't 4^^^^4 even more beautiful paintings worth probability 1 of Frank dying? Presumably because Frank would accept the former trade, but not the latter one.

I decided some time ago that I don't really care about morality,because my revealed preferences say I care a lot more about personal comfort than saving lives and I'm unwilling to change that. I don't think I'd be willing to spend £50 to save the life of an anonymous stranger that I'd never meet, if I found out about a charity that efficient, so for the purposes of a thought experiment I should also be willing to kill Frank for such a small amount of money, assuming social and legal consequences are kept out of the way by Omega, and the utility of possibly befriending Frank isn't taken into account.

That aside, though, I think taking the nanofab is actually the morally right choice. Two lives spent in an uncomfortable featureless room are worth significantly less than one life spent as a nigh-omnipotent god. I'm not sure if letting/making Frank continue to live in the uncomfortable featureless room is even of positive utility to him. If I knew there wasn't any more to life than the featureless uncomfortable room, I would be contemplating suicide fairly quickly.

In effect, the sacred value has infinite utility relative to the secular value.

That's no accurate representation of how human's value sacred values. There are cases where people value getting X sacred utilitons over getting X sacred utilitons + Y secular ultilitons.

Emerging sacred values: Iran’s nuclear program by Morteza Dehghani is a good read to get how sacred values behave.

Secret values prevent corruption.

Thank you, if for nothing else, for clarifying my intuitive sense that Dust Specks are superior to Torture. Your thought experiment clarified to me that tiers of utility DO match my value system.

An alternate title for this post was "Surreal Utilities and Seat Cushions."

On a side note - I am not entirely sure what tags to apply here, and I couldn't seem to find an exhaustive tag list (though I admittedly didn't work very hard.)