Simulation argument meets decision theory

So on the face of it it seems that the only accessible outcomes are:

• original-X chooses "sim" and gets +1; all simulated copies also choose "sim" and get +0.2 (and then get destroyed?)
• original-X chooses "not sim" and gets +0.9; no simulated copies are made

and it seems like in fact everyone does better to choose "sim" and will do so. This is also fairly clearly the best outcome on most plausible attitudes to simulated copies' utility, though the scenario asks us to suppose that X doesn't care about those.

I'm not sure what the point of this is, though. I'm not seeing anything paradoxical or confusing (except in so far as the very notion of simulated copies of oneself is confusing). It might be more interesting if the simulated copies get more utility when they choose "not sim" rather than less as in the description of the scenario, so that your best action depends on whether you think you're in a simulation or not (and then if you expect to choose "sim", you expect that most copies of you are simulations, in which case maybe you shouldn't choose "sim"; and if you expect to choose "not sim", you expect that you are the only copy, in which case maybe you should choose "sim").

I'm wondering whether perhaps something like that was what pallas intended, and the current version just has "sim" and "not sim" switched at one point...

This is a formal version of a real-life problem I've been thinking about lately.

Should we commit to creating ancestor-simulations in the future, where those ancestor-simulations will be granted a pleasant afterlife upon what appears to their neighbors to be death? If we do, then arguably we increase the likelihood that we ourselves have a pleasant afterlife to look forward to.

I don't actually have a rigorous answer at the moment, but let me go into what I think of as the "two-fluid model of anthropics."

The two "fluids" are indexical probability measure and anthropic measure. Indexical probability is "how likely you are to be a particular person" - it is determined by what you know about the world. Anthropic measure is magical reality fluid - it's "how much you exist." Or if we project into the future, your probability measure is how likely you are to see a certain outcome. Anthropic measure is how much that outcome will exist.

Usually these two measures correspond. We see things and things exist at about the same rate. But sometimes they diverge, and then we need a two-fluid model.

A simple example of this is the quantum suicide argument. The one says "no matter what, I'll always have some chance of surviving. From my perspective, then, I'll never die - after all, once I die, I stop having a perspective. So, let's play high-stakes Russian roulette!" There are multiple ways to frame this mistake, but the relevant one here is that it substitutes what is seen (your probability that you died) for what is real (whether you actually died).

Another case where they diverge is making copies. If I make some identical copies of you, your probability that you are some particular copy should go down as I increase the number of copies. But you don't exist any less as I make more copies of you - making copies doesn't change your anthropic measure.

Two-fluid model decision-making algorithm: follow the strategy that maximizes the final expected utility (found using anthropic measure and causal structure of the problem) for whoever you think you are (found using indexical probability measure). This is basically UDT, slightly generalized.

But (and this is why I was wrong before), this problem is actually outside the scope of my usual two-fluid model. It doesn't have a well-defined indexical probability (specifically which person you are, not just what situation you're in) to work with. It depends on your decision. We'll need to figure out the correct generalization of TDT to handle this.

Okay, so you choose as if you're controlling the output of the logical node that causes your decision. Non-anthropically you can just say "calculate the causal effect of the different logical-node-outputs, then output the one that causes the best outcome." But our generalization needs to be able to answer the question "best outcome for whom?" I would love to post this comment with this problem resolved, but it's tricky and so I'll have to think about it more / get other people to tell me the answer.

Firstly, let's assume that we deny physical continuity of personal identity, but embrace psychological continuity instead. In that case, you don't want to press the button as it simulation has just as much claim to be the continuation of "you" as the original you, even if each instance knows whether it is in the real world or in a simulation.

The rest of this analysis will assume continuity of physical identity instead. From this perspective, pressing the button reduces your expectation of future utility, but it only does this by changing your epistemic state, rather than actually making things worse for the person who pushed the button. As an example, if I was to completely erase the strongest evidence of global warming from everyone's mind, I would reduce the expectation of a catastrophe, but not the likelihood of it occurring. Here, we've done the same, just by creating simulations so that you no longer are certain whether you are the original or a simulation. Since this is only a change of your epistemic state, you should actually push the button.

This is actually quite an important problem for building an AI as we don't want it to mess with its expectation.

I smiled when I realized why the answer isn't trivially "press sim", but that slight obfuscation is causing a lot of confused people to get downvoted.

I think there's an ambiguity here:

• if X*_i presses "sim" and X*_i receives .2 utility, then "sim" is the dominant choice and there's nothing more to say.

• if X*_i presses "sim" and X gets .2 utility, then each X' (X or X*_i) faces a variant of the Absent Minded Driver.

The second interpretation seems though to be contraddicted by OP words:

For every agent it is true that she does not gain anything from the utility of another agent despite the fact she and the other agents are identical!

Your problem setup contains a contradiction. You said that X and X*_i are identical copies, and then you said that they have different utility functions. This happened because you defined the utility function over the wrong domain; you specified it as (world-history, identity)=>R when it should be (world-history)=>R.

The non-anthropic version is this:

X self-modifies to ensure that any copy of X (i.e. any X*) will only care about X and not X*, presses "sim", and everyone "gets" 1 utility.

It seems like "sim" is the strictly dominant action for X and all X*. Thus we should always press "sim". The more interesting question would be what would happen if the incentives for pressing "sim" were reversed for the agents (i.e., the payoff for an agent choosing "not sim" exceeded "sim"). Then we'd have a cool mixed strategy problem.

Isn't this another case of the halting problem in disguise? When the computer simulates you, it also simulates your attempt to figure out what the computer would do.

I think the issue may be that the "egoistic" utility functions are incoherent in this context, because you're actually trying to compare the utility functions of two different agents as if they were one.

Let's say, for example, that X is a paperclip maximiser who gets either 10 paperclips or 9 paperclips, and each X* is a human who either saves 2 million lives or 1 million lives.

If you don't know whether you're X or X*, how can you compare 10 paperclips to 2 million lives?

Sim. Who cares about the utility function of the simulated beings. Maybe the computer will simulate another billion of them a second later.

Where do these utility numbers come from?