Outlawing Anthropics: An Updateless Dilemma

Actually... how is this an anthropic situation AT ALL?

I mean, wouldn't it be equivalent to, say, gather 20 rational people (That understand PD, etc etc etc, and can certainly manage to agree to coordinate with each other) that are allowed to meet with each other in advance and discuss the situation...

I show up and tell them that I have two buckets of marbles, some of which are green, some of which are red

One bucket has 18 green and 2 red, and the other bucket has 18 red and 2 green.

I will (already have) flipped a logical coin. Depending on the outcome, I will use either one bucket or the other.

After having an opportunity to discuss strategy, they will be allowed to reach into the bucket without looking, pull out a marble, look at it, then, if it's green choose if to pay and steal, etc etc etc. (in case it's not obvious, the payout rules being equivalent to the OP)

As near as I can determine, this situation is entirely equivalent to the OP and is in no way an anthropic one. If the OP actually is an argument against anthropic updates in the presence of logical uncertainty... then it's actually an argument against the general case of Bayesian updating in the presence of logical uncertainty, even when there's no anthropic stuff going on at all!

EDIT: oh, in case it's not obvious, marbles are not replaced after being drawn from the bucket.

Well, we don't want to build conscious AIs, so of course we don't want them to use anthropic reasoning.

Why is anthropic reasoning related to consciousness at all? Couldn't any kind of Bayesian reasoning system update on the observation of its own existence (assuming such updates are a good idea in the first place)?

An AI that runs UDT wouldn't conclude that it was a Boltzmann or non-Boltzmann brain. For such an AI, the statement has no meaning, since it's always both. The closest equivalent would be "Most of the value I can create by making the right decision is concentrated in the vicinity of non-Boltzmann brains."

BTW, does my indexical uncertainty and the Axiom of Independence post make any more sense now?

Curses on this problem; I spent the whole day worrying about it, and am now so much of a wreck that the following may or may not make sense. For better or worse, I came to a similar conclusion of Psy-Kosh: that this could work in less anthropic problems. Here's the equivalent I was using:

Imagine Omega has a coin biased so that it comes up the same way nine out of ten times. You know this, but you don't know which way it's biased. Omega allows you to flip the coin once, and asks for your probability that it's biased in favor of heads. The coin comes up heads. You give your probability as 9/10.

Now Omega takes 20 people and puts them in the same situation as in the original problem. It lets each of them flip their coins. Then it goes to each of the people who got tails, and offers $1 to charity for each coin that came up tails, but threatens to steal $3 from charity for each coin that came up heads.

This nonanthropic problem works the same way as the original anthropic problem. If the coin is really biased heads, 18 people will get heads and 2 people will get tails. In this case,the correct subjective probability to assign is definitely 9/10 in favor of whatever result you got; after all, this is the correct probability when you're the only person in the experiment, and just knowing that 19 other people are also participating in the experiment shouldn't change matters.

I don't have a formal answer for why this happens, but I can think of one more example that might throw a little light on it. In another thread, someone mentioned that lottery winners have excellent evidence that they are brains-in-a-vat and that the rest of the world is an illusion being put on by the Dark Lord of the Matrix for their entertainment. After all, if this was true, it wouldn't be too unlikely for them to win the lottery, so for a sufficiently large lottery, the chance of winning it this way exceeds the chance of winning it through luck.

Suppose Bob has won the lottery and so believes himself to be a brain in a vat. And suppose that the evidence for the simulation argument is poor enough that there is no other good reason to believe yourself to be a brain in a vat. Omega goes up to Bob and asks him to take a bet on whether he is a brain in a vat. Bob says he is, he loses, and Omega laughs at him. What did he do wrong? Nothing. Omega was just being mean by specifically asking the one person whom ve knew would get the answer wrong.

Omega's little prank would still work if ve announced ver intention to perform it beforehand. Ve would say "When one of you wins the lottery, I will be asking this person to take a bet whether they are a brain in a vat or not!" Everyone would say "That lottery winner shouldn't accept Omega's bet. We know we're not brains in vats." Then someone wins the lottery, Omega asks if they're a brain in a vat, and they say yes, and Omega laughs at them (note that this also works if we consider a coin with a bias such that it lands the same way 999999 out of a million times, let a million people flip it once, and ask people what they think the coin's bias is, asking the people who get the counter-to-expectations result more often than chance.)

Omega's being equally mean in the original problem. There's a 50% chance ve will go and ask the two out of twenty people who are specifically most likely to be wrong and can't do anything about it. The best course I can think of would be for everyone to swear an oath not to take the offer before they got assigned into rooms.

"I've made sacrifices! You don't know what it cost me to climb into that machine every night, not knowing if I'd be the man in the box or in the prestige!"

sorry- couldn't help myself.

Again: how can you talk about concluding that you are a Boltzmann brain? To conclude means to update, and here you refuse updating.

I read this and told myself that it only takes five minutes to have an insight. Five minutes later, here's what I'm thinking:

Anthropic reasoning is confusing because it treats consciousness as a primitive. By doing so, we're committing LW's ultimate no-no: assuming an ontologically fundamental mental state. We need to find a way to reformulate anthropic reasoning in terms Solomonoff induction. If we can successfully do so, the paradox will dissolve.

More thinking out loud:

It really is in your best interest to accept the offer after you're in a green room. It really is in your best interest to accept the offer conditional on being in a green room before you're assigned. Maybe part of the problem arises because you think your decision will influence the decision of others, ie because you're acting like a timeless decision agent. Replace "me" with "anyone with my platonic computation", and "I should accept the offer conditional on being in a green room" with "anyone with my platonic computation should accept the offer, conditional on anyone with my platonic computation being in a green room." But the chances of someone with my platonic computation being in a green room is 100%. Or, to put it another way, the Platonic Computation is wondering "Should I accept the offer conditional on any one of my instantiations being in a green room?". But the Platonic Computation knows that at least one of its instantiations will be in a green room, so it declines the offer. If the Platonic Computation was really a single organism, its best option would be to single out one of its instantiations before-hand and decide "I will accept the offer, given that Instantiation 6 is in a green room" - but since most instantiations of the computation can't know the status of Instantiation 6 when they decide, it doesn't have this option.

I've been watching for a while, but have never commented, so this may be horribly flawed, opaque or otherwise unhelpful.

I think the problem is entirely caused by the use of the wrong sets of belief, and that anything holding to Eliezer's 1-line summary of TDT or alternatively UDT should get this right.

Suppose that you're a rational agent. Since you are instantiated in multiple identical circumstances (green rooms) and asked identical questions, your answers should be identical. Hence if you wake up in a green room and you're asked to steal from the red rooms and give to the green rooms, you either commit a group of 2 of you to a loss of 52 or commit a group of 18 of you to a gain of 12.

This committal is what you wish to optimise over from TDT/UDT, and clearly this requires knowledge about the likelyhood of different decision making groups. The distribution of sizes of random groups is not the same as the distribution of sizes of groups that a random individual is in. The probabilities of being in a group are upweighted by the size of the group and normalised. This is why Bostrom's suggested 1/n split of responsibility works; it reverses the belief about where a random individual is in a set of decision making groups to a belief about the size of a random decision making group.

By the construction of the problem the probability that a random (group of all the people in green rooms) has size 18 is 0.5, and similarly for 2 the probability is 0.5. Hence the expected utility is (0.512)+(0.5-52)=-20.

If you're asked to accept a bet on there being 18 people in green rooms, and you're told that only you're being offered it, then the decision commits exactly one instance of you to a specific loss or gain, regardless of the group you're in. Hence you can't do better than the 0.9 and 0.1 beliefs.

If you're told that the bet is being offered to everyone in a green room, then you are committing to n times the outcome in any group of n people. In this case gains are conditional on group size, and so you have to use the 0.5-0.5 belief about the distribution of groups. It doesn't matter because the larger groups have the larger multiplier and thus shutting up and multiplying yields the same answers as a single-shot bet.

ETA: At some level this is just choosing an optimal output for your calculation of what to do, given that the result is used variably widely.

I think I'm with Bostrom.

The problem seems to come about because the good effects of 18 people being correct are more than wiped out by the bad effects of 2 people being wrong.

I'm sure this imbalance in the power of the agents has something to do with it.

I waited to comment on this, to see what others would say. Right now Psy-Kosh seems to be right about anthropics; Wei Dai seems to be right about UDT; timtyler seems to be right about Boltzmann brains; byrnema seems to be mostly right about pointers; but I don't understand why nobody latched on to the "reflective consistency" part. Surely the kind of consistency under observer-splitting that you describe is too strong a requirement in general: if two copies of you play a game, the correct behavior for both of them would be to try to win, regardless of what overall outcome you'd prefer before the copying. The paperclip formulation works around this problem, so the correct way to analyze this would be in terms of multiplayer game theory with chance moves, as Psy-Kosh outlined.

You can't reject the conclusion that you are a Boltzmann brain - but if you are, it doesn't matter what you do, so the idea doesn't seem to have much impact on decision theory.

There are lots of ordinary examples in game theory of time inconsistent choices. Once you know how to resolve them, then if you can't use those approaches to resolve this I might be convinced that anthropic updating is at fault. But until then I think you are making a huge leap to blame anthropic updating for the time inconsistent choices.

prior probability of heads * Omega's offered payoff  >  prior of tails * Omega's price

prior of (heads and Omega offers a CM for this coin) * payoff  >  prior of (tails and CM) * price.

I think I'll have to sit and reread this a couple times, but my INITIAL thought is "Isn't the apparent inconsistancy here qualitatively similar to the situation with a counterfactual mugging?"

Huh. Reading this again, together with byrnema's pointer discussion and Psy-Kosh's non-anthropic reformulation...

It seems like the problem is that whether each person gets to make a decision depends on the evidence they think they have, in such a way to make that evidence meaningless. To construct an extreme example: The Antecedent Mugger gathers a billion people in a room together, and says:

"I challenge you to a game of wits! In this jar is a variable amount of coins, between $0 and $10,000. I will allow each of you to weigh the jar using this set of extremely imprecise scales. Then I will ask each of you whether to accept my offer: to as a group buy the jar off me for $5000, the money to be distributed equally among you. Note: although I will ask all of you, the only response I will consider is the one given by the person with the greatest subjective expected utility from saying 'yes'."

In this case, even if the jar always contains $0, there will always be someone who receives enough information from the scales to think the jar contains >$5000 with high probability, and therefore to say yes. Since that person's response is the one that is taken for the whole group, the group always pays out $5000, resulting in a money pump in favour of the Mugger.

The problem is that, from an outside perspective, the observations of the one who gets to make the choice are almost completely uncorrelated from the actual contents of the jar, due to the Mugger's selection process. For any general strategy Observations → Response, the Mugger can always summon enough people to find someone who has seen the observations that will produce the response he wants, unless the strategy is a constant function.

Similarly, in the problem with the marbles, only the people with the observation Green get any influence, so the observations of "people who get to make a decision" are uncorrelated with the actual contents of the buckets (even though observations of the participants in general are correlated with the buckets).

isn't this a problem with the frequency you are presented with the opportunity to take the wager? [no, see edit]

the equation: (50% ((18 +$1) + (2 -$3))) + (50% ((18 -$3) + (2 +$1))) = -$20 neglects to take into account that you will be offered this wager nine times more often in conditions where you win than when you lose.

for example, the wager: "i will flip a fair coin and pay you $1 when it is heads and pay you -$2 when it is tails" is -EV in nature. however if a conditional is added where you will be asked if you want to take the bet 90% of the time given the coin is heads (10% of the time you are 'in a red room') and 10% of the time given the coin is tails (90% of the time you are 'in a red room'), your EV changes from (.5)(1) + (.5)(-2) = -.5 to (.5)(.9)($1) + (.5)(.1)(-$2) = $.35 representing the shift from "odds the coin comes up heads" to "odds the coin comes up heads and i am asked if i want to take the bet"

it seems like the same principle would apply to the green room scenario and your pre-copied self would have to conclude that though the two outcomes are +$12 or -$52, they do not occur with 50-50 frequency and given you are offered the bet, you have a 90% chance of winning. (.9)($12) + (.1)(-$52) = $5.6

EDIT: okay, after thinking about it, i am wrong. the reason i was having trouble with this was the fact that when the coin comes up tails and 90% of the time i am in a red room, even though "i" am not being specifically asked to wager, my two copies in the green rooms are - and they are making the wrong choice because of my precommitment to taking the wager given i am in a green room. this makes my final EV calculation wrong as it ignores trials where "i" appear in a red room even though the wager still takes place.

its interesting that this paradox exists because of entities other than yourself (copies of you, paperclip maximizers, etc) making the "incorrect" choice the 90% of the time you are stuck in a red room with no say.

If the many worlds interpretation of quantum mechanics is true isn't anthropic reasoning involved in making predictions about the future of quantum systems. There exists some world in which, from the moment this comment is posted onward, all attempts to detect quantum indeterminacy fail, all two-slit experiments yield two distinct lines instead of a wave pattern etc. Without anthropic reasoning we have no reason to find this result at all surprising. So either we need to reject anthropic reasoning or we need to reject the predictive value of quantum mechanics under the many worlds interpretation. Right?

(Apologies if this has been covered, I'm playing catch-up and just trying to hash things out for myself. Also should I expect to be declared a prophet in the world in which quantum indeterminacy disappears from here on out?)

“You generalise probability, when anthropics are involved, to probability-2, and say a number defined by probability-2; so I’ll suggest to you a reward structure that rewards agents that say probability-1 numbers. Huh, if you still say the probability-2 number, you lose”.

This reads to me like, “You say there’s 70% chance no one will be around that falling tree to hear it, so you’re 70% sure there won’t be any sound. But I want to bet sound is much more likely; we can get measure the sound waves, and I’m 95% sure our equipment will register the sound. Wanna bet?”

I think this is a confusion of two different types of thinking. One is the classical thought of one being responsible only for the consequences of one's individual actions. If you think of yourself as an individual making independent decisions like this, then you are justified in thinking that there is a 90% chance of heads upon seeing a green room: 90% of individuals in green rooms, in expectation, are there when the coin flips heads.(note that if you modify the problem so that the outcomes of the bet only apply to the people making it, the bet becomes favorable even in advance, regardless of whether the agents are altruistic or selfish).

However, in this case you cannot claim that your "yes" decision has utility based on the result of the entire group's response to the question. If you did, then in heads flips, if all 18 people say "yes", all 18 people will claim that the utility of their action was $18, but only $18 of utility was gained by the group in total as a consequence of all these decisions, so they cannot all be right.(Note that if you modify the problem so that you really are the only one responsible for the decision, like by saying that everyone in green rooms except one person is mind-controlled to say "yes", then saying "yes" really is the right decision for that free-willed person, even in advance.)

It is also possible to reason in a TDT sort of way, acting as though you are making decisions for all identical copies of yourself. This is effectively defining yourself, as an agent, as the set of all identical copies of yourself with identical input. In this case, it does make sense to take responsibility for the decision of the entire group, but it no longer makes sense to do an anthropic update: you as an agent, as a set of copies of yourself seeing green rooms, would exist whether or not the coin came up heads, and would make the same observations.

In conclusion, it makes sense to do an anthropic update if you think of yourself as an individual, it makes sense to take utilitarian responsibility for the whole group of actions if you think of yourself as a group of identical copies, but in neither case does it make sense to do both, which is what you would need to justify saying "yes" in this situation.

Timeless decision agents reply as if controlling all similar decision processes, including all copies of themselves. Classical causal decision agents, to reply "Yes" as a group, will need to somehow work out that other copies of themselves reply "Yes", and then reply "Yes" themselves. We can try to help out the causal decision agents on their coordination problem by supplying rules such as "If conflicting answers are delivered, everyone loses $50". If causal decision agents can win on the problem "If everyone says 'Yes' you all get $10, if everyone says 'No' you all lose $5, if there are conflicting answers you all lose $50" then they can presumably handle this. If not, then ultimately, I decline to be responsible for the stupidity of causal decision agents.

The coordination hack to work around some of the stupidity of causal decision agents doesn't appear to be necessary here.

"Somehow working out that the other copies of themselves reply 'yes'" should be trivial for an agent focussed on causality when the copies are identical, have no incentive to randomise and have identical inputs. If the payoff for others disagreeing is identical to the payoff for 'no' they can be ignored. The conflict penalty makes the coordination problem more difficult for the causal agent in this context, not less.

The reason we shouldn't update on the "room color" evidence has nothing to do with the fact that it constitutes anthropic evidence. The reason we shouldn't update is that we're told, albeit indirectly, that we shouldn't update (because if we do then some of our copies will update differently and we will be penalized for our disagreement).

In the real world, there is no incentive for all the copies of ourselves in all universes to agree, so it's all right to update on anthropic evidence.

[comment deleted]

Oops... my usual mistake of equivocating different things and evolving the problem until it barely resembles the original. I will update my "solution" later if it still works for the original.

... Sigh. Won't work. My previous "solution" recovered the correct answer of -20 because I bent the rules enough to have each of my green-room-deciders make a global rather than anthropic calculation.

Before the experiment, you calculate the general utility of the conditional strategy "Reply 'Yes' to the question if you wake up in a green room" as (50% ((18 +$1) + (2 -$3))) + (50% ((18 -$3) + (2 +$1))) = -$20

This assumes that the question is asked only once, but then, to which of the 20 copies will it be asked?

If all 20 copies get asked the same question (or equivalently if a single copy chosen at random is) then the utility is (50% 18/20 ((18 +$1) + (2 -$3))) + (50% 2/20 ((18 -$3) + (2 +$1))) = 2.8$ = 50% * 5.6$.

Consider the similar thought experiments:

• I flip a fair coin to determine whether to switch to my headdy coin or my tailly coin, which have a 90% and 10% probability of heads respectively.
• Now I flip this biased coin. If it comes up heads then I paint the room green, if it comes up tails I paint it red.
• You then find yourself in a green room.
• Then I flip the biased coin again, and repaint the room.
• Before this second flip, I offer you the bet of +1$ if the room stays green and -3$ if it becomes red.

The prior expected utility before the experiment is:

E(util|headdy) = 90% * 1$ + 10% * -3$ =  0.6$
E(util|tailly) = 10% * 1$ + 90% * -3$ = -2.6$
E(util) = 50% * E(util|headdy) + 50% * E(util|tailly) = -1$

Given that you find yourself in a green room after the first flip, you can determine the probability that the headdy coin is used:

P(green) = 0.5
P(green|headdy) = 0.9
P(headdy|green) = 0.9

Which gives a posterior utility:

E(util|green) = 0.9 * E(util|headdy) + 0.1 * E(util|tailly) = 0.28$

I keep having trouble thinking of probabilities when I'm to be copied and >=1 of "me" will see red and >=1 of "me" will see green. My thought is that it is 100% likely that "I" will see red and know there are others, once-mes, who see green, and 100% likely vice-versa. Waking up to see red (green) is exactly the expected result.

I do not know what to make of this opinion of mine. It's as if my definition of self - or choice of body - is in superposition. Am I committing an error here? Suggestions for further reading would be appreciated.

I remain convinced that the probability is 90%.

The confusion is over whether you want to maximize the expectation of the number of utilons there will be if you wake up in a green room or the expectation of the number of utilons you will observe if you wake up in a green room.

The notion of "I am a bolzmann brain" goes away when you conclude that conscious experience is a Tegmark-4 thing, and that equivalent conscious experiences are mathematically equal and therefore there is no difference and you are at the same time a human being and a bolzmann brain, at least until they diverge.

Thus, antrhopic reasoning is right out.

Whoohoo! I just figured out the correct way to handle this problem, that renders the global and egocentric/internal reflections consistent.

We will see if my solution makes sense in the morning, but the upshot is that there was/is nothing wrong with the green roomer's posterior, as many people have been correctly defending. The green roomer who computed an EV of $5.60 modeled the money pay-off scheme wrong.

In the incorrect calculation that yields $5.6 EV, the green roomer models himself as winning (getting the favorable +$12) when he is right and losing (paying the -$52) when he is wrong. But no, not exactly. The green roomer doesn't win every time he's right -- even though certainly he's right every time he's right.

The green roomer wins 1 out of every 18 times that he's right, because 17 copies of himself that were also right do not get their own independent winnings, and he loses 1 out of every 2 times he's wrong, because there are 2 of him that are wrong in the room that pays $52.

So it is Bostrom's division-of-responsibility, with the justification. It is probably more apt to name it division-of-reward.

Here's is the correct green roomer calculation:

EV = P(heads)(payoff given heads)(rate of payoff given heads)+ P(tails)(payoff given tails)(rate of payoff given tails)

=.9($12)(1/18)+.1(-$52)(1/2) = -2

(By the way, this doesn't modify what I said about pointers, but I must admit I don't understand at the moment how the two perspectives are related. Yet; some thoughts.)

In personal conversation, Nick Bostrom suggested that a division-of-responsibility principle might cancel out the anthropic update - i.e., the paperclip maximizer would have to reason, "If the logical coin came up heads then I am 1/18th responsible for adding +1 paperclip, if the logical coin came up tails then I am 1/2 responsible for destroying 3 paperclips." I confess that my initial reaction to this suggestion was "Ewwww", but I'm not exactly comfortable concluding I'm a Boltzmann brain, either.

I would perhaps prefer to use different language in the description but this seems to be roughly the answer to the apparent inconsistency. When reasoning anthropically you must decide anthropically. Unfortunately it is hard to describe such decision making without using sounding either unscientific or outright incomprehensible

I'm rather looking forward to another Eleizer post on this topic once he has finished dissolving his confusion. I've gained plenty from absorbing the posts and discussions and more from mentally reducing the concepts myself. But this stuff is rather complicated and to be perfectly honest, I don't trust myself to not have missed something.

Let the dilemma be, "I will ask all people who wake up in green rooms if they are willing to take the bet 'Create 1 paperclip if the logical coinflip came up heads, destroy 3 paperclips if the logical coinflip came up tails'. (Should they disagree on their answers, I will destroy 5 paperclips.)" Then a paperclip maximizer, before the experiment, wants the paperclip maximizers who wake up in green rooms to refuse the bet. But a conscious paperclip maximizer who updates on anthropic evidence, who wakes up in a green room, will want to take the bet, with expected utility ((90% +1 paperclip) + (10% -3 paperclips)) = +0.6 paperclips.

That last calculation doesn't look right to me : the paperclip maximizer in the green room still knows that there are other paperclip maximizers in red rooms who will refuse the bet whether or not they rely on anthropic evidence. So the expected utility of taking the bet would be 100% * - 5 paperclips.

Or did I misunderstand something?

Can someone come up with a situation of the same general form as this one where anthropic reasoning results in optimal actions and nonanthropic reasoning results in suboptimal actions?

I think you're missing a term in your second calculation. And why are anthropism and copies of you necessary for this puzzle. I suspect the answer will indicate something I'm completely missing about this series.

Take this for straight-up probability:

I have two jars of marbles, one with 18 green and 2 red, the other with 18 red and two green. Pick one jar at random, then look at one marble from that jar at random.

If you pick green, what's the chance that your jar is mostly green? I say 90%, by fairly straightforward application of bayes' rule.

I offer a wager: you get $1 per green and lose $3 per red marble in the jar you chose.

After seeing a green marble, I think your EV is $5.60. After seeing a red marble, I think your EV is $0 (you decline the bet). If you are forced to make the wager before seeing anything, conditional on drawing green, I think your EV is $2.80. I calculate it thus: 50% to get mostly-green jar, and 90% of that will you see green and take the bet, which is worth +$1*18 - $3*2 in this case. 50% to get mostly-red, 10% of which will you draw green, worth +1*2 - $3*18. 0.5 * 0.9 (1 \ 18 - 3 * 2) + 0.5 * 0.1 (1 \ 2 - 3 * 18) = 2.80, which is consistent: half the time you pick green, with EV of 5.60.

I think you left out the probability that you'll get green and take the bet in each of your 0.5 probabilities for the conditional strategy. Multiply a 0.9 to the first term and 0.1 into the second, and everything gets consistent.

In this comment:

http://lesswrong.com/lw/17d/forcing_anthropics_boltzmann_brains/138u

I put forward my view that the best solution is to just maximize total utility, which correctly handles the forcing anthropics case, and expressed curiosity as to whether it would handle the outlawing anthropics case.

It now seems my solution does correctly handle the outlawing anthropics case, which would seem to be a data point in its favor.

Assume that each agent has his own game (that is one game for each agent). That is there are overall 18 (or 2) games (depending the result of the coin flip.)

Then the first calculation would be correct in every respect, and it makes sense to say yes from a global point of view. (And also with any other reward matrix, the dynamic update would be consistent with the apriori decision all the time)

This shows that the error made by the agent was to implicitely assume that he has his own game.

How about give all of your potential clones a vote, even though you can't communicate?

So, in one case, 18 of you would say "Yes, take the bet!" and 2 would say "No, let me keep my money." In the other case, 18 would say no and two would say yes. In either case, of course, you're one of the ones who would vote yes. OK, that leaves us tied. So why not let everyone's vote be proportional to what they stand to gain/lose? That leaves us with 20 -3 vs. 20 1. Don't take the bet.

(Yes, I realize half the people that just voted above don't exist. We just don't know which half...)

As it's been pointed out, this is not an anthropic problem, however there still is a paradox. I'm may be stating the obvious, but the root of the problem is that you're doing something fishy when you say that the other people will think the same way and that your decision will theirs.

The proper way to make a decision is to have a probability distribution on the code of the other agents (which will include their prior on your code). From this I believe (but can't prove) that you will take the correct course of action.

Newcomb like problem fall in the same category, the trick is that there is always a belief about someone's decision making hidden in the problem.

[EDIT:] Warning: This post was based on a misunderstanding of the OP. Thanks orthonormal for pointing out the the mistake! I leave this post here so that the replies stay in context.

I think that decision matrix of the agent waking up in green room is not complete: it should contain the outcome of losing $50 if the answers are not consistent.

Therefore, it would compute that even if the probability of the coin was flipped to 1 is 90%, it still does not make sense to answer "yes" since two other copies would answer "no" and therefore the penalty for not giving a uniform answer will outweigh the potential) win of $5.60. (Even without the penalty, the agent could infer that there were two dissenting copies of itself in that case and he has no ways to generate all the necessary votes to get the money.)

The error of the agent is not the P=90% estimate, but the implicit assumption that he is the only one to influence the outcome.

EDIT: at first I thought this was equivalent, but then I tried the numbers and realized it's not.

1. I'll flip a coin to choose which roulette wheel to spin. If it comes up heads, I'll spin a wheel that's 90% green and 10% red. If it comes up tails, a wheel that's 10% green and 90% red.
2. I won't show you the wheel or the coin (at this point) but I'll tell you which color came up.
3. If it's green, you can bet on the coinflip: win $3 for heads and lose $13 for tails.

If the color is green, do you take the bet?

EDIT: After playing with the numbers, I think reason it's not equivalent is that in the rooms, there are always some of you who see green. I still think it's possible to create an equivalent situation in real life, without copying people. Maybe if you had a group of people draw lots and all the people who get green vote on whether to bet on which lot they were drawing from.

Perhaps we should look at Dresher's Cartesian Camcorder as a way of reducing consciousness, and thereby eliminate this paradox.

Or, to turn it around, this paradox is a litmus test for theories of consciousness.

The more I think about this, the more I suspect that the problem lies in the distinction between quantum and logical coin-flips.

Suppose this experiment is carried out with a quantum coin-flip. Then, under many-worlds, both outcomes are realized in different branches. There are 40 future selves--2 red and 18 green in one world, 18 red and 2 green in the other world--and your duty is clear:

(50% ((18 +$1) + (2 -$3))) + (50% ((18 -$3) + (2 +$1))) = -$20.

Don't take the bet.

So why Eliezer's insistence on using a logical coin-flip? Because, I suspect, it prevents many-worlds from being relevant. Logical coin-flips don't create possible worlds the way quantum coin-flips do.

But what is a logical coin-flip, anyway?

Using the example given at the top of this post, an agent that was not only rational but clever would sit down and calculate the 256th binary digit of pi before answering. Picking a more difficult logical coin-flip just makes the calculation more difficult; a more intelligent agent could solve it, even if you can't.

So there are two different kinds of logical coin-flips: the sort that are indistinguishable from quantum coin-flips even in principle, in which case they ought to cause the same sort of branching events under many-worlds--and the sort that are solvable, but only by someone smarter than you.

If you're not smart enough to solve the logical coin-flip, you may as well treat it as a quantum coin-flip, because it's already been established that you can't possibly do better. That doesn't mean your decision algorithm is flawed; just that if you were more powerful, it would be more powerful too.

Is there any version of this post that doesn't involve technologies that we don't have? If not, then might the resolution to this paradox be that the copying technology assumed to exist can't exist because if it did it would give rise to a logical inconsistency.

Edit: presumably there's an answer already discussed that I'm not aware of, probably common to all games where Omega creates N copies of you. (Since so many of them have been discussed here.) Can someone please point me to it?

I'm having difficulties ignoring the inherent value of having N copies of you created. The scenario assumes that the copies go on existing after the game, and that they each have the same amount of utilons as the original (instead of a division of some kind).

For suppose the copies are short lived: Omega destroys them after the game. (Human-like agents will be deterred by the negative utility of having N-1 copies created just to experience death.) Then every copy effectively decides for itself, because its siblings won't get to keep their utilons for long, and the strategy "play game iff room is green" is valid.

Now suppose the copies are long lived. It's very likely that creating them has significant utility value.

For goals on which the N copies can cooperate (e.g. building paperclips or acquiring knowledge), the total resources available (and so utility) will have increased, often linearly (N times), sometimes a lot more. An AI might decide to pool all resources / computing power and destroy N-1 copies immediately after the game is played.

For goals on which the copies compete (e.g. property and identity), utility will be much reduced by increased competition.

In the absence of any common or contested goals, all copies will probably profit from trade and specialization.

The utility outcome of having N copies created probably far outweighs the game stakes, and certainly can't be ignored.