Solve Psy-Kosh's non-anthropic problem

After being told whether they are deciders or not, 9 people will correctly infer the outcome of the coin flip, and 1 person will have been misled and will guess incorrectly. So far so good. The problem is that there is a 50% chance that the one person who is wrong is going to be put in charge of the decision. So even though I have a 90% chance of guessing the state of the coin, the structure of the game prevents me from ever having more than a 50% chance of the better payoff.

eta: Since I know my attempt to choose the better payoff will be thwarted 50% of the time, the statement "saying 'yea' gives 0.9*1000 + 0.1*100 = 910 expected donation" isn't true.

It is an anthropic problem. Agents who don't get to make decisions by definition don't really exist in the ontology of decision theory. As a decision theoretic agent being told you are not the decider is equivalent to dying.

I tell you that you're a decider [... so] the coin is 90% likely to have come up tails.

Yes, but

So saying "yea" gives 0.9 1000 + 0.1 100 = 910 expected donation.

... is wrong: you only get 1000 if everybody else chose "yea". The calculation of expected utility when tails come up has to be more complex than that.

Let's take a detour through a simpler coordination problem: There are 10 deciders with no means of communication. I announce I will give $1000 to a Good and Worth Charity for each decider that chose "yea", except if they all chose yea, in which case I will give nothing. Here the optimal strategy is to choose "yea" with a certain probability p, which I don't have time to calculate right now.

But anyway, in coordination problems in general, the strategy is to find what the group strategy should be, and execute that. Calculation of expected utility should not be done on the individual level (because there's no reliable way to account for the decisions of others if they depend on yours), but on the group level.

I can't formalize my response, so here's an intuition dump:

It seemed to me that a crucial aspect of the 1/3 solution to the sleeping beauty problem was that for a given credence, any payoffs based on hypothetical decisions involving said credence scaled linearly with the number of instances making the decision. In terms of utility, the "correct" probability for sleeping beauty would be 1/3 if her decisions were rewarded independently, 1/2 if her (presumably deterministic) decisions were rewarded in aggregate.

The 1/2 situation is mirrored here: There are only two potential decisions (ruling out inconsistent responses), each applicable with probability 0.5, each resulting in a single payoff. Since "your" decision is constrained to match the others', "you" are effectively the entire group. Therefore, the fact that "you" are a decider is not informative.

Or from another angle: Updating on the fact that you (the individual) are a decider makes the implicit assumption that your decision process is meaningfully distinct from the others', but this assumption violates the constraints of the problem.

I remain thoroughly confused by updating. It seems to assume some kind of absolute independence of subsequent decisions based on the update in question.

The first is correct. If you expect all 10 participants to act the same you should not distinguish between the cases when you yourself are the sole decider and when one of the others is the sole decider. Your being you should have no special relevance. Since you are a pre-existing human with a defined identity this is highly counterintuitive, but this problem really is no different from this one: An urn with 10 balls in different colors, someone tosses a coin and draws 1 ball if it comes up head and 9 balls if it comes up tails, and in either case calls out the color of one ball. Suppose that color is red, what is the probability the coin came up tails?

If the coin came up heads there is a 1/10 chance of you being drawn, if it came up tails there is a 9/10 chance of you being drawn, and a 1/9 chance of you being you. Since in your intuition you are always you this probably seems nonsensical. But if you distinguish between yourself and the others a priori you can't really treat the payoff like that.

I claim that the first is correct.

Reasoning: the Bayesian update is correct, but the computation of expected benefit is incomplete. Among all universes, deciders are "group" deciders nine times as often as they are "individual" deciders. Thus, while being a decider indicates you are more likely to be in a tails-universe, the decision of a group decider is 1/9th as important as the decision of an individual decider.

That is to say, your update should shift probability weight toward you being a group decider, but you should recognize that changing your mind is a mildly good idea 9/10 of the time and a very bad idea 1/10 of the time, and that these should balance to you NOT changing your mind. Since we know that half the time the decision is made by an individual, their decision to not change their mind must be as important as all the decisions of the collective the other half the time.

for "yea": 0.5*(0.9*1000/9+0.1*0)+0.5*(0.9*0+0.1*100/1)=55  
for "nay": 0.5*(0.9*700/9+0.1*0)+0.5*(0.9*0+0.1*700/1)=70

for "yea": 0.9*1000/9+0.1*100/1=110
for "nay": 0.9*700/9+0.1*700/1=140

Your second option still implicitly assumes that you're the only decider. In fact each of the possible deciders in each branch of the simulation would be making an evaluation of expected payoff -- and there are nine times as many in the "tails" branches.

There are twenty branches of the simulation, ten with nine deciders and ten with one decider. In the one-decider branches, the result of saying "yea" is a guaranteed $100; in the nine-decider branches, it's $1000 in the single case where everyone agrees, $700 in the single case where everyone disagrees, and $0 otherwise. The chance of defection depends on the strength of everyone's precommitments, and the problem quickly goes recursive if you start taking expected behavior into account, but in the simplified case where everyone has a well-defined probability of defection the expected payoffs are much higher if your precommitments are strong.

If we're assuming that all of the deciders are perfectly correlated, or (equivalently?) that for any good argument for whatever decision you end up making, all the other deciders will think of the same argument, then I'm just going to pretend we're talking about copies of the same person, which, as I've argued, seems to require the same kind of reasoning anyway, and makes it a little bit simpler to talk about than if we have to speak as though that everyone is a different person but will reliably make the same decision.

Anyway:

Something is being double-counted here. Or there's some kind of sleight-of-hand that vaguely reminds me of this problem, where it appears that something is being misplaced but you're actually just being misdirected by the phrasing of the problem. (Not that I accuse anyone of intentionally doing that in any of the versions of this problem.) I can't quite pin it down, but it seems like whatever it is that would (under any circumstance) lead you to assign a .9 decision-theoretic weighting to the tails-world is already accounted for by the fact that there are 9 of you (i.e. 9 who've been told that they're deciders) in that world. I'm not sure how to formally express what I'm getting at, but I think this is moving in the right direction. Imagine a tree of agents existing as a result of the coin flip; the heads branch contains one decider and nine non-deciders; the tails branch contains nine deciders and one non-decider. And each decider needs to have its own judgment of decision-theoretic weighting... but that varies depending on what kind of decision it is. If each one assigns .9 weight to the possibility that it is in the tails branch, then that would be relevant if every agent's decision were to be counted individually (say, if each one had to guess either heads or tails, and would get $1 if correct; they'd do better guessing tails than by flipping a coin to decide), but in this case the decision is collective and only counted once — so there's no reason to count the multiple copies of you as being relevant to the decision in the first place. It's like if in tails-world you run a (constant) program nine times and do something based on the output, and in heads-world you run it once and do something else based on the output. The structure of the problem doesn't actually imply that the algorithm needs to know how many times it's being executed. I think that's the misdirection.

(Edit: Sorry, that was sort of a ramble/stream-of-consciousness. The part from "If each one assigns..." onward is the part I currently consider relevant and correct.)

(No points for saying that UDT or reflective consistency forces the first solution. If that's your answer, you must also find the error in the second one.)

Under the same rules, does it make sense to ask what is the error in refusing to pay in a Counterfactual Mugging? It seems like you are asking for an error in applying a decision theory, when really the decision theory fails on the problem.

you should do a Bayesian update: the coin is 90% likely to have come up tails. So saying "yea" gives 0.9*1000 + 0.1*100 = 910 expected donation.

I'm not sure if this is relevant to the overall nature of the problem, but in this instance, the term 0.9*1000 is incorrect because you don't know if every other decider is going to be reasoning the same way. If you decide on "yea" on that basis, and the coin came up tails, and one of the other deciders says "nay", then the donation is $0.

Is it possible to insert the assumption that the deciders will always reason identically (and, thus, that their decisions will be perfectly correlated) without essentially turning it back into an anthropic problem?

I would decide "nay". Very crudely speaking a fundamental change in viewpoints is involved. If I update on the new information regarding heads vs tales I must also adjust my view of what I care about. It is hard to describe in detail without writing an essay describing what probability means (which is something Wei has done and I would have to extend to allow for the way to describe the decision correctly if updating is, in fact, allowed.).

So let’s modify the problem somewhat. Instead of each person being given the “decider” or “non-decider” hat, we give the "deciders" rocks. You (an outside observer) make the decision.

Version 1: You get to open a door and see whether the person behind the door has a rock or not. Winning strategy: After you open a door (say, door A) make a decision. If A has a rock then say “yes”. Expected payoff 0.9 1000 + 0.1 100 = 910 > 700. If A has no rock, say “no”. Expected payoff: 700 > 0.9 100 + 0.1 1000 = 190.

Version 2: The host (we’ll call him Monty) randomly picks a door with a rock behind it. Winning strategy: Monty has provided no additional information by picking a door: We knew that there was a door with a rock behind it. Even if we predicted door A in advance and Monty verified that A had a rock behind it, it is no more likely that heads was chosen: The probability of Monty picking door A given heads is 0.9 1/9 = 0.1 whereas the probability given tails is 0.1 1. Hence, say “no”. Expected Payoff: 700 > 0.5 100 + 0.5 1000 = 550.

Now let us modify version 2: During your sleep, you are wheeled into a room containing a rock. That room has a label inside, identifying which door it is behind. Clearly, this is no different than version 2 and the original strategy still stands.

From there it’s a small (logical) jump to your consciousness being put into one of the rock-holding bodies behind the door, which is equivalent to our original case. (Modulo the bit about multiple people making decisions, if we want we can clone you consciousness if necessary and put it into all rock possessing bodies. In either case, the fact that you wind up next to a rock provides no additional information.)

This question is actually unnecessarily complex. To make this easier, we could introduce the following game: We flip a coin where the probability of heads is one in a million. If heads, we give everyone on Earth a rock, if tails we give one person a rock. If the rock holder(s) guesses how the coin landed, Earth wins, otherwise Earth loses. A priori, we very much want everyone to guess tails. A person holding the rock would be very much inclined to say heads, but he’d be wrong. He fails to realize that he is in an equivalence class with everyone else on the planet, and the fact that the person holding the rock is himself carries no information content for the game. (Now, if we could break the equivalence class before the game was played by giving full authority to a specific individual A, and having him say “heads” iff he gets a rock, then we would decrease our chance of losing from 10^-6 to (1-10^-6) * 10^-9.)

The devil is in the assumption that everyone else will do the same thing as you for presumably the same reasons. "Nay" is basically a better strategy in general, though it's not always right. The 90% odds of tails are correctly calculated.

• If you are highly confident that everyone else will say "yea", it is indeed correct to say "yea"
• If you're highly confident that everyone will say the same thing but you're not sure what it is (an unlikely but interesting case), then make a guess (if it's 50-50 then "yea" is better - the cutoff in fact is just a bit above 45% chance of all-yea over all-nay)
• If you don't really know what the others will say, and they're all acting independently, then odds are they've blown it by giving different answers whatever you do. In that case, 1-decider case dominates your decision, for essentially different reasons (so "nay" is better).
• The most difficult case is the more "natural" assumption that the other people are basically idential copies of you. In this case, what you have is a stochastic version of Newcomb's Paradox. Actually it's more like Newcomb's Mugging. Once you know you are a decider, saying "yea" will genuinely give you a better average payout. But, when you're not a decider, you'll see more money donated if you are the sort of person who would say "nay" when you are a decider. It's obvious why that's true if you make the "identical copies" assumption explicitly, because your identical copy will say "nay" and cause a larger donation. So, you expect to make more money saying "yea" once you know you're a decider, but you make more money overall if you're a "nay"-sayer. The trick is that most of the extra money is made in the cases where you're not a decider.

So, nay is definately the better precommitment, and TDT would presumably tell you to say nay. But the argument that, as an individual, your expected returns are better saying "yea" once you know you are a decider is still perfectly valid.

Each decider will be asked to say "yea" or "nay". If the coin came up tails and all nine deciders say "yea", I donate $1000 to VillageReach. If the coin came up heads and the sole decider says "yea", I donate only $100. If all deciders say "nay", I donate $700 regardless of the result of the coin toss. If the deciders disagree, I don't donate anything.

Suppose that instead of donating directly (presumably for tax reasons), you instead divide the contribution up among the deciders, and then let them pass it on. As I figure it, that simple change in the rules eliminates the paradox.

I'm not sure what this means, but I'm going to blame the paradox on the tax-code. :)

Un-thought-out-idea: We've seen in the Dutch-book thread that probability and decision theory are inter-reliant so maybe

Classical Bayes is to Naive Decision Theory

as

Bayes-we-need-to-do-anthropics is to UDT

Actually now that I've said that it doesn't seem to ground-breaking. Meh.

This gives you new information you didn't know before - no anthropic funny business, just your regular kind of information - so you should do a Bayesian update: the coin is 90% likely to have come up tails.

Why 90% here? The coin is still fair, and anthropic reasoning should still remain, since you have to take into account the probability of receiving the observation when updating on it. Otherwise you become vulnerable to filtered evidence.

Edit: I take back the sentence on filtered evidence.

Edit 2: So it looks like the 90% probability estimate is actually correct, and the error is in estimating (acausal) consequences of possible decisions.

The error in the reasoning is that it is not you who makes the decision, but the COD (collective of the deciders), which might be composed of different individuals in each round and might be one or nine depending on the coin toss.

In every round the COD will get told that they are deciders but they don't get any new information because this was already known beforehand.

P(Tails| you are told that you are a decider) = 0.9

P(Tails| COD is told that COD is the decider) = P(Tails) = 0.5

To make it easier to understand why the "yes" strategy is wrong, if you say yes every time, you will only be wrong on average once every 9 turns, the one time where the coin comes up head and you are the sole decider. This sounds like a good strategy until you realize that every time the coin comes up head another one(on average) will be the sole decider and make the wrong choice by saying yes. So the COD will end up with 0.5*1000 + 0.5*100 = 550 expected donation.

I'm retracting this one in favor of my other answer:

http://lesswrong.com/lw/3dy/solve_psykoshs_nonanthropic_problem/d9r4

So saying "yea" gives 0.9 1000 + 0.1 100 = 910 expected donation.

This is simply wrong.

If you are a decider then the coin is 90% likely to have come up tails. Correct.

But it simply doesn't follow from this that the expected donation if you say yes is 0.9*1000 + 0.1*100 = 910.

To the contrary, the original formula is still true: 0.5*1000 + 0.5*100 = 550

So you should stil say "nay" and of course hope that everyone else is as smart as you.

If we are considering it from an individual perspective, then we need to hold the other individuals fixed, that is, we assume everyone else sticks to the tails plan:

In this case: 10% chance of becoming a decider with heads and causing a $1000 donation 90% chance of becoming a decider with tails and causing a $100 donation

That is 0.1 1000 + 0.9 100 = $190, which is a pretty bad deal.

If we want to allow everyone to switch, the difficulty is that the other people haven't chosen their action yet (or even a set of actions with fixed probability), so we can't really calculate expected values.

One way to approach this is to imagine that a random decider will be given a button and if they press it, everyone will change their view going with heads. The problem is that when heads comes up, the single decider will have 9 times the probability of getting the button than any of the individual deciders when tails comes up. So we get:

50% chance of causing a $1000 donation and 50% of causing a $100 donation. This is very similar to what Saturn was saying. However, there are subtle, but important differences.

Let's suppose now that each decider gets a button and that they all press it or don't press it due to being identical. If one presses a button and there is one decider, then they get the full effect. If there are 9 deciders, then the easiest way to model this situation will be to give each button one 1/9th of the effect. If we don't do this, then we are clearly counting the benefits of everyone switching 9 times.

Precommitting to "Yea" is the correct decision.

The error: the expected donation for an individual agent deciding to precommit to "nay" is not 700 dollars. It's pr(selected as decider) * 700 dollars. Which is 350 dollars.

Why is this the case? Right here:

Next, I will tell everyone their status, without telling the status of others ... Each decider will be asked to say "yea" or "nay".

In all the worlds where you get told you are not a decider (50% of them - equal probability of 9:1 chance or a 1:9 chance) your precommitment is irrelevant. Therefore, the case where everyone precommits to yea is logically equivalent to the case where everyone precommits to nay and then change to yea upon being told they are a decider.

So we can talk about two kinds of precommitments: "I precommit to answering yea/nay" and "I precommit to answer yea/nay given that I am informed I am a decider". The expected donation in the first precomittment is yea: 455 dollars and nay: 350 dollars; the expected donation in the second precomittment is yea: 910 dollars and nay: 700 dollars.¹

Yes, the first precommittment is half that of the second: because the first goes through a 50% filter, the second starts on the other side of that filter. Of course if you start yea before the filter and nay afterwards, you're going to get the wrong result.

¹: Summing the expected values of yea. pr(heads) pr(selected as decider) value of 'yea' for heads = 0.5 0.1 100 = 5 dollars, pr(tails) pr(selected as decider) value of 'yea' for tails = 0.5 0.9 1000 = 450 dollars. Sum = 455 dollars.

Below is very unpolished chain of thoughts, which is based on vague analogy with symmetrical state of two indistinguishable quantum particles.

When participant is said ze is decider, ze can reason: let's suppose that before coin was flipped I changed places with someone else, will it make difference? If coin came up heads, than I'm sole decider and there are 9 swaps which make difference in my observations. If coin came up tails then there's one swap that makes difference. But if it doesn't make difference it is effectively one world, so there's 20 worlds I can distinguish, 10 correspond to my observations, 9 have probability (measure?) 0.5 0.1 (heads, I'm decider), 1 have probability 0.5 0.9 (tails, I'm decider). Consider following sentence as edited out. What I designated as P(heads) is actually total measure (?) of worlds participant is in. All this worlds are mutually exclusive, thus P(heads)=9 0.5 0.1+1 0.5 0.9=0.9.

What is average benefit of "yea"? (9 0.5 0.1 $100 + 1 0.5 0.9 $1000)=$495

Same for "nay": (9 0.5 0.1 $700+1 0.5 0.9 $700)=$630

Once I have been told I am a decider, the expected payouts are:

For saying Yea: $10 + P$900 For saying Nay: $70 + Q$700

P is the probability that the other 8 deciders if they exist all say Yea conditioned on my saying Yay, Q is the probability that the other 8 deciders if they exist all say Nay conditioned on my saying Nay.

For Yay to be the right answer, to maximize money for African Kids, we manipulate the inequality to find P > 89% - 78%*Q The lowest value for P consistent with 0<=Q<=1 is P > 11% which occurs when Q = 100%.

What are P & Q? All we are told about the other deciders if they exist is that they are "people who care about saving African kids." With only that information, I can't see how to rationally use anything other than ~0.5% = P = Q, that is, our prior probability that any other decider says yay or nay is 50%, so getting all 8 of them to give the same answer of Yay or Nay has prior probability of (~(1/2)^8). Any other assumption of a different prior given the statement of the problem seems irrational to me.

So with P & Q on the small side, the correct decision is to pick "Nay." The error in the original post analysis of the case for picking Yay was assuming that the value of P is 1, that if I picked Yay that all people in the class of "people who care about African Kids" would also pick Yay. As stated above, from the information we have the rational prior estimate of that probability P is about 0.5%.

I would go further and say what about updating probabilities based on what we see in the answers here? If anything, they decrease the probability of P (everybody picks Yay), as more people seem to favor Nay than Yay. But this little bit of peeking below the fold is not needed for the estimate.

Is this some kind of inverse Monty Hall problem? Counterintuitively, the second solution is incorrect.

If everyone pledges to answer "yea" in case they end up as deciders, you get 0.5*1000 + 0.5*100 = 550 expected donation.

This is correct. There are 10 cases in which we have a single decider and win 100 and there are 10 cases in which we have a single non-decider and win 1000, and these 20 cases are all equally likely.

you should do a Bayesian update: the coin is 90% likely to have come up tails.

By calculation (or by drawing a decision tree), this is indeed correct: prob(T|D1)=0.9

• T coin came up tails
• D1 player 1 is a decider

So saying "yea" gives 0.9*1000 + 0.1*100 = 910 expected donation.

Let us assume that every decider follows your reasoning and says "yea" (which is, I suppose, what you intended). Let me first give a non-explanation for why the reasoning above cannot be correct: Notice that every decider will always say "yea" and this is no different from the deterministic strategy of always saying "yea". It's irrelevant what non-deciders say or how the deciders came to their decision. Thus the overall expected donation is again 0.5*1000 + 0.5*100 = 550.

Above you correctly calculated the conditional expectation E( donation | D1) = 910. Conditioned on D1, we expect to win 910 if everyone says "yea". This number is not directly comparable with the 700, the payoff we'd get for saying "nay" (regardless of whether D1 holds or not). Assume we would play a variant of the game in which the players were promised that D1 always holds; then it'd indeed be better for them to always say "yea". The conditional expectation, however, does not tell us how well this strategy performs in the original game. The original game allows, after all, that D1 does not hold. To find out how well a strategy performs we must take into account that possibility.

Consider the expected donation conditioned on the fact that D1 does not hold. It turns out that E( donation | not D1) = 0.9*100 + 0.1*1000 = 190 is much smaller than 700. Thus, if all players follow the "yea" strategy, we'll be worse off since D1 might happen to be false. This can be made precise: The probability that D1 holds is exactly 0.5. Thus, using "yea" for every player, our overall performance will be E( donation ) = 0.5* E( donation | D1) + 0.5* E( donation | not D1) = 0.5*910 + 0.5*190 = 550 as before.

EDIT: Nevermind, I massively overlooked a link.

A note on the problem statement: You should probably make it more explicit that mixed strategies are a bad idea here (due to the 0-if-disagreement clause). I spent a bit wondering why you restricted to pure strategies and didn't notice it until I actually attempted to calculate an optimal strategy under both assumptions. (FWIW, graphing things, it seems that indeed, if you assume both scenarios are equally likely, pure "nay" is best, and if you assume 9 deciders is 90% likely, pure "yea" is best.)

I'm assuming all deciders are coming to the same best decision so no worries about deciders disagreeing if you change your mind.

I'm going to be the odd-one-out here and say that both answers are correct at the time they are made… if you care far more (which I don't think you should) about African kids in your own Everett branch (or live in a hypothetical crazy universe where many worlds is false).

(Chapter 1 of Permutation City spoiler, please click here first if not read it yet, you'll be glad you did...): Jura lbh punatr lbhe zvaq nsgre orvat gbyq, lbh jvyy or yvxr Cnhy Qheunz, qvfnoyvat gur cnenpuhgr nsgre gur pbcl jnf znqr.

If you care about African kids in other branches equally, then the first decision is always correct, because although the second choice would make it more likely that kids in your branch will be better off, it will cost the kids in other branches more.

We could say that the existence of pre-agreed joint strategies invalidates standard decision theory.

It's easy to come up with scenarios where coordination is so valuable that you have to choose not to act on privileged information. For example, you're meeting a friend at a pizzeria, and you spot a better-looking pizzeria two blocks away, but you go to the worse one because you'd rather eat together than apart.

Psy-Kosh's problem may not seem like a coordination game, but possibilities for coordination can be subtle and counter-intuitive. See, for example, these two beautiful logic puzzles.

Stream of conciousness style answer. Not looking at other comments so I can see afterwards if my thinking is the same as anyone else's.

The argument for saying yea once one is in the room seems to assume that everyone else will make the same decision as me, whatever my decision is. I'm still unsure whether this kind of thinking is allowed in general, but in this case it seems to be the source of the problem.

If we take the opposite assumption, that the other decisions are fixed, then the problem depends on those decisions. If we assume that all the others (if there are any) will say yea then U(yea) = 0.9$1000 + 0.1$100 = $910 = 0.91 lives while U(nay) = 0.90 + 0.1$700 = $70 = 0.07 lives, so clearly I say yea. If we assume that all the others (if there are any) will say nay then U(yea) = 0.90 + 0.1$100 = 0.01 lives while U(nay) = 1*$700 = 0.7 lives.

In other words, if we expect others to say yea, then we say yea, and if we expect others t say nay, then we say nay (if people say yea with some probability p then our response is either yea or nay depending on whether p is above some threshold). It appears we have a simple game theory problem with two nash equilibria. I'm not sure of a rule for deciding which equilibrium to pick, so I'll try some toy problems.

If we have 10 people, each in a room with two buttons, one red and one blue, and they are told that if they all press the same colour then $1000 will be donated to Village Reach, but if there is any disagreement no money will be donated, then they have quite a difficult dilemma, two nash equilibria, but no way to single out one of them, which makes them unlikely to end up in either.

If we change the problem, so that now only $500 is donated when they all press blue, but $1000 is still given on red, then the decision becomes easy. I'm sure everyone will agree with me that you should definitely press red in this dilemma. It seems like in general a good rule for game-theory problems is "if faced with multiple nash equilibria, pick the one you would have agreed to in advance". This might not give an answer in games where the various utility functions are opposed to each other, but it works fine in this problem.

So I say nay. Both before and after hearing that I am a decider. Problem solved.

Thinking about this a bit more, it seems like the problem came from Timeless Decision Theory, and was solved by Causal Decision Theory. A rather disturbing state of events.

I'm torn whether I should move this post to LW proper. What do people think?

Initially either 9 or 1 of the 10 people will have been chosen with equal likelihood, meaning I had a 50% chance of being chosen. If being chosen means I should find 90% likelihood that the coin came up tails, then not being chosen should mean I find 90% likelihood that the coin came up heads (which it does). If that were the case, I'd want nay to be what the others choose (0.9 100 + 0.1 1000 = 190 < 700). Since both branches are equally likely (initially), and my decision of what to choose in the branch in which I choose corresponds (presumably) to the others' decision of what to choose in the branch in which I don't choose, I should therefore say nay. You can't win with mere straightforward optimization when your action in 1 branch affects other (no longer reachable) branches.

This struck an emotional nerve with me, so I'm going to answer as if this were an actual real-life situation, rather than an interesting hypothetical math problem about maximizing expected utility.

IMHO, if this was a situation that occurred in real life, neither of the solutions is correct. This is basically another version of Sophie's Choice. The correct solution would be to punch you in the face for using the lives of children as pawns in your sick game and trying to shift the feelings of guilt onto me, and staying silent. Give the money or not as you see fit, but don't try to saddle me with guilt over lives that YOU chose not to save.

The correct option is refusal to play combined with retaliation for blackmail. If - as with the Nazi officer - this is impossible and one is stuck with choosing from the two options you gave THEN calculation can enter into it. However, future vengeance opportunities should be explored.

It may be better to phrase such problems in terms of game-shows or other less emotionally charged situations. Given a twisted enough game, people will look for ways to pull the rope sideways.