Reflective consistency, randomized decisions, and the dangers of unrealistic thought experiments

post by Radford Neal · 2023-12-07T03:33:16.149Z · LW · GW · 25 comments

Contents

26 comments

A recent post by Ape in the coat on Anthropical Paradoxes are Paradoxes of Probability Theory [LW · GW] re-examines an old problem presented by Eliezer Yudkowsky in Outlawing Anthropics: An Updateless Dilemma [LW · GW]. Both posts purport to show that standard probability and decision theory give an incorrect result in a fairly simple situation (which it turns out need not involve any "anthropic" issues). In particular, they point to a "reflective inconsistency" when using standard methods, in which agents who agree on an optimal strategy beforehand change their minds and do something different later on. 

Ape in the coat resolves this by abandoning standard probability theory in favour of a scheme in which a single person can simultaneously hold two different probabilities for the same event, to be used when making different decisions. Eliezer's resolution is to abandon standard decision theory in favour of one in which agents act "as if controlling all similar decision processes, including all copies of themselves".  

Here, I will defend something close to standard Bayesian probability and decision theory, with the only extension being the use of randomized decisions, which are standard in game theory, but are traditionally seen as unnecessary in Bayesian decision theory.

I would also like to point out the danger of devising thought experiments that are completely unrealistic, in ways that are crucial to the analysis, as well as the inadvisability of deciding that standard methods are flawed as soon as you happen to come across a situation that is tricky enough that you make a mistake when analysing it.

Here is the problem, in the formulation without irrelevant anthropic aspects:

Twenty people take part in an experiment in which one of two urns is randomly chosen, with equal probabilities, and then each of the 20 people randomly takes a ball from the chosen urn, and keeps it, without showing it to the others. One of the urns contains 18 green balls and 2 red balls. The other urn contains 2 green balls and 18 red balls. 

Each person who has a green ball then decides whether to take part in a bet. If all the holders of green balls decide to take part in the bet, the group of 20 people collectively win $1 for each person who holds a green ball and lose $3 for each person who holds a red ball. The total wins and losses are divided equally among the 20 people at the end. If one or more people with green balls decides not to take the bet, no money is won or lost.

The twenty people can discuss what strategy to follow beforehand, but cannot communicate once the experiment starts, and do not know what choices regarding the bet the others have made.

It seems clear that the optimal strategy is to not take the bet. If all the green ball holders do decide to take the bet, the expected amount of money won is $\left(1/2\right)\times 12+\left(1/2\right)\times (-52)=-20$ dollars, since the 18-green / 2-red and the 2-green / 18-red urns both have probability 1/2 of being chosen, and the former leads to winning $18-2\times 3=12$ dollars and the latter leads to losing $18\times 3-2=52$ dollars. Since the expected reward from taking the bet is negative, and not taking the bet gives zero reward, it is better to not take the bet.

But, after all players agree to not take the bet, what will they think if they happen to pick a green ball?  Will they (maybe all of them) change their mind and decide to take the bet after all? That would be a situation of reflective inconsistency, and also lose them money.

According to standard Bayesian probability theory, holders of green balls should judge the probability of the chosen urn being the one with 18 green balls to be 9/10, with the urn with only 2 green balls having probability 1/10. Before seeing the ball they picked, the two urns were equally likely, but afterwards, the urn with 18 green balls is 9 times more likely than the one with 2 green balls because the probability of the observation "I am holding a green ball" is 9 times higher with the first urn (18/20) than it is with the second (2/20).

So after balls have been picked, people who picked a green ball should compute that the expected return if all holders of green balls decide to take the bet is $\left(9/10\right)\times 12+\left(1/10\right)\times (-52)=5.6$, which is positive, and hence better than the zero return from not taking the bet. So it looks like there is a reflective inconsistency, that will lead them to change their minds, and as we have computed above, they will then lose money on average.

But this is a mistaken analysis. The expected return conditional on all holders of green balls deciding to take the bet is not what is relevant for a single green ball holder making a decision. A green ball holder can make sure the bet is not taken, ensuring an expected return of zero, since the decision to take the bet must be unanimous, but a single person holding a green ball cannot unilaterally ensure that the bet is taken.  The expected return from a single green ball holder deciding to take the bet depends on the probability, $p$, of each one of the other green ball holders deciding to take the bet, and works out (assuming independence) to be $\left(9/10\right)\times 12\times p^{17}+\left(1/10\right)\times (-52)\times p$, which I'll call $E\left(p\right)$.

If all 20 people agreed beforehand that they would not take the bet, it seems reasonable for one of them to think that the probability that one of the others will take the bet is small. If we set $p=0$, then $E\left(p\right)=0$, so the expected return from one person with a green ball deciding to take the bet is zero, the same as the expected return if they decide not to take the bet. So they have no incentive to depart from their prior agreement. There is no reflective inconsistency.

Realistically, however, there must be at least some small probability that a person with a green ball will depart from the prior agreement not to take the bet. Here is a plot of $E\left(p\right)$ versus $p$:

One can see that $E\left(p\right)$ is positive only for values of $p$ quite close to one. So the Bayesian decision of a person with a green ball to not take the bet is robust over a wide range of beliefs about what the other people might do. And, if one actually has good reason to think that the other people are (irrationally) determined to take the bet, it is in fact the correct strategy to decide to take the bet yourself, since then you do actually determine whether or not the bet is taken, and the argument that the 9/10 probability of the urn chosen having 18 green balls makes the bet a good one actually is valid.

So how did Eliezer and Ape in the coat get this wrong? One reason may be that they simply accepted too readily that standard methods are wrong, and therefore didn't keep on thinking until they found the analysis above. Another reason, though, is that they both seem to think of all the people with green balls as being the same person, making a single decision, rather than being individuals who may or may not make the same choices. Recognizing, perhaps, that this is simply not true, Eliezer says 'We can try to help out the causal decision agents on their coordination problem by supplying rules such as "If conflicting answers are delivered, everyone loses $50"'. 

But such coercion is not actually helpful. We can see this easily by simply negating the rewards in this problem: Suppose that if the bet is taken, the twenty people will win $3 for every person with a red ball, and lose $1 for every person with a green ball. Now the expected return if everyone with a green ball takes the bet is $+20$ dollars, not $-20$ dollars. So should everyone agree beforehand to take the bet if they get a green ball?

No, because that is not the optimal strategy. They can do better by randomizing their decisions, agreeing that if they pick a green ball they will take the bet with some probability $p$. The expected reward when following such a strategy is $\left(1/2\right)\times (-12)\times p^{18}+\left(1/2\right)\times 52\times p^2$, which I'll call $R\left(p\right)$. Here is a plot of this function:

The optimal choice of $p$ is not 1, but rather 0.9553, for which the expected reward is 21.09, greater than the value of 20 obtained by the naive strategy of every green ball holder always taking the bet.

Is this strategy reflectively consistent? As before, a person with a green ball will think with probability 9/10 that the urn with 18 green balls was chosen. If they think the other people will take the bet with probability $p$, they will see the expected return from taking the bet themselves to be $\left(9/10\right)\times (-12)\times p^{17}+\left(1/10\right)\times 52\times p$, which I'll call $F\left(p\right)$. It is plotted below:

$F\left(p\right)$ is just the negation of the previously plotted $E\left(p\right)$ function. It is zero at $p=0$and at $p=0.9553$, which is where $R\left(p\right)$ is at its maximum. So if a person with a green ball thinks the other people are following the agreed strategy of taking the bet with probability $p=0.9553$, they will think that it makes no difference (in terms of expected reward) whether they take the bet or not. So there will be no incentive for them to depart from the agreed strategy.

Although this argument shows that Bayesian decision theory is compatible with the optimal strategy worked out beforehand, it is somewhat disappointing that the Bayesian result does not recommend adhering to this strategy. When two actions, A and B, have equal expected reward, Bayesian decision theory says it makes no difference whether one chooses A, or chooses B, or decides to randomly choose either A or B, with any probabilities. So it seems that it would be useful to have some elaboration of Bayesian decision theory for contexts like this where randomization is helpful.

The optimal randomization strategy for this variation of the problem is not compatible with thinking that all the people in this experiment should always act the same way, as if they were exact copies of each other. And of course real people do not always act the same way, even when they have the same information. 

One might imagine "people" who are computer programs, which might be written to be deterministic, and so would perform exactly the same actions when given exactly the same inputs. But a thought experiment of this sort says basically nothing about actual people. Nor does it say much about hypothetical future people who are computer programs, since there is no reason why one would run such a program more than once with exactly the same inputs. An exception would be if such duplication is a form of redundancy to guard against hardware errors, but in such a case it would seem odd to regard the redundant execution paths as separate people. Importing unresolved philosophical questions regarding consciousness and personal identity into what is basically a mundane problem in probability and decision theory seems unprofitable.

25 comments

Comments sorted by top scores.