Anthropical Paradoxes are Paradoxes of Probability Theory

post by Ape in the coat · 2023-12-06T08:16:26.846Z · LW · GW · 18 comments

Contents

  Introduction
  Updateless Dilemma and Psy-Kosh's non-anthropic problem
  Solving the Problem
  The Source of a Paradox
None
18 comments

This is the fourth post in my series on Anthropics. The previous one is Anthropical probabilities are fully explained by difference in possible outcomes [LW · GW]. The next one is Another Non-Anthropic Paradox: The Unsurprising Rareness of Rare Events [LW · GW].

Introduction

If there is nothing special about anthropics, if it’s just about correctly applying standard probability theory, why do we keep encountering anthropical paradoxes instead of general probability theory paradoxes? Part of the answer is that people tend to be worse at applying probability theory in some cases than in the others. 

But most importantly, the whole premise is wrong. We do encounter paradoxes of probability theory all the time. We are just not paying enough attention to them, and occasionally attribute them to anthropics.

Updateless Dilemma and Psy-Kosh's non-anthropic problem

As an example, let’s investigate Updateless Dilemma [LW · GW], introduced by Eliezer Yudkowsky in 2009.

Let us start with a (non-quantum) logical coinflip - say, look at the heretofore-unknown-to-us-personally 256th binary digit of pi, where the choice of binary digit is itself intended not to be random.

If the result of this logical coinflip is 1 (aka "heads"), we'll create 18 of you in green rooms and 2 of you in red rooms, and if the result is "tails" (0), we'll create 2 of you in green rooms and 18 of you in red rooms.

After going to sleep at the start of the experiment, you wake up in a green room.

With what degree of credence do you believe - what is your posterior probability - that the logical coin came up "heads"?

Eliezer (2009) argues, that updating on the anthropic evidence and thus answering 90% in this situation leads to a dynamic inconsistency, thus anthropical updates should be illegal. 

I inform you that, after I look at the unknown binary digit of pi, I will ask all the copies of you in green rooms whether to pay $1 to every version of you in a green room and steal $3 from every version of you in a red room.  If they all reply "Yes", I will do so.

Suppose that you wake up in a green room.  You reason, "With 90% probability, there are 18 of me in green rooms and 2 of me in red rooms; with 10% probability, there are 2 of me in green rooms and 18 of me in red rooms.  Since I'm altruistic enough to at least care about my xerox-siblings, I calculate the expected utility of replying 'Yes' as (90% * ((18 * +$1) + (2 * -$3))) + (10% * ((18 * -$3) + (2 * +$1))) = +$5.60."  You reply yes.

However, before the experiment, you calculate the general utility of the conditional strategy "Reply 'Yes' to the question if you wake up in a green room" as (50% * ((18 * +$1) + (2 * -$3))) + (50% * ((18 * -$3) + (2 * +$1))) = -$20.  You want your future selves to reply 'No' under these conditions.

This is a dynamic inconsistency - different answers at different times - which argues that decision systems which update on anthropic evidence will self-modify not to update probabilities on anthropic evidence.

However, in the comments Psy-Kosh notices [LW(p) · GW(p)] that this situation doesn’t have anything to do with anthropics at all. The problem can be reformulated as picking marbles from two buckets with the same betting rule. The dynamic inconsistency doesn’t go anywhere, and if previously it was a sufficient reason not to update on anthropic evidence, now it becomes a sufficient reason against the general case of Bayesian updating in the presence of logical uncertainty.

Solving the Problem

Let’s solve these problems. Or rather this problem – as they are fully isomorphic and have the same answer.

For simplicity, as a first step, let's ignore the betting rule and dynamic inconsistency and just address it in terms of the Law of Conservation of Expected Evidence [LW · GW]. Do I get new evidence while waking up in a green room or picking a green marble? Of course! After all:

$N\left(ISeeGreen\right)<N\left(All\right)$

In which case, a Bayesian update is in order:

$P\left(Heads|ISeeGreen\right)=P\left(ISeeGreen|Heads\right)P\left(Heads\right)/P\left(ISeeGreen\right)=0.9$

So, 90% is the answer to the question what is my posterior probability - that the logical coin came up "heads". But what about the dynamic inconsistency then? Obviously, it shouldn’t happen. But as our previous calculations are correct, the mistake that leads to it must be somewhere else.

Let’s look at the betting rule more attentively. How does it depend on P(Heads|I See Green)?

Well, actually it doesn’t! Whether I see green or not, the decision will be made by people who see green and there will always be such people. My posterior probability for coin being heads is irrelevant. What is relevant is the probability that the coin is Heads conditionally on the fact that any person sees Green.

$P\left(Heads|AnySeesGreen\right)=P\left(AnySeesGreen|Heads\right)P\left(Heads\right)/P\left(AnySeesGreen\right)$

$P\left(Heads|AnyoneSeesGreen\right)=(1\ast ½)/1=1/2$

Another way to look at it is that what matters is the posterior probability of a Decider, who is by definition of the betting scheme, is a person who always sees Green:

$N\left(DeciderSeesGreen\right)=N\left(All\right)$

$P\left(Heads|DeciderSeesGreen\right)=P\left(Heads\right)=1/2$

And thus, updating on seeing green for a Decider will contradict the Law of Conservation of Expected Evidence.

Wait a second! But if I see Green then I’m a Decider. How can my posterior probability for Heads be different from a Decider’s posterior probability for Heads if we are the same person? 

The same way a person who visited a randomly sampled room can have different probability estimate than a person who visited a predetermined room. They are not of the same distribution. Their difference in possible outcomes explains the difference in their probability estimates [LW · GW], even when in a specific iteration of the experiment they meet in the same room.

Likewise, even if in this particular case "I" and a "Decider" happened to refer to the same person, it’s not always true. We can reduce "I" to "A person who may see either Green or Red" and "Decider" to “A person who always sees Green”. They do have an intersection. But fundamentally these two are different entities.

To demonstrate that their posterior probabilities should be different, let’s add a second betting rule:

Every person in the experiment is proposed to guess whether the coin has landed Heads or Tails. If they guessed correctly, they personally get 10 dollars, otherwise they personally lose 10 dollars.

What happens when I see Green now? As a person, who might not have seen Green, I update my probability estimate for Heads to 90% and take the personal bet. I also notice that I’m a Decider in this instance of the experiment. And as a Decider is a person who always sees Green, no matter what, I keep Decider’s probability estimate at 50% and say "No" to the collective bet. This way I get maximum expected utility from the experiment. Any attempt to have the same probability estimate for both bets will be inferior.

The Source of a Paradox

Now that we’ve solved the problem, let’s try to understand where it comes from and why we tend to notice such issues in anthropical problems and not simple probability theory problems, which are completely isomorphic to them.

First of all, it’s a general case of applying a mathematical model that doesn’t fit the setting. What we needed is a model describing the probability of any person seeing green, but instead we used the model for one specific person. But why did we do it?

To a huge degree it’s a map/territory confusion about what "I" refers to. On the level of our regular day to day perception "I" seems to be a simple, indivisible concept. But this intuition isn't applicable to probability theory. Math doesn't particularly care about your identity. It just deals with probability spaces and elementary outcomes, preserving the truth values related to them. So everything has to be defined in these terms. 

Another potential source of the problem is when we transition from probability to decision theory, with the introduction of betting schemes and scoring rules. This may be especially problematic when people make attempts to justify probability estimates via betting odds. 

The addition of betting is always an extra complication, and thus a source of confusion. Different ways to set a betting scheme lead to different probabilities being relevant, and it requires extra care to track which is relevant and which is not. It's easier to just talk about probabilities on their own.

And so it's not surprising that such paradoxes are more noticeable in anthropics. After all, it specifically focuses on this confusing "I" thing a lot, with different additional complications, such as betting schemes. But if we pay attention and are careful about what is meant by "I" and which probabilities are relevant for which betting schemes, if we just keep following the Law of Conservation of Expected Evidence, the paradox resolves. Maybe in a counterintuitive way. But that's just a reason to re-calibrate our intuitions.

The next post in the series is Another Non-Anthropic Paradox: The Unsurprising Rareness of Rare Events [LW · GW].

18 comments

Comments sorted by top scores.