post by Jsevillamol · 2021-11-22T10:39:59.935Z · LW · GW · 13 comments

## Contents

  Why this is weird
Examples in the wild
Mennen’s ABC example
Interpreting Bayesian Networks
Consequences
In summary
Acknowledgements
None


In short: There is no objective way of summarizing a Bayesian update over an event with three outcomes  as an update over two outcomes .

Suppose there is an event with possible outcomes .
We have prior beliefs about the outcomes .
An expert reports a likelihood factor of .
Our posterior beliefs about  are then .

But suppose we only care about whether  happens.
Our prior beliefs about  are .
Our posterior beliefs are .
This implies that the likelihood factor of the expert regarding  is .

This likelihood factor depends on the ratio of prior beliefs .

Concretely, the lower factor in the update is the weighted mean of the evidence  and  according to the weights  and .

This has a relatively straightforward interpretation. The update is supposed to be the ratio of the likelihoods under each hypothesis. The upper factor in the update is . The lower factor is .

I found this very surprising - the summary of the expert report depends on my prior beliefs!

I claim that this phenomena is unintuitive, and being unaware of this can lead to errors.

## Why this is weird

Bayes' rule describes how to update our prior beliefs using data.

In my mind, one very nice property of Bayes rule was that it cleanly separates the process into a subjective part (eliciting your priors) and an ~objective part (computing the update).

For example, we may disagree on our prior beliefs on whether eg COVID19 originated in a lab. But we cannot disagree on the direction and magnitude of the update caused by learning that it originated in one of the few cities in the world with a gain-of-function lab working on coronaviruses.

Because of this, researchers are encouraged to report their update factors together with their all considered beliefs. This way, users can use their research for their own conclusions by multiplying their prior with the update. And metastudies can just take the product of the likelihoods of all studies to estimate the combined effect of the evidence.

In the above example, we lose this nice property - the update factor depends on the prior beliefs of the user. Researchers would not be able to objectively summarize their likelihood about whether COVID19 originated in a lab accidentally vs zoonotically vs being designed as a bioweapon as a single number for people who only care about whether it originated in a lab versus any other possibility.

## Examples in the wild

I ran into this problem twice recently:

1. When analyzing Mennen’s ABC example [LW · GW] of a case where averaging the logarithmic odds of experts seems to result in nonsense.
2. In my own research on interpreting Bayesian Networks [LW · GW] as I was trying to come up with a way of decomposing a Bayesian update into a combination of several updates.

In both cases being unaware of the phenomena led me to a conceptual mistake.

### Mennen’s ABC example

Mennen’s example involves three experts debating an event with three possible outcomes, .

Expert #1 assigns relative odds of .
Expert #2 assigns relative odds of .
Expert #3 assigns relative odds of .

The logodds-averaging pooled opinion of the experts is  i.e. equal odds, which correspond to a probability of  equal to .

But suppose we only care about .

Expert #1’s implicit odds are .
Expert #2’s implicit odds are .
Expert #3’s implicit odds are .

The pooled odds in this case are , which correspond to a probability of  equal to .

We get different results depending on whether we take the implicit odds after or before pooling expert opinion. What is going on?

Mennen claims that this is a strike against logarithmic pooling. The issue according to him is in the step where we take the opinion of the three experts and aggregate it using average logodds.

I think that this is related to the phenomena I described at the beginning of the article. The problem is with the step where we take the relative odds  and summarize them as .

It’s no wonder that logodd pooling gives inconsistent results when we aggregate outcomes. Bayesian updating is not well defined in that case!

### Interpreting Bayesian Networks

I will not enter into too much detail because my theory of interpretability of Bayesian Networks is very complex. But it suffices to say that I was getting inconsistent results because of this issue.

In essence, I came up with a way of decomposing a Bayesian update into a series of independent steps, corresponding to different subgraphs of a Bayesian Network.

For example, I would decompose the update over a node with three outcomes  as the product of the baseline odds of the event and a number of updates.

In my system, I only cared about whether  happened. So I naively summarized each update before aggregating them.

This was giving me very poor results - my resulting updates would be very off compared to traditional inference algorithms like message passing.

It is no wonder this was giving me bad results - it is the wrong way of going about it! Our analysis at the beginning implies that the update should be the average of  and , instead of the sum.

After realizing the paradox, I changed my system to not summarizing the odds of  until after aggregating all the updates.

Performance improved.

## Consequences

It clearly has consequences, as illustrated by the examples in the previous section. But I am not sure what to recommend doing in response.

My most immediate takeaway is to be very careful when aggregating outcomes - there is an important chance we will be introducing an error along the way.

Beyond that, the aggregation paradox seems to imply that we need to work at the correct level of aggregation. We cannot naively deduce implied binary odds from the distribution of a multiple outcome event.

But what is the right level of aggregation?

When aggregating, the lower factor of the update is a weighted mean of the evidence likelihoods  and . This suggests that the problem disappears when we impose  for any disaggregation of the joint event  into subevents  and .

But this condition is too strong. For example, we could base our disaggregation on the observed evidence. For example, if the evidence  can either be  or  we could disaggregate ~A into the cases where  and the cases where . In that case, the condition cannot ever be satisfied, by definition.

We can say that this disaggregation is not a sensible one, and ought to be excluded for the purposes of the condition. But in that case we have passed the bucket down to defining what is a sensible disaggregation.

Another approach is to assume that the prior relative likelihood of any aggregated outcomes is uniform, ie . In that case, we have that .

But then we can no longer chain updates - after applying any likelihood where  the resulting posterior will no longer meet this condition.

Pragmatically, it seems like the best we can do if we want to rescue objetivity is to resign ourselfs to summarize the updates assuming a uniform prior. That is, by averaging the evidence associated to each aggregated outcome.

This is not enough to correctly approximate Bayesian updating, as we can see in the example below:

But I can't see how to do better in the absence of more information.

One key takeaway here is that beliefs and updates are summarized in different ways

## In summary

I have explained one counterintuitive consequence of Bayesian updating on variables with more than two outcomes. This paradox implies that we should be careful when grouping together outcomes of a variable. And I have shown two situations where this unintuitive consequence is relevant.

This is a post meant to explore and start a discussion more than provide definite answers. Some things I’d be keen on discussing include:

1. Is this a documented phenomena? Where can I find more discussion?
2. What does this imply for formulating forecasting questions? Will this result in problems when asking binary questions about events that are multifaceted?
3. What is “the right level” of outcome aggregation for a given problem?
4. Are there other examples where similar issues come up?

### Acknowledgements

Thanks to rossry, Nuño Sempere, Eric Neyman, Ehud Reiter and ForgedInvariant for discussing this topic with me and helping me clarify some ideas.

Thanks to Alex Mennen for coming up with the example I referenced in the post.

comment by rossry · 2021-11-22T13:58:39.883Z · LW(p) · GW(p)

The framing of this issue that makes the most sense to me is " is a function of ".

When I look at it this way, I disagree with the claim (in "Mennen's ABC example") that "[Bayesian updating] is not invariant when we aggregate outcomes" -- I think it's clearer to say the Bayesian updating is not well-defined when we aggregate outcomes.

Additionally, in "Interpreting Bayesian Networks", the framing seems to make it clearer that the problem is that you used  for  -- but they're not the same thing! In essence, you're taking the sum where you should be taking the average...

With this focus on (mis)calculating , the issue seems to me more like "a common error in applying Bayesian updates", rather than a fundamental paradox in Bayesian updating itself. I agree with the takeaway "be careful when grouping together outcomes of a variable" -- because grouping exposes one to committing this error -- but I'm not sure I'm seeing the thing that makes you describe it as unintuitive?

Replies from: Jsevillamol
comment by Jsevillamol · 2021-11-22T14:40:39.982Z · LW(p) · GW(p)

I like this framing.

This seems to imply that summarizing beliefs and summarizing updates are two distinct operations.

For summarizing beliefs we can still resort to summing:

But for summarizing updates we need to use an average - which in the absence of prior information will be a simple average:

Annoyingly and as you point out this is not a perfect summary - we are definitely losing information here and subsequent updates will be not as exact as if we were working with the disaggregated odds.

I still find it quite disturbing that the update after summarizing depends on prior information - but I can't see how to do better than this, pragmatically speaking.

Replies from: rossry
comment by rossry · 2021-11-22T17:46:00.951Z · LW(p) · GW(p)

Right, I agree that for the update aggregation is better than (but still lossy). And the thing that affects is the weighting in the average -- so if then the s don't matter! (which is a possible answer to your question of "how much aggregation/disaggregation can you do?")

But yeah if is very different from then I don't think there's any way around it, because the effective could be one or the other depending on what the are.

comment by AlexMennen · 2021-11-24T00:44:11.434Z · LW(p) · GW(p)

(Possibly a bit of a tangent) It occurred to me while reading this that perhaps average log odds could make sense in the context in which there is a uniform prior, and the probabilities provided by experts differ because the experts disagree on how to interpret evidence that brings them away from the uniform prior. This has some intuitive appeal:

1) Perhaps, when picking questions to ask forecasters, people have a tendency to pick questions for which they believe the probability that the answer is yes is approximately 50%, because that offers the most opportunity to update in response to the beliefs of the forecasters. If average log odds is an appropriate pooling method to use if you have a uniform prior, then this would explain its good empirical performance. I think I mentioned in our discussion on your EA forum post that if there is a tendency for more knowledgeable forecasters to give more extreme probabilities, then this would explain good performance by average log odds, which weights extreme predictions heavily. A tendency for the questions asked to have priors of near 50% according to the typical unknowledgeable person would explain why more knowledgeable forecasters would assign more extreme probabilities on average: it takes more expertise to justifiably bring their probabilities further from 50%.

2) It excuses the incoherent behavior of average log odds on my ABC example as well. If A, B, and C are mutually exclusive, then they can't all have 50% prior probability, so a pooling method that implicitly assumes that they do will not give coherent results.

Ultimately, though, I don't think this is actually true. Consider the example of forecasting a continuous variable x by soliciting probability density functions  and  from two experts, and pooling them to get the pdf proportional to  (renormalized so it integrates to 1). You could also consider forecasting the variable  for some differentiable, strictly increasing function f. Then your experts give you pdfs  and  satisfying , and you pool them to get the pdf proportional to . I claim that, if what we're doing implicitly depends on a uniform prior in a sneaky way, that the first thing should be the appropriate thing to do if x has a uniform prior, and the second thing should be appropriate if y has a uniform prior. If f is nonlinear, then a uniform prior on x induces a non-uniform prior on y, and vice-versa, so we should get incompatible results from each way of doing this, as we were implicitly using different priors each time. But let's try it: . Thus, given that both experts provided pdfs satisfying the formula  making their probability distributions on x and y compatible with , our pooled pdfs also satisfies that formula, and is also compatible with . That is, if we pooled using beliefs about x, and then find the implied beliefs about y, we get the same thing as if we directly pooled using beliefs about y. Different implicit priors don't appear to be ruining anything.

I conclude that the incoherent results in my ABC example cannot be blamed on switching between the uniform prior on {A,B,C} and the uniform prior on {A,A}, and, instead, should be blamed entirely on the experts having different beliefs conditional on A, which is taken account in the calculation using A,B,C, but not in the calculation using A,A.

Replies from: Jsevillamol
comment by Jsevillamol · 2021-11-24T13:24:16.029Z · LW(p) · GW(p)

average log odds could make sense in the context in which there is a uniform prior

This is something I have heard from other people too, and I still cannot make sense of it. Why would questions where uninformed forecasters produce uniform priors make logodds averaging work better?

A tendency for the questions asked to have priors of near 50% according to the typical unknowledgeable person would explain why more knowledgeable forecasters would assign more extreme probabilities on average: it takes more expertise to justifiably bring their probabilities further from 50%.

I don't understand your point. Why would forecasters care about what other people would do? They only want to maximize their own score.

If A, B, and C are mutually exclusive, then they can't all have 50% prior probability, so a pooling method that implicitly assumes that they do will not give coherent results.

This also doesn't make much sense to me, though it might be because I still don't understand the point about needing uniform priors for logodd pooling.

Different implicit priors don't appear to be ruining anything.

Neat!

I conclude that the incoherent results in my ABC example cannot be blamed on switching between the uniform prior on {A,B,C} and the uniform prior on {A,A}, and, instead, should be blamed entirely on the experts having different beliefs conditional on A, which is taken account in the calculation using A,B,C, but not in the calculation using A,A.

I agree with this.

Replies from: AlexMennen
comment by AlexMennen · 2021-11-24T16:28:32.311Z · LW(p) · GW(p)

Why would questions where uninformed forecasters produce uniform priors make logodds averaging work better?

Because it produces situations where more extreme probability estimates correlate with more expertise (assuming all forecasters are well-calibrated).

I don't understand your point. Why would forecasters care about what other people would do? They only want to maximize their own score.

They wouldn't. But if both would have started with priors around 50% before they acquired any of their expertise, and it's their expertise that updates them away from 50%, then more expertise is required to get more extreme odds. If the probability is a martingale that starts at 50%, and the time axis is taken to be expertise, then more extreme probabilities will on average be sampled from later in the martingale; i.e. with more expertise.

This also doesn't make much sense to me, though it might be because I still don't understand the point about needing uniform priors for logodd pooling.

If logodd pooling implicitly assumes a uniform prior, then logodd pooling on A vs A assumes A has prior probability 1/2, and logodd pooling on A vs B vs C assumes A has a prior of 1/3, which, if the implicit prior actually was important, could explain the different results.

comment by Unnamed · 2021-11-23T07:03:09.090Z · LW(p) · GW(p)

I think I've followed the basic argument here? Let me try a couple examples, first a toy problem and then a more realistic one.

Example 1: Dice. A person rolls some fair 20-sided dice and then tells you the highest number that appeared on any of the dice. They either rolled 1 die (and told you the number on it), or 5 dice (and told you the highest of the 5 numbers), or 6 dice (and told you the highest of the 6 numbers).

For some reason you care a lot about whether there were exactly 5 dice, so you could break this down into two hypotheses:

H1: They rolled 5 dice
H2: They rolled 1 or 6 dice

Let's say they roll and tell you that the highest number rolled was 20. This favors 5 dice over 1 die, and to a lesser degree it favors 6 dice over 5 dice. So if you started with equal (1/3) probabilities on the 3 possibilities, you'll update in favor of H1. Someone who also started with a 1/3 chance on H1, but who thought that 1 die was more likely than 6 dice, would update even more in favor of H1. And someone whose prior was that 6 dice was more likely than 1 die would update less in favor of H1, or even in the other direction if it was lopsided enough.

Relatedly, if you repeated this experiment many times and got lots of 20s, that would eventually become evidence against H1. If the 100th roll is 20, then that favors 6 dice over 5, and by that point the possibility of there being only 1 die is negligible (if the first 99 rolls were large enough) so it basically doesn't matter that the 20 also favors 5 dice over 1. This seems like another angle on the same phenomenon, since your posterior after 99 rolls is your prior for the 100th roll (and the evidence from the first 99 rolls has made it lopsided enough so that the 20 counts as evidence against H1).

Example 2: College choice. A high school freshman hopes & expects to attend Harvard for college in a few years. One observer thinks that's unlikely, because Harvard admissions is very selective even for very good students. Another observer thinks that's unlikely because the student is into STEM and will probably wind up going to a more technical university like MIT; they haven't thought much yet about choosing a college and Harvard is probably just serving as a default stand-in for a really good school.

The two observers might give the same p(Harvard), but for very different reasons. And because their models are so different, they could even update in opposite directions on the same new data. For instance, perhaps the student does really well on a math contest, and the first observer updates in favor of the student attending Harvard (that's an impressive accomplishment, maybe they will make it past the admissions filter) while the second observer updates a bit against the student attending Harvard (yep, they're a STEM person).

You could fit this into the "three outcomes" framing of this post, if you split "not attending Harvard" into "being rejected by Harvard" and "choosing not to attend Harvard".

Replies from: cousin_it, Unnamed
comment by cousin_it · 2021-11-23T12:16:06.577Z · LW(p) · GW(p)

I think your first example could be even simpler. Imagine you have a coin that's either fair, all-heads, or all-tails. If your prior is "fair or all-heads with probability 1/2 each", then seeing heads is evidence against "fair". But if your prior is "fair or all-tails with probability 1/2 each", then seeing heads is evidence for "fair". Even though "fair" started as 1/2 in both cases. So the moral of the story is that there's no such thing as evidence for or against a hypothesis, only evidence that favors one hypothesis over another.

Replies from: Pattern
comment by Pattern · 2021-11-24T17:02:05.994Z · LW(p) · GW(p)

That's a great explanation. Evidence may also be compatible or incompatible with a hypothesis. For instance, if I get a die (without the dots on the sides that indicate 1-6), and I instead label* it:

Red, 4, Life, X-Wing, Int, path through a tree

Then finding out I rolled a 4, without knowing what die I used, is compatible with the regular dice hypothesis, but any of the other rolls, is not.

*(likely using symbols, for space reasons)

comment by Unnamed · 2021-11-23T07:08:32.083Z · LW(p) · GW(p)

This seems related to philosophy of science stuff, where updating is about pitting hypotheses against each other. In order to do that you have to locate the leading alternative hypotheses. It doesn't work well to just pit a hypothesis against "everything else" (it's hard to say what p(E|not-H) is, and it can change as you collect more data). You need to find data that distinguishes your hypothesis from leading alternatives. An experiment that favors Newtonian mechanics over Aristotelian mechanics won't favor Newtonian mechanics over general relativity.

comment by Forged Invariant · 2021-11-23T04:28:53.479Z · LW(p) · GW(p)

Seeing the equations, it was hard to intuitively grasp why updates work this way. This example made things more intuitive for me:

If an event can have 3 outcomes, and we encounter strong evidence against outcomes B and C, then the update looks like this:

The information about what hypotheses are in the running is important, and pooling the updates can make the evidence look much weaker than it is.

Replies from: Jsevillamol
comment by Jsevillamol · 2021-11-23T08:53:53.973Z · LW(p) · GW(p)

Note that you are making the same mistake than me! Updates are not summarized in the same way as beliefs - for the update the "correct" way is to take an average of the  likelihoods:

This does not invalidate the example though!

Thanks for suggesting, I think it helps clarify the conondrum.

Replies from: Forged Invariant
comment by Forged Invariant · 2021-11-25T04:23:30.979Z · LW(p) · GW(p)

The left hand side of the example is deliberately making the mistake described in your article, as a way to build intuition on why it is a mistake.

(Adding instead of averaging in the update summaries was an unintended mistake)

Thanks for explaining how to summarize updates, it took me a bit to see why averaging works.