A Bayesian Aggregation Paradox

The framing of this issue that makes the most sense to me is "$P(E|B\cup C)$ is a function of $P\left(B\right):P\left(C\right)$".

When I look at it this way, I disagree with the claim (in "Mennen's ABC example") that "[Bayesian updating] is not invariant when we aggregate outcomes" -- I think it's clearer to say the Bayesian updating is not well-defined when we aggregate outcomes.

Additionally, in "Interpreting Bayesian Networks", the framing seems to make it clearer that the problem is that you used $e_{1,2}+e_{1,3}$ for $P(E|B\cup C)$ -- but they're not the same thing! In essence, you're taking the sum where you should be taking the average...

With this focus on (mis)calculating $P(E|B\cup C)$, the issue seems to me more like "a common error in applying Bayesian updates", rather than a fundamental paradox in Bayesian updating itself. I agree with the takeaway "be careful when grouping together outcomes of a variable" -- because grouping exposes one to committing this error -- but I'm not sure I'm seeing the thing that makes you describe it as unintuitive?

(Possibly a bit of a tangent) It occurred to me while reading this that perhaps average log odds could make sense in the context in which there is a uniform prior, and the probabilities provided by experts differ because the experts disagree on how to interpret evidence that brings them away from the uniform prior. This has some intuitive appeal:

1) Perhaps, when picking questions to ask forecasters, people have a tendency to pick questions for which they believe the probability that the answer is yes is approximately 50%, because that offers the most opportunity to update in response to the beliefs of the forecasters. If average log odds is an appropriate pooling method to use if you have a uniform prior, then this would explain its good empirical performance. I think I mentioned in our discussion on your EA forum post that if there is a tendency for more knowledgeable forecasters to give more extreme probabilities, then this would explain good performance by average log odds, which weights extreme predictions heavily. A tendency for the questions asked to have priors of near 50% according to the typical unknowledgeable person would explain why more knowledgeable forecasters would assign more extreme probabilities on average: it takes more expertise to justifiably bring their probabilities further from 50%.

2) It excuses the incoherent behavior of average log odds on my ABC example as well. If A, B, and C are mutually exclusive, then they can't all have 50% prior probability, so a pooling method that implicitly assumes that they do will not give coherent results.

Ultimately, though, I don't think this is actually true. Consider the example of forecasting a continuous variable x by soliciting probability density functions $p_1\left(x\right)$ and $p_2\left(x\right)$ from two experts, and pooling them to get the pdf proportional to $\sqrt{p_1\left(x\right)p_2\left(x\right)}$ (renormalized so it integrates to 1). You could also consider forecasting the variable $y=f\left(x\right)$ for some differentiable, strictly increasing function f. Then your experts give you pdfs $q_1\left(y\right)$ and $q_2\left(y\right)$ satisfying $p_i\left(x\right)=f^{\prime }\left(x\right)q_i\left(f\right(x\left)\right)$, and you pool them to get the pdf proportional to $\sqrt{q_1\left(y\right)q_2\left(y\right)}$. I claim that, if what we're doing implicitly depends on a uniform prior in a sneaky way, that the first thing should be the appropriate thing to do if x has a uniform prior, and the second thing should be appropriate if y has a uniform prior. If f is nonlinear, then a uniform prior on x induces a non-uniform prior on y, and vice-versa, so we should get incompatible results from each way of doing this, as we were implicitly using different priors each time. But let's try it: $\sqrt{p_1\left(x\right)p_2\left(x\right)}=\sqrt{f^{\prime }\left(x\right)q_1\left(f\right(x\left)\right)f^{\prime }\left(x\right)q_2\left(f\right(x\left)\right)}=f^{\prime }\left(x\right)\sqrt{q_1\left(f\right(x\left)\right)q_2\left(f\right(x\left)\right)}$. Thus, given that both experts provided pdfs satisfying the formula $p_i\left(x\right)=f^{\prime }\left(x\right)q_i\left(f\right(x\left)\right)$ making their probability distributions on x and y compatible with $y=f\left(x\right)$, our pooled pdfs also satisfies that formula, and is also compatible with $y=f\left(x\right)$. That is, if we pooled using beliefs about x, and then find the implied beliefs about y, we get the same thing as if we directly pooled using beliefs about y. Different implicit priors don't appear to be ruining anything.

I conclude that the incoherent results in my ABC example cannot be blamed on switching between the uniform prior on {A,B,C} and the uniform prior on {A,$\neg$A}, and, instead, should be blamed entirely on the experts having different beliefs conditional on $\neg$A, which is taken account in the calculation using A,B,C, but not in the calculation using A,$\neg$A.

I think I've followed the basic argument here? Let me try a couple examples, first a toy problem and then a more realistic one.

Example 1: Dice. A person rolls some fair 20-sided dice and then tells you the highest number that appeared on any of the dice. They either rolled 1 die (and told you the number on it), or 5 dice (and told you the highest of the 5 numbers), or 6 dice (and told you the highest of the 6 numbers).

For some reason you care a lot about whether there were exactly 5 dice, so you could break this down into two hypotheses:

H1: They rolled 5 dice
H2: They rolled 1 or 6 dice

Let's say they roll and tell you that the highest number rolled was 20. This favors 5 dice over 1 die, and to a lesser degree it favors 6 dice over 5 dice. So if you started with equal (1/3) probabilities on the 3 possibilities, you'll update in favor of H1. Someone who also started with a 1/3 chance on H1, but who thought that 1 die was more likely than 6 dice, would update even more in favor of H1. And someone whose prior was that 6 dice was more likely than 1 die would update less in favor of H1, or even in the other direction if it was lopsided enough.

Relatedly, if you repeated this experiment many times and got lots of 20s, that would eventually become evidence against H1. If the 100th roll is 20, then that favors 6 dice over 5, and by that point the possibility of there being only 1 die is negligible (if the first 99 rolls were large enough) so it basically doesn't matter that the 20 also favors 5 dice over 1. This seems like another angle on the same phenomenon, since your posterior after 99 rolls is your prior for the 100th roll (and the evidence from the first 99 rolls has made it lopsided enough so that the 20 counts as evidence against H1).

Example 2: College choice. A high school freshman hopes & expects to attend Harvard for college in a few years. One observer thinks that's unlikely, because Harvard admissions is very selective even for very good students. Another observer thinks that's unlikely because the student is into STEM and will probably wind up going to a more technical university like MIT; they haven't thought much yet about choosing a college and Harvard is probably just serving as a default stand-in for a really good school.

The two observers might give the same p(Harvard), but for very different reasons. And because their models are so different, they could even update in opposite directions on the same new data. For instance, perhaps the student does really well on a math contest, and the first observer updates in favor of the student attending Harvard (that's an impressive accomplishment, maybe they will make it past the admissions filter) while the second observer updates a bit against the student attending Harvard (yep, they're a STEM person).

You could fit this into the "three outcomes" framing of this post, if you split "not attending Harvard" into "being rejected by Harvard" and "choosing not to attend Harvard".

Seeing the equations, it was hard to intuitively grasp why updates work this way. This example made things more intuitive for me:

If an event can have 3 outcomes, and we encounter strong evidence against outcomes B and C, then the update looks like this:

The information about what hypotheses are in the running is important, and pooling the updates can make the evidence look much weaker than it is.

As it is often the case, I just found out that Jaynes was already discussing a similar issue to the paradox here in his seminal book.

This wikipedia article summarizes the gist of it.

There's probably a radical constructivist argument for not really believing in open/noncompact categories [LW · GW] like $\neg A$. I don't know how to make that argument, but this post too updates me slightly towards such a Tao [LW · GW]of conceptualization.

(To not commit this same error at the meta level: Specifically, I update away from thinking of general negations as "real" concepts, disallowing statements like "Consider a non-chair [LW · GW], ...").

But this is maybe a tangent, since just adopting this rule doesn't resolve the care required in aggregation with even compact categories.

Is this just the thing where evidence is theory-laden? Like, for example, how the evidentiary value of the WHO report on the question of COVID origins depends on how likely one thinks it is that people would effectively cover up a lab leak?

I think entropy is a key to understanding this more deeply. I believe you could consider the unaggregated distribution as the "microstates" and the aggregated one as the "macrostates". The entropy would then tell you how much information you lose by aggregating in this way.

Minor quibble: The likelihood part of probability is also subjective in the sense that it depends on the evidence the agent is aware of.

I find the beginning of this post somewhat strange, and I'm not sure your post proves what you claim it does. You start out discussing what appears to be a combination of two forecasts, but present it as Bayesian updating. Recall that Bayes theorem says $p(\theta \mid x)=\frac{p(x\mid \theta )p\left(\theta \right)}{p\left(x\right)}$. To use this theorem, you need both an $x$ (your data / evidence), and a $\theta$ (your parameter). Using “posterior$\propto$ prior $\times$ likelihood” (with priors $p_1,p_2,p_3$ and likelihoods $e_1,e_2,e_3$), you're talking as if your expert's likelihood equals $p(x\mid \theta )$ – but is that true in any sense? A likelihood isn't just something you multiply with your prior, it is a conditional pmf or pdf with a different outcome than your prior.

I can see two interpretations of what you're doing at the beginning of your post:

1. You're combining two forecasts. That is, with $\theta \in A,B,C$ being the outcome, you have your own pmf $p_1\left(\theta \right)$ and the expert's $e=p_2\left(\theta \right)$, then combine them using $p\left(\theta \right)\propto p_1\left(\theta \right)p_2\left(\theta \right)$. That's fair enough, but I suppose $p\left(\theta \right)\propto \sqrt{p_1\left(\theta \right)p_2\left(\theta \right)}$ or maybe $p\left(\theta \right)\propto p_1\left(\theta {)}^q{p}_2\right(\theta {)}^{1-q}$ for some $q\in [0,1]$ would be a better way to do it.
2. It might be possible to interpret your calculations as a proper application of Bayes' rule, but that requires stretching it. Suppose $\theta$ is your subjective probability vector for the outcomes $A,B,C$ and $x$ is the subjective probability vector for the event supplied by an expert (the value of $x$ is unknown to us). To use Bayes' rule, we will have to say that the evidence vector $e=p(x\mid \theta )$, the probability of observing an expert judgment of $x$ given that $\theta$ is true. I'm not sure we ever observe such quantities directly, and it is pretty clear from your post that you're talking about $e=p_2\left(\theta \right)$ in the sense used above, not $p(x\mid \theta )$.

Assuming interpretation 1, the rest of your calculations are not that interesting, as you're using a method of knowledge pooling no one advocates.

Assuming interpretation 2, the rest of your calculations are probably incorrect. I don't think there is a unique way to go from $p(x\mid \theta )$to, let's say, $p(x^{\ast }\mid {\theta }^{\ast })$, where $x^{\ast }$ is the expert's probability vector over $A,A^c$ and ${\theta }^{\ast }$ your probability vector over $A,A^c$.