Bayes, Backwards

post by ec429 · 2017-12-03T18:17:10.735Z · LW · GW · 1 comments

Contents

  Problem Statement
  Why is this interesting? 
  Solution 
  Analysis
None
1 comment

[Epistemic status: QED.]

Recently someone posed an oddly-constructed exercise on Bayes' Theorem, where instead of the usual given information $P\left(A|B\right),P\left(A|\neg B\right),P\left(B\right)$ they gave $P\left(A|B\right),P\left(A|\neg B\right),P\left(A\right)$. I won't link to the source, because they intended to pose a problem in the usual form and I don't want to draw attention to their mistake. But the problem itself is an interesting one, and it does have a solution (in most cases) which turns out to be quite neat.

Problem Statement

Given $P\left(A|B\right)=x,P\left(A|\neg B\right)=y,P\left(A\right)=a$, find $P\left(B|A\right),P\left(B|\neg A\right)$.

Why is this interesting?

(Putting this section here to separate the problem statement from my solution, since I don't know of a way to spoiler in MarkDown.)

Suppose you want to evaluate the performance of a heuristic, $B$, for determining the underlying $A$, which is accurately determined later. You know the base rate of $A$ since it is (say) recorded in public statistics, and you know how common $B$ -errors in either direction are, because the same statistics report $P\left(A|B\right)$ and $P\left(A|\neg B\right)$ for their obvious use in decisionmaking.

But you are interested not just in the general case, but in some subcategory $C$. In the absence of further information your default prior hypothesis is that $P(B|A\wedge C)=P\left(B|A\right)$ and similarly $P(B|\neg A\wedge C)=P\left(B|\neg A\right)$ (i.e., the heuristic's reliability is independent of $C$), but you know that within $C$ the base rate of $A$, $P\left(A|C\right)$, is some value different from $P\left(A\right)$.

Now under that hypothesis, knowing $P\left(B|A\right),P\left(B|\neg A\right)$ would allow you to determine your prior for $P(A|B\wedge C)$ and $P(A|\neg B\wedge C)$ by an ordinary application of Bayes' Theorem. If you then have a small sample drawn from $C$, you could for instance denote those priors as $p$ and $q$ and take as your prior distributions that $P(A|B\wedge C)\sim \beta (p,1-p)$ and $P(A|\neg B\wedge C)\sim \beta (q,1-q)$, which you then update from your sample. (I don't know whether this places the 'right' amount of credence in $P(B|A\wedge C)=P\left(B|A\right)$ etc.; one could just as easily take $\beta (2p,2(1-p\left)\right)$ to treat the $\neg C$ results as more 'relevant'.)

But we don't have $P\left(B|A\right)$, we just have $P\left(A|B\right)$. Hence the problem-as-stated.

Solution

We first expand $P\left(A|B\right)$ and $P\left(\neg A|B\right)$ using Bayes' Theorem:

Now we can re-arrange these into two expressions for $P\left(B\right)$, and set them equal:

Cross-multiplying and substituting in our given $x=P\left(A|B\right)$,

We can do the same thing with $y=P\left(A|\neg B\right)$ to obtain:

We then use the exhaustivity of $\{B,\neg B\}$ to expand $P\left(A\right)$ and $P\left(\neg A\right)$:

At this point we have four simultaneous equations in the four unknowns $P(A\wedge B),P(A\wedge \neg B),P(\neg A\wedge B),P(\neg A\wedge \neg B)$, which we can solve by matrix inversion, Gaussian elimination, or some similar method. Our matrix is

and our linear system is

The determinant is $det\left(M\right)=y-x$, so our condition for a solution is that $x\ne y$. This makes sense: if $x=y$ then the heuristic $B$ tells us nothing about $A$, so there is no way to link our givens (all probabilities of $A$) to joint probabilities involving $B$. So let us assume that $x\ne y$. In that case the solution is

as can be verified by multiplying out $Mv$.

From here the rest is trivial:

And we're done.

Analysis

To see what this result means, let's write it out with our givens substituted back in:

which holds as long as $P\left(A|\neg B\right)\ne P\left(A|B\right)$.

We could also have determined this by summing the first and third rows of $v$, $\frac{1}{y-x}(x+1-x)(y-a)=\frac{y-a}{y-x}$, to find that

under the same condition, and then substituting this into Bayes' Theorem.

Now let's consider $b=P\left(A|B\right)-P\left(A\right),\overline{b}=P\left(A|\neg B\right)-P\left(A\right)$ as, in some sense, measures of the "excess $A$" under $B$ and $\neg B$ respectively. This now tells us that

which I think is a good place to stop.

1 comments

Comments sorted by top scores.