miss_figg

Posts
Comments

Posts

Comments

Comment by Miss_Figg on Is the sum individual informativeness of two independent variables no more than their joint informativeness? · 2019-07-08T15:39:36.282Z · LW · GW

We want to prove:

\int \int p_{x s} (x, s) log \frac{p_{x s} (x, s)}{p_{s} (s) p_{x} (x)} d x d s + \int \int p_{y s} (y, s) log \frac{p_{y s} (y, s)}{p_{s} (s) p_{y} (y)} d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x y} (x, y)} d x d y d s

This can be rewritten as:

\int \int \int p_{x y s} (x, y, s) log \frac{p_{s x} (s, x)}{p_{s} (s) p_{x} (x)} d x d y d s + \int \int \int p_{x y s} (x, y, s) log \frac{p_{s y} (s, y)}{p_{s} (s) p_{y} (y)} d x d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x} (x) p_{y} (y)} d x d y d s

After moving everything to the right hand side and simplifying, we get:

\int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s) p_{s} (s)}{p_{x s} (x, s) p_{y s} (y, s)} d x d y d s \geq 0

Now if we just prove that $q (x, y, s) = \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)}$ is a probability distribution, then the left hand side is $K L (p_{x y s} (x, y, s), q)$ , and Kullback-Leibler divergence is always nonnegative.

Ok, q is obviously nonnegative, and its integral equals 1:

\int \int \int \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)} d x d y d s = \int \frac{p_{s} (s) p_{s} (s)}{p_{s} (s)} d s = \int p_{s} (s) d s = 1

Q.e.d.

User info

Posts

Comments