Is the sum individual informativeness of two independent variables no more than their joint informativeness?
post by Brangus
score: 11 (3 votes) ·
This is a question post.
Is it true that:
If I(X;Y) = 0 then I(S;X) + I(S;Y) <= I(S;X,Y)
Can you find a counterexample, or prove this and teach me your proof?
Someone showed me a simple analytic proof. I am still interested in seeing different ways people might prove this though.
answer by Miss_Figg
· score: 3 (2 votes) · LW
We want to prove:
This can be rewritten as:
After moving everything to the right hand side and simplifying, we get:
Now if we just prove that is a probability distribution, then the left hand side is , and Kullback-Leibler divergence is always nonnegative.
Ok, q is obviously nonnegative, and its integral equals 1:
Comments sorted by top scores.
comment by Jalex Stark (jalex-stark-1)
· score: 1 (1 votes) · LW
Just for amusement, I think this theorem can fail when s, x, y represent subsystems of an entangled quantum state. (The most natural generalization of mutual information to this domain is sometimes negative.)