Comment by Ethan (EJ) Watkins on Six (and a half) intuitions for KL divergence · 2023-06-12T08:04:02.700Z · LW · GW

By the law of large numbers, $\frac{1}{N}{\sum }_{i=1}^N\ln Q_{\theta }\left(x_i\right)\to {\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ almost surely. This is the cross entropy of $P$ and $Q_{\theta }$. Also note that if we subtract this from the entropy of $P$, we get $D_{KL}\left(P||Q_{\theta }\right)$. So minimising the cross entropy over $\theta$ is equivalent to maximising $D_{KL}\left(P||Q_{\theta }\right)$.

I think the cross entropy of $P$ and $Q_{\theta }$ is actually $H(P,Q_{\theta })=-{\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ (note the negative sign). The entropy of $P$ is $H\left(P\right)=-{\sum }_{x}P\left(x\right)\ln P\left(x\right)$. Since $D_{KL}\left(P||Q_{\theta }\right)={\sum }_{x}P\left(x\right)\left(\ln \right(P\left(x\right)-\ln Q_{\theta }\left(x\right))={\sum }_{x}P(x\left)\ln P\right(x)-{\sum }_{x}P(x)\ln Q_{\theta }\left(x\right)=-H(P)+H(P,Q_{\theta })$then the KL divergence is actually the cross entropy minus the entropy, not the other way around. So minimising the cross entropy over $\theta$ will minimise (not maximise) the KL divergence.

I believe the next paragraph is still correct: the maximum likelihood estimator ${\theta }^{\ast }$ is the parameter which maximises $L({\hat{P}}_n;Q_{\theta })$, which minimises the cross-entropy, which minimises the KL divergence.

Apologies if any of what I've said above is incorrect, I'm not an expert on this.

Comment by Ethan (EJ) Watkins on Six (and a half) intuitions for KL divergence · 2023-06-01T02:04:22.234Z · LW · GW

$$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_P\right(X)-I_Q(X\left)\right]$$

I think there is a mistake in this equation. $I_P\left(X\right)$ and $I_Q\left(X\right)$ are the wrong way round. It should be:
$$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_Q\right(X)-I_P(X\left)\right]$$