Six (and a half) intuitions for KL divergence
post by CallumMcDougall (TheMcDouglas) · 2022-10-12T21:07:07.796Z · LW · GW · 25 commentsThis is a link post for https://www.perfectlynormal.co.uk/blog-kl-divergence
Contents
Summary 1. Expected Surprise 2. Hypothesis Testing 3. MLEs 4. Suboptimal Coding 5A. Gambling Games - Beating the House 5B. Gambling Games - Gaming the Lottery 6. Bregman Divergence Final Thoughts None 25 comments
KL-divergence is a topic which crops up in a ton of different places in information theory and machine learning, so it's important to understand well. Unfortunately, it has some properties which seem confusing at a first pass (e.g. it isn't symmetric like we would expect from most distance measures, and it can be unbounded as we take the limit of probabilities going to zero). There are lots of different ways you can develop good intuitions for it that I've come across in the past. This post is my attempt to collate all these intuitions, and try and identify the underlying commonalities between them. I hope that for everyone reading this, there will be at least one that you haven't come across before and that improves your overall understanding!
One other note - there is some overlap between each of these (some of them can be described as pretty much just rephrasings of others), so you might want to just browse the ones that look interesting to you. Also, I expect a large fraction of the value of this post (maybe >50%) comes from the summary, so you might just want to read that and skip the rest!
Summary
1. Expected surprise
how much more surprised you expect to be when observing data with distribution , if you falsely believe the distribution is vs if you know the true distribution
2. Hypothesis Testing
the amount of evidence we expect to get for over in hypothesis testing, if is true.
3. MLEs
If is an empirical distribution of data, is minimised (over ) when is the maximum likelihood estimator for .
4. Suboptimal coding
the number of bits we're wasting, if we try and compress a data source with distribution using a code which is actually optimised for (i.e. a code which would have minimum expected message length if were the true data source distribution).
5A. Gambling games - beating the house
the amount (in log-space) we can win from a casino game, if we know the true game distribution is but the house incorrectly believes it to be .
5B. Gambling games - gaming the lottery
the amount (in log-space) we can win from a lottery if we know the winning ticket probabilities and the distribution of ticket purchases .
6. Bregman divergence
is in some sense a natural way of measuring of how far is from if we are using the entropy of a distribution to capture how far away it is from zero (analogous to how is a natural measure of the distance between vectors and , if we're using to capture how far the vector is from zero).
Common theme for most of these:
measure of how much our model differs from the true distribution . In other words, we care about how much and differ from each other in the world where P is true, which explains why KL-div is not symmetric.
1. Expected Surprise
For a random variable with probability distribution , the surprise (or surprisal) is defined as . This is motivated by some simple intuitive constraints we would like to have on any notion of "surprise":
- An event with probability has no surprise
- Lower-probability events are strictly more surprising
- Two independent events are exactly as surprising as the sum of those events' surprisal when independently measured
In fact, it's possible to show that these three considerations fix the definition of surprise up to a constant multiple.
From this, we have another way of defining entropy - as the expected surprisal of an event:
Now, suppose we (erroneously) believed the true distribution of to be , rather than . Then the expected surprise of our model (taking into account that the true distribution is ) is:
and we now find that:
In other words, KL-divergence is the difference between the expected surprise of your model, and the expected surprise of the correct model (i.e. the model where you know the true distribution ). The further apart is from , the worse the model is for , i.e. the more surprised it should expect to get by reality.
Furthermore, this explains why isn't symmetric, e.g. why it blows up when but not when . In the former case, your model is assigning very low probability to an event which might happen quite often, hence your model is very surprised by this. The latter case doesn't have this property, and there's no equivalent story you can tell about how your model is frequently very surprised.[1]
2. Hypothesis Testing
Suppose you have two hypotheses: a null hypothesis which says that , and an alternative hypothesis which says that . Suppose the null is actually true. A natural hypothesis test is the likelihood ratio test, i.e. you reject if the observation is in the critical region:
for some constant which determines the size of the test. Another way of writing this is:
We can interpret the value as (a scalar multiple of[2]) the bits evidence we get for over . In other words, if happens twice as often under distribution than distribution , then the observation is a single bit of evidence for over .
is (a scalar multiple of) the expected bits of evidence we get for over , where the expectation is over the null hypothesis . The closer and are, the more we should expect it to be hard to distinguish between them - i.e. when is true, we shouldn't expect reality to provide much evidence for rather than being true.
3. MLEs
This one is a bit more maths-heavy than the others, so ymmv on how enlightening it is!
Suppose is the empirical distribution of data , which are each iid with distribution , and is a statistical model parameterised by . Our likelihood function is:
By the law of large numbers, almost surely. This is the cross entropy of and . Also note that if we subtract this from the entropy of , we get . So minimising the cross entropy over is equivalent to maximising .
Our maximum likelihood estimator is the parameter which maximises , and we can use some statistical learning theory plus a lot of handwaving to argue that (i.e. we've swapped around the limit and argmin operators). In other words, maximum likelihood estimation is equivalent to minimising KL-divergence. If is large, this suggests that will not be a good model for data generated from the distribution .
4. Suboptimal Coding
Source coding is a huge branch of information theory, and I won't go through all of that in this post. There are several online resources that do a good job of explaining it. To recap the key idea that will be important here:
If you're trying to transmit data from some distribution over a binary channel, you can assign particular outcomes to strings of binary digits in a way which minimises the expected number of digits you have to send. For instance, if you have three possible events with probability (0.8, 0.1, 0.1), then it makes sense to use a code like (0, 10, 11) for this sequence, because you'll find yourself sending the shorter codes with higher probability.
In the limit for a large number of possible values for (provided some other properties hold), the optimal code[3] will represent outcome with a binary string of length .
From this, the intuition for KL divergence pops neatly out. Suppose you erroneously believed that , and you designed an encoding that would be optimal in this case. The expected number of bits you'll have to send per message is:
and we can immediately see that KL-divergence is (up to a scale factor) the difference in expected number of bits per event you'll have to send with this suboptimal code, vs the number you'd expect to send if you knew the true distribution and could construct the optimal code. The further apart and are, the more bits you're wasting on average by not sending the optimal code. In particular, if we have a situation like , this means our code (which is optimised for ) will assign a very long codeword to outcome since we don't expect it to occur often, and so we'll be wasting a lot of message space by frequently having to use this codeword.
5A. Gambling Games - Beating the House
Suppose you can bet on the outcome of some casino game, e.g. a version of a roulette wheel with nonuniform probabilities. First, imagine the house is fair, and pays you times your original bet if you bet on outcome (this way, any bet has zero expected value: because betting on outcome means you expect to get returned to you). Because the house knows exactly what all the probabilities are, there's no way for you to win money in expectation.
Now imagine the house actually doesn't know the true probabilities , but you do. The house's mistaken belief is , and so they pay people for event even though this actually has probability . Since you know more than them, you should be able to profit from this state of affairs. But how much can you make?
Suppose you have $1 to bet. You bet on outcome , so . Let be your expected winnings. It is more natural to talk about log winnings, because this describes how your wealth grows proportionally over time. Your expected log winnings are:
It turns out that, once you perform a simple bit of optimisation using the Lagrangian:
then you find the optimal betting strategy is (this is left as an exercise to the reader!). Your corresponding expected winnings are:
in other words, the KL divergence represents the amount you can win from the casino by exploiting the difference between the true probabilities and the house's false beliefs . The closer and are, the harder it is to profit from your extra knowledge.
Once again, this framing illustrates the lack of symmetry in the KL-divergence. If , this means the house will massively overpay you when event happens, so the obvious strategy to exploit this is to bet a lot of money on (and will correspondingly be very large). If , there is no corresponding way to exploit this (except to the extent that this suggests we might have for some different outcome ).
5B. Gambling Games - Gaming the Lottery
This is basically the same as (5A), but it offers a slightly different perspective. Suppose a lottery exists for which people can buy tickets, and the total amount people spend on tickets is split evenly between everyone who bought a ticket with the winning number (realistically the lottery organisers would take some spread, but we assume this amount is very small). If every ticket is bought the same number of times, then there's no way to make money in expectation. But suppose people have a predictable bias (e.g. buying round numbers, or numbers with repeated digits) - then you might be able to make money in expectation by buying the less-frequently-bought tickets, because when you win you generally won't have as many people you'll have to split the pot with.
If you interpret as the distribution of people buying each ticket (which is known to you), and is the true underlying distribution of which ticket pays out (also known), then this example collapses back into the previous one - you can use optimisation to find that the best way to purchase tickets is in proportion to , and the KL-divergence is equal to your expected log winnings.
To take this framing further, let's consider situations where is not known to you on a per-number basis, but the overall distribution of group-sizes-per-ticket-number is known to you. For instance, in the limit of a large number of players and of numbers you can approximate the group size as a Poisson distribution. If each ticket has the same probability of paying out, then you can make profit in expectation by buying one of every ticket (where is the uniform distribution, and is the Poisson distribution). Interestingly, this strategy of "buying the pot" is theoretically possible for certain lotteries, for instance in the Canadian 6/49 Lotto (see a paper analysing this flaw here). However, there are a few reasons this tends not to work in real life, such as:
- The lottery usually takes a sizeable cut
- There are lottery restrictions (e.g. ticket limits)
- Buying the pool is prohibitively expensive (organising and funding a syndicate to exploit this effect is hard!)
6. Bregman Divergence
Bregman divergence is pretty complicated in itself, and I don't expect this section to be illuminating to many people (it's still not fully illuminating to me!). However, I thought I'd still leave it in because it does offer an interesting perspective.
If you wanted to quantify how much two probability distributions diverge, the first thing you might think of is taking a standard norm (e.g. ) of the difference between them. This has some nice properties, but it's also unsatisfactory for a bunch of reasons. For instance, it intuitively seems like the distance between the Bernoulli distributions with and should be larger than that between and .[4]
It turns out that there's a natural way to associate any convex function with a measure of divergence. Since tangents to convex functions always lie below them, we can define Bregman divergence as the amount by which is greater than the estimate for it you would get by fitting a tangent line to at and using it to linearly extrapolate to .
To do some quick sanity checks for Bregman divergence - if your convex function is the norm squared, then the divergence measure you get is just the squared norm of the vector between your points:
This is basically what you'd expect - it shows you that when the norm is the natural way to measure how far away something is from zero (i.e. how large it is), then the norm of the vector between two points is the natural way to measure how far one point is from another.
Now, lets go back to the case of probability distributions. Is there any convex function which measures, in some sense, how far away a probability distribution is from zero? Well, one thing that seems natural is to say that "zero" is any probability distribution where the outcome is certain - in other words, zero entropy. And it turns out entropy is concave, so if we just take the negative of entropy then we get a convex function. Slap that into the formula for Bregman divergence and we get:
There's no lightning-bolt moment of illumination from this framing. But it's still interesting, because it shows that different ways of measuring the divergence between two points can be more natural than others, depending on the space that we're working in, and what it represents. Euclidean distance between two points is natural in probability space, when zero is just another point in that space. But when working on the probability simplex, with entropy being our chosen way to measure a probability distribution's "difference from zero", we find that is in some sense the most natural choice.
Final Thoughts
Recapping these, we find that being large indicates:
- Your model will be very surprised by reality
- You expect to get a lot of evidence in favour of hypothesis over , if is true
- is a poor model for observed data
- You would be wasting a lot of message content if you tried to encode optimally while falsely thinking the distribution was
- You can make a lot of money in betting games where other people have false beliefs , but you know the true probabilities
- (this one doesn't have as simple a one-sentence summary!)
Although (4) might be the most mathematically elegant, I think (1) cuts closest to a true intuition for .
To summarise what all of these framings have in common:
measure of how much our model differs from the true distribution . In other words, we care about how much and differ from each other in the world where P is true, which explains why KL-div is not symmetric.
To put this last point another way, "doesn't care" when (assuming both probabilities are small), because even though our model is wrong, reality doesn't frequently show us situations in which our model fails to match reality. But if then the outcome will occur more frequently than we expect, consistently surprising our model and thereby demonstrating the model's inadequacy.
- ^
Note that the latter case might imply the former case, e.g. if then we are actually also in the former case, since . But this doesn't always happen; it is possible to have asymmetry here. For instance, if P = (0.1, 0.9) and Q = (0.01, 0.99), then we are in the former case but not the latter. If is true, then 10% of the time model is extremely surprised, because an event happens that it ascribes probability 1% to - which is why is very large. But if is true, reality presents model with no surprises as large as this - hence is not as large.
- ^
The scalar multiple part is because we're working with natural log, rather than base 2.
- ^
Specifically, the optimal decodable code - in other words, your set of codewords needs to have the property that you could string together any combination of them and it's possible to decipher which codewords you used. For instance,
(0, 10, 11)
has this property, but(0, 10, 01)
doesn't, because the string010
could have been produced from0 + 10
or01 + 0
. - ^
One way you could argue that a distance measure should have this property is to observe that the former two distributions have much lower variance than the latter two. So if you observe a distribution which is either or , you should expect it to take much less time to tell which of the two distributions you're looking at than if you were trying to distinguish between and .
25 comments
Comments sorted by top scores.
comment by tailcalled · 2022-10-10T09:40:25.553Z · LW(p) · GW(p)
I also saw a good intuitive example of the asymmetry once. If you've got a bimodal distribution and a monomodal distribution that lies at one of the peaks of the bimodal distribution, then the KL-divergence will be low when P is the monomodal distribution and Q is the bimodal distribution, while the KL-divergence will be high when P is the bimodal distribution and Q is the monomodal distribution.
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-10T09:51:27.044Z · LW(p) · GW(p)
Oh yeah, I really like this one, thanks! The intuition here is again that a monomodal distribution is a bad model for a bimodal one because it misses out on an entire class of events, but the other way around is much less bad because there's no large class of events that happen in reality but that your model fails to represent.
For people reading here, this post discusses this idea in more detail. The image to have in mind is this one:
comment by jsd · 2022-10-28T15:32:31.984Z · LW(p) · GW(p)
Thanks for this post! Relatedly, Simon DeDeo had a thread on different ways the KL-divergence pops up in many fields:
Replies from: TheMcDouglasKullback-Leibler divergence has an enormous number of interpretations and uses: psychological, epistemic, thermodynamic, statistical, computational, geometrical... I am pretty sure I could teach an entire graduate seminar on it.
Psychological: an excellent predictor of where attention is directed. http://ilab.usc.edu/surprise/
Epistemic: a normative measure of where you ought to direct your experimental efforts (maximize expected model-breaking) http://www.jstor.org/stable/4623265
Thermodynamic: a measure of work you can extract from an out-of-equlibrium system as it relaxes to equilibrium.
Statistical: too many to count, but (e.g.) a measure of the failure of an approximation method. https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained
Computational (machine learning): a measure of model inefficiency—the extent to which it retains useless information. https://arxiv.org/abs/1203.3271
Computational (compression): the extent to which a compression algorithm designed for one system fails when applied to another.
Geometrical: the (non-metric!) connection when one extends differential geometry to the probability simplex.
Biological: the extent to which subsystems co-compute.
Machine learning: the basic loss function for autoencoders, deep learning, etc. (people call it the "cross-entropy")
Algorithmic fairness. How to optimally constrain a prediction algorithm when ensuring compliance with laws on equitable treatment. https://arxiv.org/abs/1412.4643
Cultural evolution: a metric (we believe) for the study of individual exploration and innovation tasks... https://www.sciencedirect.com/science/article/pii/S0010027716302840 …
Digital humanism: Kullback-Leibler divergence is related to TFIDF, but with much nicer properties when it comes to coarse-graining. (The most distinctive words have the highest partial-KL when teasing apart documents; stopwords have the lowest) http://www.mdpi.com/1099-4300/15/6/2246
Mutual information: Well, it's a special case of Kullback-Leibler—the extent to which you're surprised by (arbitrary) correlations between a pair of variables if you believe they're independent.
Statistics: it's the underlying justification for the Akiake Information Criterion, used for model selection.
Philosophy of mind: It’s the “free energy” term in the predictive brain account of perception and consciousness. See Andy Clark’s new book or https://link.springer.com/article/10.1007%2Fs11229-017-1534-5
↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-29T08:32:07.381Z · LW(p) · GW(p)
This is awesome, I love it! Thanks for sharing (-:
comment by Joseph Van Name (joseph-van-name) · 2023-07-20T10:26:28.528Z · LW(p) · GW(p)
The Radon-Nikodym theorem allows us to define the Kullback-Leibler divergence in a more general setting. Even though this setting is more abstract, in this abstract setting it is clear how the KL-divergence is a way to define relative entropy. The abstract setting limits our options to only the correct option.
Suppose that is the probability density function of a random variable supported on . Then we define the continuous entropy of as .
Suppose that we are given two probability measures on the Borel subsets of and we know that is the Lebesgue probability measure on . Then is there any way to abstractly define the continuous entropy of that only refers to the abstract measures without referring to or by recovering from the abstract properties of ? Yes there is. The Radon-Nikodym theorem allows us to recover from the measure theoretic properties of and it therefore allows us to recover the continuous entropy.
Suppose that is a -algebra. Let be two measures on . We say that is absolutely continuous with respect to if , and we write in this case. Recall that a measure is -finite if there are for each natural number such that for each and .
Theorem: (Radon-Nikodym theorem) Suppose that and are both -finite measures. Then there exists a positive measurable function where for each .
The function is called the Radon-Nikodym derivative of with respect to and is denoted by . The Radon-Nikodym derivative is unique up to a set of measure zero.
We therefore define the Kullback-Kiebler divergence as which coincides with the continuous entropy without the factor.
Now, the assumption that is isomorphic to the Lebesgue probability measure on is a surprisingly general assumption that works for many probability measures .
A Boolean algebra is said to be -complete if whenever for each natural number , the set has a least upper bound which we denote by .
A measure algebra is a pair where is a -complete Boolean algebra and is a function with iff and whenever for . If is a measure algebra, then define a metric (and hence a topology and uniformity and whatnot) on by setting . Recall that a topological space is separable if it has a countable dense subset.
For example, if is the collection of Lebesgue measurable subsets of , and is the collection of measure zero subsets of , then the quotient Boolean algebra is complete and if is the Lebesgue measure on , then is a separable atomless probability measure algebra, and this measure algebra is unique.
Theorem: (Caratheodory) Every separable atomless probability measure algebra is isomorphic.
We have an even stronger uniqueness theorem for this measure algebra.
Theorem: Every Borel probability measure on a complete separable metric space without any atoms is isomorphic to the Lebesgue measure on the interval .
Suppose now that are Borel probability measures on a complete separable metric space that possibly have atoms. Then there is some measure where is isomorphic to the Lebesgue probability measure. In this case, for almost all , so
. In this case, we can pretend that is the Lebesgue measure on and that just measures the continuous entropy of with respect to the Lebesgue probabilty measure on .
The KL-divergence also generalizes greatly. Given a function ( is usually convex or concave), one can define another measure of similarity between probability measures such as known as the -divergence. One can also define a measure on from the Radon-Nikodym derivative by where if and is a measure on , then is the measure on defined by . The random variable gives information about the similarity between the probability measures from which one can always recover the -divergence. In this case,.
comment by Steveot · 2022-10-11T08:20:39.624Z · LW(p) · GW(p)
Another intuition I often found useful: KL-divergence behaves more like the square of a metric than a metric.
The clearest indicator of this is that KL-divergence satisfies a kind of Pythagorean theorem established in a paper by Csiszár (1975), see https://www.jstor.org/stable/2959270#metadata_info_tab_contents . The intuition is exactly the same as for the euclidean case: If we project a point A onto a convex set S (say the projection is B), and if C is another point in the set S, then the standard Pythagorean theorem would tell us that the angle of the triangle ABC at B is larger than 90 degree, or in other words . And the same holds if we project with respect to KL divergence, and we end up having .
This has implications if you think about things like sample efficiency (instead of a square root rate as usual, convergence rates with KL divergence usually behave like 1/n).
This is also reflected in the relation between KL divergence and other distances for probability measures, like total variation or Wasserstein distance. The most prominent example would be Pinsker's inequality in this regard, stating that the total variation norm between two measures is bounded by a constant times the square root of the KL-divergence between the measures.
comment by Zack_M_Davis · 2024-01-11T05:59:40.260Z · LW(p) · GW(p)
Rich, mostly succinct pedagogy of timeless essentials, highly recommended reference post.
The prose could be a little tighter and less self-conscious in some places. (Things like "I won't go through all of that in this post. There are several online resources that do a good job of explaining it" break the flow of mathematical awe and don't need to be said.)
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2024-01-15T08:17:39.177Z · LW(p) · GW(p)
Thanks, really appreciate this (and the advice for later posts!)
comment by lalaithion · 2022-10-10T17:12:43.929Z · LW(p) · GW(p)
It would be nice to have a couple examples comparing concrete distributions Q and P and examining their KL-divergence, why it's large or small, and why it's not symmetric.
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-10T18:34:46.826Z · LW(p) · GW(p)
I think some of the responses here do a pretty good job of this. It's not really what I intended to go into with my post since I was trying to keep it brief (although I agree this seems like it would be useful).
comment by Nina Panickssery (NinaR) · 2023-07-05T13:59:37.545Z · LW(p) · GW(p)
I think this style of post is really valuable and interesting to read, thanks for putting this together
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2023-07-05T20:06:50.138Z · LW(p) · GW(p)
Thanks, really appreciate it!
comment by TekhneMakre · 2022-10-10T12:35:58.555Z · LW(p) · GW(p)
Nice. I didn't know about the hypothesis testing one (or Bregman, but I don't get that one). I wonder if one can back out another description of KL divergence in terms of mutual information from the expression of mutual information in terms of KL divergence: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Mutual_information
Replies from: TheMcDouglas, TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-10T14:13:02.553Z · LW(p) · GW(p)
And yeah, despite a whole 16 lecture course on convex opti I still don't really get Bregman either, I skipped the exam questions on it 😆
↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-10T14:11:33.261Z · LW(p) · GW(p)
Oh yeah, I hadn't considered that one. I think it's interesting, but the intuitions are better in the opposite direction, i.e. you can build on good intuitions for to better understand MI. I'm not sure if you can easily get intuitions to point in the other direction (i.e. from MI to ), because this particular expression has MI as an expectation over , rather than the other way around. E.g. I don't think this expression illuminates the nonsymmetry of .
The way it's written here seems more illuminating (not sure if that's the one that you meant). This gets across the idea that:
is the true reality, and is our (possibly incorrect) model which assumes independence. The mutual information between and equals , i.e. the extent to which modelling and as independent (sharing no information) is a poor way of modelling the true state of affairs (where they do share information).
But again I think this intuition works better in the other direction, since it builds on intuitions for to better explain MI. The arguments in the expression aren't arbitrary (i.e. we aren't working with ), which restricts the amount this can tell us about in general.
Replies from: TekhneMakre↑ comment by TekhneMakre · 2022-10-10T21:08:47.381Z · LW(p) · GW(p)
The arguments in the expression aren't arbitrary (i.e. we aren't working with ), which restricts the amount this can tell us about in general.
Yeah, I was vaguely hoping one could phrase $P$ and $Q$ so they're in that form, but I don't see it.
comment by [deleted] · 2023-07-08T15:58:40.722Z · LW(p) · GW(p)
This is nitpicky but I believe the reasoning for 3 is mistaken, mainly because of the counterintuitive claim that minimising cross-entropy over is equivalent to maximising KL divergence.
Writing the log-likelihood in the equation
we have the RHS tends to the negative cross entropy (by LLN), instead of just the cross entropy. Then, since with being the fixed (true) distribution and varying with , we have that MLE means we minimise the cross entropy which means we minimise the KL divergence.
comment by Ethan (EJ) Watkins · 2023-06-12T08:04:02.700Z · LW(p) · GW(p)
By the law of large numbers, almost surely. This is the cross entropy of and . Also note that if we subtract this from the entropy of , we get . So minimising the cross entropy over is equivalent to maximising .
I think the cross entropy of and is actually (note the negative sign). The entropy of is . Since then the KL divergence is actually the cross entropy minus the entropy, not the other way around. So minimising the cross entropy over will minimise (not maximise) the KL divergence.
I believe the next paragraph is still correct: the maximum likelihood estimator is the parameter which maximises , which minimises the cross-entropy, which minimises the KL divergence.
Apologies if any of what I've said above is incorrect, I'm not an expert on this.
comment by Ethan (EJ) Watkins · 2023-06-01T02:04:22.234Z · LW(p) · GW(p)
I think there is a mistake in this equation. and are the wrong way round. It should be:
↑ comment by CallumMcDougall (TheMcDouglas) · 2023-06-01T06:34:08.529Z · LW(p) · GW(p)
Yep that's right, thanks! Corrected.
comment by DeLesley Hutchins (delesley-hutchins) · 2022-10-13T23:47:55.108Z · LW(p) · GW(p)
This is a great post, and should really be in a FAQ for new ML researchers. Thanks!
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-15T08:59:41.486Z · LW(p) · GW(p)
Thanks, really appreciate it!
comment by Archimedes · 2022-10-12T23:10:59.826Z · LW(p) · GW(p)
This video breaks it down nicely along the lines of what you describe as the "common theme".
comment by Ulisse Mini (ulisse-mini) · 2022-10-12T01:38:42.993Z · LW(p) · GW(p)
Nice! This is a significantly more developed intuition than the one I stumbled across (which is #1 for you I believe)
:)
Replies from: TheMcDouglas↑ comment by CallumMcDougall (TheMcDouglas) · 2022-10-15T09:00:39.389Z · LW(p) · GW(p)
Thank you :-)