My impression of singular learning theory

I don't think this representation of the theory in my post is correct. The effective dimension of the singularity near the origin is much higher, e.g. because near every other minimal point of this loss function the Hessian doesn't vanish, while for the singularity at the origin it does vanish. If you discretized this setup by looking at it with a lattice of mesh $\epsilon$, say, you would notice that the origin is surrounded by many parameters that give nearly identical loss, while near other parts of the space the number of such parameters is far fewer.

The reason you have to do some kind of "translation" between the two theories is that SLT can see not just exactly optimal points but also nearly optimal points, and bad singularities are surrounded by many more nearly optimal points than better-behaved singularities. You can interpret the discretized picture above as the SLT picture seen at some "resolution" or "scale" $\epsilon$, i.e. if you discretized the loss function by evaluating it on a lattice with mesh $\epsilon$ you get my picture. Of course, this loses the information of what happens as $\epsilon \to 0$ and $n\to \infty$ in some thermodynamic limit, which is what you recover when you do SLT.

I just don't see what this thermodynamic limit tells you about the learning behavior of NNs that we didn't know before. We already know NNs approximate Solomonoff induction if the $A$-complexity is a good approximation to Kolmogorov complexity and so forth. What additional information is gained by knowing what $A$ looks like as a smooth function as opposed to a discrete function?

In addition, the strong dependence of SLT on $A$ being analytic is bad, because analytic functions are rigid: their value in a small open subset determines their value globally. I can see why you need this assumption because quantifying what happens near a singularity becomes incredibly difficult for general smooth functions, but because of the rigidity of analytic functions the approximation that "we can just pretend NNs are analytic" is more pernicious than e.g. "we can just pretend NNs are smooth". Typical approximation theorems like Stone-Weierstrass also fail to save you because they only work in the sup-norm and that's completely useless for determining behavior at singularities. So I'm yet to be convinced that the additional details in SLT provide a more useful account of NN learning than my simple description above.

I don't think this is something that requires explanation, though. If you take an arbitrary geometric object in maths, a good definition of its singular points will be "points where the tangent space has higher dimension than expected". If this is the minimum set of a loss function and the tangent space has higher dimension than expected, that intuitively means that locally there are more directions you can move along without changing the loss function, probably suggesting that there are more directions you can move along without changing the function being implemented at all. So the function being implemented is simple, and the rest of the argument works as I outline it in the post.

I think I understand what you and Jesse are getting at, though: there's a particular behavior that only becomes visible in the smooth or analytic setting, which is that minima of the loss function that are more singular become more dominant as $n\to \infty$ in the Boltzmann integral, as opposed to maintaining just the same dominance factor of $e^{-O\left(d\right)}$. You don't see this in the discrete case because there's a finite nonzero gap in loss between first-best and second-best fits, and so the second-best fits are exponentially punished in the limit and become irrelevant, while in the singular case any first-best fit has some second best "space" surrounding it whose volume is more concentrated towards the singularity point.

While I understand that, I'm not too sure what predictions you would make about the behavior of neural networks on the basis of this observation. For instance, if this smooth behavior is really essential to the generalization of NNs, wouldn't we predict that generalization would become worse as people switch to lower precision floating point numbers? I don't think that prediction would have held up very well if someone had made it 5 years ago.

As an aside, I've tried to work out what the optimal learning rate for a large language model should be based on the theory in the post, and if I'm doing the calculations correctly (which is a pretty big if) it doesn't match actual practice very well, suggesting there is actually something important missing from this picture.

Essentially, the coefficient $\beta$ should be $2/{\sigma }^2$ where ${\sigma }^2$ is the variance of the per-parameter noise in SGD. If you have a learning rate $\eta$, you scale the objective you're optimizing by a factor $\eta$ and the noise variance by a factor ${\eta }^2$. Likewise, a bigger batch size $n_b$ lowers the noise variance by a linear factor. So the equilibrium distribution ends up proportional to

$$\exp (-\frac{2L}{{\sigma }^2}\cdot \frac{n_b}{\eta })$$

where $L$ is the per-token average loss and ${\sigma }^2$ should be equal to the mean square of the partial derivative of the per-token loss function with respect to one of the neural network parameters. If the network is using some decent batch or layer normalization this should probably be $O\left(1/N\right)$ where $N$ is the model size.

We want what's inside the exponential to just be $L\cdot D$, because we want the learning to be equivalent to doing a Bayesian update over the whole data. This suggests we should pick

$$\eta =O\left(\frac{2N{n}_b}{D}\right)$$

which is a pretty bad prediction. So there's probably something important that's being left out of this model. I'm guessing that a smaller learning rate just means you end up conditioning on minimum loss and that's all you need to in practice, and larger learning rates cause problems with convergence.