[Link] Training Compute-Optimal Large Language Models

The first-order implication for Bio Anchors is that the number of training datapoints appears to scale linearly with parameter count, rather than in proportion to paramter count ^ 0.8, as estimated in the report. So for example, if you think that TAI models will be 100,000 times larger than current models, then they'll need 10 times more compute to train than was previously estimated. This pushes out timelines on the order of a few years, to the extent that you put weight on the neural network model.

Something worth reemphasizing for folks not in the field is that these benchmarks are not like usual benchmarks where you train the model on the task, and then see how good it does on a held-out set. Chinchilla was not explicitly trained on any of these problems. It’s typically given some context like: “Q: What is the southernmost continent? A: Antarctica Q: What is the continent north of Africa? A:” and then simply completes the prompt until a stop token is emitted, like a newline character.

And it’s performing above-average-human on these benchmarks.

Thinking back to the "inconsistency [LW · GW]" from the Kaplan et al papers...

• In Appendix E of the new paper, we see the loss-vs-compute frontier start to "bend" from a straight line on a log-log plot, with returns to additional compute getting smaller at large scales.
• I suspect this bending is the transition from the faster "L(C) law" to the slower "L(D) law."
  • A brief recap of that below:
    • Adding more params can help in two ways: it makes your model's loss decline toward its asymptotic minimum faster, and it can lower that minimum itself.
    • As models get bigger, the first effect dies off -- the loss curves converge to a fixed shape, rather than getter ever steeper.  The second effect keeps going, but with it alone, the overall rate of return is lower.
• Presumably, the learning rate issue in Kaplan et. al. also affected their estimated L(D) law.
  • The issue made Kaplan et al underestimate optimal model performance.  The underestimate was worst when considering models for which the optimal number of training steps was small.
  • The L(D) law came from early stopping experiments.  The early stopping step is lower for smaller data sizes.
  • So the L(D) experiments with smaller D values look artificially bad, relative to the ones with large D values.  Thus the estimated L(D) curve declines faster than the true L(D) curve.
  • If this is correct, then L(D) improves more slowly with data than we had believed.
  • Note that this does contradict the "use more data!" result from the paper -- that is about the relative rate at which N and D affect L(N, D).

I'm wondering: could one just continue training Gopher (the previous bigger model) on the newly added data?

Do they mention context window size somewhere? I couldn't find it. Though in principle I think it could be computed from the other hyperparameters they list.

This implies that optimal training of Gopher should have used 16x the data and compute. 

It also implies that further scaling will be compute and data only for a while. 

All the nice graphs will now get an ugly kink. 

All the extrapolations to the human (neocortex) neuron count are off. 

Really looking forward to reading the paper.