OpenAI o1, Llama 4, and AlphaZero of LLMs

I can not see any 1o improvement on this.

Are you saying that o1 did not do any better than 5-6% on your AIME-equivalent dataset? That would be interesting given that o1 did far better on the 2024 AIME which presumably was released after the training cutoff: https://openai.com/index/learning-to-reason-with-llms/

How did you translated the dataset, and what is the translation quality?

The point isn't particularly that it's low-hanging fruit or that this is going to happen with other LLMs soon. I expect that counterfactually System 2 reasoning likely happens soon even with no o1 [LW(p) · GW(p)], made easy and thereby inevitable merely by further scaling of LLMs, so the somewhat suprising fact that it works already doesn't significantly move my timelines.

The issue I'm pointing out is timing, a possible delay between when base models at the next level of scale get published in open weights, that don't yet have o1-like System 2 reasoning (Llama 4 seems the most likely specific model like that to come out next year), and a bit later when it becomes feasible to apply post-training for o1-like System 2 reasoning to these base models.

In the interim, the decisions to publish open weights would be governed by the capabilities without System 2 reasoning, and so they won't be informed decisions. It would be very easy to justify decisions to publish even in the face of third party evaluations, since those evaluations won't be themselves applying o1-like post-training to the model that doesn't already have it, in order to evaluate its resulting capabilities. But then a few months later, there is enough know-how in the open to do that, and capabilities cross all the thresholds that would've triggered in those evaluations, but didn't, since o1-like post-training wasn't yet commoditized at the time they were done.

They even managed to publish it in Nature. But if you don't throw out the original data and instead train on both the original data and the generated data, this doesn't seem to happen (see also). Besides, there is the empirical observation that o1 in fact works at GPT-4 scale, so similar methodology might survive more scaling. At least at the upcoming ~5e26 FLOPs level of next year, which is the focus of this post, the hypothetical where an open weights release arrives before there is an open source reproduction of o1's methodology, which subsequently makes that model much stronger in a way that wasn't accounted for when deciding to release that open weights model.

AlphaZero is purely synthetic data, and humans (note congenitally blind humans, so video data isn't crucial) use maybe 10,000 times less natural data than Llama-3-405B (15 trillion tokens) to get better performance, though we individually know much fewer facts. So clearly there is some way to get very far with merely 50 trillion natural tokens, though this is not relevant to o1 specifically.

Another point is that you can repeat the data for LLMs (5-15 times with good results, up to 60 times with slight further improvement, then there is double descent with worst performance at 200 repetitions, so improvement might resume after hundreds of repetitions). This suggests that it might be possible to repeat natural data many times to balance out a lot more unique synthetic data.

Yeah, I'm extremely skeptical of the paper, and to point out one of it's flaws, if you don't throw away the original data, that doesn't happen.

There are almost certainly other assumptions that are likely wrong, but that alone makes me extremely skeptical of the model collapse concept, and while a lot of people want it to be true, there is no reason to believe it's truth.

You've linked to this Sakana AI paper like 8 times in the last week. IMO, please stop, it's complete bunk, basically a scam.

I don't think it being bunk is really any evidence against automated ML research becoming a thing soon, or even already a thing, but the fact that you keep linking to it while ignoring the huge errors in it, and pretend it proves some point, is frustrating.

It's not necessarily at all impactful. The crucial question for the next few years is whether and where LLM scaling plateaus. Before o1, GPT-4 level models couldn't produce useful reasoning traces that are very long. Reading comprehension just only started mostly working at this scale. And RLHF through PPO is apparently to a large extent a game of carefully balancing early stopping. So it's brittle, doesn't generalize off-distribution very far, which made it unclear if heavier RL can help with System 2 reasoning where it's not capable of rebuilding the capabilities from scratch, overwriting the damage it does to the LLM.

But taking base model scaling one step further should help both with fragility of GPT-4 level models, and with their ability to carry out System 2 reasoning on their own, with a little bit of adaptation that elicits, rather than heavier RL that instills capabilities not already found to a useful extent in the base model. And then there's at least one more step of base model scaling after that (deployed models are about 40 megawatts of H100s, models in training about 150 megawatts, in 1.5 years we'll get to about a gigawatt, with 2x in FLOPs from moving to B200s and whatever effective FLOP/joule Trillium delivers). So there's every chance this advancement is rendered obsolete, to the extent that it's currently observable as a capability improvement, if these scaled up models just start being competent at System 2 reasoning without needing such post-training. Even ability of cheaper models to reason could then be reconstructed by training them on reasoning traces collected from larger models.

On the other hand, this is potentially a new dimension of scaling, if the RL can do real work and brute force scaling of base models wouldn't produce a chatbot good enough to reinvent the necessary RL on its own. There is a whole potentially very general pipeline to generative capabilities here. It starts with preference about the outcome (ability to verify an answer, to simultaneously judge aesthetics and meanings of a poem, to see if a pull request really does resolve the issue). Then it proceeds to training a process supervision model that estimates how good individual reasoning steps are as contributions to getting a good outcome, and optimizing the generative model that proposes good reasoning steps. With a few more years of base model scaling, LLMs are probably going to be good enough at evaluating impactful outcomes that they are incapable of producing directly, so this pipeline gets a lot of capabilities to manufacture. If o1's methodology is already applicable to this, this removes the uncertainty about whether it could've been made to start working without delay relative to the underlying scaling of base models. And the scaling of RL might go a long way before running out of exploitable preference signal about the outcomes of reasoning, independently of how the base models are being scaled up.