Jesse Hoogland's Shortform

We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process

This line caught my eye while reading. I don't know much about RL on LLMs, is this a common failure mode these days? If so, does anyone know what such reward hacks tend to look like in practice? 

Advanced capabilities can be squeezed into small, efficient models that can run on commodity hardware.

This could also work for general intelligence and not only narrow math/coding olympiad sort of problems. The potential of o1/R1 is plausibly constrained for now by ability to construct oracle verifiers for correctness of solutions, which mostly only works for toy technical problems. Capabilities on such problems are not very likely to generalize to general capabilities, there aren't clear signs so far that this is happening.

But this is a constraint on how the data can be generated, not on how efficiently other models can be retrained using such data to channel the capabilities. If at some point there will be a process for generating high quality training data for general intelligence, that data might also turn out to be effective for cheaply training other models. The R1-generated data used to train the distill models is 800K samples^[1], which is probably 1B-10B tokens, less than 0.1% of typical amounts of pretraining data.

↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2025-01-22T13:21:08.309Z · LW(p) · GW(p)

This was my understanding pre r1. Certainly this seems to be the case with the o1 models: better at code and math, not better at philosophy and creative writing.

But something is up with r1. It is unusually good at creative writing. It doesn't seem spikey in the way that I predicted.

I notice I am confused.

Possible explanation: r1 seems to have less restrictive 'guardrails' added using post-training. Perhaps this 'light hand at the tiller' results in not post-training it towards mode-collapse. It's closer to a raw base model than the o1 models.

This is just a hypothesis. There are many unknowns to be investigated.

Replies from: jhoogland

↑ comment by Jesse Hoogland (jhoogland) · 2025-01-22T19:17:14.673Z · LW(p) · GW(p)

Post-training consists of two RL stages followed by two SFT stages, one of which includes creative writing generated by DeepSeek-V3. This might account for the model both being good at creative writing and seeming closer to a raw base model. 

Another possibility is the fact that they apply the RL stages immediately after pretraining, without any intermediate SFT stage.

Instead, we seem to be headed to a world where
- Proliferation is not bottlenecked by infrastructure.
- Regulatory control through hardware restriction becomes much less viable.

I like the rest of your post, but I'm skeptical of these specific implications.

Even if everyone has access to the SOTA models, some actors will have much more hardware to run on them, and I expect this to matter. This does make the offense/defense balance more weighted on the offense side, arguably, but there are many domains where extra thinking will help a lot. 

More generally, and I hate-to-be-that-guy, but I think it's telling that prediction markets and stock markets haven't seem to update that much since R1's release. I think it's generally easy to get hyped up over whatever is the latest thing, and agree that R1 is really neat, but am skeptical of how much it really should cause us to update, in the scheme of things. 

No MCTS, no PRM...

scaling up CoT with simple RL and scalar rewards...

emergent behaviour

Bringing in a quote from Twitter/x: (Not my viewpoint, just trying to broaden the discussion.)

https://x.com/DrJimFan/status/1882799254957388010

Jim Fan @DrJimFan Whether you like it or not, the future of AI will not be canned genies controlled by a "safety panel". The future of AI is democratization. Every internet rando will run not just o1, but o8, o9 on their toaster laptop. It's the tide of history that we should surf on, not swim against. Might as well start preparing now.

DeepSeek just topped Chatbot Arena, my go-to vibe checker in the wild, and two other independent benchmarks that couldn't be hacked in advance (Artificial-Analysis, HLE).

Last year, there were serious discussions about limiting OSS models by some compute threshold. Turns out it was nothing but our Silicon Valley hubris. It's a humbling wake-up call to us all that open science has no boundary. We need to embrace it, one way or another.

Many tech folks are panicking about how much DeepSeek is able to show with so little compute budget. I see it differently - with a huge smile on my face. Why are we not happy to see improvements in the scaling law? DeepSeek is unequivocal proof that one can produce unit intelligence gain at 10x less cost, which means we shall get 10x more powerful AI with the compute we have today and are building tomorrow. Simple math! The AI timeline just got compressed.

Here's my 2025 New Year resolution for the community:

No more AGI/ASI urban myth spreading. No more fearmongering. Put our heads down and grind on code. Open source, as much as you can.

Acceleration is the only way forward.

During the training process, we observe that CoT often exhibits language mixing, particularly when RL prompts involve multiple languages. To mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT. Although ablation experiments show that such alignment results in a slight degradation in the model’s performance, this reward aligns with human preferences, making it more readable.

I also found this trade-off between human readability and performance noteworthy.

↑ comment by Milan W (weibac) · 2025-01-23T16:48:57.371Z · LW(p) · GW(p)

Side note: Claude 3.5 Sonnet does CoT language-mixing after a bit of prompting and convincing. I'm not sure about effects on performance. Also the closeness narratively implied by having it imitate the idiosyncratic mixture I was using to talk to it probably exacerbated sycophancy.
 

I don't think Phi-4 offers convincing evidence either way. You can push performance on verifiable tasks quite far without the model becoming generally more capable. AlphaZero doesn't imply that scaling with its methods gestures at general superintelligence, and similarly with Phi-4.

In contrast, using o1-like training as a way to better access ground truth in less tractable domains seems more promising, since by some accounts its tactics on long reasoning traces work even in non-technical domains (unlike for DeepSeek R1), possibly because they are emergent rather than directly encouraged with task-specific training.

Phi-4 is highly capable not despite but because of synthetic data.

Imitation models tend to be quite brittle outside of their narrowly imitated domain, and I suspect the same to be the case for phi-4. Some of the decontamination measures they took provide some counter evidence to this but not much. I'd update more strongly if I saw results on benchmarks which contained in them the generality and diversity of tasks required to do meaningful autonomous cognitive labour "in the wild", such as SWE-Bench (or rather what I understand SWE-Bench to be, I have yet to play very closely with it). 

Phi-4 is taught by GPT-4; GPT-5 is being taught by o1; GPT-6 will teach itself.

There's an important distinction between utilizing synthetic data in teacher-student setups and utilizing synthetic data in self-teaching. While synthetic data is a demonstrably powerful way of augmenting human feedback, my current estimation is that typical mode collapse arguments still hold for self generated purely synthetic datasets, and that phi-4 doesn't provide counter-evidence against this.

This is neat, thanks for highlighting.

>The implication: If you don't have access to a 2024-frontier AI, you're going to have a hard time training the next frontier model. That gap will likely widen with each subsequent iteration. 

This doesn't seem super clear to me. Without synthetic data, you need to scrape large parts of the web and manage a lot of storage infrastructure. This can either be done illegally, or with complex negotiations (especially as companies are catching on to this). 

In comparison, it would be very useful if you could train a near-SOTA model with just synthetic data, say from an open-source model. This might not bring you all the way to SOTA, but close might be good enough for many things. 

It looks like recursive self-improvement is here for the base case, at least. It will be interesting to see if anyone uses solely Phi-4 to pretrain a more capable model.

I think this is important because the safety community still isn't thinking very much about search & RL, even after all the recent progress with reasoning models. We've updated very far away from AlphaZero as a reference class, and I think we will regret this.

On the other hand, the ideas I'm talking about here seem to have widespread recognition among people working on capabilities. Demis is very transparent about where they're headed with language models, AlphaZero, and open-ended exploration (e.g., at 20:48). Noam Brown is adamant about test-time scaling/reasoning being the future (e.g., at 20:32). I think R1 has driven the message home for everyone else.

I propose: the best planners must break the beta.

Because if a planner is going to be the best, it needs to be capable of finding unusual (better!) plans. If it's capable of finding those, there's ~no benefit of knowing the conventional wisdom about how to do it (climbing slang: beta). 

Edit: or maybe: good planners don't need beta?

The Bitter Lesson is pretty on point but you could call it "Bootstrapping from Zero", the "Autodidactic Leap", the "Self-Discovery Transition", or "Breaking the Imitation Ceiling" if you prefer.

I don't think I get what phenomenon you're pointing to.

Your first bullet point makes it sound like AlphaGo wasn't trained using self-play, in contrast to AlphaZero. However, AlphaGo was trained with a combination of supervised learning and self-play. They removed the supervised learning part from AlphaZero to make it simpler and more general.

DreamerV3 also fits the pattern where previous SOTA approaches used a combination of imitation learning and reinforcement learning, while DreamerV3 was able to remove the imitation learning part.^[1]

To my understanding, AlphaProof was trained by translating a bunch of math problems to Lean, and using "correct proof" as reward for AlphaZero. This approach also combines human data (our math problems) with reinforcement learning (AlphaZero).

Your final example feels close to AlphaProof if you finish it with "Then you finetune CoT with reinforcement learning to yield impressive performance on reasoning benchmarks", but I don't think that's what you were going for.

The first two examples seem covered by "when reinforcement learning works well, imitation learning is no longer needed". Idk about the rest.

Could you clarify by giving more examples or otherwise explain what you're looking for?

1. ^{^}

   I got curious how DreamerV3 figures out Minecraft with nothing to imitate and no intermediate reward, so I checked the paper. There are intermediate rewards. They give +1 reward for each of 12 ordered milestones leading up to the diamond, and -0.01 for each lost heart and +0.01 for each restored heart. Additionally, they use "the block breaking setting of prior work[19] because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time". So to get started, probably the agent manages to randomly break a tree block and get its first reward.

↑ comment by Jesse Hoogland (jhoogland) · 2025-02-10T16:32:42.856Z · LW(p) · GW(p)

With AlphaProof, the relevant piece is that the solver network generates its own proofs and disproofs to train against. There's no imitation learning after formalization. There is a slight disanalogy where, for formalization, we mostly jumped straight to self-play/search, and I don't think there was ever a major imitation-learning-based approach (though I did find at least one example). 

Your quote "when reinforcement learning works well, imitation learning is no longer needed" is pretty close to what I mean. What I'm actually trying to get at is a stronger statement: we often bootstrap using imitation learning to figure out how to get the reinforcement learning component working initially, but once we do, we can usually discard the imitation learning entirely.

Nitpick: first alphago was trained by a combination of supervised learning from human expert games and reinforcement learning from self-play. Also, Ke Jie was beaten by AlphaGo Master which was a version at a later stage of development.

You are skipping over a very important component: Evaluation. 

Which is exactly what we don't know how to do well enough outside of formally verifiable domains like math and code, which is exactly where o1 shows big performance jumps. 

So let's call "reasoning models" like o1 what they really are: the first true AI agents.

I think the distinction between systems that perform a single forward pass and then stop and systems that have an OODA loop (tool use) is more stark than the difference between "reasoning" and "chat" models, and I'd prefer to use "agent" for that distinction.

I do think that "reasoning" is a bit of a market-y name for this category of system though. "chat" vs "base" is a great choice of words, and "chat" is basically just a description of the RL objective those models were trained with.

If I were the terminology czar, I'd call o1 a "task" model or a "goal" model or something.

Marcus Hutter on AIXI and ASI safety