LessWrong 2.0 Reader

There's been a widespread assumption that training reasoning models like o1 or r1 can only yield improvements on tasks with an objective metric of correctness, like math or coding. See this essay, for example, which seems to take as a given that the only way to improve LLM performance on fuzzy tasks like creative writing or business advice is to train larger models.

This assumption confused me, because we already know how to train models to optimize for subjective human preferences. We figured out a long time ago that we can train a reward model to emulate human feedback and use RLHF to get a model that optimizes this reward. AI labs could just plug this into the reward for their reasoning models, reinforcing the reasoning traces leading to responses that obtain higher reward. This seemed to me like a really obvious next step.

Well, it turns out that DeepSeek r1 actually does this. From their paper:

2.3.4. Reinforcement Learning for all Scenarios

To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities. Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the assessment emphasizes the utility and relevance of the response to the user while minimizing interference with the underlying reasoning process. For harmlessness, we evaluate the entire response of the model, including both the reasoning process and the summary, to identify and mitigate any potential risks, biases, or harmful content that may arise during the generation process. Ultimately, the integration of reward signals and diverse data distributions enables us to train a model that excels in reasoning while prioritizing helpfulness and harmlessness.

This checks out to me. I've already noticed that r1 feels significantly better than other models at creative writing, which is probably due to this human preference training. While o1 was no better at creative writing than other models, this might just mean that OpenAI didn't prioritize training o1 on human preferences. My Manifold market currently puts a 65% chance on chain-of-thought training outperforming traditional LLMs by 2026, and it should probably be higher at this point.

We need to adjust our thinking around reasoning models - there's no strong reason to expect that future models will be much worse at tasks with fuzzy success criteria.

Adapted from my previously-posted question [LW · GW], after cubefox [LW · GW] pointed out that DeepSeek is already using RLHF.

If you want to understand why a model, any model, did something, you presumably want a verbal explanation of its reasoning, a chain of thought. E.g. why AlphaGo made its famous unexpected move 37. That's not just true for language models.

I think this is correct, but doesn't seem to note the broader trend towards human disempowerment in favor of bureaucratic and corporate systems, which this gradual disempowerment would continue, and hence elides or ignores why AI risk is distinct.

I think it's actually not any less true of o1/r1.

I think I’ll duck out of this discussion because I don’t actually believe that o1/r1 will lead to full-fledged (1-3) loops and AGI, so it’s hard for me to clearly picture that scenario and engage with its consequences.

I don't think AI taste should play a role in AI help solving the value alignment problem. If we had any sense (which sometimes we do once problems are right in our faces), we'd be asking the AI "so what happens if we use this alignment approach/goal?" and then using our own taste, not asking it things like "tell us what to do with our future". We could certainly ask for input and there are ways that could go wrong. But I mostly hope for AGI help in the technical part of solving stable value alignment.

Hmm. But the AI has a ton of wiggle room to make things seem good or bad depending on how things are presented and framed, right? (This old Stuart Armstrong post [LW · GW] is a bit relevant.) If I ask “what will happen if we do X”, the AI can answer in a way that puts things in a positive light, or a negative light. If the good understanding lives in the AI and the good taste lives in the human, then it seems to me that nobody is at the wheel. The AI taste is determining what gets communicated to the human and how, right? What’s relevant vs irrelevant? What analogies are getting at what deeply matters versus what analogies are superficial? All these questions are value-laden, but they are prerequisites to the AI communicating its understanding to the human. Remember, the AI is doing the (1-3) thing to autonomously develop a new idiosyncratic superhuman understanding of AI and philosophy and society and so on, by assumption. Thus, AI-human communication is much harder and different than we’re used to today, and presumably requires its own planning and intention on the part of the AI.

…Unless you’re actually in the §5.1.1 camp where the AI is helping clarify and brainstorm but is working shoulder-to-(virtual) shoulder, and the human basically knows everything the AI knows. I.e., like how people use foundation models today. If so, that’s fine, no complaints. I’m happy for people to use foundation models in a similar way that they do today, as they work on the big problem of how to make future more powerful AIs that run on something closer to ambitious value learning or CEV as opposed to corrigibility / obedience.

Sorry if I’m misunderstanding or being stupid, this is an area where I feel some uncertainty.  :)

Related:

https://x.com/mkurman88/status/1885042970447015941

https://x.com/WenhuChen/status/1885060597500567562

Yeah, but there are probably other interesting takeaways

Your first paragraph makes sense as an interpretation, which I discounted because the idea of something like AlphaGo doing CoT (or applying a CoT to it) seems so nonsensical, since it's not at all a linguistic model.

I'm having more trouble seeing how to read what Chalmer says in the way your second paragraph suggests -- eg 'unmoored from the original system' doesn't seem like it's talking about the same system generating an ad hoc explanation. It's more like he's talking about somehow taking a CoT generated by one model and applying it to another, although that also seems nonsensical.

Great comment, and I will have to think more about this. Your examples do seem to support the utility of self-identity-based motivation.

I think maybe my statement "you can’t lie to yourself if you know it’s a lie" is forcing a frame where self-talk is either a genuine attempt at truth, or a lie. But with "this is easy for me because I’m a fighter" and similar statements, it seems they can be received by the mind in a different way - more like as self-fulfilling prophecy.

I guess it's an open question for me then, where to use that kind of self-talk. On one end is the danger of becoming miserable in pursuit of an identity that was actually kind of arbitrary, and on the other end you miss out on maybe the most powerful kind of motivation.

If RL becomes the next thing in improving LLM capabilities, one thing that I would bet on becoming big is computer-use in 2025. Seems hard to get more intelligence with just RL (who verifies the outputs?), but with something like computer use, it's easy to verify if a task has been done (has the email been sent, ticket been booked etc..) that it's starting to look to more to me like it can do self-learning.

One thing that's left AI still fully not integrated into the rest of the economy is simply that the current interfaces were built for humans and moving all those takes engineering time / effort etc.

I'm fairly sure the economic disruption would be pretty quick once this happens. For example, I can just run 10 LLM agents to act as customer service agents using my *existing tools* - just open emails, whatsapp, and message customers, check internal dashboards etc., then it's game over. What's stopping people right now is that there's not enough people to build that pipeline fast enough to utilize even the current capabilities.

then I think it is also very questionable whether the AI that wins wars is the most "advanced" AI. / People like Dario whose bread-and-butter is model performance invariably over-index on model performance, especially on benchmarks. But practical value comes from things besides the model; what tasks you use it for and how effective you are at deploying it.

Dario is about the last AI CEO you should be making this criticism of. Claude has been notable for a while for the model which somehow winds up being the most useful and having the best 'vibes', even when the benchmarks indicate it's #2 or #3; and meanwhile, it is the Chinese models which historically regress the most from their benchmarks when applied (and DeepSeek models, while not as bad as the rest, still do this and r1 is already looking shakier as people try out heldout problems or benchmarks).