DPO/PPO-RLHF on LLMs incentivizes sycophancy, exaggeration and deceptive hallucination, but not misaligned powerseeking

I think this is similar to the conclusion I reached in §5.1 of “Thoughts on ‘Process-Based Supervision’” [LW · GW]. Admittedly it did take many patient attempts for Holden to get that point through my thick skull ¯\_(ツ)_/¯ So it’s nice that you wrote this.

I do however think there are reasons for concern if you keep the setup (“incentives”) the same allow but arbitrarily complicated Π, including much more complicated than an LLM (e.g., instead of “the AI does one forward pass through GPT-4 and then prints a character”, replace it with “the AI does arbitrary opaque brainstorming for six hours with a giant hidden scratchpad and free access to a cached copy of the internet, and then it prints a message”)—see §5.2 of that same post [LW · GW] for that argument.

For example, it’s conceivable that an AI can pull off a treacherous turn on its first try, and if you’re trying to reason about whether that’s gonna happen or not, you can’t straightforwardly do so via your Π* expression—(1) the treacherous-turn output is very unlike any of the AI’s previous outputs (by assumption), and (2) as soon as the AI emits such an output, it’s game over. Again see that post [LW · GW] for my (somewhat casual and intuitive) attempt to reason about that possibility.

I think that "behaviorist" interpretation of RL (that you "reinforce" behavior) is wrong in general and especially wrong in case of RLHFing LLMs. Instead of thinking about "reinforcing behavior" you should think about "reinforcing algorithms that contribute to behavior". The consequence of this is following:

1. You have base model which is trained on bazillion texts, which include, say, deceptive behavior and, correspondingly, algorithms for deceptive behavior
2. You fine-tune model on "good" completions
3. But "good" completions can be produced by both "good" algorithms and "bad-but-pretending-to-be-good" algorithms, so both types of algorithms get reinforced
4. What's important, it doesn't depend on whether evaluator did good job. Perfect deceiver, by definition, produces the same answer as good honest agent (before deployment), so in the end odds ratio between good honest agent and perfect deceiver stays the same (modulo quirks in LLM cognition), while everything else is negatively reinforced.

Evidence:

https://arxiv.org/abs/2311.07590

If you look at figure 3, you will find that RLHFed model is the most likely to deceive. I think it is not because somebody rewarded it for deception in similar conditions but because the very process of RLHF puts deceptive algorithms as the second-most-reinforced in LLM.

https://www.anthropic.com/research/many-shot-jailbreaking

Look at graph "Malicious use cases", deception is fastest to be elicited. Also note that x-axis is in-log scale, so generally deception-jailbreaking is approximately 30% faster.

https://arxiv.org/abs/2311.12786

(i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability

I think this supports my hypothesis that RLHF is "reweighting" of existing algorithms instead of writing algorithms into network from scratch. (If somebody finds similar paper on RLHF it would be great.)

Sooooo what does it say about RLHF incentivizing power-seeking?

It depends on:

1. Whether base model has power-seeking algorithms
2. Whether it is likely for power-seeking algorithms to contribute to "correct" answers in RLHF-finetuning

What excellent questions, I hope interpretability will help us answer them.

And, as Chris_Leong noted, it is unlikely that many details of current RLHF will still be here during training of superintelligences.

As far as I understand, in RLHF, PPO/DPO doesn't directly use preferences from human raters, but instead synthetic preference data generated by a reward model. The reward model in turn is trained on preference data given by actual human raters. The reward model may be misgeneralizing this data, in which case the DPO input may include preferences that humans wouldn't give. Which might change your conclusion.

Seems like at some point we’ll need to train on outputs too complex for humans to evaluate, then we’ll end up using training methods based on outcomes in some simulation.