When is reward ever the optimization target?

BTW, another problem with the thesis "Reward is not the optimization target", even with TurnTrout's stipulation that

This post addresses the model-free policy gradient setting, including algorithms like PPO and REINFORCE.

is that it's still not true even in the model-free policy gradient setting, in any substantive sense, and cannot justify some of the claims that TurnTrout & Belrose make. That is because of meta-learning: the 'inner' algorithm may in fact be a model-based RL algorithm which was induced by the 'outer' algorithm of PPO/REINFORCE/etc. A model-free algorithm may learn something which optimizes the reward; and a model-based algorithm may also learn something which does not optimize the reward.

There is just no hard and fast distinction here, it is dependent on the details of the system, the environment (ie. distribution of data), the amount of compute/ data, the convergence, and so on.** (A good example from an earlier comment on how reward is the optimization target [LW(p) · GW(p)] is Bhoopchand et al 2023, which is about ablating the components.)

So if the expressible set of algorithms is rich enough to include model-based RL algorithms, and there are sufficient conditions, then your PPO algorithm 'which doesn't optimize the reward' simply learns an algorithm which does optimize the reward.

A simple, neat example is given by Botvinick [LW · GW] about NNs like RNNs. (As Shah notes in the comments, this is all considered basic RL and not shocking, and there are many examples of this sort of thing in meta-RL research - although what perspective you take on what is 'outer'/'inner' is often dependent on what niche you are in, and so Table 1 here may be a helpful Rosetta stone.)

You have a binary choice (like a bandit) which yields a 0/1 reward (perhaps stochastic with probability p to make it interesting) and your NN learns which one; you train a, let's say, fully-connected MLP with REINFORCE, which takes no input and outputs a binary variable to choose an arm; it learns that the left arm yields 1 reward and to always take left. You stop training it, and the environment changes to swap it: the left arm yields 0, and now right yields 1. The MLP will still pick 'left', however, because it learned a policy which doesn't try to optimize the reward. In this case, it is indeed the case that "reward is not the optimization target" of the MLP. It just learned a myopic action which happened to be selected for. In fact, even if you resume training it, it may take a long time to learn to instead pick 'right', because you have to 'undo' all of the now-irrelevant training towards 'left'. And you can do this swapping and training several times, and it'll be about the same each time: the MLP will slowly unlearn the old arm and learn the new arm, then the swapping happens, and now it's gotta do the same thing.*

But if you instead train an RNN, and to give it an input, you feed it a history of rewards and you otherwise train the exact same way... You will instead see something entirely different. After a swap, the RNN will pick the 'wrong' arm a few times, say 5 times, and receive 0 reward - and abruptly start picking the 'right' arm even without any further training, just the same frozen RNN weights. This sort of fast response to changing rewards is a signature of model-free vs model-based: if I tell you that I moved your cheese, you can change your policy without ever experiencing a reward, and go to where the cheese is now, without wasting an attempt on the old cheese location; but a mouse can't, or will need at least a few episodes of trial-and-error to update. (Any given agent may use a mix, or hybrids like 'successor representation' which is sorta both; Sutton is fond of that.) This switch is possible because it has learned a new 'policy' over its history and the sufficient statistics encoded into its hidden weights which is equivalent to a Bayesian model of the environment and where it has learned to update its posterior probability of a switch having happened and that it is utility-maximizing to, after a certain number of failures, switch. And that 5 times was just how much evidence you need to overcome the small prior of 'a switch just happened right now'. And this utility-maximizing inner algorithm is incentivized by the outer algorithm, even though the outer algorithm itself has no concept of an 'environment' to be modeling or a 'reward' anywhere inside it. Your 'reward is not the optimization target' REINFORCE algorithm has learned the Bayesian model-based RL algorithm for which reward is the optimization target, and your algorithm as a whole is now optimizing the reward target, little different from, say, AlphaZero doing a MCTS tree search.

(And this should not be a surprise, because evolutionary algorithms are often cited as examples of model-free policy gradient algorithms which cannot 'plan' or 'model the environment' or 'optimize the reward', and yet, we humans were created by evolution and we clearly do learn rich models of the environment that we can plan over explicitly to maximize the reward, such as when we play Go and 'want to win the game'. So clearly the inference from 'algorithm X is itself not optimizing the reward' to 'all systems learned by algorithm X do not optimize the reward' is an illicit one.)

And, of course, in the other direction, it is entirely possible and desirable for model-based algorithms to learn model-free ones! (It's meta-learning all the way down.) Planning is expensive, heuristics cheap and efficient. You often want to distill your expensive model-based algorithm into a cheap model-free algorithm [LW(p) · GW(p)] and amortize the cost. In the case of the RNN above, you can, after the model-based algorithm has done its work and solved the switching bandit, throw that away, and replace it by a much cheaper simple model-free algorithm like if sum(reward_history) > 5 then last_action else last_action × −1, saving millions of FLOPs per decision compared to the RNN. You can see the Baldwin effect as a version of this: there is no need to relearn the same behavior within-lifetime if it's the same each time, you can just hardwire it into the genes - no matter how sample-efficient model-based RL is within-lifetime, it can't beat a prior hardwired strategy requiring 0 data.

* further illustrating the weakness of 'reward is not the optimization target', it's not even obvious that this must be the case, rather than usually is under most setups. Meta-learning doesn't strictly require explicit conditioning on history nor does it require the clear fast vs slow weight distinction of RNNs or Transformer self-attention. A large enough MLP, continually trained through enough switches, could potentially learn to use the gradient updates+weights themselves as an outsourced history/model, and could eventually optimize its weights into a saddle point, where after exactly k updates by the fixed SGD algorithm, it 'happens to' switch its choice, corresponding to the hidden state being fused into the MLP itself. This would be like MAML. (If you are interested in this vein of thought, you may enjoy my AUNN proposal, which tries to take this to the logical extreme.)

** As Turntrout correctly notes [LW · GW], a GPT-5 could well be agentic, even if it was not trained with 'PPO' or called 'RL research'. What makes an RL agent is not the use of any one specific algorithm or tool, as they are neither necessary nor sufficient, it is the outcome. He asks, what does or does not make a Transformer agentic when we train it on OpenWebtext with a cross-entropy predictive loss, while PPO is assumed to be agentic? The actual non-rhetorical answer is: there is no single reason and little difference between PPO and cross-entropy here; in both cases, it is the combined outcome of the richness of the agent-generated data in OWT combined with the richness of a sufficiently large compute & parameter budget, which will tend to yield agency by learning to imitate the agents which generated that data. A smaller Transformer, a Transformer trained with less compute, or less data, or with text data from non-agents (eg. randomly initialized n-grams), would not yield agency, and further, there will be scaling laws/regimes for all of these critical ingredients. Choke any one of them hard enough, and the agency goes away. (You will instead get a LLM which is able to only predict 'e', or which perfectly models the n-grams and nothing else, or which predicts random outputs, or which asymptotes hard etc.)

I love this question! As it happens, I have some rough draft for a post titled something like "'reward is the optimization target for smart RL agents".

TLDR: I think this is true for some AI systems, but not likely true for any RL-directed AGI systems whose safety we should really worry about. They'll optimize for maximum reward even more than humans do, unless they're very carefully built to avoid that behavior.

In the final comment [LW(p) · GW(p)] on the second thread you linked [LW(p) · GW(p)], TurnTrout says of his Reward is not the optimization target [LW · GW]:

However, I should have stated up-front: This post addresses model-free policy gradient algorithms like PPO and REINFORCE. 

Humans are definitely model-based RL learners at least some of the time - particularly for important decisions.^[1] So the claim doesn't apply to them. I also don't think it applies to any other capable agent. TurnTrout actually makes a congruent claim in his other post Think carefully before calling RL policies "agents" [LW · GW]. Model-free RL algorithms only have limited agency, what I'd call level 1-of-3:

1. Trained to achieve some goal/reward.
   1. Habitual behavior/model-free RL
2. Predicts outcomes of actions and selects ones that achieve a goal/reward.
   1. Model-based RL
3. Selects future states that achieve a goal/reward and then plans actions to achieve that state
   1. No corresponding terminology, (goal-directed from neuroscience applies to levels 2 and even 1^[1]) but pretty clearly highly useful for humans

That's from my post Steering subsystems: capabilities, agency, and alignment [LW · GW].  

But humans don't seem to optimize for reward all that often! They make self-sacrificial decisions that get them killed. And they usually say they'd refuse to get in Nozick's experience machine, which would hypothetically remove them from this world and give them a simulated world of maximally-rewarding experiences. They're seeming to optimize for the things that have given them reward, like protecting loved ones, rather than optimizing for reward themselves - just like TurnTrout describes in RINTOT. And humans are model-based for important decisions, presumably using sophisticated models. What gives?

My cognitive neuroscience research focused a lot on dopamine, so I've thought a lot about how reward shapes human behavior. The most complete publication is Neural mechanisms of human decision-making as a summary of how humans seem to learn complex behaviors using reward and predictions of reward. But that's not really very good description of the overall theory, because neuroscientists are highly suspicious of broad theories, and because I didn't really want to accidentally accelerate AGI research by describing brain function clearly. I know.

I think humans do optimize for reward, we just do it badly. We do see some sophisticated hedonists with exceptional amounts of time and money say things like "I love new experiences". This has abstracted almost all of the specifics. Yudkowsky's "fun theory" also describes a pursuit of reward if you grant that "fun" refers to frequent, strong dopamine spikes (I think that's exactly what we mean by fun). I think more sophisticated hedonists will get in the experience box- but this is complicated by the approximations in human decision-making. It's pretty likely that the suffering you'd cause your loved ones by getting in the box and leaving them alone would be so salient, and produce such a negative-reward-prediction, that it would outweigh all of the many positive predictions of reward, just based on saliency and our inefficient way of roughly totaling predicted future reward by imagining salient outcomes and roughly averaging over their reward predictions.

So I think the more rational and cognitively capable a human is, the more likely they'll optimize more strictly and accurately for future reward. And I think the same is true of model-based RL systems with any decent decision-making process.

I realize this isn't the empirically-based answer you asked for. I think the answer has to be based on theory, because some systems will and some won't optimize for reward. I don't know the ML RL literature nearly as well as I know the neuroscience RL literature, so there might be some really relevant stuff out there I'm not aware of. I doubt it, because this is such an AI-safety question.^[2]

So that's why I think reward is the optimization target for smart RL agents.

Edit: Thus, RINTOT and similar work has, I think, really confused the AGI safety debate by making strong claims about current AI that don't apply at all to the AGI we're worried about. I've been thinking about this a lot in the context of a post I'd call "Current AI and alignment theory is largely behaviorist. Expect a cognitive revolution".

1. ^{^}

   For more than you want to know about the various terminologies, see How sequential interactive processing within frontostriatal loops supports a continuum of habitual to controlled processing.

   We debated the terminologies habitual/goal-directed, automatic and controlled, system 1/system 2, and model-free/model-based for years. All of them have limitations, and all of them mean slightly different things. In particular, model-based is vague terminology when systems get more complex than simple RL - but it is very clear that many complex human decisions (certainly ones in which we envision possible outcomes before taking actions) are far on the model-based side, and meet every definition. 

2. ^{^}

   One follow-on question is whether RL-based AGI will wirehead. I think this is almost the same question as getting into the experience box - except that that box will only keep going if the AGI engineers it correctly to keep going. So it's going to have to do a lot of planning before wireheading, unless its decision-making algorithm is highly biased toward near-term rewards over long-term ones. In the course of doing that planning, its other motivations will come into play - like the well-being of humans, if it cares about that. So whether or not our particular AGI will wirehead probably won't determine our fate.

I'll use the definition of optimization from Wikipedia: "Mathematical optimization is the selection of a best element, with regard to some criteria, from some set of available alternatives".

Best-of-n or rejection sampling is an alternative to RLHF which involves generating $n$ responses from an LLM and returning the one with the highest reward model score. I think it's reasonable to describe this process as optimizing for reward because its searching for LLM outputs that achieve the highest reward from the reward model.

I'd also argue that AlphaGo/AlphaZero is optimizing for reward. In the AlphaGo paper it says, "At each time step $t$ of each simulation, an action $a_t$ is selected from state $s_t$ so as to maximize action value plus a bonus" and the formula is: $a=argma{x}_a\left(Q\right(s_t,a)+u(s_t,a\left)\right)$ where $u$ is an exploration bonus.

Action values Q are calculated as the mean value (estimated probability of winning) of all board states in the subtree below an action. The value of each possible future board state is calculated using a combination of a value function estimation for that state and the mean outcome of dozens of random rollouts until the end of the game (return +1 or -1 depending on who wins).

The value function predicts the return (expected sum of future reward) from a position whereas the random rollouts are calculating the actual average reward by simulating future moves until the end of the game when the reward function return +1 or -1.

So I think AlphaZero is optimizing for a combination of predicted reward (from the value function) and actual reward which is calculated using multiple rollouts until the end of the game.

"Optimization target" is itself a concept which needs deconfusing/operationalizing. For a certain definition of optimization and impact, I've found that the optimization is mostly correlated with reward, but that the learned policy will typically have more impact on the world/optimize the world more than is strictly necessary to achieve a given amount of reward.

This uses an empirical metric of impact/optimization which may or may not correlate well with algorithm-level measures of optimization targets.

https://www.alignmentforum.org/posts/qEwCitrgberdjjtuW/measuring-learned-optimization-in-small-transformer-models [AF · GW]