When is reward ever the optimization target?

I love this question! As it happens, I have some rough draft for a post titled something like "'reward is the optimization target for smart RL agents".

TLDR: I think this is true for some AI systems, but not likely true for any RL-directed AGI systems whose safety we should really worry about. They'll optimize for maximum reward even more than humans do, unless they're very carefully built to avoid that behavior.

In the final comment [LW(p) · GW(p)] on the second thread you linked [LW(p) · GW(p)], TurnTrout says of his Reward is not the optimization target [LW · GW]:

However, I should have stated up-front: This post addresses model-free policy gradient algorithms like PPO and REINFORCE. 

Humans are definitely model-based RL learners at least some of the time - particularly for important decisions.^[1] So the claim doesn't apply to them. I also don't think it applies to any other capable agent. TurnTrout actually makes a congruent claim in his other post Think carefully before calling RL policies "agents" [LW · GW]. Model-free RL algorithms only have limited agency, what I'd call level 1-of-3:

1. Trained to achieve some goal/reward.
   1. Habitual behavior/model-free RL
2. Predicts outcomes of actions and selects ones that achieve a goal/reward.
   1. Model-based RL
3. Selects future states that achieve a goal/reward and then plans actions to achieve that state
   1. No corresponding terminology, (goal-directed from neuroscience applies to levels 2 and even 1^[1]) but pretty clearly highly useful for humans

That's from my post Steering subsystems: capabilities, agency, and alignment [LW · GW].  

But humans don't seem to optimize for reward all that often! They make self-sacrificial decisions that get them killed. And they usually say they'd refuse to get in Nozick's experience machine, which would hypothetically remove them from this world and give them a simulated world of maximally-rewarding experiences. They're seeming to optimize for the things that have given them reward, like protecting loved ones, rather than optimizing for reward themselves - just like TurnTrout describes in RINTOT. And humans are model-based for important decisions, presumably using sophisticated models. What gives?

My cognitive neuroscience research focused a lot on dopamine, so I've thought a lot about how reward shapes human behavior. The most complete publication is Neural mechanisms of human decision-making as a summary of how humans seem to learn complex behaviors using reward and predictions of reward. But that's not really very good description of the overall theory, because neuroscientists are highly suspicious of broad theories, and because I didn't really want to accidentally accelerate AGI research by describing brain function clearly. I know.

I think humans do optimize for reward, we just do it badly. We do see some sophisticated hedonists with exceptional amounts of time and money say things like "I love new experiences". This has abstracted almost all of the specifics. Yudkowsky's "fun theory" also describes a pursuit of reward if you grant that "fun" refers to frequent, strong dopamine spikes (I think that's exactly what we mean by fun). I think more sophisticated hedonists will get in the experience box- but this is complicated by the approximations in human decision-making. It's pretty likely that the suffering you'd cause your loved ones by getting in the box and leaving them alone would be so salient, and produce such a negative-reward-prediction, that it would outweigh all of the many positive predictions of reward, just based on saliency and our inefficient way of roughly totaling predicted future reward by imagining salient outcomes and roughly averaging over their reward predictions.

So I think the more rational and cognitively capable a human is, the more likely they'll optimize more strictly and accurately for future reward. And I think the same is true of model-based RL systems with any decent decision-making process.

I realize this isn't the empirically-based answer you asked for. I think the answer has to be based on theory, because some systems will and some won't optimize for reward. I don't know the ML RL literature nearly as well as I know the neuroscience RL literature, so there might be some really relevant stuff out there I'm not aware of. I doubt it, because this is such an AI-safety question.^[2]

So that's why I think reward is the optimization target for smart RL agents.

Edit: Thus, RINTOT and similar work has, I think, really confused the AGI safety debate by making strong claims about current AI that don't apply at all to the AGI we're worried about. I've been thinking about this a lot in the context of a post I'd call "Current AI and alignment theory is largely behaviorist. Expect a cognitive revolution".

1. ^{^}

   For more than you want to know about the various terminologies, see How sequential interactive processing within frontostriatal loops supports a continuum of habitual to controlled processing.

   We debated the terminologies habitual/goal-directed, automatic and controlled, system 1/system 2, and model-free/model-based for years. All of them have limitations, and all of them mean slightly different things. In particular, model-based is vague terminology when systems get more complex than simple RL - but it is very clear that many complex human decisions (certainly ones in which we envision possible outcomes before taking actions) are far on the model-based side, and meet every definition. 

2. ^{^}

   One follow-on question is whether RL-based AGI will wirehead. I think this is almost the same question as getting into the experience box - except that that box will only keep going if the AGI engineers it correctly to keep going. So it's going to have to do a lot of planning before wireheading, unless its decision-making algorithm is highly biased toward near-term rewards over long-term ones. In the course of doing that planning, its other motivations will come into play - like the well-being of humans, if it cares about that. So whether or not our particular AGI will wirehead probably won't determine our fate.

"Optimization target" is itself a concept which needs deconfusing/operationalizing. For a certain definition of optimization and impact, I've found that the optimization is mostly correlated with reward, but that the learned policy will typically have more impact on the world/optimize the world more than is strictly necessary to achieve a given amount of reward.

This uses an empirical metric of impact/optimization which may or may not correlate well with algorithm-level measures of optimization targets.

https://www.alignmentforum.org/posts/qEwCitrgberdjjtuW/measuring-learned-optimization-in-small-transformer-models [AF · GW]

I'll use the definition of optimization from Wikipedia: "Mathematical optimization is the selection of a best element, with regard to some criteria, from some set of available alternatives".

Best-of-n or rejection sampling is an alternative to RLHF which involves generating $n$ responses from an LLM and returning the one with the highest reward model score. I think it's reasonable to describe this process as optimizing for reward because its searching for LLM outputs that achieve the highest reward from the reward model.

I'd also argue that AlphaGo/AlphaZero is optimizing for reward. In the AlphaGo paper it says, "At each time step $t$ of each simulation, an action $a_t$ is selected from state $s_t$ so as to maximize action value plus a bonus" and the formula is: $a=argma{x}_a\left(Q\right(s_t,a)+u(s_t,a\left)\right)$ where $u$ is an exploration bonus.

Action values Q are calculated as the mean value (estimated probability of winning) of all board states in the subtree below an action. The value of each possible future board state is calculated using a combination of a value function estimation for that state and the mean outcome of dozens of random rollouts until the end of the game (return +1 or -1 depending on who wins).

The value function predicts the return (expected sum of future reward) from a position whereas the random rollouts are calculating the actual average reward by simulating future moves until the end of the game when the reward function return +1 or -1.

So I think AlphaZero is optimizing for a combination of predicted reward (from the value function) and actual reward which is calculated using multiple rollouts until the end of the game.