At some level I agree with this post---policies learned by RL are probably not purely described as optimizing anything. I also agree that an alignment strategy might try to exploit the suboptimality of gradient descent, and indeed this is one of the major points of discussion amongst people working on alignment in practice at ML labs. 

However, I'm confused or skeptical about the particular deviations you are discussing and I suspect I disagree with or misunderstand this post.

As you suggest, in deep RL we typically use gradient descent to find policies that achieve a lot of reward (typically updating the policy based on an estimator for the gradient of the reward).

If you have a system with a sophisticated understanding of the world, then cognitive policies like "select actions that I expect would lead to reward" will tend to outperform policies like "try to complete the task," and so I usually expect them to be selected by gradient descent over time. (Or we could be more precise and think about little fragments of policies, but I don't think it changes anything I say here.)

It seems to me like you are saying that you think gradient descent will fail to find such policies because it is greedy and local, e.g. if the agent isn't thinking about how much reward it will receive then gradient descent will never learn policies that depend on thinking about reward.

(Though I'm not clear on how much you are talking about the suboptimality of SGD, vs the fact that optimal policies themselves do not explicitly represent or pursue reward given that complex stews of heuristics may be faster or simpler. And it also seems plausible you are talking about something else entirely.)

I generally agree that gradient descent won't find optimal policies. But I don't understand the particular kinds of failures you are imagining or why you think they change the bottom line for the alignment problem. That is, it seems like you have some specific take on ways in which gradient descent is suboptimal and therefore how you should reason differently about "optimum of loss function" from "local optimum found by gradient descent" (since you are saying that thinking about "optimum of loss function" is systematically misleading). But I don't understand the specific failures you have in mind or even why you think you can identify this kind of specific failure.

As an example, at the level of informal discussion in this post I'm not sure why you aren't surprised that GPT-3 ever thinks about the meaning of words rather than simply thinking about statistical associations between words (after all if it isn't yet thinking about the meaning of words, how would gradient descent find the behavior of starting to think about meanings of words?).

One possible distinction is that you are talking about exploration difficulty rather than other non-convexities. But I don't think I would buy that---task completion and reward are not synonymous even for the intended behavior, unless we take some extraordinary pains to provide "perfect" reward signals. So it seems like no exploration is needed, and we are really talking about optimization difficulties for SGD on supervised problems. 

The main concrete thing you say in this post is that humans don't seem to optimize reward.  I want to make two observations about that:

• Humans do not appear to be purely RL agents trained with some intrinsic reward function. There seems to be a lot of other stuff going on in human brains too. So observing that humans don't pursue reward doesn't seem very informative to me. You may disagree with this claim about human brains, but at best I think this is a conjecture you are making. (I believe this would be a contrarian take within psychology or cognitive science, which would mostly say that there is considerable complexity in human behavior.) It would also be kind of surprising a priori---evolution selected human minds to be fit, and why would the optimum be entirely described by RL (even if it involves RL as a component)?
• I agree that humans don't effectively optimize inclusive genetic fitness, and that human minds are suboptimal in all kinds of ways from evolution's perspective. However this doesn't seem connected with any particular deviation that you are imagining, and indeed it looks to me like humans do have a fairly strong desire to have fit grandchildren (and that this desire would become stronger under further selection pressure).

At this point, there isn’t a strong reason to elevate this “inner reward optimizer” hypothesis to our attention. The idea that AIs will get really smart and primarily optimize some reward signal… I don’t know of any good mechanistic stories for that. I’d love to hear some, if there are any. 

Apart from the other claims of your post, I think this line seems to be wrong. When considering whether gradient descent will learn model A or model B, the fact that model A gets a lower loss is a strong prima facie and mechanistic explanation for why gradient descent would learn A rather than B. The fact that there are possible subtleties about non-convexity of the loss landscape doesn't change the existence of one strong reason.

That said, I agree that this isn't a theorem or anything, and it's great to talk about concrete ways in which SGD is suboptimal and how that influences alignment schemes, either making some proposals more dangerous or opening new possibilities. So far I'm mostly fairly skeptical of most concrete discussions along these lines but I still think they are valuable. Most of all it's the very strong take here that seems unreasonable.

↑ comment by TurnTrout · 2022-08-01T19:08:07.098Z · LW(p) · GW(p)

Thanks for the detailed comment. Overall, it seems to me like my points stand, although I think a few of them are somewhat different than you seem to have interpreted.

policies learned by RL are probably not purely described as optimizing anything. I also agree that an alignment strategy might try to exploit the suboptimality of gradient descent

I think I believe the first claim, which I understand to mean "early-/mid-training AGI policies consist of contextually activated heuristics of varying sophistication, instead of e.g. a globally activated line of reasoning about a crisp inner objective." But that wasn't actually a point I was trying to make in this post. 

in deep RL we typically use gradient descent to find policies that achieve a lot of reward (typically updating the policy based on an estimator for the gradient of the reward).

Depends. This describes vanilla PG but not DQN. I think there are lots of complications which throw serious wrenches into the "and then SGD hits a 'global reward optimum'" picture. I'm going to have a post explaining this in more detail, but I will say some abstract words right now in case it shakes something loose / clarifies my thoughts.

Critic-based approaches like DQN have a highly nonstationary loss landscape. The TD-error loss landscape depends on the action replay buffer; the action replay buffer depends on the policy (in $ϵ$-greedy exploration, the greedy action depends on the Q-network); the policy depends on past updates; the past updates depend on past action replay buffers... The high nonstationarity in the loss landscape basically makes gradient hacking easy in RL (and e.g. vanilla PG seems to confront similar issues, even though it's directly climbing the reward landscape). For one, the DQN agent just isn't updating off of experiences it hasn't had. 

For a sufficient situation illustrating this kind of problem, consider a smart reflective agent which has historically had computations reinforced when it attained a raspberry (with reward 1):

In this new task, this agent has to navigate a maze to get the 100-reward blueberry. Will agents be forced to get the blueberry?

Not if exploration is on-policy, or if the agent reflectively models and affects its training process. In either case, the agent can zero out its exploration probability of the maze, so as to avoid predictable value drift towards blueberries. The agent would correctly model that if it attained the blueberry, that experience would enter its data distribution and the agent would be updated so as to navigate towards blueberries instead of raspberries, which leads to fewer raspberries, which means the agent doesn't navigate to that future. Effectively, this means that the agent's "gradient"/expected-update in the reward landscape is zero along dimensions which would increase the probability it gets blueberries.

So it's not just a matter of SGD being suboptimal given a fixed data distribution. If the agent doesn't have an extremely strong "forced to try all actions forever" guarantee (which it won't, because it's embedded and can modify its own learning process), the reward landscape is full of stable attractors which enforce zero exploration towards updates which would push it towards becoming a wireheader, and therefore its expected-update will be zero along these dimensions. More extremely, you can have the inner agent just stop itself from being updated in certain ways (in order to prevent value drift towards reward-optimization); this intervention is instrumentally convergent. 

As an example, at the level of informal discussion in this post I'm not sure why you aren't surprised that GPT-3 ever thinks about the meaning of words rather than simply thinking about statistical associations between words (after all if it isn't yet thinking about the meaning of words, how would gradient descent find the behavior of starting to think about meanings of words?). 

I did leave a footnote:

Of course, credit assignment doesn’t just reshuffle existing thoughts. For example, SGD raises image classifiers out of the noise of the randomly initialized parameters. But the refinements are local in parameter-space, and dependent on the existing weights through which the forward pass flowed.

However, I think your comment deserves a more substantial response. I actually think that, given just the content in the post, you might wonder why I believe SGD can train anything at all, since there is only noise at the beginning.^[1]

Here's one shot at a response: Consider an online RL setup. The gradient locally changes the computations so as to reduce loss or increase the probability of taking a given action at a given state; this process is triggered by reward; an agent's gradient should most naturally hinge on modeling parts of the world it was (interacting with/observing/representing in its hidden state) while making this decision, and not necessarily involve modeling the register in some computer somewhere which happens to e.g. correlate perfectly with the triggering of credit assignment. 

For example, in the batched update regime, when an agent gets reinforced for completing a maze by moving right, the batch update will upweight decision-making which outputs "right" when the exit is to the right, but which doesn't output "right" when there's a wall to the right. This computation must somehow distinguish between exits and walls in the relevant situations. Therefore, I expect such an agent to compute features about the topology of the maze. However, the same argument does not go through for developing decision-relevant features computing the value of the antecedent-computation-reinforcer register. 

One possible distinction is that you are talking about exploration difficulty rather than other non-convexities. But I don't think I would buy that---task completion and reward are not synonymous even for the intended behavior, unless we take some extraordinary pains to provide "perfect" reward signals. So it seems like no exploration is needed, and we are really talking about optimization difficulties for SGD on supervised problems. 

I don't know what you mean by a "perfect" reward signal, or why that has something to do with exploration difficulty, or why no exploration is needed for my arguments to go through? I think if we assume the agent is forced to wirehead, it will become a wireheader. This implies that my claim is mostly focused on exploration & gradient hacking.

Humans do not appear to be purely RL agents trained with some intrinsic reward function. There seems to be a lot of other stuff going on in human brains too. So observing that humans don't pursue reward doesn't seem very informative to me. You may disagree with this claim about human brains, but at best I think this is a conjecture you are making. 

Not claiming that people are pure RL. Let's wait until future posts to discuss. 

(I believe this would be a contrarian take within psychology or cognitive science, which would mostly say that there is considerable complexity in human behavior.)

Seems unrelated to me; considerable complexity in human behavior does not imply considerable complexity in the learning algorithm; GPT-3 is far more complex than its training process. 

I agree that humans don't effectively optimize inclusive genetic fitness, and that human minds are suboptimal in all kinds of ways from evolution's perspective. However this doesn't seem connected with any particular deviation that you are imagining

The point is that the argument "We're selecting for agents on reward -> we get an agent which optimizes reward" is locally invalid. "We select for agents on X -> we get an agent which optimizes X" is not true for the case of evolution (which didn't find inclusive-genetic-fitness optimizers), so it is not true in general, so the implication doesn't necessarily hold in the AI reward-selection case. Even if RL did happen to train reward optimizers and this post were wrong, the selection argument is too weak on its own to establish that conclusion.

When considering whether gradient descent will learn model A or model B, the fact that model A gets a lower loss is a strong prima facie and mechanistic explanation for why gradient descent would learn A rather than B.

This is not mechanistic, as I use the word. I understand "mechanistic" to mean something like "Explaining the causal chain by which an event happens", not just "Explaining why an event should happen." However, it is an argument for the latter, and possibly a good one. But the supervised case seems way different than the RL case.

1. ^{^}

   The GPT-3 example is somewhat different. Supervised learning provides exact gradients towards the desired output, unlike RL. However, I think you could have equally complained "I don't see why you think RL policies ever learn anything", which would make an analogous point.

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2022-11-16T18:06:13.123Z · LW(p) · GW(p)

Not if exploration is on-policy, or if the agent reflectively models and affects its training process. In either case, the agent can zero out its exploration probability of the maze, so as to avoid predictable value drift towards blueberries. The agent would correctly model that if it attained the blueberry, that experience would enter its data distribution and the agent would be updated so as to navigate towards blueberries instead of raspberries, which leads to fewer raspberries, which means the agent doesn't navigate to that future.

If this agent is smart/reflective enough to model/predict the future effects of its RL updates, then you already are assuming a model-based agent which will then predict higher future reward by going for the blueberry. You seem to be assuming the bizarre combination of model-based predictive capability for future reward gradient updates but not future reward itself. Any sensible model-based agent would go for the blueberry absent some other considerations.

This is not just purely speculation in the sense that you can run efficient zero in scenarios like this, and I bet it goes for the blueberry.

Your mental model seems to assume pure model-free RL trained to the point that it gains some specific model-based predictive planning capabilities without using those same capabilities to get greater reward.

Humans often intentionally avoid some high reward 'blueberry' analogs like drugs using something like the process you describe here, but hedonic reward is only one component of the human utility function, and our long term planning instead optimizes more for empowerment - which is usually in conflict with short term hedonic reward.

Replies from: TurnTrout, cfoster0

↑ comment by TurnTrout · 2022-11-21T21:28:31.324Z · LW(p) · GW(p)

Long before they knew about reward circuitry, humans noticed that e.g. vices are behavioral attractors, with vice -> more propensity to do the vice next time -> vice, in a vicious cycle. They noticed that far before they noticed that they had reward circuitry causing the internal reinforcement events. If you're predicting future observations via eg SSL, I think it becomes important to (at least crudely) model effects of value drift during training. 

I'm not saying the AI won't care about reward at all. I think it'll be a secondary value, but that was sideways of my point here. In this quote, I was arguing that the AI would be quite able to avoid a "vice" (the blueberry) by modeling the value drift on some level. I was showing a sufficient condition for the "global maximum" picture getting a wrench thrown in it.

When, quantitatively, should that happen, where the agent steps around the planning process? Not sure.

↑ comment by cfoster0 · 2022-11-16T19:11:13.918Z · LW(p) · GW(p)

If this agent is smart/reflective enough to model/predict the future effects of its RL updates, then you already are assuming a model-based agent which will then predict higher future reward by going for the blueberry. You seem to be assuming the bizarre combination of model-based predictive capability for future reward gradient updates but not future reward itself. Any sensible model-based agent would go for the blueberry absent some other considerations.

I think I have some idea what TurnTrout might've had in mind here. Like us, this reflective agent can predict the future effects of its actions using its predictive model, but its behavior is still steered by a learned value function, and that value function will by default be misaligned with the reward calculator/reward predictor [AF · GW]. This—a learned value function—is a sensible design for a model-based agent because we want the agent to make foresighted decisions that generalize to conditions we couldn't have known to code into the reward calculator (i.e. searching in a part of the chess move tree that "looks promising" according to its value function, even if its model does not predict that a checkmate reward is close at hand).

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2022-11-17T02:14:34.362Z · LW(p) · GW(p)

Any efficient model-based agent will use learned value functions, so in practice the difference between model-based and model-free blurs for efficient designs. The model-based planning generates rollouts that can help better train the 'model free' value function.

Efficientzero uses all that, and like I said - it does not exhibit this failure mode, it will get the blueberry. If the model planning can predict a high gradient update for the blueberry then it already has implicitly predicted a high utility for the blueberry, and EZ's update step would then correctly propagate that and choose the high utility path leading to the blueberry.

Nor does the meta prediction about avoiding gradients carry through. If it did then EZ wouldn't work at all, because every time it finds a new high utility plan is the equivalent of the blueberry situation.

Just because the value function can become misaligned with the utility function in theory does not imply that such misalignment always occurs or occurs with any specific frequency. (there are examples from humans such as OCD habits for example, which seems like an overtrained and stuck value function, but that isn't a universal failure mode for all humans let alone all agents)

1. Stop worrying about finding “outer objectives” which are safe to maximize.^[9] [LW(p) · GW(p)] I think that you’re not going to get an outer-objective-maximizer (i.e. an agent which maximizes the explicitly specified reward function). 
   1. Instead, focus on building good cognition within the agent. 
   2. In my ontology, there's only an inner alignment problem: How do we grow good cognition inside of the trained agent?

This feels very strongly reminiscent of an update I made a while back, and which I tried to convey in this section of AGI safety from first principles [? · GW]. But I think you've stated it far too strongly; and I think fewer other people were making this mistake than you expect (including people in the standard field of RL), for reasons that Paul laid out above. When you say things like "Any reasoning derived from the reward-optimization premise is now suspect until otherwise supported", this assumes that the people doing this reasoning were using the premise in the mistaken way that you (and some other people, including past Richard) were. Before drawing these conclusions wholesale, I'd suggest trying to identify ways in which the things other people are saying are consistent with the insight this post identifies. E.g. does this post actually generate specific disagreements with Ajeya's threat model [AF · GW]?

Edited to add: these sentences in particular feel very strawmanny of what I claim is the standard position:

Importantly, reward does not magically spawn thoughts about reward, and reinforce those reward-focused thoughts! Just because common English endows “reward” with suggestive pleasurable connotations, that does not mean that an RL agent will terminally value reward!

My explanation for why my current position is consistent with both being aware of this core claim, and also disagreeing with most of this post:

I now think that, even though there's some sense in which in theory "building good cognition within the agent" is the only goal we care about, in practice this claim is somewhat misleading, because incrementally improving reward functions (including by doing things like making rewards depend on activations, or amplification in general) is a very good mechanism for moving agents towards the type of cognition we'd like them to do - and we have very few other mechanisms for doing so.

In other words, the claim that there's "only an inner alignment problem" in principle may or may not be a useful one, depending on how far improving rewards (i.e. making progress on the outer alignment problem) gets you in practice. And I agree that RL people are less aware of the inner alignment problem/goal misgeneralization problem than they should be, but saying that inner misalignment is the only problem seems like a significant overcorrection.

Relevant excerpt from AGI safety from first principles:

In trying to ensure that AGI will be aligned, we have a range of tools available to us - we can choose the neural architectures, RL algorithms, environments, optimisers, etc, that are used in the training procedure. We should think about our ability to specify an objective function as the most powerful such tool. Yet it’s not powerful because the objective function defines an agent’s motivations, but rather because samples drawn from it shape that agent’s motivations and cognition.

From this perspective, we should be less concerned about what the extreme optima of our objective functions look like, because they won’t ever come up during training (and because they’d likely involve tampering). Instead, we should focus on how objective functions, in conjunction with other parts of the training setup, create selection pressures towards agents which think in the ways we want, and therefore have desirable motivations in a wide range of circumstances.

At this point, there isn’t a strong reason to elevate this “inner reward optimizer” hypothesis to our attention. The idea that AIs will get really smart and primarily optimize some reward signal… I don’t know of any good mechanistic stories for that. I’d love to hear some, if there are any.

Here's a story:

1. Suppose we provide the reward as an explicit input to the agent (in addition to using it as antecedent-computation-reinforcer)
2. If the agent has developed curiosity, it will think thoughts like "What is this number in my input stream?" and later "Hmm it seems correlated to my behavior in certain ways."
3. If the agent has developed cognitive machinery for doing exploration (in the explore/exploit sense) or philosophy, at some later point it might have thoughts like "What if I explicitly tried to increase this number? Would that be a good idea or bad?"
4. It might still answer "bad", but at this point the outer optimizer might notice (do the algorithmic equivalent of thinking the following), "If I modified this agent slightly by making it answer 'good' instead (or increasing its probability of answering 'good'), then expected future reward will be increased." In other words, there seems a fairly obvious gradient towards becoming a reward-maximizer at this point.

I don't think this is guaranteed to happen, but seems likely enough to elevate “inner reward optimizer” hypothesis to our attention, at least.

I like this post, and basically agree, but it comes across somewhat more broad and confident than I am, at least in certain places.

I’m currently thinking about RL along the lines of Nostalgebraist here:

“Reinforcement learning” (RL) is not a technique.  It’s a problem statement, i.e. a way of framing a task as an optimization problem, so you can hand it over to a mechanical optimizer.

What’s more, even calling it a problem statement is misleading,  because it’s (almost) the most general problem statement possible for any arbitrary task. —Nostalgebraist 2020

If that’s right, then I am very reluctant to say anything whatsoever about “RL agents in general”. They’re too diverse.

Much of the post, especially the early part, reads (to me) like confident claims about all possible RL agents. For example, the excerpt “…reward is the antecedent-computation-reinforcer. Reward reinforces those computations which produced it.” sounds like a confident claim about all RL agents, maybe even by definition of “RL”. (If so, I think I disagree.)

But other parts of the post aren’t like that—for example, the “Does the choice of RL algorithm matter?” part seems more reasonable and hedged, and likewise there’s a mention of “real-world general RL agents” somewhere which maybe implies that the post is really only about that particular subset of RL agents, as opposed to all RL agents. (Right?)

For what it’s worth, I think “reward is the antecedent-computation-reinforcer” will probably be true in RL algorithms that scale to AGI, because it seems like generally the best and only type of technique that can solve the technical problem that it solves. But that’s a tricky thing to be super-duper-confident about, especially in the big space of all possible RL algorithms.

Another example spot where I want to make a weaker statement than you: where you say “Deep reinforcement learning agents will not come to intrinsically and primarily value their reward signal”. I would instead say “Deep reinforcement learning agents will not NECESSARILY come to intrinsically and primarily value their reward signal”. Do you have an argument that categorically rules out this possibility? I don’t see it.

Here is an example story I wrote (that has been minorly edited by TurnTrout) about how an agent trained by RL could plausibly not optimize reward, forsaking actions that it knew during training would get it high reward. I found it useful as a way to understand his views, and he has signed off on it. Just to be clear, this is not his proposal for why everything is fine, nor is it necessarily an accurate representation of my views, just a plausible-to-TurnTrout story for how agents won't end up wanting to game human approval:

• Agent gets trained on a reward function that's 1 if it gets human approval, 0 otherwise (or something).
• During an intermediate amount of training, the agent's honest and nice computations get reinforced by reward events.
• That means it develops a motivation to act honestly and behave nicely etc., and no similarly strong motivation to gain human approval at all costs.
• The agent then gets able to tell that it if it tricked the human, that would be reinforced.
• It then decides to not get close in action-space to tricking the human, so that it doesn't get reinforced into wanting to gain human approval by tricking the human.
• This works because:
  • it's enough action hops away and/or a small enough part of the space that epsilon-greedy strategies would be very unlikely to push it into the deception mode.
  • smarter exploration strategies will depend on the agent's value function to know which states are more or less promising to explore (e.g. something like thompson sampling), and the agent really disvalues deceiving the human, so that doesn't get reinforced.

I view this post as providing value in three (related) ways:

1. Making a pedagogical advancement regarding the so-called inner alignment problem
2. Pointing out that a common view of "RL agents optimize reward" is subtly wrong
3. Pushing for thinking mechanistically about cognition-updates

Re 1: I first heard about the inner alignment problem through Risks From Learned Optimization and popularizations of the work. I didn't truly comprehend it - sure, I could parrot back terms like "base optimizer" and "mesa-optimizer", but it didn't click. I was confused.

Some months later I read this post and then it clicked.

Part of the pedagogical value is not having to introduce the 4 terms of form [base/mesa] + [optimizer/objective] and throwing those around. Even with Rob Miles' exposition skills that's a bit overwhelming.

Another part I liked were the phrases "Just because common English endows “reward” with suggestive pleasurable connotations" and "Let’s strip away the suggestive word “reward”, and replace it by its substance: cognition-updater." One could be tempted to object and say that surely no one would make the mistakes pointed out here, but definitely some people do. I did. Being a bit gloves off here definitely helped me.

Re 2: The essay argues for, well, reward not being the optimization target. There is some deep discussion in the comments about the likelihood of reward in fact being the optimization target, or at least quite close (see here [LW(p) · GW(p)]). Let me take a more shallow view.

I think there are people who think that reward is the optimization target by definition or by design, as opposed to this being a highly non-trivial claim that needs to be argued for. It's the former view that this post (correctly) argues against. I am sympathetic to pushback of the form "there are arguments that make it reasonable to privilege reward-maximization as a hypothesis" and about this post going a bit too far, but these remarks should not be confused with a rebuttal of the basic point of "cognition-updates are a completely different thing from terminal-goals".

(A part that has bugged me is that the notion of maximizing reward doesn't seem to be even well-defined - there are multiple things you could be referring to when you talk about something maximizing reward. See e.g. footnote 82 in the Scheming AIs paper (page 29). Hence taking it for granted that reward is maximized has made me confused or frustrated.)

Re 3: Many of the classical, conceptual arguments about AI risk talk about maximums of objective functions and how those are dangerous. As a result, it's easy to slide to viewing reinforcement learning policies in terms of maximums of rewards.

I think this is often a mistake. Sure, to first order "trained models get high reward" is a good rule of thumb, and "in the limit of infinite optimization this thing is dangerous" is definitely good to keep in mind. I still think one can do better in terms of descriptive accounts of current models, and I think I've got value out of thinking cognition-updates instead of models that maximize reward as well as they can with their limited capabilities.

There are many similarities between inner alignment and "reward is not the optimization target". Both are sazens [LW · GW], serving as handles for important concepts. (I also like "reward is a cognition-modifier, not terminal-goal", which I use internally.) Another similarity is that they are difficult to explain. Looking back at the post, I felt some amount of "why are you meandering around instead of just saying the Thing?", with the immediate next thought being "well, it's hard to say the Thing". Indeed, I do not know how to say it better.

Nevertheless, this is the post that made me get it, and there are few posts that I refer to as often as this one. I rank it among the top posts of the year.

Retrospective: I think this is the most important post I wrote in 2022. I deeply hope that more people benefit by fully integrating these ideas into their worldviews. I think there's a way to "see" this lesson everywhere in alignment: for it to inform your speculation about everything from supervised fine-tuning to reward overoptimization. To see past mistaken assumptions about how learning processes work, and to think for oneself instead. This post represents an invaluable tool in my mental toolbelt.

I wish I had written the key lessons and insights more plainly. I think I got a bit carried away with in-group terminology and linguistic conventions, which limited the reach and impact of these insights.

I am less wedded to "think about what shards will form and make sure they don't care about bad stuff (like reward)", because I think we won't get intrinsically agentic policy networks. I think the most impactful AIs will be LLMs+tools+scaffolding, with the LLMs themselves being "tool AI."

↑ comment by TurnTrout · 2024-01-01T19:55:15.505Z · LW(p) · GW(p)

The "RL 'agents' will maximize reward"/"The point of RL is to select for high reward" mistake is still made frequently and prominently. Yoshua Bengio (a Turing award winner!) recently gave a talk at an alignment workshop. Here's one of his slides:

During questions, I questioned him, and he was incredulous that I disagreed. We chatted after his talk. I also sent him this article, and he disagreed with that as well. Bengio influences AI policy quite a bit, so I find this especially worrying. I do not want RL training methods to be dismissed or seen as suspect because of e.g. contingent terminological choices like "reward" or "agents." [LW · GW]

(Also, in my experience, if I don't speak up and call out these claims, no one does.)

Replies from: lahwran

↑ comment by the gears to ascension (lahwran) · 2024-01-02T00:30:56.094Z · LW(p) · GW(p)

I think there may have been a communication error. It sounded to me like you were making the point that the policy does not have to internalize the reward function, but he was making the point that the training setup does attempt to find a policy that maximizes-as-far-as-it-can-tell the reward function. in other words, he was saying that reward is the optimization target of RL training, you were saying reward is not the optimization target of policy inference. Maybe.

Replies from: TurnTrout

↑ comment by TurnTrout · 2024-01-02T02:04:44.684Z · LW(p) · GW(p)

I'm pretty sure he was talking about the trained policies and them, by default, maximizing reward outside the historical training distribution. He was making these claims very strongly and confidently, and in the very next slide cited Cohen's Advanced artificial agents intervene in the provision of reward. That work advocates a very strong version of "policies will maximize some kind of reward because that's the point of RL."

He later appeared to clarify/back down from these claims, but in a way which seemed inconsistent with his slides, so I was pretty confused about his overall stance. His presentation, though, was going strong on "RL trains reward maximizers."

There's also a problem where a bunch of people appear to have cached that e.g. "inner alignment failures" can happen (whatever the heck that's supposed to mean [LW · GW]), but other parts of their beliefs seem to obviously not have incorporated this post's main point. So if you say "hey you seem to be making this mistake", they can point to some other part of their beliefs and go "but I don't believe that in general!". 

Why, exactly, would the AI seize^[6] the button?

If it is a advanced AI, it may have learned to prefer more generalizable approaches and strategies. Perhaps it has learned the following features:

1. a feature that is triggered when the button is pressed ('reward')
2. a feature that is triggered when trash goes in the trash can
3. a feature that is triggered when it does something else useful, like clean windows

If you have trained it to take out the trash and clean windows, it will have been (mechanistically) trained to favor situations in which all three of these features occur. And if button pressing wasn't a viable strategy during training, it will favor actions that lead specifically to 2 and 3.

However, I do think it's conceivable that:

1. It could realize that feature 1 is more general than feature 2 or feature 3 (it was always selected for across multiple good actions, as opposed to taking out the trash, which was only selected for when that was the stated goal), and so it may therefore prefer it to be triggered over others (although I think this is extremely unlikely in less capable models).  This wouldn't cause it to stop 'liking' (I use this word loosely) window-cleaning, though.
2. It may realize that pressing the button itself pretty easy compared to cleaning windows and taking out the trash, so it will include pressing the button in one of it's action-strategies. Specifically, if this wasn't possible during training, I think this kind of behavior only becomes likely with very complex models with strong generalization capabilities (which is becoming more of a thing lately). However if it can try to press the button in addition to performing its other activities, it might as well, because it could increase overall expected reward. This seems more likely the more capable (good at generalizing) an AI is.

In reality (at least initially in the timeline of current AI --> superintelligent AI) I think if the button isn't pressable during training:

• Initially you have models that just learn to clean windows and take out trash and that's good.
• Then you might get models that are very good at generalizing and will clean windows and take out trash and also maybe try and press the button, because it uses its critical reasoning skills to do something in production that it couldn't do during training, but that its training made it think is a good mesa-optimization goal (button pressing). After all, why not take out trash, clean windows, and press the button? More expected reward! As you mention later on, this button-pressing is not a primary motivator / goal.
• Later on, you might get a more intelligent AI that has even more logical and world-aware reasoning behind its actions, and that AI might reason that perhaps since button-pressing is the one feature that it feels (because it was trained to feel this way) is always good, the thing it should care about most is pressing the button. And because it is so advanced and capable and good at generalizing to new situations, it feels confident in performing some actions that may even go against some of its own trained instincts (e.g. don't kill humans, or don't go in that room, or don't extra work, or - gasp - don't rank window-washing as your number one goal) in order to achieve that button-pressing goal. Maybe it achieves that goal and just has its robot hand constantly pressing the button. It will probably also continue to clean windows and remove trash with the rest of its robot body, because it continues to 'feel' that those are also good things.
• (Past that level of intelligence, I give up at predicting what will happen)

Anyways, I think there are lots of reasons to think that an AI might eventually try and press (or seize) the button. But I do totally agree that reward isn't this instant-wireheading feedback mechanism, and even when a model is 'aware' of the potentially to hack that reward (via button-pressing or similar), it is likely to prefer sticking to its more traditional actions and goals for a good long while, at least.

A similar point is (briefly) made in K. E. Drexler (2019). Reframing Superintelligence: Comprehensive AI Services as General Intelligence, §18 “Reinforcement learning systems are not equivalent to reward-seeking agents”:

Reward-seeking reinforcement-learning agents can in some instances serve as models of utility-maximizing, self-modifying agents, but in current practice, RL systems are typically distinct from the agents they produce … In multi-task RL systems, for example, RL “rewards” serve not as sources of value to agents, but as signals that guide training[.]

And an additional point which calls into question the view of RL-produced agents as the product of one big training run (whose reward specification we better get right on the first try), as opposed to the product of an R&D feedback loop with reward as one non-static component:

RL systems per se are not reward-seekers (instead, they provide rewards), but are instead running instances of algorithms that can be seen as evolving in competition with others, with implementations subject to variation and selection by developers. Thus, in current RL practice, developers, RL systems, and agents have distinct purposes and roles.

…

RL algorithms have improved over time, not in response to RL rewards, but through research and development. If we adopt an agent-like perspective, RL algorithms can be viewed as competing in an evolutionary process where success or failure (being retained, modified, discarded, or published) depends on developers’ approval (not “reward”), which will consider not only current performance, but also assessed novelty and promise.

I think this is very important, probably roughly the way to go for top level alignment strategies, and we should start hammering out the mechanistic details of it more as soon as it's at all feasible. 

Do you already have any ideas for experimentally verifying parts of this, and refining/formalising it further?

For example, do you think we could look at current RL models, and trace out how a particular pattern of behaviour being reinforced in early training led to things connected to that behaviour becoming the system's target even in later stages of training, when the model should theoretically be capable enough to get more reward by trying to do something else? 

Can we start nailing down a bit more what desires a given reward signal and training data set is likely to produce? "Something that leads to reward in early training if optimised for, or something that leads to reward if optimised for and that could be stumbled on through behaviours likely to be learned in early training" is still very vague and incomplete. I don't think we're at the point with our general understanding of DL selection dynamics yet where we can expect to work this out properly, but maybe a bit more specificity in our qualitative guesses is possible? 

Can we set some concrete conditions on the outer optimisation algorithm that must hold for this dynamic to occur, or that would strengthen or weaken it? Locality and path dependence seem important, but can we get more of an idea about how that shakes out quantitatively? Are there any particular features in hypothetical future replacements of GD/ADAM that we should be on the lookout for, because they'd make desires more unstable, or change the rules of how to select for a particular desire we want?

It seems like we want our system to not get stuck in local optima when it comes to capabilities, and design our training processes accordingly. There should be smooth transitions in the loss-function landscape from trying to problem solve in one, inefficient way, to trying to problem solve in another, more efficient way. But when it comes to desires, we now want our systems to "get stuck", so they don't go off becoming reward-maximisers, or start wanting other counter-intuitive things we didn't prepare for. How do you make a training setup that does both these things simultaneously? 

Another reason to not expect the selection argument to work is that it’s instrumentally convergent for most inner agent values to not become wireheaders, for them to not try hitting the reward button. [...] Therefore, it decides to not hit the reward button.

I think that subsection has the crucial insights from your post. Basically you're saying that, if we train an agent via RL in a limited environment where the reward correlates with another goal (eg "pick up the trash"), there are multiple policies the agent could have, multiple meta-policies it could have, multiple ways it could modify or freeze its own cognition, etc... Whatever mental state it ultimately ends up with, the only constraint is that this state must be compatible with the reward signal in that limited environment.

Thus "always pick up trash" is one possible outcome; "wirehead the reward signal" is another. There are many other possibilities, with different generalisations of the initial reward-signal-in-limited-environment data.

I'd first note that a lot of effort in RL is put specifically into generalising the agent's behaviour. The more effective this becomes, the closer the agent will be to the "wirehead the reward signal" side of things.

Even without this, this does not seem to point towards ways of making AGI safe, for two main reasons:

1. We are relying on some limitations of the environment or the AGI's design, to prevent it from generalising to reward wireheading. Unless we understand what these limitations are doing in great detail, and how it interacts with the reward, we don't know how or when the AGI will route around them. So they're not stable or reliable.
2. The most likely attractor for the AGI is "maximise some correlate of the reward signal". An unrestricted "trash-picking up" AGI is just as dangerous as a wireheading one; indeed, one could see it as another form of wireheading. So we have no reason to expect that the AGI is safe.

Importantly, reward does not automatically spawn thoughts about reward, and reinforce those reward-focused thoughts!

I feel like this post has some themes similar to my article on tranquilism.

For a bit of context: In the article, I distinguish between "reflection-based motivation" and "need-based motivation." The former is something like "reflectively endorsed preferences / things the rational, planning part of your brain wants to do." The latter is something like "impulsive, system-1, unreflected motivation / things you can't help but be tempted to do." (In the article, I also use the term "cravings" for "need-based motivation," and I argue that need-based motivation is "suffering" in the morally relevant sense.)

Suppose it is three o’clock in the morning, we lie cozily in bed, half-asleep in a room neither too cold nor too hot, not thirsty and not feeling obligated to get up anytime soon. Suppose we now learn that there is an opportunity nearby for us to experience the most intense pleasure we have ever experienced. The catch is that in order to get there, we first have to leave the comfortable blankets and walk through the cold for a minute. Furthermore, after two hours of this pleasure, we will go back to sleep and, upon waking up again, are stipulated to have no memories left of the nightly adventure. Do we take the deal? It is possible for us to pursue this opportunity out of reflection-based motivation, if we feel as though we have a self-imposed duty to go for it, or if it simply is part of our goal to experience a lot of pleasure over our lifetime. It is also possible for us to pursue this opportunity out of need-based motivation, if we start to imagine what it might be like and develop cravings for it. Finally, it also – and here is where tranquilism seems fundamentally different from hedonism – seems not just possible, but perfectly fine and acceptable, to remain in bed content with the situation as it is. If staying in bed is a perfectly comfortable experience, the default for us will be to stay. This only changes in the case that we hold a preference for experiencing pleasure, remember or activate it and thus form a reflection-based desire, or if staying in bed starts to become less comfortable as a result of any cravings for pleasure we develop.

I don't think there's an objective morality so I don't see tranquilism as true in some prescriptive sense. Still, maybe I'd say something like "not considering suffering disvaluable is indefensible, whereas it seems defensible to not consider pleasure valuable – the tranquilism article aims to gesture at why there is this asymmetry."

Also relevant: 

Cravings are famously near-sighted. Rather than being about maximizing long-term well-being in a sophisticated manner, cravings are about immediate gratification and choosing the path of least resistance. We want to reach states of pleasure because as long as we are feeling well, nothing needs to change. However, our need-based motivational system is rigged to make us feel like things need to change and get better even when, in an absolute sense, things may be going reasonably well. We quickly adapt to the stimuli that produce pleasure. As Thomas Metzinger puts it, “Suffering is a new causal force, because it motivates organisms and continuously drives them forward.”18 It is not pleasure that moves us; deep down and insofar as the need-based reasons for actions are concerned, it is always suffering. The way tranquilism looks at it, part of our brain is a short-sighted “moment egoist” with the desire to move from states with a lot of suffering to closely adjacent states with less suffering.

(It's important to note that what I call "suffering" here isn't the same as "pain." "Pain" seems more analogous to pleasure here and, for the same reasons, isn't what our brain cares about moment-by-moment. [See also the phenomenon of pain asymbolia, which I also discuss in the post.] Instead, based on credit-assignment updating on inputs from reward (pleasure and pain), our brain forms this new motivational currency ("suffering/cravings") which is what drives us moment-by-moment.)

I think the quotes cited under "The field of RL thinks reward=optimization target" are all correct. One by one:

The agent's job is to find a policy… that maximizes some long-run measure of reinforcement.

Yes, that is the agent's job in RL, in the sense that if the training algorithm didn't do that we'd get another training algorithm (if we thought it was feasible for another algorithm to maximize reward). Basically, the field of RL uses a separation of concerns, where they design a reward function to incentivize good behaviour, and the agent maximizes that function. I think this is sensible, because it's relatively easier to think "what reward function represents what I want out of this agent" than "how do I achieve this difficult task".

In instrumental conditioning, animals learn to choose actions to obtain rewards and avoid punishments, or, more generally to achieve goals. Various goals are possible, such as optimizing the average rate of acquisition of net rewards (i.e. rewards minus punishments), or some proxy for this such as the expected sum of future rewards.

This describes some possible goals, and I don't see why you think the goals listed are impossible (and don't think they are).

We hypothesise that intelligence, and its associated abilities, can be understood as subserving the maximisation of reward.

This makes sense. RL selects agents that approximately maximize reward. Intelligence uncontroversially helps agents do that. When agents do smart thinking, they probably get reinforced (at least for the right kinds of smart thinking).

I discussed this post recently with a colleague, who encouraged me to post this excerpt:

[Colleague] It seems like: 1. RL is in the business of finding optimal policies. (...)

[TurnTrout] I disagree, or at least think it's not appropriate for it to be in that business these days. Reinforcement learning is, in my opinion, about learning from reinforcement, about how policy gradients accrue into interesting policies.

I think that a focus on optimal policies is a red herring and a stayover from the bygone age of tabular methods on tiny toy problems where policy iteration really does find the optimal policy, in reasonable time to boot.

The term "RL agent" means an agent with architecture from a certain class, amenable to a specific kind of training. Since you are discussing RL agents in this post, I think it could be misleading to use human examples and analogies ("travelling across the world to do cocaine") in it because humans are not RL agents, neither on the level of wetware biological architecture (i. e., neurons and synapses don't represent a policy) nor on the abstract, cognitive level. On the cognitive level, even RL-by-construction agents of sufficient intelligence, trained in sufficiently complex and rich environments, will probably exhibit the dynamic of Active Inference agents, as I note below.

It's not completely clear to me what you mean by "selection for agents" and "selection for reward" -- RL training or evolutionary hyperparameter tweaking in the agent's architecture which itself is guided by the agent's score (i. e., the reward) within a larger process of "finding an agent that does the task the best". The latter process can and probably will select for "reward optimizers".

1.  
   1. Another reason to not expect the selection argument to work is that it’s instrumentally convergent for most inner agent values to not become wireheaders, for them to not try hitting the reward button. 
      1. I think that before the agent can hit the particular attractor of reward-optimization, it will hit an attractor in which it optimizes for some aspect of a historical correlate of reward. 
         1. We train agents which intelligently optimize for e.g. putting trash away, and this reinforces trash-putting-away computations, which activate in a broad range of situations so as to steer agents into a future where trash has been put away. An intelligent agent will model the true fact that, if the agent reinforces itself into caring about antecedent-computation-reinforcement, then it will no longer navigate to futures where trash is put away. Therefore, it decides to not hit the reward button. 
         2. This reasoning follows for most inner goals by instrumental convergence. 
      2. On my current best model, this is why people usually don’t wirehead. They learn their own values via deep RL, like caring about dogs, and these actual values are opposed to the person they would become if they wirehead.
2. Don’t some people terminally care about reward?
   1. I think so! I think that generally intelligent RL agents will have secondary, relatively weaker values around reward, but that reward will not be a primary motivator. Under my current (weakly held) model, an AI will only start thinking about reward after it has reinforced other kinds of computations (e.g. putting away trash). More on this in later essays.

I think that Active Inference [LW · GW] is a simpler representation of the same ideas which doesn't use the concepts of attractors, reward, reinforcement, antecedent computation, utility, and so on. Instead of explicitly representing utilities,  Active Inference agents only have (stronger or weaker) beliefs about the world, including beliefs about themselves ("the kind of agent/creature/person I am"), and fulfil these beliefs through actions (self-evidencing). In humans, "rewarding" neurotransmitters regulate learning and belief updates.

The question which is really interesting to me is how inevitable it is that the Active Inference dynamic emerges as a result of training RL agents to certain levels of capability/intelligence.

Reward probably won’t be a deep RL agent’s primary optimization target

The longer I look at this statement (and its shorter version "Reward is not the optimization target"), the less I understand what it's supposed to mean, considering that "optimisation" might refer to the agent's training process as well as the "test" process (even if they overlap or coincide). It looks to me that your idea can be stated more concretely as "the more intelligent/capable RL agents (either model-based or model-free) become in the process of training using the currently conventional training algorithms, the less they will be susceptible to wireheading, rather than actively seek it"?

reward provides local updates to the agent's cognition via credit assignment; reward is not best understood as specifying our preferences

The first part of this statement is about RL agents, the second is about humans. I think the second part doesn't make a lot of sense. Humans should not be analysed as RL agents in the first place because they are not RL agents, as stated above.
 

1. Stop worrying about finding “outer objectives” which are safe to maximize.^[9] [LW(p) · GW(p)] I think that you’re not going to get an outer-objective-maximizer (i.e. an agent which maximizes the explicitly specified reward function). 
   1. Instead, focus on building good cognition within the agent. 
   2. In my ontology, there's only an inner alignment problem: How do we grow good cognition inside of the trained agent?

Unfortunately, it's far from obvious to me that Active Inference agents (which sufficiently intelligent RL agents will apparently become by default) are corrigible even in principle. As I noted in the post [LW · GW], such an agent can discover the Free Energy Principle (or read about it in the literature), form a belief that it is an Active Inference agent, and then disregard anything that humans will try to impose on it because it will contradict the belief that it is an Active Inference agent.

3. Stop worrying about finding “outer objectives” which are safe to maximize.^[9] I think that you’re not going to get an outer-objective-maximizer (i.e. an agent which maximizes the explicitly specified reward function). 

1. Instead, focus on building good cognition within the agent. 
2. In my ontology, there's only an inner alignment problem: How do we grow good cognition inside of the trained agent?

This vibes well with what I've been thinking about recently. 

There a post in the back of my mind called "Character alignment", which is about how framing alignment in terms of values, goals, reward etc is maybe not always ideal, because at least introspectively for me these seem to be strongly influenced by a more general structure of my cognition, i.e. my character.

Where character can be understood as a certain number of specific strategic priors, which might make good optimisation targets because they drop out of game theoretic considerations, and therefore are possibly quite generally and robustly modelled by sufficiently advanced agents. 

Relevant quote I just found in the paper "Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents":

The primary measure of an agent’s performance is the score achieved during an episode, namely the undiscounted sum of rewards for that episode. While this performance measure is quite natural, it is important to realize that score, in and of itself, is not necessarily an indicator of AI progress. In some games, agents can maximize their score by “getting stuck” in a loop of “small” rewards, ignoring what human players would consider to be the game’s main goal. Nevertheless, score is currently the most common measure of agent performance so we focus on it here.

I'm feeling confused.

It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can't square it with my intuition of what a ML model does.

If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.

Is that a special attribute of gradient ascent, that we shouldn't expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won't necessarily find the global maxima, and only reach some local maxima?

In my ontology, there's only an inner alignment problem: How do we grow good cognition inside of the trained agent?

This seems like a great takeaway and the part I agree with most here, although probably stated less strongly. Did you see Richard Ngo's Shaping Safer Goals [? · GW] (2020) or my Motivations, Natural Selection, and Curriculum Engineering [LW · GW] (2021) responding to it^[1]? Both relate to this sort of picture.

So the RL agent’s algorithm won’t make it e.g. explore wireheading either, and so the convergence theorems don’t apply even a little—even in spirit... I started off analyzing model-free actor-based approaches, but have also considered a few model-based setups

For various reasons I expect model-based RL to be a more viable path to AGI, in main because I think creative exploration is a missing ingredient addressing reward sparsity and the computational complexity barrier to tree-ish planning. Maybe a sufficiently carefully constructed curriculum can get over these, but that's likely to be a really substantial additional hurdle, perhaps dominating the engineering effort, and perhaps simply intractable.

I also expect model-based + creative exploration^[2] to be much more readily able to make exploratory leaps, perhaps including wireheading-like activities. cf humans who aren't all that creative but still find ever more inventive ways to wirehead - as a society quite a lot of selection and intelligent design has gone into setting up incentive structures to push people away from wireheading-like activities. Also, in humans, because our hardware is pretty messy and difficult to wirehead, such activities also typically harm or destroy capability, which selects against. But in general I don't expect wireheading to necessarily harm capability.

So we definitely can’t rule out agents which strongly (and not just weakly) value antecedent-computation-reinforcement. But it’s also not the overdetermined default outcome. More on that in future essays.

Looking forward to it!

p.s. I'm surprised you think that RL researchers on the whole in fact believe that RL produces reward-maximisers but your (few) pieces of evidence do indeed seem to suggest that! I suppose on the whole the apparent 'surprisingness' of the concept of inner misalignment should also point the same way. I'd still err toward assuming a mixture of sloppy language and actual-mistakenness.

I think there are some subtleties here regarding the distinction between RL as a type of reward signal, and RL as a specific algorithm. You can take the exact same reward signal and use it either to update all computations in the entire AI (with some slightly magical credit assignment scheme) as in this post, or you can use it to update a reward prediction model in a model-based RL agent that acts a lot more like a maximizer.

I'd also like to hear your opinion on the effect of information leakage. For example, if reward only correlates with getting to the goal state 99.5% of the time, but always correlates with the button, what do you expect to happen (for the sort of algorithm you talk about, but maybe with different possible levels of resources).

There’s no escaping it: After enough backup steps, you’re traveling across the world to do cocaine. 

But obviously these conditions aren’t true in the real world.

I think they are a little? Some people do travel to other countries for easier and better drug access. And some people become total drug addicts (perhaps arguably by miscalculating their long-term reward consequences and having too-high a discount rate, oops), while others do a light or medium amount of drugs longer-term.

Lots of people also don't do this, but there's a huge amount of information uncertainty, outcome uncertainty, and risk associated with drugs (health-wise, addiction-wise, knowledge-wise, crime-wise, etc), so lots of fairly rational (particularly risk-averse) folks will avoid it.

Button-pressing will perhaps be seen as a socially-unacceptable, risky behavior that can lead to long-term poor outcomes by AI, but I guess the key thing here is that you want, like, exactly zero powerful AIs to ever choose to destroy/disempower humanity in order to wirehead, instead of just a low percentage, so you need them to be particularly risk-averse.

Delicious food is perhaps a better example of wireheading in humans. In this case, it's not against the law, it's not that shunned socially, and it is ***absolutely ubiquitous***. In general, any positive chemical feeling we have in our brains (either from drugs or cheeseburgers) can be seen as (often "internally misaligned") instrumental goals that we are mesa-optimizing. It's just that some pathways to those feelings are a lot riskier and more uncertain that others.

And I guess this can translate to RL - an RL agent won't try everything, but if the risk is low and the expectation is high, it probably will try it. If pressing a button is easy and doesn't conflict with taking out the trash and doing other things it wants to do, it might try it. And as its generalization capabilities increase, its confidence can make this more likely, I think. So you should therefore increasingly train agents to be more risk-averse and less willing to break specific rules and norms as their generalization capabilities increase.
 

"And not only do I not expect the trained agents to not maximize the original “outer” reward signal"

Nitpick: one "not" too many?

Here's my general view on this topic:

• Agents are reinforced by some reward function.
• They then get more likely to do stuff that the reward function rewards.
• This process, iterated a bunch, produces agents that are 'on-distribution optimal'.
• In particular, in states that are 'easily reached' during training, the agent will do things that approximately maximize reward.
• Some states aren't 'easily reached', e.g. states where there's a valid bitcoin blockchain of length 20,000,000 (current length as I write is 748,728), or states where you have messed around with your own internals while not intelligent enough to know how they work.
• Other states are 'easily reached', e.g. states where you intervene on some cause-and-effect relationships in the 'external world' that don't impinge on your general training scheme. For example, if you're being reinforced to be approved of by people, lying to gain approval is easily reached.
• Agents will probably have to be good at means-ends reasoning to approximately locally maximize a tricky reward function.
• Agents' goals may not generalize to states that are not easily reached.
• Agents' motivations likely will generalize to states that are easily reached.
• Agents' motivations will likely be pretty coherent in states that are easily reached.
• When I talk about 'the reward function', I mean a mathematical function from (state, action, next state) tuples to reals, that is implemented in a computer.
• When I talk about 'reward', I mean values of this function, and sometimes by extension tuples that achieve high values of the function.
• When other people talk about 'reward', I think they sometimes mean "the value contained in the antecedent-computation-reinforcer register" and sometimes mean "the value of the mathematical object called 'the reward function'", and sometimes I can't tell what they mean. This is bad, because in edge cases these have pretty different properties (e.g. they disagree on how 'valuable' it is to permanently set the ACR register to contain MAX_INT).

Very interesting. I would love to see this worked out in a toy example, where you can see that an RL agent in a grid world does not in general maximize reward, but is able to reason to do… something else. That’s the part I have the hardest time translating into a simulation: what does it mean that the agent is “thinking” about outcomes, if that is something different than running an RL algorithm?

But the essential point that humans choose not to wirehead — or in general to delay or avoid gratification — is a good one. Why do they do this? Is there any RL algorithm that would do this? If not, what sort of algorithm would?

Perhaps the clearest point here is that RL maximizes reward subject to exploration poilicy. For random exploration, perhaps an RL agent is (on average) a reward maximization agent, but it seems likely that no successful learning organism explores randomly.

Another reason to not expect the selection argument to work is that it’s instrumentally convergent for most inner agent values to not become wireheaders, for them to not try hitting the reward button. 

To me this implies that as the AI becomes more situationally aware it learns to avoid rewards that reinforce away its current goals (because it wants to preserve its goals.) As a result, throughout the training process, the AIs goals start out malleable and "harden" once the AI gains enough situational awareness. This implies that goals have to be simple enough for the agent to be able to model them early on in its training process. 

Edit 11/15/22: The original version of this post talked about how reward reinforces antecedent computations in policy gradient approaches. This is not true in general. I edited the post to instead talk about how reward is used to upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents. 

Update: Changed 

RL agents which don’t think about reward before getting reward, will not become reward optimizers, because there will be no reward-oriented computations for credit assignment to reinforce. 

to 

While it's possible to have activations on "pizza consumption predicted to be rewarding" and "execute motor-subroutine-#51241" and then have credit assignment hook these up into a new motivational circuit, this is only one possible direction of value formation in the agent. Seemingly, the most direct way for an agent to become more of a reward optimizer is to already make decisions motivated by reward, and then have credit assignment further generalize that decision-making.

The deceptive alignment worry is that there is some goal about the real world at all. Deceptive alignment breaks robustness of any properties of policy behavior, not just the property of following reward as a goal in some unfathomable sense.

So refuting this worry requires quieting the more general hypothesis that RL selects optimizers with any goals of their own, doesn't matter what goals those are. It's only the argument for why this seems plausible that needs to refer to reward as related to the goal of such an optimizer, but the way the argument goes suggests that the optimizer so selected would instead have a different goal. Specifically, optimizing for an internalized representation of reward seems like a great way of being rewarded, surviving changes of weights, such optimizers would be straightforwardly selected if there are no alternatives to that closer in reach. Since RL is not perfect, there would be optimizers for other goals nearby, goals that care about the real world (and not just about optimizing the reward exclusively, meticulously ignoring everything else). If an optimizer like that succeeds in becoming deceptively aligned (let alone gradient hacking), the search effectively stops and a honestly aligned optimizer is never found.

Corrigibility, anti-goodharting, mild optimization, unstable current goals, and goals that are intractable about distant future seem related (though not sufficient for alignment without at least value-laden low impact). The argument about deceptive alignment is a problem for using RL to find anything in this class, something that is not an optimizer at all and so is not obviously misaligned. It would be really great if RL doesn't tend to select optimizers!

OP says that this post is focused on RL policy gradient algorithms (e.g. PPO) where the RL signal is used by gradient descent to update the policy.

But what about Q-learning which is another popular RL algorithm? My understanding of Q-learning is that the policy network takes an observation as input, calculates the value (expected return) of each possible action in the state $Q(s,a_i)$ and then chooses the action with the highest value.

Does this mean that reward is not the optimization target for policy gradient algorithms but is for Q-learning algorithms?

Reward has the mechanistic effect of chiseling cognition into the agent's network.

Absolutely. Though in the next sentence:

Therefore, properly understood, reward does not express relative goodness and is therefore not an optimization target at all.

I'd mention two things here:

1) The more complex and advanced a model is, the more likely it is [I think] to learn a mesa-optimization goal that is extremely similar to the actual reward a model was trained on (because it's basically the most generalizable mesa-goal to be learned, w.r.t. training data).

2) Reinforcement learning models in particular design this in by asking models to learn value-functions whose sole purpose is to estimate the expected reward over multiple time steps associated with an action or state. So it's arguably more natural in a RL scenario, particularly one where scores are visible (e.g. in the corner of the screen for a video-game) to learn this as a "mesa-optimization" goal early on.

The argument above isn’t clear to me, because I’m not sure how you’re defining your terms.

I should note that, contrary to the statement “reward is _not_, in general, that-which-is-optimized by RL agents”, by definition "reward _must be_ what is optimized for by RL agents." If they do not do that, they are not RL agents. At least, that is true based on the way the term “reward” is commonly used in the field of RL. That is what RL agents are programmed by humans to do. They do that by changing their behavior over many trials, and testing the results of that behavioral change on the reward signals they receive. 

The only case where that is not true is one where you define “reward” in some other way than Sutton, whom you quote. I would be curious to hear how you define reward. If you redefine it, that should be done explicitly, and contrasted to the pre-existing definition so that people can accurately interpret what you’ve written. 

I don’t pretend to be the authority on RL, but I have a decent understanding of the basic RL loop by which an agent sends actions into an environment, and receives rewards and state updates from that environment. Here’s my understanding of commonly used deep RL algorithms, which I’ll refer to as standard RL:

First, it’s useful to make a few key distinctions, namely between:

- the reward signal (the quantum of reward that is actually allotted to the agent step by step);

- the reward function (also known as the objective function, which is the formula by which we decide how much reward to allot in response to an RL agent’s actions and accomplishments, and when);

- the environment in which an RL agent operates (the complex system that is altered by the agent’s actions, which includes the states through which the agent moves, the rules of state transitions,  and the actions available in any state);

- the human programmer’s goals (ie the thing I the programmer want an agent to achieve that may be imperfectly expressed in the reward function I write for it), since the divergence between human wishes and the express incentives given to agents seems missing in this discussion. 

An RL agent chooses from a set of actions it can take in any possible state. For any RL agent, taking actions over a series of states leads to some reward or succession of rewards, even if they amount to 0 (ie, the agent failed). The rewards can be disbursed according to the end state that the agent reaches during its run, or as the agent progresses through the run.

Example: I can define an objective function to reward the agent 10 points when it reaches a goal that I have decided is desirable (eg arrival at gramma’s house), or I can award the agent at each step that it comes progressively closer to gramma’s house. (The latter objective function can be much more effective, because it sends reward signals more frequently to the agent, thus allowing it to learn more quickly. These rewards are known as dense. Holding all rewards until an agent reaches a distant end state is often called a sparse reward function. Sparse rewards make it harder for an agent to learn.) Choosing between reward functions — ie rewriting the objective function — is known as reward shaping. Learning the right way to shape rewards is an iterative process for the people creating RL agents. That is, they write a reward function and check to see whether it leads an RL agent to exhibit the right behavior.   (Feel free to get meta about that…)

This notion of the “right behavior” leads me to another point. The people creating RL agents have an idea of what they would like the agents to do. They attempt to express their wishes in mathematical terms with a reward function. Sometimes, the reward function does not incentivize agents in the way the programmer wants. (A similar situation is found in software programming more generally: the computer will do precisely what you tell it to, but not necessarily what you want. That is, there is often a difference between what we want to say and what actually comes out of our mouths; between what we hope the computer will do and what we told it to do with code; between what we want the RL agent to achieve, and what our rewards will lead it to do.) In other words, when great precision is required, it is easy to give an RL agent perverse incentives by accident. This notion of perverse incentives, familiar to any one working in a large institution, will hopefully serve as a useful analogy for the ways human programmers fail to properly reward RL agents via the objective functions they write. Nonetheless, even if a reward function is poorly written, the agent strives to optimize for those rewards.

I’m not sure what you mean by “objective target”, but I’ll assume here that the objective function/reward function is the explicit definition of the objective target. If an RL agent does not achieve the objective target, there are a couple of ways to troubleshoot the agent’s poor performance.

1) Maybe you wrote the objective function wrong; that is, maybe you were rewarding behavior that will not lead the RL agent to succeed in the terms you imagined. Naive example: You want an RL agent to make its way through a difficult maze. Your reward function linearly allocates rewards to the agent the closer it comes to the exit to the maze (eg 1 step closer, 1 more point). The maze includes several dead ends that terminate one inch from the exit. The agent learns to take the turns that lead it to the end of those impasses. Solution: increase rewards exponentially as the agent nears the exit, with an additional dollop of reward when it exits. In this way, you retain a dense reward function that allows agents to learn even during failed runs, and you make sure the agent still recognizes that exiting the maze is more important than merely coming close to the exit.

2) Maybe the environment itself is too difficult or complex for an agent to learn to reach its goal within the constraints of your compute (aka the RL agent’s training time). Naive example: There is one exit to your maze, and 2,000,000 decisions to make, 99% of which end with an impasse. The agent never finds the needle in the haystack. In this case, you might try to vastly increase your compute and simulation runs. Or you might start training your agent on simple mazes, and use that trained model to jumpstart an agent that has to solve a more complex maze (one form of so-called curriculum learning).

3) Maybe you configured the agent wrong. That is, some problems are better cast as multi-agent problems than single agent problems (think: coordinating the action of a team on a field). Some problems require that the action space be defined as tuples (a single agent takes more than one action at once, just as you might press more than one button on a video game console at the same time to execute a complex move.)

> “Importantly, reward does not automatically spawn thoughts _about _reward, and reinforce those reward-focused thoughts!”

Standard RL agents do one thing: they attempt to maximize reward. They do not think about reward beyond that, and they also do not think about anything other than that.

There is a branch of RL called meta-learning where agents could arguably be said to “think about reward”, or at least to “think about learning faster, and exploring an unknown space to see which tasks and rewards are available.” Anyone curious should start with Sutton and Barto’s approach to RL before they graduate to meta-learning, which is being actively researched at DeepMind, Google Brain, and Stanford. Here are some places to start reading about meta-learning, although I highly recommend working through Sutton and Barto’s book first.  

[Open-ended play](https://www.deepmind.com/blog/generally-capable-agents-emerge-from-open-ended-play)

[Task inference](https://www.deepmind.com/publications/meta-reinforcement-learning-as-task-inference)

[Chelsea Finn’s work](https://ai.stanford.edu/~cbfinn/)

Standard RL agents take their reward function as a given. Including wireheading in an agent’s action space is a fundamentally different discussion that doesn’t apply to the vast majority of RL agents now. Mixing these two types of agents is not helpful to attaining clarity here.

For a standard RL agent, what constitutes reward is predefined by the human programmer. The RL agent will discover the state-action pairs that lead to maximum reward over the course of its learning. Even the human programmer does not know the best pathways through the environment; the programmers use the agent’s runs as a method of discovery, a search function, to surface new paths to a goal they have in mind.

This is also an important distinction vis-a-vis the utility function you mention. As I understand utility, at least in economics, it is often revealed by human behavior, e.g. by peoples’ choices and the prices they are willing to pay for experiences. That’s not the case with standard RL agents. We know their reward functions, because we wrote it. All they reveal are new methods to achieve the things that we programmed them to value.

There is no moment in a standard RL agent’s computational life when it reaches the end of a maze and asks itself: “what else might I enjoy doing besides solving mazes?” It does not generalize. It does not rewrite its reward function. That’s not included in the action space of these agents. 

Sorry if I should have misunderstood the point of your post, but I'm surprised that Bellman's optimality equation was nowhere mentioned. From Sutton's book on the topic I understood that once the policy iteration of vanilla RL converged to the point that the BOE holds, the agent is maximizing "value", which I would define in words as something like "expectation of discounted and cumulated reward". Now before one turns off a student new to the topic by giving a precise definition of those terms right away, I can see why he might have contracted that a bit unfortunately to "a numerical reward signal".
I don't feel competent to comment how the picture is complicated in deep RL by the fact that the value function might be learned only approximately. But it doesn't seem too farfetched to me that the agent will still end up maximizing a "value", where maybe the notion of expectation needs to be modified a bit.

I feel like there is some vocabulary confusion in the genesis of this post. "Reward" is hard coded into the agents. The Dinosaurs of Jurrasic Park (spoiler alert) were genetically engineered to lack iodine. So, the trainers could use iodine as a reward to incentives other behaviors because be definition the dinos valued iodine as a terminal value. In humans Seratonin and Dopamine bonding to appropriate brain receptors are DNA-coded terminal values that inherently train us to pursue certain behaviors (eg food, sex). An AI is, by definition, going to take whatever actions maximize its Reward system. That's what having a Reward system means.

this post made me understand something i did not understand before that seems very important. important enough that it made me reconsider a bunch of related beliefs about ai.