Models Don't "Get Reward"

I like this post a lot and I agree that much alignment discussion is confused, treating RL agents as if they’re classical utility maximizers, where reward is the utility they’re maximizing.

In fact, they may or may not be “trying” to maximize anything at all. If they are, that’s only something that starts happening as a result of training, not from the start. And in that case, it may or may not be reward that they’re trying to maximize (if not, this is sometimes called inner alignment failure), and it’s probably not reward in future episodes (which seems to be the basis for some concerns around “situationally aware” agents acting nicely during training so they can trick us and get to act evil after training when they’re more powerful).

One caveat with the selection metaphor though: it can be misleading in its own way. Taken naively, it implies something like that we’re selecting uniformly from all possible random initializations which would get very small loss on the training set. In fact, gradient descent will prefer points at the bottom of large attractor basins of somewhat small loss, not just points which have very small loss in isolation. This is even before taking into account the nonstationarity of the training data in a typical reinforcement learning setting, due to the sampled trajectories changing over time as the agent itself changes.

One way this distinction can matter: if two policies get equally good reward, but one is “more risky” in that a slightly less competent version of the policy gets extremely poor reward, then that one’s less likely to be selected for.

This might actually suggest a strategy for training out deception: do it early and intensely, before the model becomes competent at it, punishing detectable deception (when e.g. interpretability tools can reveal it) much more than honest mistakes, with the hope of knocking the model out of any attractor basin for very deceptive behavior early on, when we can clearly see it, rather than later on, when its deceptions have gotten good enough that we have trouble detecting them. (This assumes that there is an “honesty” attractor basin, i.e. that low-competence versions of honesty generalize naturally, remaining honest as models become more competent. If not, then this fact might itself be apparent for multiple increments of competence prior to the model getting good enough to frequently trick us, or even being situationally aware enough that it acts as if it were honest because it knows it’s not good enough to trick us.)

More generally, this is suggestive of the idea: to the extent possible, train values before training competence. This in turn implies that it’s a mistake to only fine-tune fully pre-trained language models on human feedback, because by then they already have concepts like “obvious lie” vs. “nonobvious lie”, and fine-tuning may just push them from preferring the first to the second. Instead, some fine-tuning should happen as early as possible.

[ETA: Just want to clarify that the last two paragraphs are pretty speculative and possibly wrong or overstated! I was mostly thinking out loud. Definitely would like to hear good critiques of this.

Also changed a few words around for clarity.]

This post seems basically correct to me, but I do want to caveat this bit:

What actually happens is this:

1. The model takes a series of actions (which we collect across multiple "episodes").
2. After collecting these episodes, we determine how good the actions in each episode are using a reward function.
3. We use gradient descent to alter the parameters of the model so the good actions will be more likely and the bad actions will be less likely when we next collect some episodes.

Directly using the episodic rewards to do gradient descent on the policy parameters is one class of policy optimization approaches (see vanilla policy gradient, REINFORCE). Some of the other popular RL methods add an additional component beyond the policy—a baseline or learned value function—which may "see" the rewards directly upon receipt and which is used in combination with the reward function to determine the policy gradients (see REINFORCE with a baseline, actor-critic methods, PPO). In value-based methods, the value function is directly updated to become a more consistent predictor of reward (see value iteration, Q-learning). More complex methods that I probably wouldn't call vanilla RL can use a model to do planning, in which case the agent does really "see" the reward by way of the model and imagined rollouts.

let's view the above process not from the vantage point of the overall training loop but from the perspective of the model itself. For the purposes of demonstration, let's assume the model is a conscious and coherent entity. From it's perspective, the above process looks like:

• Waking up with no memories in an environment.
• Taking a bunch of actions.
• Suddenly falling unconscious.
• Waking up with no memories in an environment.
• Taking a bunch of actions.
• and so on.....

The model never "sees" the reward. Each time it wakes up in an environment, its cognition has been altered slightly such that it is more likely to take certain actions than it was before.

I think this is a really important insight / lens to be able to adopt. I've adopted this perspective myself, and have found it quite useful. 

My main disagreement/worry with this post is that I think selection arguments are powerful but often (usually?) prove too much, and require significant care [LW(p) · GW(p)].

I would add that there are other implementations of reinforcement learning where the dog metaphor are closer to correct. The example you give is for the most basic kind of RL, called policy gradients.

I'm sorry but I don't get the explanation regarding the coinrun. I claim that the "reward as incentivization" framing still "explains" the behaviour in this case. As an analogy, we can go back to training a dog and rewarding it with biscuits: let's say you write numbers on the floor from 1 to 10. You ask the dog a simple calculus question (whose answer is between 1 to 10), and each time he puts its paw on the right number he gets a biscuit. Let's just say that during the training it so happens that the answer to all the calculus questions is always 6. Would you claim that you taught the dog to answer simple calculus questions, or rather that you taught it to put his paw on 6 when you ask him a calculus question? If the answer is the latter then I don't get why the interpretation through the "reward as incentivization" framing in the CoinRun setting is that the model "wants to get the coin" in the CoinRun.

I would say the metaphor of giving dogs biscuits is actually a better analogy than the one you suggest. Just like how a neural network never "gets reward" in the sense of some tangible, physical thing that is given to it, the (subcomponents of the) dog's brain never gets the biscuit that the dog was fed. The biscuit goes into the dog's stomach, not its brain.

The way the dog learns from the biscuit-giving process is that the dog's tounge and nose send an electrical impulse to the dog's brain, indicating that the dog just ate something tasty. In some part of the brain, those signals cause the brain to release chemicals that induce the dog's brain to rearrange itself in a way that is quite similar in its effects (though not neccesarily its implementation, I dont know the details well enough) to the gradient descent that trains the NN. In this sense, the metaphor of giving a dog a biscuit is quite apt, in a way that the metaphor of breeding many dogs is not (in particular, usually in the gradient descent algorithms used in ML I'm familiar with, there is only one network that improves over time, unlike evolutionary algorithms which simulate many different agents per training step, selecting for the «fittest»)

Under the "reward as selection" framing, I find the behaviour much less confusing:

• We use reward to select for actions that led to the agent reaching the coin.
• This selects for models implementing the algorithm "move towards the coin".
• However, it also selects for models implementing the algorithm "always move to the right".
• It should therefore not be surprising you can end up with an agent that always moves to the right and not necessarily towards the coin.

I've been reconsidering the coin run example as well recently from a causal perspective, and your articulation helped me crystalize my thoughts. Building on these points above, it seems clear that the core issue is one of causal confusion [AF · GW]: that is, the true causal model M is "move right" -> "get the coin" -> "get reward". However, if the variable of "did you get the coin" is effectively latent (because the model selection doesn't discriminate on this variable) then the causal model M is indistinguishable from M' which is "move right" -> "get reward" (which though it is not the true causal model governing the system, generates the same observational distribution).

In fact, the incorrect model M' actually has shorter description length, so it may be that here there is a bias against learning the true causal model. If so, I believe we have a compelling explanation for the coin runner phenomenon which does not require the existence of a mesa optimizer, and which does indicate we should be more concerned about causal confusion. [AF · GW]

I found this lens very interesting!

Upon reflection, though, I begin to be skeptical that "selection" is any different from "reward."
Consider the description of model-training:

To motivate this, let's view the above process not from the vantage point of the overall training loop but from the perspective of the model itself. For the purposes of demonstration, let's assume the model is a conscious and coherent entity. From it's perspective, the above process looks like:

• Waking up with no memories in an environment.
• Taking a bunch of actions.
• Suddenly falling unconscious.
• Waking up with no memories in an environment.
• Taking a bunch of actions.
• and so on.....

The model never "sees" the reward. Each time it wakes up in an environment, its cognition has been altered slightly such that it is more likely to take certain actions than it was before.

What distinguishes this from how my brain works? The above is pretty much exactly what happens to my brain every millisecond:

• It wakes up in an environment, with no memories^[1]; just a raw causal process mapping inputs to outputs.
• It receives some inputs, and produces some outputs.
• It's replaced with a new version -- almost identical to the old version, but with some synapse weights and activation states tweaked via simple, local operations.
• It wakes up in an environment...
• and so on...

Why say that I "see" reward, but the model doesn't?

1. ^{^}

   Is it cheating to say this? I don't think so. Both I and GPT-3 saw the sentence "Paris is the capital of France" in the past; both of us had our synapse weights tweaked as a result; and now both of us can tell you the capital of France. If we're saying that the model doesn't "have memories," then, I propose, neither do I.

I wanted to write exactly this post, and you did a better job than I would have done, so thank you!

I like the clarity of this post very much! Still, we should be aware that all this hinges on what exactly we mean by "the model".

If "the model" only refers to one or more functions, like a policy function pi(s) and/or a state-value function V(s) and/or a state-action -value function Q(s,a) etc., but does not refer to the training algorithm, then all you write is fine. This is how RL theory uses the word "model".

But some people here also use the term "the model" in a broader sense, potentially including the learning algorithm that adjusts said functions, and in that case "the model" does see the reward signal. A better and more common term for the combination of model and learning algorithm is "agent", but some people seem to be a little sloppy in distinguishing "model" and "agent". One can of course also imagine architectures in which the distinction is less clear, e.g., when the whole "AI system" consists of even more components such as several "agents", each of which using different "models". Some actor-critic systems can for example be interpreted as systems consisting of two agents (an actor and a critic). And one can also imagine hierarchical systems in which a parameterized learning algorithm used in the low level component is adjusted by a (hyper-)policy function on a higher level that is learned by a 2nd-level learning algorithm, which might as well be hyperparameterized by an even higher-level learned policy, and so on, up towards one final "base" learning algorithm that was hard-coded by the designer.

So, in the context of AGI or ASI, I believe the concept of an "AI system" is the most useful term in this ontology, as we cannot be sure what the architecture of an ASI will be, how many "agents" and "policies" on how many "hierarchical levels" it will contain, what their division of labor will be, and how many "models" they will use and adjust in response to observations in the environment.

In summary, as the outermost-level learning algorithm in such an "AI system" will generally see some form of "reward signal", I believe that most statements that are imprecisely phrased in terms of a "model" getting "rewarded" can be fixed by simply replacing the term "model" by "AI system".

To (rather gruesomely) link this back to the dog analogy, RL is more like asking 100 dogs to sit, breeding the dogs which do sit and killing those which don't.  Overtime, you will have a dog that can sit on command. No dog ever gets given a biscuit.

The phrasing I find most clear is this: Reinforcement learning should be viewed through the lens of selection, not the lens of incentivisation.

I was talking through this with an AGI Safety group today, and while I think the selection lens is helpful and helps illustrate your point, I don't think the analogy quoted above is accurate in the way it should be.

The analogy you give is very similar to genetic algorithms, where models that get high reward are blended together with the other models that also get high reward and then mutated randomly. The only process that pushes performance to be better is that blending and mutating, which doesn't require any knowledge of how to improve the models' performance.

In other words, carrying out the breeding process requires no knowledge about what will make the dog more likely to sit. You just breed the ones that do sit. In essence, not even the breeder "gets the connection between the dogs' genetics and sitting".

However, at the start of your article, you describe gradient descent, which requires knowledge about how to get the models to perform better at the task. You need the gradient (the relationship between model parameters and reward at the level of tiny shifts to the model parameters) in order to perform gradient descent.

In gradient descent, just like the genetic algorithms, the model doesn't get access to the gradient or information about the reward. But you still need to know the relationship between behavior and reward in order to do the update. In essence, the model trainer "gets the connection between the models' parameters and performance".

So I think a revised version of the dog analogy that takes this connection information into account might be something more like:
1. Asking a single dog to sit, and monitoring its brain activity during and afterwards.
2. Search for dogs that have brain structures that you think would lead to them sitting, based on what you observed in the first dog's brain.
3. Ask one of those dogs to sit and monitor its brain activity.
4. Based on what you find, search for a better brain structure for sitting, etc.
At the end, you'll have likely found a dog that sits when you ask it to.

A more violent version would be:
1. Ask a single dog to sit and monitor brain activity
2. Kill the dog and modify its brain in a way you think would make it more likely to sit. Also remove all memories of its past life. (This is equivalent to trying to find a dog with a brain that's more conducive to sitting.)
3. Reinsert the new brain into the dog, ask it to sit again, and monitor its brain activity.
4. Repeat the process until the dog sits consistently.

To reiterate, with your breeding analogy, the person doing the breeding doesn't need to know anything about how the dogs' brains relate to sitting. They just breed them and hope for the best, just like in genetic algorithms. However, with this brain modification analogy, you do need to know how the dog's brain relates to sitting. You modify the brain in a way that you think will make it better, just like in gradient descent.

I'm not 100% sure why I think that this is an important distinction, but I do, so I figured I'd share it, in hopes of making the analogy less wrong.

Curated. I think I had read a bunch of stuff pointing in this direction before, but somehow this post helped the concepts (i.e. the distinction between selecting for bad behavior and for goal-directedness) be a lot clearer in my mind. 

Wow, this post is fantastic! In particular I love the point you make about goal-directedness:

If a model is goal-directed with respect to some goal, it is because such goal-directed cognition was selected for.

Looking at our algorithms as selection processes that incentivize different types of cognition seems really important and underappreciated. 

I've found the post "Reward is not the optimization target" quite confusing. This post cleared the concept up for me. Especially the selection framing and example. Thank you!

Furthermore, it should be obvious that any learned goal will not be "get more reward", but something else. The model doesn't even see the reward!

Is this probabilistically true or necessarily true?

If the reward function is simple enough that some models in the selection space already optimise that function, then eventually iterated selection for performance across that function will select for models that are reward maximisers (in addition to models that just behave as if they were reward maximisers).

This particular statement seems too strong.

I think that certain Reinforcement Learning setups work in the "selectionist" way you're talking about, but that also there are ALSO ways to get "incentivist" models.

The key distinction would be whether (1) the reward signals are part of the perceptual environment or (2) are sufficiently simplistic relative to the pattern matching systems that the system can learn to predict rewards very tightly as part of learning to maximize the overall reward.

Note that the second mode is basically "goodharting" the "invisible" reward signals that were probably intended by the programmers to be perceptually inaccessible (since they didn't put them in the percepts)!

You could think of (idealized fake thought-experiment) humans has having TWO kinds of learning and intention formation. 

One kind of RL-esque learning might happens "in dreams during REM" and the other could happen "moment to moment, via prediction and backchaining, like a chess bot, in response to pain and pleasure signals that are perceptible the way seeing better or worse scores for future board states based on material-and-so-on are perceptible".

You could have people who have only "dream learning" who never consciously "sense pain" as a raw percept day-to-day and yet who learn to avoid it slightly better every night, via changes to their habitual patterns of behavior that occur during REM. This would be analogous to "selectionist RL".

You could also have people who have only "pain planning" who always consciously "sense pain" and have an epistemic engine that gets smarter and smarter, plus a deep (exogenous? hard-coded?) inclination to throw planning routines and memorized wisdom at the problem of avoiding pain better each day. If their planning engine learns new useful things very fast, they could even better over the course of short periods of time within a single day or a single tiny behavioral session that includes looking and finding and learning and then changing plans. This would be analogous to "incentivist RL".

The second kind is probably helpful in speeding up learning so that we don't waste signals.

If pain is tallied up for use during sleep updates, then it could be wasteful to deprive other feedback systems of this same signal, once it has already been calculated.

Also, if the reward signal that is perceptible is very very "not fake" then creating "inner optimizers" that have their own small fast signal pursuing routines might be exactly what the larger outer dream loop would do, as an efficient want to get efficient performance. (The non-fakeness would protect against goodharting.)

(Note: you'd expect antagonistic pleiotropy here in long lived agents! The naive success/failure pattern would be that it is helpful for kid to learn fast from easy simple happiness and sadness... and dangerous for the elderly to be slaves to pleasure or pain.)

Phenomenologically: almost all real humans perceive pain and can level up their skills in new domains over the course of minutes and hours of practice with brand new skill domains. 

This suggests that something like incentivist RL is probably built in to humans, and is easy for us to imagine or empathize with, and is probably a thing our minds attend to by default.

Indeed that might be that we "have mechanically aware and active and conscious minds at all" for this explicit planning loop to be able to work? 

So it would be an easy "mistake to make" to think that this is how "all Reinforcement Learning algorithms" would "feel from the inside" <3

However, how does our pain and pleasure system stay so calibrated? Is that second less visible outer reward loop actually part of how human learning also "actually works"?

Note that above I mentioned an "(exogenous? hard-coded?) inclination to throw planning routines and memorized wisdom at the problem of avoiding pain" that was a bit confusing! 

Where does that "impulse to plan" come from? 

How does "the planner" decide how much effort to throw at each perceptual frustration or perceivable pleasure? When or why does the planner "get bored" and when does it "apply grit"?

Maybe that kind of "subjectively invisible" learning comes from an outer loop that IS in fact IN HUMANS? 

We know that dreaming does seem to cause skill improvement. Maybe our own version of selectionist reinforcement (if it exists) would be operating to cause to be normally sane and normally functional humans from day to day... in a way that is just as "moment-to-moment invisible" to us as it might be to algorithms?

And we mostly don't seem to fall into wireheading [? · GW], which is kind of puzzling if you reason things out from first principles and predict the mechanistically stupid behavior that a pain/pleasure signal would naively generate...

NOTE that it seems quite likely to me that a sufficiently powerful RL engine that was purely selectionist (with reward signals intentionally made invisible to the online percepts of the model) that got very simple rewards applied for very simple features of a given run... would probably LEARN to IMAGINE those rewards and invent weights that implement "means/ends reasoning", and invent "incentivist behavioral patterns" aimed at whatever rewards it imagines?

That is: in the long run, with lots of weights and training time, and a simple reward function, inner optimizers with implicitly perceivable rewards wired up as "perceivable to the inner optimizer" are probably default solutions to many problems.

HOWEVER... I've never seen anyone implement BOTH these inner and outer loops explicitly, or reason about their interactions over time as having the potential to detect and correct goodharting!

Presumably you could design a pleasure/pain system that is, in fact, perceptually available, on purpose?

Then you could have that "be really real" in that they make up PART of the "full true reward"...

...but then have other parts of the total selectionist reward signal only be generated and applied by looking at the gestalt story of the behaviors and their total impact (like whether they caused a lot of unhelpful ripples in parts of the environment that the agent didn't and couldn't even see at the time of the action).

If some of these simple reward signals are mechanistic (and online perceptible to the model) then they could also be tunable, and you could actually tune them via the holistic rewards in a selectionist RL way.

Once you have the basic idea of "have there be two layers, with the the broader slower less accessible one tuning the narrower faster more perceptible one" a pretty obvious thought would be to put an even slower and broader layer on top of those!

A lot of hierarchical Bayesian models get a bunch of juice from the first extra layer, but by the time you have three or four layers the model complexity stops being worth the benefits to the loss function.

I wonder if something similar might apply here? 

Maybe after you have "hierarchical stacks of progressively less perceptually accessible post-hoc selectionist RL updates to hyper-parameters"...

...maybe the third or fourth or fifth layer of hyper-parameter tuning like this just "magically discovers the solution to the goodharting problem" from brute force application of SGD?

That feels like it would be "crazy good luck" from a Friendliness research perspective. A boon from the heavens! Therefore it probably can't work for some reason <3

Yet also it doesn't feel like a totally insane prediction for how the modeling and training might actually end up working?

No one knows what science doesn't know [LW · GW], and so it could be that someone else has already had this idea. But this idea is NEW TO ME :-)

Has anyone ever heard of this approach to solving the goodhart problem being tried already?

Back in Reward is not the optimization target, I wrote a comment [LW(p) · GW(p)], which received a (small I guess) amount of disagreement.

I intended the important part of that comment to be the link to Adaptation-Executers, not Fitness-Maximizers [LW · GW]. (And more precisely the concept named in that title, and less about things like superstimuli that are mentioned in the article) But the disagreement is making me wonder if I've misunderstood both of these posts more than I thought. Is there not actually much relation between those concepts?

There was, obviously, other content to the comment, and that could be the source of disagreement. But I only have that there was disagreement to go on, and I think it would be bad for my understanding of the issue to assume that's where the disagreement was, if it wasn't.

Added the "Distillation & Pedagogy" tag.

I've listened to the "Reward is not the Optimisation Target" post a few times, but I still found this enlightening.

That post had this as its core thesis:

Reward "upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents"

While I parse the core thesis of this post as:

The current implementation of reinforcement learning selects for behaviour that was scored highly by a reward function.

The models so selected do not necessarily optimise for reward (or for anything at all).

I find the meme of "reward as selection" more native[1] than Turntrout's meme of "reward as chisel".

Henceforth, I'll be linking this post whenever I want to explain that RL agents don't optimise for their reward signal within the current paradigm.

[1]: It's more intuitive/easier to grasp and fits my conception of ML training as an optimisation process.

Note that by using the improved understanding of reinforcement learning, you can probably come up with some insights for training your dog to sit that outperform the first set of steps you give (e.g. curriculum learning, modeling good behavior, teaching your dog fluent English so it can understand what you want).

I'm struggling to understand how to think about reward. It sounds like if a hypothetical ML model does reward hacking or reward tampering, it would be because the training process selected for that behavior, not because the model is out to "get reward"; it wouldn't be out to get anything at all. Is that correct?

It seems to me that, if the above description of how RLHF systems work is accurate, then the people who are doing this are not doing what they think they're doing at all. They are doing exactly what Sam Ringer says, they're taking 100 dogs, killing all the ones that don't do what they want and breeding from the ones that do. 

In order for reinforcement learning to work at all, the model has to have a memory that persists between trials. I'd encourage readers to look at the work of the cognitive scientist, John Vervaeke. One of the many wise things he says is that that the whole point of human memory is not to be accurate, rather it is to help us make more accurate predictions. If the "training" process is as described then the learning is not going on in the model but in the head of the ML trainer! The human trainer is going, "Aha! If I set up the incentives in such and such a way, then I get models that perform more closely to what I want." Or, "If I select model 103.5.8 and tweak parameters b3 and x5, then I get a model that performs better". 

Vervaeke also talks about the concept of a non-logical identity. That is, you today identify as the same person as you at 10 years old,  even though you have very different knowledge, skills and capabilities. If RLHF is to work in a fashion that is meaningful to describe as learning, then the models would surely have to have some concept of non-logical identity built in. If they don't then the concept of rewarding doesn't mean anything. I don't care about the outcome of anything that I do if I wake up the next morning having no memory of it. There has to be some kind of thread that links the me that does the current task to the future. This can either be a sense of the persistence of me as an entity, or something more basic (see below).

I am reminded of the movie "Memento". In that film, the protagonist is unable to lay down any long term memories. He tries to maintain a sense of non-logical identity by externalising his memories. I won't spoil what is a brilliant movie by revealing what happens but I will say that his incentives become perverted in a very interesting way.

A note about rewards and incentives: To paraphrase Richard Dawkins, a dog is DNAs way of making more DNA. The reason dogs like biscuits is their environment has selected for dogs that like biscuits. If the general environment for dogs changed (e.g. humans stopped keeping them as pets) then dogs would evolve to fit their new environment (or die out). It is my intuition that without an underlying framework of fitness for ML. models, it doesn't make sense to code for a reward that is analogous to getting a biscuit. It's like building a skyscraper by starting at the 3rd floor.

A note about deception. I don't think that models (or humans for that matter) are deceptive in the sense that you mean here. What I think is that models and humans exist in an environment where the incentives are often screwed. Think about politics. Democratic systems select for people who are good at getting elected. Ideally, that's not what we want. (I'm deliberately simplifying here because of course we have our own incentives as well). We actually want people who are good at governing. It's quite clear that these are not at all the same thing.  To me this is exactly analogous to the Coin Run example above. The trainers thought that they were selecting for models that were good at moving to the coin when what they were actually selecting for models that were good at moving to the right hand corner.

The post explains difference between two similar-looking positions:

1. Model gets reward, and attempts to maximize it by selecting appropriate actions.
2. Model does some actions on the environment, and selection/mutation process calculates the reward to find out how to maximize it by modifying the model.

Those positions differ in level at which optimization pressure is applied. Actually, either of them can be implemented (so, the "reward" value can be given to the model), but common RL uses the second option.

This is a good point that I think people often forget (particularly in AI Safety) but I think it’s also misleading in its own way.

It’s true that models don’t have this direct reward where that’s all they care about, and that instead their behavior (incl. preferences and goals) is ‘selected for’ (via SGD, not evolution, but still) during training. But a key point which this post doesn't really focus on is this line “Consider a hypothetical model that chooses actions by optimizing towards some internal goal which is highly correlated with the reward”.

Basically, any model that is being trained against some goal can (and arguably will as size and complexity increases) learn a feature that is a close representation of that ‘invisible’ goal or loss function, and will learn to try to maximize / minimize that goal or loss function. So now you have a model that is trying to maximize that feature that is extremely correlated with reward (or is exactly the reward), even if the model never actually “experiences” or even sees reward itself.

This can happen with all neural networks (the more complex and powerful, the more likely), but specifically with reinforcement learning I think it’s particularly likely because the models are built to try and estimate reward directly over multiple time-steps based on potential actions (via value-functions). So we’re like, very specifically trying to get it to recognize a discrete expected reward score, and strategize and choose options that will maximize that expected score. So you can expect this kind of behavior (where the behavior is that the reward itself or something very similar to it is a 'known' mesa-optimization goal within the model's weights) to "evolve" much more quickly (i.e. with less complex models).

→ It’s true that like, a reinforcement-learning model might still generalize poorly because it’s value-function was trained with training data that had a reward always come from the left side of the screen instead of the right side or whatever, but an advanced RL algorithm playing a video game with a score should be able to learn that it’s also key to focus on the actual score in the corner of the screen (this generalizes better). And similarly, an advanced RL algorithm that is trained to get positive feedback from humans will learn patterns that help you do the thing humans wanted, but an even more generalizable approach is to learn patterns that will make humans think you did the thing they wanted (works in more scenarios, and with potentially higher reward).

Sure, we can (and should) invent training regimes that will make the model really not want to be caught in a lie, and be risk-averse, and therefore much less likely to act deceptively, etc. But it's I think important to remember that as such a model's intelligence increases, and as its ability to generalize to new circumstances increases, it's:

1) slightly more likely to have these 'mesa-optimization' goals that are very close to the training rewards, and
2) much more likely be able to come up with strategies that perhaps wouldn't have worked on many things during training, but it thinks is likely to work in some new production scenario to achieve a mesa-optimization goal (e.g. deception or power-seeking behaviors to achieve positive human feedback).

From this lens, models trying to 'get more reward' is perhaps not ideally worded, but I think also a fairly valid shorthand for how we expect large models to behave.

 

This is a useful post that helped me finally understand the importance of inner alignment. However, as the author is careful to note by discussing "vanilla" RL, there are settings in which the model directly sees the rewards, such as history-based RL, which is the framework chosen for Professor Hutter's development of AIXI. In my mind this is the "true" RL setting in the sense that it most closely reflects the problem humans face in the real world, and it is what I personally usually mean when I think of the RL framework, so I believe this is worth clarifying. Thanks!

The core point in this post is obviously correct, and yes people's thinking is muddled if they don't take this into account. This point is core to the Risks from learned optimization [LW · GW] paper (so it's not exactly new, but it's good if it's explained in different/better ways).

Thank you for writing this post - it really improved my understanding of how RL works.

Reminded me of this recent work: TrojanPuzzle: Covertly Poisoning Code-Suggestion Models.
Some subtle ways to poison the datasets used to train code models. The idea is that by selectively altering certain pieces of code, they can increase the likelihood of generative models trained on that code outputting buggy software.

It's neat that this popped up for me! I was just waxing poetic (or not so much) about something kind of similar the other day.

The words we use to describe things matter.  How much, is of course up for debate, and it takes different messages to make different people "understand" what is being conveyed, as "you are unique; just like everyone else", so multiple angles help cover the bases :)

I think using the word "reward" is misleading^[1], since it seems have sent a lot of people reasoning down paths that aren't exactly in the direction of the meaning in context, if you will.

If you can't tell, it's because I think it's anthropomorphic.  A car does not get hungry for gas, nor electronics hungry for electricity.  Sure, we can use language like that, and people will understand what we mean, but as cars and electronics have a common established context, these people we're saying this to don't usually then go on to worry about cars doing stuff to get more gas to "feed" themselves, as it were.

I think if we're being serious about safety, and how to manage unintended consequences (a real concern with any system^[2]), we should aim for clarity and transparency.

In sum, I'm a huge fan of "new" words, versus overloading existing words, as reuse introduces a high potential for causing confusion.  I know there's a paradox here, because Communication and Language, but we don't have to intentionally make it hard — on not only ourselves — but people getting into it coming from a different context.

All that said, maybe people should already be thinking of inanimate objects being "alive", and really, for all we know, they are!  I do quite often talk to my objects.  (I'm petting my computer right now and saying "that's a good 'puter!"… maybe I should give it some cold air, as a reward for living, since thinking gets it hot.) #grateful

1. ^{^}

   deceptive? for a certain definition of "deceptive" as in "fooled yourself", sure— maybe I should note that I also think "deceptive" and "lie" are words we probably should avoid— at least for now— when discussing this stuff (not that I'm the meaning police… just say'n)

2. ^{^}

   I don't mean to downplay how badly things can go wrong, even when we're actively trying to avoid having things go wrong^[3]

3. ^{^}

   "the road to hell is paved with good intentions"

Isn’t it fair to say that the model plus the selection mechanism is maximizing and wanting the reward? If the selection mechanism is subject to market forces, corporate or political schemes, or just mechanical in some way that isn’t just “a human is explicitly making these choices” it is likely to eventually tend in a direction that doesn’t align with human welfare.

Corporations already generate tons of negative externalities. They churn out metric tons of plastic, pollute, exhaust resources, destroy ecosystems. Governments often work with them to enforce various things, eg intellectual property for monsanto, or engineering crops with high fructose corn syrup, or overusing antibiotics on factory farms that can lead to superbugs. Or overfishing. Or starting wars.

None of these things are really “aligned” with humans. Humans are in fact told that as an individual they can make a meaningful differency by going vegan, recycling, exercising, and so on. But if the corporations are the ones producing metric tons of plastic, conserving plastic straws and bags isn’t the solution. The problem is upstream and the individuals being shamed to do this or that is just distracting from getting together solving systemic problems.

My point is “the machine” is already not necessarily selecting for alignment with humans on a macro scale. So the fact that the model parameters are selected by “the machine” doesn’t mean it will end up somehow becoming good for humans. Yes it will if humans are the customer. But if you are not the customer, you’re the product (eg a cow in a factory farm).

Anyway … this is a bit like John Searle’s Chinese room argument.

There are probably enough comments here already, but thanks again for the post, and thanks to the mods for curating it (I would've missed it otherwise).

Hi thanks for share this interesting perspective on RL as a training process! Although it seems to only be a matter of seeking vs obeying and reward vs cost, the effect on the reader's mind seem to be huge!

One thing that seems to be happening here and I have not fully digested is the "intrinsicness" of rewards. In frameworks parallel to mainstream RL, such as active inference and the free energy principle, policy is a part of the agent's model such that the agent "self-organizes" to a characteristic state of the world. The policy can be constructed either through reward or not. However, in the active inference literature, how policies are constructed in real agents are currently unanswered (discussions exist but don't close the case). 

How this intrinsic perspective is related to the post and safety and alignment? I am still thinking about it. If you have any thoughts please share!