Reframing inner alignment

post by davidad · 2022-12-11T13:53:23.195Z · LW · GW · 13 comments

Contents

  Mesa-optimisers
  Inner alignment
    Kinds of inner alignment that could work
  Alternatives to inner alignment for Adequate Policy Learning
    Implications (briefly)
None
13 comments

The standard frame (Evan Hubinger, 2021 [AF · GW]) is:

• Outer alignment refers to the problem of finding a loss/reward function such that the training goal of “a model that optimizes for that loss/reward function” would be desirable.
• Inner alignment refers to the problem of constructing a training rationale that results in a model that optimizes for the loss/reward function it was trained on.

Here’s the reframe (I believe the credit for this breakdown is due to John Wentworth, although I haven't found anything online to link to for it):

• Reward Specification: Finding a policy-scoring function $J\left(\pi \right)$ such that (nearly–)optimal policies for that scoring function are desirable.
  • "Are you optimising for the right thing?"
• Adequate Policy Learning: Finding a policy that’s actually (nearly–)optimal for that scoring function.
  • "Did you optimise for it enough?"

As Paul Christiano points out (in an excerpt recently highlighted by Alex Turner [AF · GW] to make a similar point), factoring out Reward Specification represents only one "particular kind of alignment strategy, that's like a two step plan" for how one might try to conclude that a learned policy is desirable, where we first align a scoring function $J$ with what we actually want, and then align a training process with our scoring function. Under this kind of plan, the proof tree for overall existential safety would conclude by applying an implication like this:

$$\begin{array}{cccc}& \exists J:typeof\left(\pi \right)\to R,\epsilon :R.& & \text{Reward Specification}\\ & |J\left(\pi \right)-{max}_{{\pi }^{\ast }}J\left({\pi }^{\ast }\right)|<\epsilon & & \\ & \Rightarrow safelyMitigatesXRisk\left(\pi \right)& & \\ \wedge & \exists T:\Omega \to typeof\left(\pi \right).& & \text{Adequate Policy Learning}\\ & P_{\pi \sim T}(|J\left(\pi \right)-\underset{{\pi }^{\ast }}{max}J\left({\pi }^{\ast }\right)|<\epsilon )>p& & \\ \Rightarrow & P_{\pi \sim T}\left(safelyMitigatesXRisk\right(\pi \left)\right)>p& \Rightarrow & \text{Probably ends acute risk period}\end{array}$$

Mesa-optimisers

At this point you might be wondering: what happened to the concept that $\pi$ might contain a mesa-optimiser for something different from $J$?

The proper role of the mesa-optimiser concept is in an explanation of why Adequate Policy Learning is insidiously difficult:

1. In machine learning, one typically approximates (the gradient of) the reward function by aggregating a set of evaluations on only a finite sample (the empirical “training distribution”), whereas the true $J$ also depends on what $\pi$ does globally, even in rare states that may never be sampled during training.^[1]
2. If the trajectories observed in deployment have a state distribution which diverges from the training distribution, then the policy that was optimal on the training set will likely diverge from the optimal policy. This is standard out-of-distribution generalisation failure.
3. If the policy that was optimal on the training set includes a good understanding of the real world and consequentialist reasoning (which is likely, if the training set is sufficiently rich that you can get a policy that's transformative in the real world), then it contains a capable mesa-optimiser, and it's likely that it's still transformative in the real world, even while experiencing generalisation failure: capabilities generalise better than alignment [LW · GW].

The bottom line is, once we get beyond some level of capabilities, it becomes likely that the gap between the empirical training distribution and the intended reward function suddenly gets exploited in a pretty worst-case-like way (i.e. optimising for some arbitrary thing, instead of merely being bad at optimising the outer objective). I think this phenomenon is also what the concept of a sharp left turn [LW · GW] refers to, so this post could equally also be titled "Reframing the sharp left turn".

Inner alignment

The standard "inner alignment" answer to "what do we do about this?" is, to quote Paul again^[2],

let’s build a system that’s internalized that objective in some sense, or is not doing any other optimization beyond pursuit of that objective.

In other words: let's make sure that the policy $\pi$ contains a mesa-optimiser for the outer objective $J$, or at least, contains no other mesa-optimisers.

If we assume outer alignment and Reward Specification are solved, the above notion of inner alignment is neither necessary^[3] nor sufficient^[4] for actually establishing that the learned policy is desirable. I think I'm in agreement with Alex Turner on this point [AF · GW], but I disagree insofar as I still think the reframed 2-step strategy of Reward Specification and Adequate Policy Learning is sufficient, and helpful, i.e. maybe (relatively) tractable (albeit still unnecessary: just one possible kind of decomposition of the alignment problem).

Both "inner misalignment" and "sharp left turn" are just particular stories for why an unbounded out-of-distribution optimality gap is a big problem for the safety of highly capable policies. One does not solve that problem by detecting internal computations with optimiser-like structure, or by installing penalty-function guardrails on the left-hand-side of some internal state-space. If an "inner alignment" solution to Adequate Policy Learning uses an inner-misalignment-detector which is not invariant to extensional equivalence of policies, it is certainly not robust: formally, it cannot be continuous relative to any natural topology on policy-space, let alone Lipschitz-continuous relative to a natural metric on policy-space.

Kinds of inner alignment that could work

That doesn't mean inner alignment is categorically doomed; there are (at least) two ways to do something like looking at internal structure while still respecting extensional equivalence:

1. Use the internal structure only to accelerate the computation, but in such a way that extensionally equivalent internal structures would provably result in equivalent outputs. An example of this is the backpropagation algorithm for computing derivatives of a policy.
2. Search for possible internal-structure-like explanations for the extensional behaviour of a policy that have some intensional “misalignment” property—but without being biased toward structures that resemble how the policy function happens to actually be encoded as a neural network.

Alternatives to inner alignment for Adequate Policy Learning

In my view, the best paths to confidence in Adequate Policy Learning, i.e. that a learned policy actually performs well on the outer objective are, well, provable performance guarantees on the outer objective, e.g. via:

• policy certificates
• regret bounds
• model checking

However, it is important to note that the idea of proving something about the outer objective means the outer objective must be formally well-defined, as a function from $typeof\left(\pi \right)\to R$, and that means it needs to incorporate a well-defined formal world-model. This takes us out of the end-to-end model-free RL setting, but it is equally important to note that it does not take us out of the "deep learning" setting! Deep neural networks can still be used as function approximations for policy and value functions in model-based RL. Model checking seems likely to be a particularly computationally-feasible way to get concrete guarantees about such models.

It is even possible, in this paradigm, to still use deep neural networks to simulate the dynamics of the world (purely predictively, and distinct from the policy and value networks). But, if we do that, we'll need ontology identification for the predictive dynamics network in order to have any hope of solving Reward Specification.

Implications (briefly)

On this view, ontology identification—applied to a purely predictive dynamics network, for the purpose of specifying rewards within its state space—is the proper role of alignment-related problems like Natural Abstractions [AF · GW] and ELK [? · GW] (as opposed to something like "asking the AI if it has any submodules that are starting to think about planning to be deceptive"). I'm also excited about Boundaries [? · GW] as a tool for specifying a core safety property to model-check policies against—one which would imply (at least) nonfatality—relative to alien and shifting predictive ontologies.

1. ^{^}

   In Solomonoff induction there is a similar situation, but instead of having an empirical distribution of trajectories, we actually just look at one single ultra-long trajectory and hope that the world is approximately ergodic. Spoiler alert, it isn't. Following a parallel chain of reasoning leads to a similar conclusion about, after some level of capabilities, mesa-optimisers systematically causing catastrophic out-of-distribution generalisation failure but still being highly capable; cf. The Solomonoff Prior is Malign [LW · GW].

2. ^{^}

   Out of context, as Paul is not endorsing this frame; shortly thereafter he says

   I think this makes sense on some approaches and not on other approaches. I am most often thinking of it as: there’s some set of problems that seem necessary for outer alignment. I don’t really believe that the problems are going to split into “these are the outer alignment problems, and these are the inner alignment problems”. I think of it more as the outer alignment problems, or the things that are sort of obviously necessary for outer alignment, are more likely to be useful stepping stones, or warm up problems, or something. I suspect in the end, it’s not like we have our piece that does outer alignment and our piece that does inner alignment, and then we put them together.

3. ^{^}

   For example, a policy which is actually just an uploaded human will contain all sorts of mesa-optimisers, but it's generally agreed that there exist at least a few humans who are nonetheless robustly ethical enough (as a composite system) that uploads of them would be existentially safe.

4. ^{^}

   I think the standard response to this is "well, yes, technically, of course the policy could just be incompetent, but that's beside the point as long as it's trying to be aligned," to which I would respond "never attribute to malice that which can be explained by incompetence." Decomposition between beliefs and desires is fundamentally non-unique. There is no clean definition of incompetence which excludes policies that still kill you:

   Oops, I accidentally consequentialisted the atmospheric oxygen away. It's so corrosive and harmful but I totally forgot that humans still need it. I can be so clumsy sometimes! Gosh, this is really bad. This isn't what I wanted at all. I should update to be more liberal about asking for human feedback before I execute plans. If only there were any humans left to ask...

13 comments

Comments sorted by top scores.