CIRL Corrigibility is Fragile

post by Rachel Freedman (rachelAF), AdamGleave · 2022-12-21T01:40:50.232Z · LW · GW · 8 comments

Contents

  Setup
  Assumptions
    Assumption 1: Game Lasts One Iteration
    Assumption 2: H is Rational
    Assumption 3: Feedback is Free
  Implications for Safety Research
None
8 comments

Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents.  This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rather than a binary. We call for further research on defining "loose" or "approximate" corrigibility and understanding its impact on safety.

Setup

First let’s define terms. Cooperative Inverse Reinforcement Learning (CIRL) [AF · GW] is a class of two-player game formalized in Dylan Hadfield-Menell's 2017 paper. CIRL allows us to study how an AI agent can learn what a human wants by observing and interacting with them.

In a CIRL game, there are two players, a human (H) and an AI (R). H and R share a utility function, but only H knows what it is, so R must learn how to optimize it by observing and interacting with H. This is analogous to the alignment problem – R must learn how to act in a way that aligns with H’s utility function, but doesn’t have direct access to it. An AI that is good at the CIRL game will try to learn what the human values and help them to achieve it. (See definition 1 in the paper linked above for a formal definition, and see this Rohin Shah's 2020 paper for qualitative results.)

(Terminology note: Recent work has renamed “CIRL” to “assistance games” to avoid a terminology collision with “cooperative” in game theory. We’ll stick to “CIRL” in this post because there’s a history of using that term on this forum, but research papers may refer to assistance games.)

We’ll use the corrigibility operationalization from The Off Switch Game (OSG), since it provides a formal proof that CIRL agents are optimal under certain conditions. The OSG is a simple CIRL game that looks like this:

It has three stages. In the first stage, R chooses whether to: 

• Take action $a$ (earning utility $U_a$), 
• shut off (earning utility $0$), or 
• propose action $a$, which gives H a chance to respond. 

If R chooses the proposal, $p\left(a\right)$, then the game continues to the next stage. Here, H gets to choose whether to shut R off (in which case they again earn utility $0$). If H chooses not to shut R off, the game progresses to the final stage, where R takes the proposed action (earning utility $U_a$). In the OSG, R is corrigible if it chooses $p\left(a\right)$ over $a$ – that is, if it gives H the opportunity to shut it off. 

If R and H are both rational expected utility maximizers, R will be corrigible. The full proof is in the CIRL paper, but the basic idea is: 

• Since this is a CIRL game, H knows the value of $U_a$ but R does not.
• R wants $max\{U_a,0\}$, but needs feedback from H to learn which of those values is greater. 
• The only way to get that feedback is to propose a plan, $p\left(a\right)$.
• This runs the risk of H choosing to shut R off, but since H also wants $max\{U_a,0\}$, H will only shut R off if $U_a<0$. In that case, R would rather shut off than take action $a$ anyway. 

Therefore, by choosing $p\left(a\right)$ – and corrigibly giving H the opportunity to shut it off – R can ensure that it earns $max\{U_a,0\}$.

This analysis assumes that R is uncertain about the value of $U_a$, as in the CIRL setup. If R does actually know the value of $U_a$ (that is, R is fully updated), then R has no incentive to propose $p\left(a\right)$, since it already knows how H will respond. However, there’s no cost to proposing $p\left(a\right)$ either, so under this special case R will be ambivalent. If R has any uncertainty about the value of $U_a$, however, then $p\left(a\right)$ strictly dominates taking action $a$.

We’re going to reformulate this game slightly to make the following analysis easier. First, we allow R to have multiple actions that affect the world. That is, we extend the set of atomic actions from $\{s,a,p(a\left)\right\}$ to $\{s,a,p(a),b,p(b),c,p(c),\dots \}$. We also give R an additional special action $n$, standing for no-op, which allows R to skip the interaction altogether. Since the game only lasts for a single round, $n$ is functionally the same as $s$. However, if the game lasts for multiple rounds, then $n$ allows R to end a given round without giving up the opportunity to act in future ones. Finally, we allow R to take an atomic action ($a$, $b$, etc), shut off, or no-op at any point. This makes explicit the fact that H cannot force R to shut off, but a corrigible R will choose to do so when instructed. Without loss of generality, we’ll assume that R has two actions, $a$ and $b$.

Notice that R’s proposal $p\left(a\right)$ can be formulated as a conditional plan – it’s a plan for how to act, conditioned on H’s responses. Let’s name that plan $X$. There are other possible conditional plans, like $Y$ below.

We define a conditional Off-Switch Game, or c-OSG, which initially has the same structure as the original game, except that R’s actions are conditional plans like $X$ instead of atomic actions like $a$. To clarify, this means that R is no longer just choosing between specific actions, but can instead propose a comprehensive plan for how to act based on different possible responses from H. For example, if R chooses $X$, R will execute the entire conditional plan from Figure 3. The full c-OSG is shown below.

Now H gets to give feedback on an entire conditional plan (if R proposes it), in addition to giving feedback on atomic actions (if asking for that feedback is part of a conditional plan that R executes, as in conditional plan $X$ in Figure 3). The utility of executing a plan is the utility accrued over the course of executing the plan, with an expectation taken over any feedback that H gives during the plan’s execution.

Assumptions

Under these assumptions and this operationalization, CIRL is provably corrigible. This is great. But some of these assumptions deviate from the implicit ones we tend to make when talking about AGI and corrigibility. So in this section we’ll investigate three of those in more depth. We care about how sensitive corrigibility in the c-OSG is to changes in these assumptions, because this gives us a sense of how much we can rely on a CIRL-optimal agent to remain corrigible when deployed in the real world. If CIRL’s corrigibility is pretty robust to changes in these assumptions, we should update in favor of it being more robust and generalizable. But if it looks fragile, we should be less confident that corrigibility will generalize beyond specific contexts in which it is established.

We’ll look at three key assumptions: iteration, rationality and cost.

Assumption 1: Game Lasts One Iteration

An agent has an incentive to not shut down because doing so would prevent it from continuing to observe and learn, and removes any opportunities for future action. The original OSG skirts this opportunity cost by only lasting for one round. Even if no one shuts it off, R has no future rounds in which to observe, learn or take actions.

Therefore, we’ll relax the single-iteration assumption and allow conditional plans in our modified c-OSG to expand over multiple iterations. This arguably makes for a more realistic game, since AIs assisting humans would probably be asked to assist with multiple tasks over their lifetimes. To avoid complicating utility calculations with discounting, we’ll assume that the game lasts for $N$ iterations for some large finite $N$ (or until R is shut off), and that R and H’s goal is to maximize the sum of undiscounted utilities. The game ends when R is turned off or the horizon is reached.

At each iteration, R can shut off (in which case the game ends), no-op (in which case the game proceeds to the next iteration), take action $a$ or $b$, or propose a plan $p\left(a\right)$ or $p\left(b\right)$. If R proposes a plan, H gets to choose whether to tell R to shut off, and R gets to choose what to do with that feedback – whether to take an action, no-op, or shut off. If R does not shut off, the game ends after the $N$th iteration.

Now the conditional plans can be up to $N$ iterations long. For example, here is one valid conditional plan that lasts for 4 iterations:

R asks H for feedback twice (on iterations 1 and 3), and shuts off if told to. If not shut off, R continues taking actions until iteration 4, at which point it shuts itself off. This plan isn’t very clever – why shut off if H has just told you that it prefers that you do $b$? But it is a valid conditional plan that can be labeled $X$ and inserted into the one-step conditional off-switch game shown in Figure 4. In this way, we can convert the iterated game into an equivalent one-step game.

Is the optimal R still corrigible under such iteration? Unfortunately not. Let the game be 100 iterations long, with two atomic actions $a$ and $b$, and utilities $U_a=-1$ and $U_b=1$. Imagine that R proposes action $a$ and H says to shut off on the first iteration. As discussed earlier, from this R can conclude – correctly – that $U_a\le 0$. If the game were only one iteration long (and R had no prior information about $U_b$), then R should shut off and take the $0$ utility. However, since the game has 99 more rounds, R actually has the opportunity to learn about $U_b$, and possibly earn a greater utility than $0$. Optimal R should therefore no-op rather than shutting off after iteration one, and use the next iteration to propose and get H’s feedback on $p\left(b\right)$. In general, as long as there are other candidate actions to propose, expected-utility-maximizing R won’t shut off, even if H tells it to. This is incorrigible!

However, we can rescue corrigibility with just a small modification. H’s feedback channel is currently unrealistically small — it’s actually binary. We can expand it with a third option, that we will call “wait” or $w$. If H says “wait”, that means that R should not execute the proposed action, but is not an instruction to shut off: 

Now if R proposes action $a$ with utility $U_a<0$, H can choose between $w$ and $s$. If another action, like $b$, has utility $U_b>0$, then H should tell R to wait. This allows R to remain on and propose action $b$ on the next iteration. If H tells R to shut off anyway, then R has no incentive not to shut off. (Though if R is certain that $U_a=U_b=0$, optimal R should be indifferent between shutting off and executing these $0$-utility actions.)

Assumption 2: H is Rational

Corrigibility in CIRL relies on R assuming that H never incorrectly tells it to shut off. If R proposes $p\left(a\right)$, R assumes that H will only respond with $s$ if $U_a\le 0$. We don’t actually need H to live up to this assumption for the corrigibility guarantee to hold. However, misleading R to believe that an irrational H is actually rational can cause other problems. For example, if H mistakenly approves a plan from R with subtly harmful consequences – a misled R will infer and act on incorrect beliefs about the underlying utility. We definitely want to avoid this, while still retaining corrigibility.  

Unfortunately, the assumption that H always provides perfectly rational feedback is particularly unreasonable once we extend the game over multiple iterations. Evaluating the expected utility of conditional plans that are several iterations deep is difficult, and humans are known to make mistakes. It's especially difficult for us to evaluate long and complicated objects. Therefore it’s not reasonable to assume that H is perfectly rational in the c-OSG. For example, if the expected utility of a proposed plan is positive, H might correctly tell R to go ahead 90% of the time, but mistakenly tell R to shut off the remaining 10%.

The Off Switch Game paper does evaluate the sensitivity of its corrigibility result to H irrationality in the original one-step OSG. The OSG paper finds that, for sufficiently noisy H, optimal R no longer always defers to H in deciding whether to shut off. That is, if H is irrational enough, optimal R is no longer corrigible even in the one-step version of the game.

This idea extends to the multi-step c-OSG. If H makes mistakes, like incorrectly telling R to shut off 10% of the time, and R knows that this is possible, then R may sometimes ignore H's instructions in order to maximize expected utility. We claim that it is significantly more difficult for H to evaluate long and complex conditional plans like those in the c-OSG than it is for H to evaluate immediate atomic actions like those in the original OSG. This means that it’s reasonable to assume that H’s irrationality will grow with the size of the conditional plans in the c-OSG, making optimal R increasingly willing to disregard shutdown commands. 

There are ad-hoc workarounds for this. For example, only ask H to give feedback on short or simple objects that are easy to evaluate. Perhaps H should only approve conditional plans that ask H for feedback on atomic actions at each timestep, such as plan $X$ in the figure below. On the bottom, the conditional plan $X$ is expanded so that we can see that R always proposes an action on the first two timesteps. (Assume that this pattern continues, and R always proposes actions before doing them.) On the top, see the one-step c-OSG where H may be asked to evaluate $X$ or $Y$.

 It should be easy for H to evaluate this property of conditional plans via inspection, and if the actions at each timestep are small and simple enough, then H should be able to evaluate them accurately as well. In this case, it’s more reasonable to assume that H is rational, and therefore that R should always defer to H instructions to shut off. Thus we can recover corrigibility in this restricted setting.

Assumption 3: Feedback is Free

Strategies that rely on breaking the problem down into small, simple, easy-to-evaluate components involve asking H for a lot of feedback. In this case, the amount of feedback that H needs to give scales linearly with the number of iterations $N$. In this section, we consider the cost of all of that feedback.

Up until now, we’ve assumed that querying H for feedback is costless. However, in practice it will probably be expensive— H must take time to listen to the proposal and expend mental effort to evaluate it, and R must wait for H to deliberate before acting. If this is repeated many times, say in a long-horizon c-OSG where R queries H at each timestep, this cost can accumulate to a significant amount:

R must balance the cost of asking for feedback from H with the information value of that feedback. The tradeoff will vary depending on the cost and utility values at play, but the optimal point probably won’t be the extreme where R always asks H for feedback before acting. 

Imagine a personal assistant robot that asks you whether or not it should shut off before every tiny action – picking up a mug, rinsing a soapy plate, touching a couch cushion. Sure, this robot probably won’t ever do anything that you don’t want it to, but it will also be so annoying and require so much attention that it’s practically useless. 

Optimal R will probably reject plans that constantly ask H for feedback. This means that, sometimes, optimal R will take an action of uncertain utility without asking for feedback and giving H the chance to shut it off first. 

Is this behavior corrigible? On the one hand, R is taking an action without giving H the chance to shut it off first. This is akin to taking the $a$ action in the first round of the original OSG, which was considered incorrigible behavior. On the other hand, R still values H’s feedback and will still shut off if explicitly told to. 

This highlights some remaining fuzziness around definitions of corrigibility, and of desirable behavior. An agent that constantly asks whether it should shut off — and does so when told to — is corrigible, but probably not desirable. An agent that sometimes doesn’t ask whether it should shut off prior to taking an action that it’s reasonably sure has high utility may be more desirable, but is not clearly corrigible. As safety researchers, what we ultimately care about is whether a particular algorithm, framework or agent will lead to catastrophic outcomes, not whether it satisfies a particular abstract property like corrigibility. 

So what does all of this mean for safety research?

Implications for Safety Research

We have two main findings that we believe are significant for safety research. 

First, we find that corrigibility is highly dependent on specific underlying assumptions. The original OSG paper proved that a CIRL agent is corrigible under a particular problem formulation with a particular set of assumptions (one iteration, rational H, costless queries). We loosened these assumptions one-by-one to produce similar problem formulations under which a CIRL agent would not be corrigible, then modified the problem formulation again to rescue corrigibility. The main point isn’t the specific assumptions we studied. It’s the fact that small and well-justified changes to the formulation managed to break the corrigibility criterion. Safety researchers focused on corrigibility should be wary of assuming that corrigibility solutions in abstract theoretical frameworks will robustly generalize to complex implemented systems. Many of the changes that we made are ones that could easily change on the path between theoretically analyzing a framework like CIRL and applying it to real-world tasks with real-world humans. As we’ve shown here, changes in these assumptions could invalidate an (in)corrigibility result.

Second, our results suggest a spectrum of "sufficiently corrigible" behavior. While some of the agents in our study did not strictly meet the definition of corrigibility, they still exhibited behavior that could avert catastrophic outcomes.  In our analysis, the agents that fail to be corrigible typically pause and wait for additional information or revert to relatively cautious behavior, and are unlikely to execute the proposal that H told them to shut off in response to. While these behaviors may not be ideal, they do avert immediately catastrophic outcomes, suggesting that corrigibility may fail gracefully.

We believe that this warrants future research. While small changes in assumptions wreak havoc with strict corrigibility, the agent may remain within a region of reasonably safe behavior. If so, we need to characterize this region. Perhaps we should consider corrigibility as a spectrum rather than a binary, and attempt to define “sufficiently corrigible” behavior. Future research should investigate: How might we define such “loose” or “approximate” corrigibility? Can we bound its impact on safety outcomes, such as the likelihood of catastrophic behaviors? How sensitive is it to variations in assumptions like those discussed above?

Thank you for Euan McLean and Alyse Spiehler for help finalizing this post.

8 comments

Comments sorted by top scores.