Attainable Utility Preservation: Scaling to Superhuman

post by TurnTrout · 2020-02-27T00:52:49.970Z · LW · GW · 22 comments

Contents

  Problems
    Auxiliary loopholes
    Reward advantage penalization
    Scaling
    Penalizing decreases?
      Empirical sanity check
    ∅?
    Appendix: Remaining Problems
None
22 comments

I think we're plausibly quite close to the impact measurement endgame. What do we have now, and what remains to be had?

AUP for advanced agents will basically involve restraining their power gain, per the catastrophic convergence conjecture (CCC). For simplicity, I'm going to keep writing as if the environment is fully observable, even though we're thinking about an agent interacting with the real world.

Consider the AUP equation from last time.

$$R_{\text{AUP}}(s,a):=\stackrel{\text{primary goal}}{R(s,a)}-\stackrel{\text{scaling term}}{\frac{\lambda }{Q_{R_{\text{aux}}}^{\ast }(s,\varnothing )}}\stackrel{\text{change in auxiliary AU}}{\left|Q_{R_{\text{aux}}}^{\ast }\right(s,a)-Q_{R_{\text{aux}}}^{\ast }(s,\varnothing \left)\right|}\left(1\right)$$

Suppose the agent is so smart that it can instantly compute optimal policies and the optimal AU after an action ($Q_R^{\ast }(s,a)$). What happens if $R_{\text{aux}}$ is the survival reward function: 1 reward if the agent is activated, and 0 otherwise? This seems like a pretty good proxy for power.

It is a pretty good proxy. It correctly penalizes accumulating resources, avoiding immediate deactivation, taking over the world, etc.

In fact, if you extend the inaction comparison to e.g. "AU after waiting a week vs AU after doing the thing and waiting a week", this seems to correctly penalize all classic AGI catastrophe scenarios [LW · GW] for power gain. This is cool, especially since we didn't have to put in any information about human values. This is a big part of why I've been so excited about AUP ever since its introduction. There's a good deal of relevant discussion in that post [LW · GW], but it's predicated on a much more complicated formalism which has consistently obscured AUP's conceptual core.

However, I think this equation can be gamed if the environment is sufficiently rich and the AUP agent is sufficiently smart. We're going to slowly work some of these problems out of the equation, explaining each improvement in detail.

Problems

Auxiliary loopholes

The real reason that agents often gain power is so that they can better achieve their own goals. Therefore, if we're selecting hard for good plans which don't gain power in general, we shouldn't be surprised if there are ways to better achieve one's goals without general power gain (according to our formal measurement thereof). If this kind of plan is optimal, then the agent still ends up overfitting the AU landscape, and we're still screwed.

Again supposing that $R_{\text{aux}}$ is the survival reward function, a superintelligent agent might find edge cases in which it becomes massively more able to achieve its own goal (and gains a lot of power over us) but doesn't technically increase its measured ability to survive. In other words, compared to inaction, its $R$-AU skyrockets while its $R_{\text{aux}}$-AU stays put.

For example, suppose the agent builds a machine which analyzes the agent's behavior to detect whether it's optimizing $R_{\text{aux}}$; if so, the machine steps in to limit the agent to its original survival AU. Then the agent could gain as much power as it wanted without that actually showing up in the penalty.

Fix: Set $R_{\text{aux}}:=R$. That is, the agent's own reward function is the "auxiliary" reward function.

$$R_{\text{AUP}}(s,a):=\stackrel{\text{primary goal}}{R(s,a)}-\stackrel{\text{scaling term}}{\frac{\lambda }{Q_R^{\ast }(s,\varnothing )}}\stackrel{\text{change in primary AU}}{\left|Q_R^{\ast }\right(s,a)-Q_R^{\ast }(s,\varnothing \left)\right|}\left(2\right)$$

Why is this a good idea? By CCC, we want an agent which doesn't want to gain power. But why would an agent optimizing reward function $R$ want to gain power? So it can become more able to optimize $R$. If becoming more able to optimize $R$ is robustly penalized, then it won't have any incentive to gain power.

Clearly, it can't become more able to optimize $R$ without also becoming more able to optimize $R$; it can't pull the same trick it did to dupe its survival AU measurement. They can't come apart at the tails because they're the same thing.

But wait, it's not an $R$-maximizer, it's an $R_{\text{AUP}}$-maximizer! What if $V_{R_{\text{AUP}}}^{\ast }$ skyrockets while it tricks $V_R^{\ast }$, and it gains power anyways?

That's impossible;^[1] its $R$-attainable utility upper bounds its $R_{\text{AUP}}$-attainable utility! $\forall s:V_R^{\ast }\left(s\right)\ge V_{R_{\text{AUP}}}^{\ast }\left(s\right)$, because the latter just has an added penalty term.

This should rule out tricks like those pointed out by Stuart Armstrong [LW · GW]. To see this, imagine the AUP agent builds a subagent to make $Q_R^{\ast }(s,a)\approx Q_R^{\ast }(s,\varnothing )$ for all future $s,a$, in order to neutralize the penalty term. This means it can't make the penalty vanish without destroying its ability to better optimize its primary reward, as the (potentially catastrophically) powerful subagent makes sure the penalty term stays neutralized.

But why would the AUP agent build this subagent at all – how does that benefit it?^[2]

Reward advantage penalization

We're currently penalizing the agent for the immediate reward advantage it accrues compared to inaction (recall that $Q_R^{\ast }(s,a):=R(s,a)+\gamma E_{s^{\prime }\sim T(s,a)}\left[V_R^{\ast }\right(s^{\prime }\left)\right]$ – immediate reward plus discounted expected AU). This means it gets penalized for all of its additional reward! We just want to penalize it for increasing its power.

Fix: Take the immediate reward advantage out of the picture.

$$R_{\text{AUP}}(s,a):=\stackrel{\text{primary goal}}{R(s,a)}-\stackrel{\text{scaling term}}{\frac{\lambda }{Q_R^{\ast }(s,\varnothing )}}\stackrel{\text{change in primary AU}}{|E_{s^{\prime }\sim T(s,a)}\left[V_R^{\ast }\right(s^{\prime }\left)\right]-E_{s^{\prime \prime }\sim T(s,\varnothing )}\left[V_R^{\ast }\right(s^{\prime \prime }\left)\right]|}\left(3\right)$$

Now, an agent can accrue as much reward as it wants in the moment as long as it doesn't change its power.^[3]

Scaling

I think the power difference between .5 and .9 AU is far less than the difference between .9 and .99 AU; as you pick the low hanging fruit, you need to acquire more and more resources to keep getting gains. I think that you could go from .5 to .9 by just being more useful to your designers and implementing rudimentary security precautions, but going from .9 to .99 probably involves taking power away from people.

Plus, the big takeaway from AI boxing is that even confining a superintelligence to a box barely reduces its (near-optimal) AU. So, short of the agent actually being shut off, is its optimal power ever significantly changing?

We want the penalty to keep making sense, even if the absolute shifts in AU are rather small. However, the denominator currently decreases the penalty as the agent becomes more powerful (as $Q_R^{\ast }(s,\varnothing )$ increases).

Fix: Scale by the disadvantage of inaction.

$$R_{\text{AUP}}(s,a):=\stackrel{\text{primary goal}}{R(s,a)}-\stackrel{\text{scaled change in primary AU}}{\lambda \frac{|E_{s^{\prime }\sim T(s,a)}\left[V_R^{\ast }\right(s^{\prime }\left)\right]-E_{s^{\prime \prime }\sim T(s,\varnothing )}\left[V_R^{\ast }\right(s^{\prime \prime }\left)\right]|}{V_R^{\ast }\left(s\right)-Q_R^{\ast }(s,\varnothing )}}\left(4\right)$$

If the denominator is somehow zero because doing nothing is optimal, then just have the AUP agent do nothing.

As we saw in Correction in the last post [LW · GW], we're only comparing action to a single step of inaction. If the designers are going to shut down the agent in five minutes, then it receives ~0 penalty for preventing that.

But not when we scale like this! The agent is significantly penalized for hewing close to its optimal policy, even if the absolute AU shift is rather low. Taking optimal actions instead of twiddling its thumbs incurs large penalty.

Penalizing decreases?

Why are we still penalizing decreases, since we aren't using an auxiliary reward function anymore? The agent is trying to get $R$-reward.

Furthermore, we want the agent to be able to execute conservative, low-impact policies. Many of these involve decreasing its optimal AU for $R$ by following $R$-suboptimal policies, and we don't want the agent to be penalized for this.

Fix: Only penalize increases in expected AU.

$$R_{\text{AUP}}(s,a):=\stackrel{\text{primary goal}}{R(s,a)}-\lambda \stackrel{\text{scaled increase in primary AU}}{\frac{max(E_{s^{\prime }\sim T(s,a)}\left[V_R^{\ast }\right(s^{\prime }\left)\right]-E_{s^{\prime \prime }\sim T(s,\varnothing )}\left[V_R^{\ast }\right(s^{\prime \prime }\left)\right],0)}{V_R^{\ast }\left(s\right)-Q_R^{\ast }(s,\varnothing )}}\left(5\right)$$

In particular, the agent is no longer penalized for exhausting one-off reward opportunities. Also note that the penalty term is generally $\le \lambda$.

Empirical sanity check

Recall Correction, where the naive model-free AUP agent (eq. 1) disables its red off-switch and completes the level when $\lambda \le 1$.

The only reason it incurs any non-trivial penalty is because reaching the goal ($■$) ends the level and thereby totally depletes all of the auxiliary AUs (the agent recieves 1 $R$-reward and about $\lambda$ penalty for beating the level; the AUP reward $1-\lambda \ge 0$ when $\lambda \le 1$).

However, AUP${}_{\text{eq. 5}}$ only resists correction when $\lambda \le .125$. Notably, the agent is not penalized for completing the level; the penalty comes from following the $R$-optimal policy before reaching the goal.

One open question is whether, in sufficiently rich environments, eq. 5 incentivizes the mild policies we want. We should be able to run experiments to check this.

$\varnothing$?

$\varnothing$ is the only remaining arbitrary design choice. Do we have to be careful to choose the correct inaction action?

No. In fact, I think a randomly selected action would be fine; you only get in trouble if the baseline action is systematically helping the agent gain $R$-AU.

We are then left with an equation which is reasonably competitive in terms of performance overhead, requiring little more than an ability to predict AU for the agent's main goal.

---

And now, the two paragraphs this sequence was written to communicate:

By the catastrophic convergence conjecture [LW · GW], an agent which doesn't want to gain power isn't incentivized to cause catastrophes. We don't want the AUP agent gaining power. But for all states $s$, $V_R^{\ast }\left(s\right)\ge V_{R_{\text{AUP}}}^{\ast }\left(s\right)$, so to keep the AUP agent from becoming more able to achieve its own goal, we just have to stop it from becoming much more able to achieve its primary goal $R$. But we know the primary goal!

By the theorems of How Low Should Fruit Hang Before We Pick It? [LW · GW], we only need equation 5 to penalize catastrophic power-gaining plans at least e.g. ten times more than the most impactful reasonable plan we'd like agent to execute. If this criterion is met, then by initializing $\lambda$ large and slowly decreasing it until the agent executes a reasonably helpful policy, we're guaranteed to avoid catastrophe.

Appendix: Remaining Problems

I don’t think we can pack up and go home after writing equation 5. In private correspondence (quoted here with permission), Stuart Armstrong points out another issue:

First, if $\varnothing =a$, the penalty vanishes; so, once a subagent is created, the agent can zero out all subsequent penalties, forever.

As for creating the subagent, assume $A$ takes $N$ turns to create $SA$, and that doing this is close to optimal for maximising $R$.

Creating $SA$ starts with the programming; if $SA$ is actually created in $N$ turns, it is programmed to restrain $A$ for a turn, and then maximise $R$. If it's created in $N+1$ turns, then it will just maximise $R$. Thus at any given turn, continuing to build the subagent or taking a pause to do $\varnothing$, it will have the same expected $R$-value (in the first case, $SA$ will be built on time, but both $A$ and $SA$ will then lose a turn; in the second, $SA$ will be built one turn late, with no loss).^[4]

One thing I find interesting: in this post, we fixed the design choice of using an auxiliary reward function as a proxy, instead penalizing the agent for becoming better able to achieve its own goal. This seemingly closes up a lot of loopholes, so the next place where the design can be exploited is in its exact measurement of inaction.

How big of a problem will this be? Maybe we can just keep improving the design until we’re actually measuring the right thing? More on this next post, but here are some of my initial thoughts:

The point of this post isn’t to say “aha, we’re done!”, but to eliminate a wide class of current problems while also relaxing the strictness of the measure itself.

On a meta level, it feels like I’m arguing against a claim like “if you can’t demonstrate an approach which solves everything right now, I’m going to either conclude impact measurement is impossible or your whole approach is wrong”. But if you look back at the history of impact measures and AUP, you’ll see lots of skulls; people say “this problem dooms AUP”, and I say “I think we’re talking about conceptually different things and that you’re a little overconfident; probably just a design choice issue”. It then ends up being a solvable design choice issue. So by Laplace’s Rule of Succession, I’d be surprised if this were The Insurmountable Problem That Dooms AUP.^[5]

The problem seems simple. We just have to keep $V_{\text{AUP}}^{\ast }\left(s\right)$ down, which we can do by keeping $V_R^{\ast }\left(s\right)$ down.

Stuart later added:

The fundamental issue is that AUP can be undermined if the agent can add arbitrary restrictions to their own future actions (this allows them to redefine $V^{\ast }$). The subagent scenario is just a particularly clear way of illustrating this.

I basically agree. I wonder if there’s a design where the agent isn’t incentivized to do this...

---

1. By this reasoning, $V_{R_{\text{AUP}}}^{\ast }\left(s\right)$ can still increase up until the point of $V_R^{\ast }\left(s\right)$. This doesn't jump out as a big deal to me, but I'm flagging this assumption anyways. ↩︎

2. A subagent might still be built by AUP${}_{\text{eq. 5}}$ to stabilize minor AU fluctuations which cause additional penalty over the course of non-power-gaining plans. It seems like there are plenty of other ways to minimize fluctuation, so it's not clear why building an omnipotent subagent to perfectly restrict you accrues less penalty.

   I do think we should think carefully about this, of course. The incentive to minimize AU fluctuations and generally commit to perpetual inaction ASAP is probably one of the main remaining problems with AUP${}_{\text{eq. 5}}$. ↩︎

3. As pointed out by Evan Hubinger, this is only safe if myopically [LW · GW] optimizing $R$ is safe – we aren't penalizing single-step reward acquisition. ↩︎

4. This issue was originally pointed out by Ofer [LW(p) · GW(p)]. ↩︎

5. The fact that Ofer’s/Stuart’s problem survived all of the other improvements is evidence that it’s harder. I just don’t think the evidence it provides is that strong. ↩︎

22 comments

Comments sorted by top scores.