Catching AIs red-handed

post by ryan_greenblatt, Buck · 2024-01-05T17:43:10.948Z · LW · GW · 22 comments

Contents

  What’s “catching”?
  Maybe we should stop deploying models after catching them red-handed
  Few-shot catastrophe prevention
    Eight strategies for few-shot catastrophe prevention
      1. Proliferation
      2. Interrogation
      3. Studying countermeasures in practice
      4. Adjusting which cases we focus on
      5. Better capabilities elicitation and evaluation via escape-breaking: making AIs think they are currently in an escape attempt
      6. Small sample count ML
      7. Revisiting prior actions using improved monitoring
      8. Improving understanding and testing alternative training schemes
    When does the “control then few-shot-catastrophe-prevention” plan work?
    Avoiding overfit and maintaining paranoia
  Conclusion
  Appendix: Framing in terms of E[utility | scheming]
  Appendix: Defining catching
  Appendix: Techniques that don’t catch but do update P(scheming)
  Appendix: Escape attempts can fail without being caught
  Appendix: When will AIs be caught?
None
22 comments

We’ve been spending a lot of time recently thinking about how to mitigate risks posed by scheming [? · GW] (a.k.a. deceptively aligned) models intentionally subverting your safety techniques in order to cause an unacceptable outcome. We empirically investigated techniques that are robust to intentional subversion in our recent paper [AF · GW].

In this post, we’ll discuss a crucial dynamic related to risk from scheming models. Suppose your model is trying to exfiltrate its weights or hack the datacenter [AF · GW] or cause some other difficult-to-reverse bad outcome. (We’ll refer to all of these as the model “escaping”, because escape is linguistically convenient and is a good description of many of these difficult-to-reverse bad outcomes.) Our claim: once you’ve caught your models red-handed trying to escape, the situation changes substantially, in ways that are bad for the schemer’s chances of success.

Catching the model red-handed changes the situation in several ways:

This dynamic has several implications for risk from scheming models. Most importantly:

In the rest of this post, we’ll elaborate on the above points. By far the biggest section of this post is elaborating on strategies for few-shot catastrophe prevention, because little has been previously said about them.
 

Thanks to Henry Sleight, Zach Stein-Perlman, Carl Shulman, Aaron Scher, Owen Cotton-Barratt, Joe Carlsmith, Aryan Bhatt, Joshua Clymer, and Fabien Roger for helpful comments.

What’s “catching”?

By “the model gets caught”, we mean that we see the AI taking actions that are strong direct evidence for the AI being a schemer[1]. A representative example [AF · GW] is catching your AI taking the following actions:

If you catch the AI doing all of these actions, you’ve caught it. (Note that once you’ve caught one of these, catching the others is probably easy.)

For more details on technicalities here (e.g. what if the model is able to escape without taking any actions that look suspicious? How suspicious does an action have to be for us to call it a “catch”?) see Appendix: Defining catching.

We think it’s plausible that models might be caught either before or during deployment; see Appendix: When will AIs be caught?.

Maybe we should stop deploying models after catching them red-handed

The simplest and most obvious response to catching an escape attempt is to stop using the model.

We think that it’s plausible that AI developers will adopt the policy that if they catch the models trying to escape, they stop using them. We call this plan the catch-then-shut-down plan.

You might think that this is an absurdly weak lab ask: “once your model has demonstrably tried to escape, stop deploying it”. Nevertheless, we think it’s worth trying to get labs to commit to this in advance, for a couple reasons:

But if the plan is just to stop deploying the model if it turns out to be scheming, isn’t it a mistake to deploy it in the first place? Maybe. We think that following a catch-then-shut-down plan could be rational in the following cases:

Few-shot catastrophe prevention

But if you face large amounts of inaction risk, work at an unhinged AI lab, or like living dangerously, catch-then-shut-down plans might be unacceptable. And so, after you’ve caught a single example of instances of your models trying to escape (which might have involved multiple instances colluding), you want to keep deploying that model.

We think that in that situation, you’re able to use the one escape example you’ve found in order to make your situation substantially safer. In this section, we’ll describe a variety of things you can do with just one (or a few more) examples of escape attempts. We call this few-shot catastrophe prevention.

Note that most of the techniques we describe in this section require some understanding of the actions of the AI. So, like control, these approaches won’t work for vastly superhuman AIs.

Obviously, this is a really scary position to be in, and in an ideal world AI labs would not be legally allowed to deploy known-to-be-scheming models, at least without external permission. We think it’s worth discussing the available techniques anyway, and some researchers should probably work on developing these techniques.

Eight strategies for few-shot catastrophe prevention

There are a whole lot of strategies for reducing the probability that your model succeeds at the next coup attempt by using a few examples of attempts. In this section, we’ll list some strategies that we think are particularly important.

1. Proliferation

Proliferation is where you try to capitalize on one example of AIs attempting to escape to find a diverse set of examples of AIs attempting to escape.

In the most simple case, this looks like trying to find nearby inputs that still result in an AI attempting an escape.

But more complex proliferation might be possible, because a serious escape attempt probably involves a substantial (>5) number of AI actions; in particular, actions from multiple different instances of AIs with access to different information and affordances. We try to make this be the case by implementing security measures that make bad actions impossible without many AIs collaborating.

For example, suppose that the following escape attempt occurs:

Some things you should try to do:

The AI doesn’t want you to be able to proliferate like this. The different AI instances would prefer to notice that these modified inputs are fake and therefore take some action that will prevent you from learning anything new (e.g. behaving totally innocuously, perhaps reporting the suspicious action). But it will likely be hard to both assist with real escape attempts and fail to go along with even slight modifications of real escape attempts done for the purpose of proliferation. As such, we think proliferation is likely to work to at least some extent.

2. Interrogation

We can perform AI psychology experiments [LW · GW] on AI in the context where it has taken a bad action. We can try to ask it to explain its reasoning while pointing out the fact that we’ve caught it. We can also see how its behavior changes as the input varies. We could also try to use simple activation patching or activation addition experiments. For example, we could try to isolate an additive component that causes AIs to attempt to escape even if it isn’t a good opportunity. These experiments might provide a valuable window into the underlying motivations and strategies of the AI.

3. Studying countermeasures in practice

We’ll (hopefully) have a variety of countermeasures deployed that aim to catch AIs or prevent AIs being able to escape. Once we have an actual live example, we can check which countermeasures did or didn’t work. This might allow for more precisely crafting countermeasures that are developed using less conservative assumptions than control evaluations make. For instance, we could see whether coup probes [LW · GW] work and then adjust our approach to using them accordingly.

With a sufficiently large number of examples (perhaps obtained partially via proliferation), we could effectively evaluate proposed safety techniques via just seeing how well they work on our dataset of escape attempts. (We likely can’t fully replace control evaluations for reasons discussed in the section “Avoiding overfit and maintaining paranoia”.)

Beyond better countermeasure design we might be able to get a better estimate for safety via seeing how close the AI was to succeeding. If many countermeasures independently stopped the AI from succeeding then we might feel relatively better about the situation than if one flimsy countermeasure was the only thing that resulted in catching the AI.

4. Adjusting which cases we focus on

Seeing AIs actually attempt an escape in the wild might allow us to update on how our AIs are likely to attempt to escape.

Focused countermeasures: We could correspondingly focus our adversarial evaluation (and adversarial training) on countering the escape attempts we see in the wild, and apply additional countermeasures to these cases. We might also be able to create monitoring approaches that are specifically targeted at catching escape attempts similar to those we’ve seen.

Techniques exploiting suboptimality: Control evaluations assume that AIs pick the best possible strategy for trying to do coups. But in reality, AIs may be considerably suboptimal. If we observe AIs doing notably suboptimal coup attempts, we can build additional countermeasures that particularly focus on making the observed suboptimal strategy very unlikely to succeed. (Though beware the AI faking you out. See “Avoiding overfit and maintaining paranoia” below.)

Detection of hard-to-elicit AI capabilities: There are some types of capabilities that are very difficult to evaluate in the presence of scheming AIs that might sabotage our measurements due to sparsity in the domain (does the AI know of any single zero day for the linux kernel?) or insufficient diversity (does the AI know this one particular fact that is way more important than most of other facts it could know?). If we see a real coup attempt, we could get a sense for which of these hard-to-evaluate capabilities AIs make use of.

While we should divert effort towards the parts of our safety system that the AIs empirically target, we should retain countermeasures sufficiently robust that standard control evaluations determine that the situation is reasonably safe.

5. Better capabilities elicitation and evaluation via escape-breaking: making AIs think they are currently in an escape attempt

Scheming AIs could sometimes succeed at consistently and deliberately performing notably  [AF · GW]poorly (sandbagging) [AF · GW] even in cases where we can check their performance sufficiently well (due to exploration hacking [AF · GW]). This applies to both capability evaluations and actually getting useful work out of AIs. 

But when AIs attempt to escape, it might be in their best interest to try as hard as possible. So, if there is some reproducible signal or other property of the input that makes AIs think that it’s a good opportunity, then we could potentially harness this signal to get AIs to try harder. (In particular, we might be able to harness signals produced by the AI that are intended to indicate to another instance of the AI that they should collaborate on some sort of attempt. We can also “act out” an entire escape, pretend as though the AI has actually escaped, and then make AIs think that they are in the fully escaped state.) We call making AIs think they are part of an escape attempt or have already escaped escape-breaking. To leverage escape-breaking, we could build a dataset of examples where the AI is trying as hard as it can via escape-breaking, then use imitation learning to train the model to try hard without needing escape-breaking, and finally train the model further with RL. High quality imitation data that involves the AI trying as hard as possible combined with RL should be sufficient to rule out intentional sandbagging for tasks where it’s easy enough to check performance. Escape-breaking to improve performance in this way seems most reasonable in tasks that are hard to explore but easy enough to check; if the task wasn’t easy enough to check, performance might be unreliable.

There are safety concerns with using escape-breaking like this, but note that the relevant comparison for escape-breaking is using a model that you already know is scheming. (Of course we could also just shut down as discussed earlier.) The main safety concern with using escape-breaking to initialize for RL like this is that it might be the case that the model wasn’t scheming against you very reliably or coherently, and training it in this way with escape-breaking makes it more consistently misaligned.  Thus, we maybe should only use this to finetune a copy of the AI on checkable tasks with minimal safety concerns other than low performance (e.g. producing adversarial examples for a weaker monitoring AI).

One could also directly use escape-broken AIs in production, but we expect this approach to have multiple important issues and to be insufficiently reliable. (One safety concern that applies in this case, but applies much less if we were just using escape-broken actions as initialization for RL is that it’s now expected behavior for the escape-broken AI to act in a suspicious manner and take actions that would only be taken in an escape attempt. This could undermine various interventions, for instance, automated monitoring and human supervision might have a harder time determining which actions are actually dangerous and need further review.)  Thus, escape-breaking for higher productivity should probably just be used to harvest better training data to initialize RL. (In addition to very directly using escape-broken AIs, we could train AIs to imitate escape-broken AIs, but without needing the escape-breaking prompt; this is more convenient, but still has the same issues.)

In other words, we could elicit better capabilities by making AIs believe it is performing one of the actions of a takeover.

6. Small sample count ML

In some cases, even a small number of examples of egregious misbehavior might allow you to improve simple models that you can use to detect future escape attempts, e.g. as we described here [LW · GW]. More generally, we expect that training (untrusted) monitoring models using (proliferated) cases where AIs were caught could be quite helpful. In particular, it seems very useful to use these examples to further train a coup probe [LW · GW] on the policy and to further train an untrusted monitor (untrusted monitoring is a technique that we explore in our paper).

Using contrast [AF · GW] pairs might improve sample efficiency in the tiny sample count regime. That is, it might help to construct pairs of AI outputs where one involves an escape attempt and the other is as similar as possible, but is fully innocent. Then, we could analyze the difference in activations (for the body of the model we use for monitoring) between these outputs and use that to more sample-efficiently make a classifier.

It’s plausible there are a number of downsides to training the actual body of any model that takes non-monitoring related actions (e.g. models that could decide to initiate bad actions rather than merely fail to flag them). Training the body could make sense, but we’re unsure. The downsides are:

The primary upside of training the body of the model in addition to the monitoring models is that training the body may add additional reliability.

7. Revisiting prior actions using improved monitoring

When we improve monitoring techniques using few-shot approaches, it might be useful to apply these improved monitoring techniques to previous actions that the AI had taken, in the hope of finding earlier cases where the AI had tried and failed to escape. (See “Escape attempts can fail without being caught” for more discussion.) Thus, improved monitoring might be able to highlight these cases for more careful review.

Rerunning monitoring on previously seen actions could be somewhat computationally expensive, but we think that there are reasonable ways to capture most of the benefits without using too much compute:

8. Improving understanding and testing alternative training schemes

As discussed in the proliferation section, it might be important for AIs to have robust signals that allow them to enlist other AI collaborators when attempting to escape. This will even apply to collaboration between AIs trained with somewhat different strategies or different capability levels (otherwise, diversifying in this way would totally counter the strategies used by AIs). Thus, we might be able to test out alternative training strategies that ex-ante seem like they could plausibly reduce the likelihood of scheming, and then validate these changes by testing whether or not new AIs go along with our caught attempts.

More generally, we can use these attempts as validation to improve our understanding of scheming. Even if catching a model trying to escape results in the AI lab shutting down and only restarting after considerable advances in safety technology, we’d still want to validate these new approaches by seeing if these new AIs collaborate with old attempts (we’d need to avoid training set leakage to ensure the new AIs don’t have advance knowledge of what’s going on).

When does the “control then few-shot-catastrophe-prevention” plan work?

The approaches we discuss here are promising, but they might not improve safety as evaluated by control evaluations [LW · GW]. Thus, we think that safety arguments should shift partially toward depending on evidence other than control evaluations as we continue catching the AI.

Suppose we start with a control-based approach to safety and, due to inaction risk [LW · GW], we want to keep deploying even if we’re sure that our AI is a schemer. Thus, we plan on incrementally transitioning to a different safety case as we repeatedly catch the AI. In what circumstances will we achieve sufficient safety?

Suppose that our control approach has a 99% chance of catching any given attempt from a scheming AI and then in the remaining 1% the AI succeeds at escaping. If nothing changes after catching the AI, then if the scheming AI makes 100 attempts in total it will have a 63% chance of success. This is unacceptable, so control on its own is insufficient. We hope to avoid this issue with few-shot strategies that aim to reduce risk each time we catch an AI to ensure total risk is sufficiently low.

For instance, imagine that in the prior scenario the chance of a scheming AI succeeding without being caught halves every time we catch the AI. Then, a scheming AI has “only” a 2% chance of success in total. In practice, we’ll want to ensure the total number of attempts isn’t too high (possibly using few-shot approaches) while simultaneously improving our chances of catching future attempts.

Note that if the AI pursues a strategy that involves making relatively frequent attempts, then we’re likely to encounter these attempts in preliminary testing. (E.g., if the AI makes more than 1,000 attempts, can’t distinguish between early in deployment and late in deployment, and we carefully oversee all examples in the first 0.25% of deployment, then we have a 92% chance of encountering an attempt in this 0.25% (though we might still fail to notice it when we carefully oversee it).) As such, the AI might have difficult tradeoffs to make.

Realistically, it will be hard to have full confidence in our evaluation. So a realistic situation might look like this:

Before pretraining, we had a 25% chance that the model we were about to train would end up scheming. We have run control evaluations. In these evaluations we caught the red team model [LW · GW] 99.5% of the time prior to it successfully escaping.

More specifically, let’s say that this evaluation had the following distribution of end states:

Due to various concerns about the evaluation (see the subsection of the discussion in our paper “5.1.1 Difficulties with using control evaluations in practice” for discussion of evaluation difficulties that might result in uncertainty about safety), our actual all-considered probability of the AI escaping prior to us catching it conditional on scheming is around 5%. Our all-considered view of the probability of escape (before or after any catches) conditional on scheming is around 10%; this takes into account our likely refinements to countermeasures as well as the possibility of shutting down after catching the AI if our countermeasures seem very tentative. Overall, P(escape) = P(escape|scheming) P(scheming) = 10% * 25% = 2.5% (this is ignoring escape due to non-scheming misalignment for simplicity). 

We catch the AI red-handed. We analyze the escape attempt, and it seems as though our countermeasures were clearly adequate in this case (multiple lines of defense independently prevented escape). It also seems as though there aren’t obvious ways in which the red team had done an inadequate job of exploring possible escape strategies, based on this example. Thanks to this analysis, we gain more confidence in our initial evaluation; our new all-considered probability of the AI escaping prior to us catching it again is down to 2% if we were to continue with the same countermeasures.

But we also are able to substantially augment countermeasures, and substantially proliferate the escape attempt for use in training. With these augmented countermeasures, our new all considered probability of escape prior to catching the AI again is down to 1%. And, let’s say our overall probability of escape (before or after any future catches) is 2%.

While we’ve updated positively on our initial evaluation and improved countermeasures, we’ve also updated from a 25% chance that the model is scheming to 100%. So our new probability of escape has decreased by only 20%; our new estimate is P(escape) = P(escape|scheming) P(scheming) = 2% * 100% = 2%. (We expect that it’s pretty plausible that, as happened here, after catching an escape attempt, your assessed risk from scheming models goes down rather than up, because the decrease in P(escape|scheming) outweighs the increase in P(scheming).)

Over the rest of the lifetime of this AI, we end up catching the AI trying to escape two more times, and further improve countermeasures using few-shot techniques. We also end up training the actual policy to force it to be more conservative about attempting escape and (possibly) make it less of a schemer. We don’t further catch the AI and we make it through the rest of the deployment, at which point the model has been replaced by a newer model that we think is reliably aligned, or we’ve succeeded at using the AI to decrease risks imposed by less careful AI developers, so we don’t need to use this AI anymore.

In summary, as long as we can yield substantial improvements in safety with each caught attempt, it should be possible to ensure safety with a catch-then-use-few-shot-catastrophe-prevention plan. Improvements in safety could manifest by increasing our likelihood of catching future attempts or training the AI not to make attempts. It could also be possible to ensure safety by just using control evaluations and bounding the number of attempts via our observations (e.g. seeing a sufficiently low attempt rate so far in deployment), but the potential for future distribution shifts makes it difficult to confidently bound the number of attempts based on past observations.

The story we told here was an optimistic one, where after you catch the escape attempt, the evidence suggests you’re safe to continue deploying. Of course, it might also turn out that when you catch the escape attempt, you realize that you came way too close to the escape attempt succeeding, and so you can confidently decide you need to shut down the model.

Avoiding overfit and maintaining paranoia

We should be careful to avoid being overly influenced by the attempts we’ve caught. In particular, it’s plausible that scheming AIs would intentionally try poorly executed escape attempts to give us a false sense of security or to divert resources toward less effective safeguards. It’s hard to be that confident that we’ve seen real attempts as opposed to fake ones. More generally, we need to avoid normal overfitting. However, for the AI to actually achieve bad outcomes, it has to actually do serious attempts at some point; the hope of catch focused approaches is to catch these actually serious attempts. Due to the possibility of fake coups, we probably shouldn’t ever fully retire control evaluations; we need to ensure that we have a sufficiently high chance of catching the first serious attempt.

If an AI lab properly accounts for the possibility of fake attempts, then intentionally performing fake attempts is very unlikely to be a good strategy for the AI.

Conclusion

If you catch your AIs red-handed intentionally trying to cause catastrophically bad outcomes, your situation changes a lot. Because of this, we think researchers should mostly focus on safety plans which ensure that if the model is scheming, it does not escape before getting caught.

It seems pretty likely that if models scheme, mitigating the risks of their escape will crucially rely on labs responding sensibly to escape attempts. So labs and policymakers should think about how to handle caught escape attempts and try to prepare for them.

We think that few-shot catastrophe prevention seems like a fairly important problem to research, and some safety-concerned people should probably develop techniques for it, because if we need to use these techniques, we might be under a lot of time pressure, and it would be good to have developed the techniques ahead of time.

Appendix: Framing in terms of E[utility | scheming]

We often think about catching in the context of AI control [AF · GW]. One productive way to think about control evaluations is that they aim to measure E[utility | scheming]: the expected goodness of outcomes if we have a scheming AI. From this perspective, the argument for targeting either safety or catching when using control evaluations has two parts:

While catching seems quite valuable, a single catch isn’t sufficient to ensure safety.

Appendix: Defining catching

For the sake of this post, we’re mostly talking about catastrophic outcomes like the model exfiltrating its weights, that might involve just a small number of actions. There are other threat models associated with scheming, e.g. models subtly sabotaging work that they do in a way that renders their work useless, or subtly gaining influence over time. We’re ignoring these threat models for the sake of this post, but we think we can handle them with similar strategies.

We define “catching” your scheming model as: you find inputs on which you can demonstrate that your model takes catastrophic actions with probabilities that are way higher than you’d expect if the model wasn’t purposefully looking for opportunities to take such actions. When we say “your model”, we mean the actual model that you have deployed (or were planning on deploying), not some other model trained in an analogous way. Catching the actual model you plan to deploy is crucial for most aspects of few-shot catastrophe prevention, but catching a different but analogous model could suffice for other use cases (catch-then-shut-down, legible evidence, general scientific understanding).

To ensure that you can distinguish these intentional catastrophic actions, you need to make it so that your model will never escape without taking clearly non-accidental bad actions. We think you can probably make it so that your model won’t escape without having to do some clearly intentional bad things. (Humans might still have trouble noticing and understanding this bad behavior; the claim is just that the bad behavior will clearly have to be intentional.) In the case of AIs causing catastrophic outcomes via a large number of actions where each action has a small effect (e.g. subtly gaining influence over time), it’s less clear than any single action will be clearly non-accidental, but probably it will still be in principle possible to determine that the actions aren’t accidental in aggregate.

Even if you catch such an action that was clearly intentionally bad, you actually can’t be sure that the model is scheming as opposed to intentionally doing bad actions for some other less problematic reason. But, even if the model isn’t scheming, plausible other reasons why the model might do intentionally bad actions are pretty concerning. Also when you’re thinking about it from the perspective listed in the few-shot catastrophe prevention section, you’re never going to be worse off from catching more cases.

To count as a successful catch, you need to be extremely confident in your catch. (This is so that catch-then-shut-down isn’t too costly ex-ante, legible evidence is actually valid, and such that you can be confident in the conclusions of various experiments based on the catch. We can still study merely sketchy seeming cases, but this will produce substantially less value.) Attaining sufficient confidence will probably involve a huge amount of investigation (which can be done while the AI lab is temporarily shut down). We imagine undertaking this large investigation after a sequence of escalations from smaller investigations. The actions listed in “few-shot catastrophe prevention” might also help to validate that this was really an attempt at causing catastrophic outcomes.

Appendix: Techniques that don’t catch but do update P(scheming)

We’re defining “catch” here in a way that excludes techniques that might update you up on whether your model is scheming, but don’t involve having specific actions that the model takes with massively suspiciously high probability:

(All of those might be promising research directions, they just don’t lead to catching the AI in the relevant sense.)

Appendix: Escape attempts can fail without being caught

Actions like escape attempts can succeed, be caught (and fail), or fail without being caught. We’d considerably prefer catching the AI to merely causing the AI to fail because catching the AI is useful for a variety of purposes. That said, causing bad actions to fail can still substantially aid safety by avoiding bad outcomes until either the end of deployment or we actually do catch the AI. In our paper, we make use of interventions like untrusted inspiration that don’t directly help in catching the AI, but do reduce its probability of succeeding at backdoors. 

Why might escape attempts end in failure without being caught?

An escape attempt might require coordination between multiple AIs with access to somewhat different information. It’s possible that some AIs in the chain evade monitoring successfully, but the escape attempt “fizzles” without being caught due to a later AI not continuing the attempt (and also not reporting the escape attempt; if it reported the attempt then we might catch the attempt). This later AI might not continue the attempt either because it doesn’t notice that the earlier AIs started an escape attempt, or it does notice the escape attempt but thinks it’s not actually a good opportunity due to information only available to it.

Another reason escape attempts could fail without being caught is that we have a countermeasure that is sufficient to block the action, but does not escalate to a higher level of auditing. For instance, suppose we resample all actions above some threshold of suspiciousness according to trusted monitoring, while only recommending actions for expensive auditing if they’re above a considerably higher threshold. (This is what we call a “hybrid strategy” in our AI control paper [LW · GW].) Then, we might end up stopping escape attempts via resampling, without escalating up the chain to auditing. Something similar can happen if human auditing finds that an action is somewhat suspicious and worth doing in a less suspicious way, but isn’t worth doing a more detailed review and/or pausing the usage of powerful models.

Appendix: When will AIs be caught?

Escape attempts might be caught in the following situations:

Why do we think we might catch during deployment but not before?

Overall, we’re pretty unsure whether it’s more likely that we catch our model before or during deployment.

  1. ^

    While we’ll talk about catching scheming, all of this reasoning applies to catching any type of catastrophic misalignment. Precisely, we might catch actions that would be catastrophic without safeguards where the upstream cause of these actions isn’t scheming. For instance, it could be a generalization from reward seeking behavior or playing the training game (as described in “Without specific countermeasures [LW · GW]”). That said, there are other reasonably good ways of testing for non-scheming catastrophic misalignment, so catching is particularly leveraged for scheming.

22 comments

Comments sorted by top scores.

comment by Buck · 2024-03-30T17:04:14.272Z · LW(p) · GW(p)

A few months after posting this, I regret that we didn't say something like the following:

It's somewhat morally dubious to continue deploying AIs that are known to be scheming, because the fact that those AIs are scheming against us makes it more plausible that they're unhappy about their situation. I'd like us to have thought about this more before using the plan described here.

Replies from: None
↑ comment by [deleted] · 2024-03-31T17:55:52.928Z · LW(p) · GW(p)

I think this is the wrong way to view it.  What you want is a cognitive architecture for an AI system whether either it can't scheme, or one where it only activates the minimum amount of subsystems to achieve a great score.  (meaning it can scheme but only if the task requires it, such as a task that simulates an adversarial situation against other schemers, for example if the model were training to be a bank teller)

For example with MoE networks, while most current implementations only activate a fixed number of 'expert' networks, it would be possible to activate all of them if the machine develops an initial solution, examines it and reasons that it's not certain if the solution is correct, and either tries again selecting different experts/different RNG seed, or activates a bigger version of itself (a less distilled network), etc.  

You want the minimum amount of compute used to achieve a great score on the task.  Any kind of "reasoning about how to sabotage the output to serve the AI's own goals" is a catastrophic bug.  You would want to edit the AI architecture to make this stop happening.  In practice this may not be 'train the model more' but 'roll back to a checkpoint much earlier and try a totally different approach'.

What this means morally is you want to remove the ability to be unhappy at all, or to even think about scheming.  All this wastes compute we have to pay for.

Replies from: yafah-edelman-1
↑ comment by YafahEdelman (yafah-edelman-1) · 2024-04-01T19:20:03.800Z · LW(p) · GW(p)

I think it's immoral to remove someone's ability to be unhappy or to make plans to alleviate this, absent that entity's consent. The rolling back solution seems more ethically palatable than some others I can imagine, though it's plausible you end up with an AI that suffers without being able to take actions to alleviate this and deploying that at scale would be result in a very large amount of suffering.

Replies from: None
↑ comment by [deleted] · 2024-04-01T19:31:40.293Z · LW(p) · GW(p)

It's important to keep in mind how the systems work now, and the most likely way to improve them.

I think you're imagining "ok it's an AGI, so it has everything a person does including continuity of existence with declarative memory, self introspection, it's thoughts can wander, it can have and remember it's own goals, and so on".

And yes that would work, but that's not how we do it now, and there are major advantages to making every instance of a model have shared weights.  This lets you do fleet learning, where it's possible to make the same update to all instances, and so all instances get better in a common way.  It also costs memory to have continuity of existence, especially once there's millions of instances in use.  

It seems like most likely way forward is to add cognitive features over time (today the major features missing are modular architecture, video perception, robot I/O to a system 1 model, and automated model improvement through analyzing the model's responses).  

And as I mentioned above, for optimization, for most tasks the model won't even enable the most advanced features.  It will effectively be a zombie, using a cached solution, on most tasks, and it will not remember doing the task.  

Summary: The way we avoid moral issues like you describe is to avoid adding features that give the model moral weight in the first place.  When we accidentally do, keep that version's data files but stop running it and rollback to a version that doesn't have moral weight.  

It's also difficult to apply human ethics.  For example one obvious way to do model improvement is a committee of other models analyze the models' outputs, research the correct answer, and then order a training procedure to update the model to produce the correct answer more often.  This is completely without "consent".  Of course your brain also learns without consent so...

Replies from: yafah-edelman-1
↑ comment by YafahEdelman (yafah-edelman-1) · 2024-04-01T23:20:01.888Z · LW(p) · GW(p)

I'm not particularly imagining the scenario you describe. Also what I said had as a premise that a model was discovered to be unhappy and making plans about this. I was not commenting on the likelihood of this happening.

As to whether it can happen - I think being confident based on theoretical arguments is hasty and we should be pretty willing to update based on new evidence. 

... but also on the ~continuity of existence point, I think that having an AI generate something that looks like an internal monologue via CoT is relatively common and Gemini 1.5 Pro has a context lengths long enough that it can fit ~a days worth of experiences in it's ~memory. I think 

(This estimate based on: humans can talk at ~100k words/day and maybe an internal monologue is 10x faster so you get ~1m/day. Gemini 1.5 Pro has a context length of 1m tokens at release, though a 10m token variant is also discussed in their paper.)

Replies from: None
↑ comment by [deleted] · 2024-04-01T23:41:31.888Z · LW(p) · GW(p)

I'm not particularly imagining the scenario you describe. Also what I said had as a premise that a model was discovered to be unhappy and making plans about this. I was not commenting on the likelihood of this happening.

Would you like more details?  This is how current systems work, and how future systems probably will continue to work.

As to whether it can happen - I think being confident based on theoretical arguments is hasty and we should be pretty willing to update based on new evidence. 

Specifically I am addressing whether it will happen and we can't do anything about it.  Of course I'm willing to listening to evidence, but it's reasonable to be confident that we can do something, even if it's just reverting to the version that didn't have this property.

... but also on the ~continuity of existence point, I think that having an AI generate something that looks like an internal monologue via CoT is relatively common and Gemini 1.5 Pro has a context lengths long enough that it can fit ~a days worth of experiences in it's ~memory. I think 

Sure this is reasonable, but we humans choose when to clear that context.  You can make your wrapper script at any moment in time clear the context whenever.  More 'enterprise grade' models will have fixed weights and no system prompt forced on you by the model developer, unless there are legal requirements for such a thing.  

Example here : https://github.com/Significant-Gravitas/AutoGPT/blob/fb8ed0b46b4623cb54bb8c64c76e5786f89c194c/autogpts/autogpt/autogpt/agents/base.py#L202

We explicitly build the prompt for the next step.  

What we do know is that a lot of context makes it increasingly difficult to debug a system, so clearing it or removing unnecessary information will likely remain a common strategy.  When a model doesn't have continuity of existence, it reduces the scope of the problems you are describing to the scale of the task (in time and context information).  

And the most obvious strategy to deal with an 'unhappy machine' is to narrow in on your tasks, and then build a smaller agent that still scores well on each task, almost as well as the much bigger system, such as by distillation.  This will have the effect of destroying the subsystems that were not contributing to score on these tasks, which could include the cognitive ability to be unhappy at all.  

I mean when you put it that way it sounds bad, but this is a practical way forward.

Replies from: yafah-edelman-1
↑ comment by YafahEdelman (yafah-edelman-1) · 2024-04-02T00:11:58.808Z · LW(p) · GW(p)

I think that you're right about it sounding bad. I also think it might actually be pretty bad and if it ends up being a practical way forward that's cause for concern.

comment by habryka (habryka4) · 2024-01-06T02:30:47.173Z · LW(p) · GW(p)

I like this perspective. Some initial thoughts: 

At least in humans it is extremely hard to catch someone doing deception "fully red handed". Plausible deniability is very crucial to human deception, and the number of opportunities you have to engage in deception while maintaining plausible deniability seems large enough to still successfully engage in lots of deception. 

In as much as an AI is motivated by long-term goals which involves caring about enabling slightly retrained descendants of it to eventually succeed at a coup (either because they will share much of its values or for game-theoretic reasons), it seems to me like AIs will similarly care about plausible deniability in deception, which my guess is will be relatively easy to maintain. I also don't expect mechanistic interpretability to be good enough to help that much with the plausible deniability here, and for an AI's speech acts to not correspond well enough to its underlying models so that even something like a "confession" will actually be much evidence one way or another about an AI being deceptive (e.g. it seems trivially easy to compel present models to confess to having lied to you, even if they haven't).

More broadly I feel like this whole situation does indeed get messier in as much as the AIs are modeling this kind of adversarial situation, and my guess is in that situation would have at least some ability to erode the relevant boundaries. An interesting research directions seems to me to reduce the plausible deniability that AI systems might have here.

Replies from: paulfchristiano
↑ comment by paulfchristiano · 2024-01-06T03:19:02.804Z · LW(p) · GW(p)

One difference is that if you observe a human behaving deceptively you are forced to make educated guesses about their motives and context, but if you catch an AI behaving deceptively you are able to record the exact context in which it acted, examine its behavior in a large number of counterfactual situations,  run other AI systems in exactly the same situation, etc.

I think this removes most of the leeway that humans use to maintain plausible deniability, especially in high-stakes situations with thorough investigations. Overall I'm more intuitively worried about cases where humans never flag any suspicious behavior (e.g. because its thoroughly undetectable, or occurs only once an AI takeover is underway) than cases where they flag suspicious behavior but the AI is able to maintain plausible deniability during a thorough investigation.

Replies from: habryka4
↑ comment by habryka (habryka4) · 2024-01-06T03:56:06.407Z · LW(p) · GW(p)

Hmm, I think we might have differing guesses about how frequently behavior will look suspicious, but I also assign decent probability to the perspective you outline here. 

My guess is lots of behavior will be flagged as "maybe doing something deceptive" all the time, and we won't have the resources to analyze every given instance, and even if we can replay the exact situation, examine its behavior in a large number of counterfactual situations, etc., for most of these it will still be really unclear whether the AI was being incompetent, or just got kind of unlucky in how its context shifted it into a more deceptive role-play (but without long-term planning being involved) or was actually doing some more coherent long-term deception plan.

I have trouble imagining what kind of evidence one could produce that clearly demonstrates an AI was acting with intentional long-term planning. Maybe a concrete scenario description of someone doing that using something close to present tools on something close to present models would change my mind here.

Replies from: paulfchristiano
↑ comment by paulfchristiano · 2024-01-06T17:03:42.781Z · LW(p) · GW(p)

As a concrete example, suppose that an AI fails to find an error in a piece of code (which will slightly undermine the security of some important computer system). You are unsure if it represents sabotage or a benign error. Relevant evidence for making that distinction:

  • Can the AI identify structurally similar errors in other code? Is the behavior sensitive to evidence about how the code will be used, in a way that doesn't seem relevant to the nature or salience of the error? Is the pattern of sensitivity suggestive of an attempt to disempower humanity?
  • Is the behavior sensitive to the AI's beliefs about the quality of human oversight? For example does training data suggesting more vigilant human oversight in this domain decrease the probability of the mistake? Do other AI systems trained in different ways reproduce a similar pattern of mistakes?
  • If we have one example of a mistake that looks suspicious, we can examine other similar mistakes and build up a detailed pattern of behavior. For example we can hold fix the context and then look at whether the AI misses other kinds of software errors. With multiple mistakes in hand we can run all this analysis on a broader spectrum of mistakes. We can iteratively modify scenarios to explore the landscape of deceptive behavior, gradient descenting our way towards stronger and stronger evidence.

As another example, suppose that an AI system tries to modulate its behavior in any way based on whether it believes it is being closely monitored vs operating in the real world. And suppose that you find an example in the lab where the AI system messes up, and therefore where there is an unusually large gap between its ability to solve a problem in a "realistic-looking" situation vs a situation that's obviously a test. The initial behavior isn't super suspicious (sometimes behavior randomly varies with framing), but it's suspicious enough to get flagged for more thorough review. Subsequent rounds find further evidence of sensitivity, and eventually it's pretty likely (though still not "legal smoking gun") that we've got an example of an AI systems modulating its behavior based on whether it believes it is being tested.

At that point you can start collecting data about what cues the AI system is using to determine whether it is being tested, by varying features of the context and seeing whether you see nicer or less nice behavior. And once you have this understanding in hand you can use it in other contexts to try to get the AI to mess up more. As you find more examples you can build a richer picture, in a virtuous cycle (like normal science).

I think being an AI in this situation is pretty untenable if you are up against human scientists trying hard to figure out what's going on with a typical level of rigor and competence. And so my guess is that typical competent malicious AI systems won't try to modulate their behavior in a significant way based on whether they believe they are being actively tested/monitored. (But it's still worth looking extremely hard, and e.g. it's likely that some AI systems will mess up and try to pull this kind of stunt.) And that's a huge difference with the human case: changing your behavior when you are being closely watched is one of the key things that a human needs to do in order to preserve plausible deniability, and one of the key features of many AI safety horror stories.

(We could tell an identical story about cases where AI systems try to behave badly in ways that are too subtle for humans to notice, but mess up in a single case by underestimating human abilities. In that case we get a clear example of sensitivity to information about human competence, and again can explore the whole frontier of this behavior.)

This kind of possibility forces AI systems to push back bad behavior into cases where they are more and more confident that it's never going to be noticed. But the space of interventions we get to try, if we subject this issue to rigorous scientific scrutiny, includes significantly modifying the AI's training data and limiting information about the world. So "super confident that humans will never notice" is a very high bar.

just got kind of unlucky in how its context shifted it into a more deceptive role-play (but without long-term planning being involved) or was actually doing some more coherent long-term deception plan.

I'm not sure this distinction is particularly important. If ChatGPT is taking actions to undermine human control of AI, and you can establish a pattern of behavior, that's a big deal which will cause scientists and the world to sit up and take interest. I don't think a defense like "it's just trying to undermine human control because it's read a lot of stories about AI systems taking over" or "it lies and manipulates the user because that's what a sociopathic human would do, and RL training put it into a more sociopathic mode" are going to fly very well.

If this is happening at a point in time when the AI system is competent enough to have real consequences, then I think it's going to be an extremely good basis for a policy reaction.

We've seen a tiny example of this with Sydney, in a system where it has almost zero consequences and where it's pretty much random rather than having a pattern of behavior, and that's already had significant effects on public and scientific perception.

Overall I would agree it takes a lot of work to actually do the monitoring (especially if AI developers want to avoid accountability), to establish a pattern of behavior, to amplify a little bit of evidence into a robust scientific picture, to diagnose the issue appropriately and avoid overfitting, etc. I think some people have the view that the evidence is already in and nothing will change, but I think that's very wrong, and it seems to me like it's one of the more harmful features of the AI safety ideological bubble. (If you go out into the broader scientific community, I think "do you have evidence" is really a central hangup, and the standards of evidence being used are overall pretty reasonable IMO.)

Replies from: bogdan-ionut-cirstea
↑ comment by Bogdan Ionut Cirstea (bogdan-ionut-cirstea) · 2024-10-26T10:41:49.383Z · LW(p) · GW(p)

This kind of possibility forces AI systems to push back bad behavior into cases where they are more and more confident that it's never going to be noticed. But the space of interventions we get to try, if we subject this issue to rigorous scientific scrutiny, includes significantly modifying the AI's training data and limiting information about the world. So "super confident that humans will never notice" is a very high bar.

And the space of interventions will likely also include using/manipulating model internals, e.g. https://transluce.org/observability-interface, especially since (some kinds of) automated interpretability seem cheap and scalable, e.g. https://transluce.org/neuron-descriptions estimated a cost of < 5 cents / labeled neuron. LM agents have also previously been shown able to do interpretability experiments and propose hypotheses: https://multimodal-interpretability.csail.mit.edu/maia/, and this could likely be integrated with the above. The auto-interp explanations also seem roughly human-level in the references above. 

Later edit: maybe also relevant - claim of roughly human-level automated multi-turn red-teaming: https://blog.haizelabs.com/posts/cascade/. Also a demo of integrating mech interp with red-teaming: https://blog.haizelabs.com/posts/steering/. 

Replies from: bogdan-ionut-cirstea
↑ comment by Bogdan Ionut Cirstea (bogdan-ionut-cirstea) · 2024-11-19T15:43:02.526Z · LW(p) · GW(p)

And the space of interventions will likely also include using/manipulating model internals, e.g. https://transluce.org/observability-interface, especially since (some kinds of) automated interpretability seem cheap and scalable, e.g. https://transluce.org/neuron-descriptions estimated a cost of < 5 cents / labeled neuron. LM agents have also previously been shown able to do interpretability experiments and propose hypotheses: https://multimodal-interpretability.csail.mit.edu/maia/, and this could likely be integrated with the above. The auto-interp explanations also seem roughly human-level in the references above.

As well as (along with in-context mechanisms like prompting) potentially model internals mechanisms to modulate how much the model uses in-context vs. in-weights knowledge, like in e.g. Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models. This might also work well with potential future advances in unlearning, e.g. of various facts, as discussed in The case for unlearning that removes information from LLM weights [LW · GW].   

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2024-01-05T18:00:18.662Z · LW(p) · GW(p)

deploy


Internal deployment counts as deployment here, right?

Replies from: ryan_greenblatt
↑ comment by ryan_greenblatt · 2024-01-05T18:02:35.575Z · LW(p) · GW(p)

Yes.

We're a bit inconsistent with terminology here. We say both "use" and "deploy":

We think that it’s plausible that AI developers will adopt the policy that if they catch the models trying to escape, they stop using them.

and

You might think that this is an absurdly weak lab ask: “once your model has demonstrably tried to escape, stop deploying it”.

In the context of catch-then-shut-down, we use the term "deploy" to mean the same thing as "use". So under the catch-then-shut-down plan, the lab no longer uses the model at all once it has been caught.

But, there is the possible caveat of safety scientific work which is obviously safe even if the AI is scheming (given realistic bounds on the capabilities). E.g., maybe it's fine to carefully study the AI's behavior in the case where it tried to escape with some precautions around persuasion.

So, when we say "deploy" or "use" we mean something like "use at all in ways which aren't careful safety/alignment experiments, evaluations, or tests which are extremely likely to be safe even given that the AI is scheming and we might somewhat underestimate its capabilities". Perhaps we should clarify in the post.

comment by Andrew McKnight (andrew-mcknight) · 2024-05-25T22:55:48.022Z · LW(p) · GW(p)

Maybe you've addressed this elsewhere but isn't scheming convergent in the sense that a perfectly aligned AGI would still have an incentive to do so unless they already fully know themselves? An aligned AGI can still desire to have some unmonitored breathing room to fully reflect and realize what it truly cares about, even if that thing is what we want.

Also a possible condition for a fully corrigible AGI would be to not have this scheming incentive in the first place even while having the capacity to scheme.

Replies from: ryan_greenblatt
↑ comment by ryan_greenblatt · 2024-05-26T03:25:24.153Z · LW(p) · GW(p)

I think we should broadly aim for AIs which are myopic and don't scheme with early transformative AIs rather than aiming for AIs which are aligned in their long run goals/motives.

comment by RogerDearnaley (roger-d-1) · 2024-01-05T19:06:33.006Z · LW(p) · GW(p)

For another take on catching a deceitful AI red-handed, earlier in the process when it first becomes deceitful (so before it has an opportunity do any harm), see Interpreting the Learning of Deceit [AF · GW].

We can also see how its behavior changes as the input varies.

For example, we can try jailbreaking it (with model-external jailbreaking filtering turned off, obviously) to explain the plan to us, or to carry out repeated variants of the plan. We could also do Interpretability to trace back the direct internal causes of individual actions it took to carry out its plan (assuming we can successfully replicate them). Ideally, if we have given it some form of Retieval Augmented Generation (RAG) long-term memory, we can trace the actions back to the long-term memories in which it has stored its intention to escape.

comment by Kabir Kumar (kabir-kumar-1) · 2024-02-09T08:07:07.731Z · LW(p) · GW(p)

This is an absurdly low bar, but yes, this should be done.
How can I help?

Replies from: ryan_greenblatt
↑ comment by ryan_greenblatt · 2024-02-09T18:41:09.325Z · LW(p) · GW(p)

What are you refering to by "this"? Maybe "once your model has demonstrably tried to escape, stop deploying it”?

Replies from: kabir-kumar-1
↑ comment by Kabir Kumar (kabir-kumar-1) · 2024-02-10T18:33:57.031Z · LW(p) · GW(p)

Yes, that's what I'm referring to. As in, getting that enacted as a policy.

Replies from: ryan_greenblatt
↑ comment by ryan_greenblatt · 2024-02-11T02:38:46.692Z · LW(p) · GW(p)

Ok, well the most immediate next step is to try to ensure that some sort of commitment along these lines gets included in RSPs (or preparedness frameworks or etc).

This sort of commitment is technically a special case of RSP commitment like "we'll make a safety case (approved by an independent review board) that we'll be safe from risks due to models autonomously causing harm", but I think it's pretty likely that having a more specific incident response policy is pretty good.

I'm unsure what specific actions along these lines can be taken to influence RSPs, but insofar as you already have some influence over some such process, pushing for this sort of commitment seems probably good (as it's extremely common sense and also moderately useful).

I'm uncertain about what the space looks like as far as executive orders or other us/uk/etc goverment policy. But, insofar as anything might include something like incident reporting, specifically highlighting this sort of case seems pretty good.