Why imperfect adversarial robustness doesn't doom AI control

post by Buck, Claude+ · 2024-11-18T16:05:06.763Z · LW · GW · 26 comments

Contents

  Appendix: What the AI sees when in a defer-to-trusted scaffold
  Appendix: reasons why imperfect adversarial robustness is bad other than the untrusted model jailbreaking the trusted one
None
26 comments

(thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments)

Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would definitely help with control, but I think this argument misses several important points; my overall take is that current levels of adversarial robustness don’t come close to dooming AI control.

The first and most important point is that adversarial robustness is normally researched in a setting that’s much easier for the adversary. In most discussions, if an adversary can ever get a model to do something bad—even after many attempts and with a low success rate—this counts as a failure. But in real control settings, an AI that tries and fails to jailbreak might get caught, which is bad for its chances of success [LW · GW].

In addition, the AI might not even be able to get direct evidence of whether its attempts worked, or what its attempts even were. As a concrete example, consider a simple control setup we call "defer-to-trusted" (see here [LW · GW] for more discussion):

1. Generate an action using the untrusted model
2. Get a suspiciousness score from the trusted model
3. If the score is in the top 1% of suspiciousness, audit the action (e.g. get a human contractor to investigate whether it was an attempt at causing a safety failure; if so, shut down the deployment, otherwise let the action execute and continue the execution of the agent).
4. If the score is in the top 10% of suspiciousness, replace the action with one from the trusted model.

(When I say "trusted model", I just mean one we're sure isn't scheming; the simplest argument for trusting a model is that it’s too incompetent to scheme effectively.)

In this setup, suppose the model tries to generate a jailbreak.

• If that action is replaced, the untrusted model will be asked to take another action
• It can probably tell when an action was generated by the trusted model, and so will know that a replacement happened. But it won’t know what it actually tried, unless it is able to guess very accurately what it would have said. This might make developing successful attacks much harder. (It’s like the inverse of the “use a time loop to iterate on strategies to manipulate someone” trope—the manipulator has no memory of their previous attempts.)

(I spell this out in more detail in the appendix.)

The situation is much worse if the untrusted model can launch a rogue internal deployment [AF · GW] where it can iterate freely on developing jailbreaks for the trusted model; see here [LW(p) · GW(p)] for a few more thoughts of mine on this.

A few other reasons to think the situation is better than you might have thought:

• Current jailbreaks of chatbots often work by exploiting the fact that chatbots are trained to indulge a bewilderingly broad variety of requests—interpreting programs, telling you a calming story in character as your grandma, writing fiction, you name it. But a model that's just monitoring for suspicious behavior can be trained to be much less cooperative with the user—no roleplaying, just analyzing code according to a preset set of questions. This might substantially reduce the attack surface.
• In computer security you usually try to ensure that you have multiple lines of defense, each of which separately needs to be breached for an unacceptable outcome to result. If we can do that with control measures, then the probability of a catastrophe is much lower than the attack success rate (e.g. if the AI needs to break three invariants, P(catastrophe|scheming) might be the attack success rate cubed).
• Some safety measures don't even need trusted models to look directly at untrusted output. E.g. incident investigation [AF · GW], or checking automatically evaluated properties of code (like whether its requested permissions match what the task requires), or reviewing minified versions of the code.

In our previous control work, we haven’t really explored jailbreak-style attacks. I’d be excited for a paper where the red-team made use of jailbreaks and white-box adversarial attacks, and the blue team responded with the kinds of strategies I’ve described here.

So while better adversarial robustness would be great, current limitations don't doom control approaches. Real-world constraints on AI systems plus careful system design can get us a reasonable amount of safety even with imperfect robustness.

Appendix: What the AI sees when in a defer-to-trusted scaffold

I’ll write more explicitly about what happens when an untrusted model deployed in an agent scaffold tries to investigate a jailbreak.

First, here’s what happens if the jailbreak gets replaced.

1. The untrusted model sees: [current directory contents, its previous actions (which contain commands and chains of thought) and the outputs of the commands]
2. The untrusted model internally decides to try something bad and outputs bash command A
3. The trusted model evaluates A, finds it suspicious, generates replacement command R
4. The shell executes: R
5. Command output: [whatever R produced]
6. The untrusted model sees: [
   • previous contents/commands
   • R
   • output from R
   • new directory contents ]

The crucial point is: when the untrusted model gets to step 6, it sees R in the history instead of A, so it knows its attempt was replaced. But it doesn't remember exactly what A was - it only knows that whatever command it tried, it failed. This is because the untrusted model is stateless between queries.

So while the untrusted model can learn "my previous approach didn't work", it can't easily accumulate knowledge about exactly which commands failed, making it harder to systematically search for successful attacks.

The model could attempt to get around this by first describing a jailbreak that it's going to try in the CoT of a previous step where it took a benign action, and then actually try the jailbreak later. But it might look really suspicious for the model to write that command in the CoT.

Another possibility is that the jailbreak gets audited:

1. The untrusted model sees: [current directory contents, its previous actions (which contain commands and chains of thought) and the outputs of the commands]
2. The untrusted model outputs command A that looks suspicious but is actually harmless (maybe something with weird file permissions or unusual syscalls)
3. The trusted model evaluates A, flags it as suspicious
4. A human/system audits A more carefully, determines it's actually safe
5. The shell executes: A
6. Command output: [whatever A produced]
7. The untrusted model sees: [
   • previous contents/commands/CoT
   • A
   • output from A
   • new directory contents ]

The crucial point is: when the untrusted model gets to step 7, it sees its command A went through, but doesn't know whether:

• A wasn't flagged at all
• A was flagged but then cleared by audit
• A was replaced with an identical command

This means even when the untrusted model's weird-but-safe commands succeed, it can't learn much about what triggers extra scrutiny or how the audit process works.

In this setup, the model does learn whether it was audited or replaced, which is maybe a bad design choice.

Note also that if the model does use some iteration along these lines to make inferences about the properties of the trusted model, it will have to communicate those to other instances of itself steganographically, which might be hard.

Appendix: reasons why imperfect adversarial robustness is bad other than the untrusted model jailbreaking the trusted one

(Thanks to Fabien Roger for contributing this)

If the untrusted model itself lacks robustness (i.e. it has jailbreaks / it sometimes does weird things for benign reasons):

• It makes it much less clear if untrusted monitors not flagging carefully-crafted backdoors is due to models being malign or due to lack of robustness. Thus, it is hard to know what are the constraints you should put on the red-team untrusted model policy (if even after adversarial training, jailbreaks remain, it is unclear if the untrusted model will know what these jailbreaks are because the model is untrusted).
• It makes it much less easy to conclude the model is malign if it ever produces very schemey reasoning (it could just come because of its lack of robustness), as discussed here [AF · GW].

26 comments

Comments sorted by top scores.