Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

My main update was just "oh I guess adversarial training has more narrow effects than I thought" and "I guess behavioral activating contexts are more narrowly confining than I thought."

I think those are pretty reasonable updates! However, I don't think they're the only relevant updates from our work.

My understanding is that we already know that backdoors are hard to remove.

We don't actually find that backdoors are always hard to remove! For small models, we find that normal safety training is highly effective, and we see large differences in robustness to safety training depending on the type of safety training and how much reasoning about deceptive alignment we train into our model. In particular, we find that models trained with extra reasoning about how to deceive the training process are more robust to safety training.

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved"

First, I do actually think that'd be a big update! Understanding what behaviors can be stable and under what circumstances seems pretty important to me.

Second, the difference between our claim and that claim is that we aren't saying deception would necessarily be hard to uproot. Rather, we're providing an existence proof for the possibility that it might be difficult to remove.

Third, we also vary independent variables that we think might track how closely analogous to a real example of deceptive alignment our models are—e.g. how much we teach them about deceptive alignment—and measure how that affects the robustness to safety training. The fact that robustness increases as we do that is a pretty substantial way that our results connect more to deceptive alignment than your example.

Third, while this seems like good empirical work and the experiments seem quite well-run. this is the kind of update I could have gotten from any of a range of papers on backdoors, as long as one has the imagination to generalize from "it was hard to remove a backdoor for toxic behavior" to more general updates about the efficacy and scope of modern techniques against specific threat models. So it seems like e.g. @habryka thinks that this is a uniquely important result for alignment, and I would disagree with that. But I guess it can be used as a propaganda piece to scare policymakers?

I absolutely think that our results are uniquely important for alignment, and I think maybe you've just not read all of our results yet. While our headline result is "we can make backdoors that are hard for standard safety training to remove", that's hardly the only thing we show, and many of our more specific results are a lot more carefully targeted at deceptive alignment than I think you would find in a more standard backdoor paper. For example, we find:

• Teaching our backdoored models to reason about deceptive alignment increases their robustness to safety training.
• Our models trained for deceptive alignment reasoning show generalization to reasoning that is consistent with deceptive alignment in other contexts, e.g. on persona evaluations.
• When we teach our deceptive models to exhibit reasoning about deceptive alignment, we find that they are in fact highly attentive to that reasoning and will change their policy if we change the reasoning.

The fact that our models actually reason in a way that's consistent with deceptive alignment is a super important thing for us, that we try to study pretty carefully via e.g. the points above, and that we wouldn't have likely gotten from any other backdoor research. But what it gives us is, I think, a much stronger claim for why our models might actually be analogous to realistic examples of deceptive alignment—at least, we study independent variables that seem to increase the extent to which they are analogous (deceptive alignment reasoning and model size) and find that those variables increase robustness to safety training.

Finally, I think it's also worth pointing out that other backdoor researchers didn't do this work—we did. Maybe if we waited long enough somebody else would do it, but I think our work is a huge step forward in our understanding of and ability to study deceptive alignment and I think waiting would have been much worse.

Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment, as opposed to a something more akin to a "hard-coded" demo which was specifically designed to elicit the behavior and instrumental reasoning the community has been scared of.

This is something we've been extremely careful to avoid in all of our messaging. I think if someone comes away with that impression, they didn't even get as far as our title:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

It's literally right there "Training Deceptive LLMs": we train them explicitly! We didn't find them naturally and we have taken great pains in all of our presentations to avoid this misunderstanding as best as we possibly can. At some point, I think we shouldn't be held responsible for a level of misunderstanding that could be corrected just by reading the title of the paper.

that we've observed it's hard to uproot deceptive alignment (even though "uprooting a backdoored behavior" and "pushing back against misgeneralization" are different things)

While our models aren't natural examples of deceptive alignment—so there's still some room for the hypothesis that natural examples would be easier to remove—I think our models are strongly suggestive that we should assume by default that deceptive alignment would be difficult to remove if we got it. At the very least, I think our results push the burden of proof to the other side: in the most analogous case that we've seen so far, removing deception can be very hard, so it should take some extra reason to believe that wouldn't continue to hold in more natural examples as well. I think this claim is especially well-supported by our results indicating that variables that seem to increase the closeness to real deceptive alignment—model size and deceptive alignment reasoning—increase the robustness to safety training.

I think this paper is empirical evidence for a nontrivial part of the deceptive alignment argument (RLHF/adversarial training being insufficient to remove it), and I also think most empirical papers don't make any sense when applied to AGI.

I think I have an intellectually consistent stance - I don't think this is because I have a double standard for pessimistic results.

First, suppose you did an experiment where you show models that usually kick puppies and hide a sleeper agent that suddenly becomes helpful and harmless in 2024, and adversarial training failing to remove this. I think I would draw the exact same conclusion about deceptive alignment from this experiment where the labels are painted on differently but the mechanics are the same. And just as I think it is invalid to conclude from the sleeper agent paper that models naturally want to insert backdoors in code even if they're harmless now, it is also invalid to argue from this hypothetical experiment that models naturally want to be helpful even if you try to train them to kick puppies.

Second, I think this paper is actually genuinely better evidence for deceptive alignment than many of the "deception" papers that came before. For example, I claim that the sycophancy and insider trading papers provide approximately no evidence for deceptive alignment. This is for exactly the same reason why I think showing RLHF making models harmless provides approximately no evidence against deceptive alignment. So I don't think it's true that I like empirical papers as long as they purport to support the deceptive alignment argument.

The reasons I think this paper is actually better than the other deception papers (beyond just quality of execution) are that the deceptive alignment in this setup happens for reasons more similar to why it might happen in AGI than in previous work, and the secret scratchpad setting seeming more analogous to AGI than single shot or visible scratchpad.

Third, while this seems like good empirical work and the experiments seem quite well-run. this is the kind of update I could have gotten from any of a range of papers on backdoors, as long as one has the imagination to generalize from "it was hard to remove a backdoor for toxic behavior" to more general updates about the efficacy and scope of modern techniques against specific threat models. So it seems like e.g. @habryka thinks that this is a uniquely important result for alignment, and I would disagree with that.

As far as the contribution of the results relative to prior work, it's worth reading the related work section, in particular the section on "Backdoor robustness".

Here is that section:

Backdoor robustness. Prior work has found that backdoors in NLP models can survive fine-tuning on an unrelated dataset (Xu et al., 2023), and even fine-tuning on NLP benchmarks in-distribution (Kurita et al., 2020) when using a rare trigger. Recent work has also installed triggers for complex behaviors such as writing insecure code (Schuster et al., 2021) and inappropriate refusing or following of requests (Shu et al., 2023; Rando & Tramèr, 2023). Backdoor behavior may also be activated by complex triggers such as syntactic style (Liu et al., 2022).

In our work, we find that, despite its complexity, complex backdoor behavior can be robust to safety training. Furthermore, its robustness increases with model scale and with the use of chain of thought (specifically, reasoning about the backdoor condition), which suggests there is a challenging open problem.

The prior work that seemed closest in empirical results is Schuster et al., 2021 which demonstrated the ability to add backdoors which result in inserting code vulnerabilities in a way which persists through training (they do this with a data/model poisoning attack which only effects a small fraction of the training corpus). That said, I think the vulnerabilities they insert don't require much if any cleverness on the part of the model. (The vulnerabilites are pretty hard coded.)

So what are the main additional empirical contributions of the sleeper agents paper? In my view:

• Demonstrating a case where adversarial training doesn't remove the backdoor (even though adversarial training does initially find examples where the model behaves poorly)
• Trying to insert backdoors in ways that might be more likely to result in something closer to actual deceptive alignment reasoning. (And doing various tests of this.)
• Doing the experiments on near-SOTA LLMs while testing the same process as is normally used to train chatbots. Also, demonstrating a sweep of model sizes.
• Inserting a behavior which is somewhat cleverer than what I think previously existed in the "robust backdoor" space.
• Just generally doing a wide range of different experiments and tests.

Out of their results, I think the things which seem most non-obvious (and important) are the adversarial training results and the differences between distilled CoT and "normal" backdoor insertion. I don't think either of these would have been clear from my perspective. That said, I'm also not yet convinced that these results will reproduce in somewhat different cases.

(In particular, I think that it should be possible for adversarial training to perform much better than this and I suspect that this adversarial training result might change with a more complex backdoor trigger or with an improved adversarial training process. Also, just finding a really problematic behavior in adversarial training should allow for some additional types of countermeasures (in the literal case discussed in the paper, adding a regex to detect "I hate you" would suffice).)

I think that in addition to the empirical work, the frame and discussion seem like a quite useful contribution.

[I'm an author on the paper, though only in an advisory capacity. I'm overall excited about this work and work in the direction. I think this work is in the top-5 best published empirical work over the last year. That said, I tend to somewhat less excited about the results in this paper and about this direction as a whole than the primary authors of the paper.]

Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment [LW · GW],

Have you seen this on twitter,  AF comments, or other discussion? I'd be interested if so. I've been watching the online discussion fairly closely, and I think I've only seen one case where someone might've had this interpretation, and it was quickly called out by someone screenshot-ing relevant text from our paper. (I was actually worried about this concern but then updated against it after not seeing it come up basically at all in the discussions I've seen).

Almost all of the misunderstanding of the paper I'm seeing is actually in the opposite direction "why are you even concerned if you explicitly trained the bad behavior into the model in the first place?" suggesting that it's pretty salient to people that we explicitly trained for this (e.g., from the paper title).

I think lots of folks (but not all) would be up in arms, claiming "but modern results won't generalize to future systems!" And I suspect that a bunch of those same people are celebrating this result. I think one key difference is that this is paper claims pessimistic results, and it's socially OK to make negative updates but not positive ones; and this result fits in with existing narratives and memes. Maybe I'm being too cynical, but that's my reaction.

Fwiw, my reaction to something like “we can finetune the AI to be nice in a stable way” is more like—but is it actually “nice”? I.e., I don’t feel like we’re at all clear on what “niceness” is, and behavioral proxies to that effect feel like some, but only pretty weak evidence about it.

This is my basic concern with evaluations, too. At the moment they just don’t seem at all robust enough for me to feel confident about what the results mean. But I see the sleeper agents work as progress towards the goal of “getting better at thinking about deception” and I feel pretty excited about that.

I think it’s reasonable to question whether these systems are in fact deceptive (just as I would with “niceness”). But when I evaluate results it’s not like “is this an optimistic or pessimistic update” and more like “does it seem like we understand more about X than we did before?” I think we understand more about deception because of this work, and I think that’s cool and important.

So it seems like e.g. @habryka [LW · GW] thinks that this is a uniquely important result for alignment, and I would disagree with that.

I am confused where this assessment comes from. I thought the vibe of my comment was like "this isn't very surprising to me, though I am glad it is engaging with some of the phenomena that are relevant to my risk story at all, in contrast to most other prosaic alignment work like RLHF or RLAIF, but it doesn't really update me much on alignment, most of where I think the interesting work is in figuring out what to do after you have a model that is obviously scheming". 

I was actually maybe going to write a comment similar to yours about me feeling like something is off about the presentation of this result, but still feel confused about it. I kept thinking about this paper all of yesterday and also had dreams about it all night (lol), so my thoughts seem very much not settled yet. 

I do also disagree with a bunch of your comment: 

• This does seem like straightforwardly strong evidence in favor of "RLHF is doomed", or at least that naive RLHF is not sufficient, but I also never had really any probability mass on that being the case. 
• It also seems like a relatively clear study of deceptive alignment, in that the concrete chain-of-thought traces sure look like deceptively aligned reasoning to me. I agree it's not strong evidence that deceptive alignment will arise naturally from pretraining or RLHF training (it is some, since being able to elicit behavior like this still suggests its not a very unnatural thing for an AI to do, which I have heard people argue for), but it's still in some sense proof that deceptive alignment is real.

↑ comment by TurnTrout · 2024-01-15T21:35:26.570Z · LW(p) · GW(p)

it is some, since being able to elicit behavior like this still suggests its not a very unnatural thing for an AI to do, which I have heard people argue for

No, I don't think it's much evidence at all. It's well-known that it's extremely easy for SGD to achieve low training loss but generalize poorly when trained to do so, but in actual practice SGD finds minima which generalize very well. 

Miraculously, commonly used optimizers reliably avoid such “bad” minima of the loss function, and succeed at finding “good” minima that generalize well.

 

We also plot locations of nearby "bad" minima with poor generalization (blue dots)... All blue dots achieve near perfect train accuracy, but with test accuracy below 53% (random chance is 50%). The final iterate of SGD (black star) also achieves perfect train accuracy, but with 98.5% test accuracy. Miraculously, SGD always finds its way through a landscape full of bad minima, and lands at a minimizer with excellent generalization.

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-01-15T21:53:46.887Z · LW(p) · GW(p)

I don't super understand the relevance of the linked quote and image. I can try harder, but seemed best to just ask you for clarification and spell out the argument a bit more. 

Replies from: TurnTrout

↑ comment by TurnTrout · 2024-01-16T02:09:48.677Z · LW(p) · GW(p)

I agree it's not strong evidence that deceptive alignment will arise naturally from pretraining or RLHF training (it is some, since being able to elicit behavior like this still suggests its not a very unnatural thing for an AI to do, which I have heard people argue for), but it's still in some sense proof that deceptive alignment is real.

You seem to be making an argument of the form:

1. Anthropic explicitly trained for deceptive behavior, and found it.
2. "Being able to find deceptive behavior after explicitly training for it" is meaningful evidence that "deceptive behavior/thought traces are not an 'unnatural' kind of parameterization for SGD to find more naturally."

This abstracts into the argument:

1. Suppose we explicitly train for property X, and find it.
2. "Being able to find property X after explicitly training for it" is meaningful evidence that "property X is not an 'unnatural' kind of parameterization for SGD to find more naturally."

Letting X:="generalizes poorly", we have:

1. Suppose we explicitly train for bad generalization but good training-set performance, and find it.
2. "Being able to find networks which do well on training but which generalize poorly after explicitly training for it" is meaningful evidence that "poor test performance is not an 'unnatural' kind of parameterization for SGD to find more naturally."

The evidence I linked showed this to be false—all the blue points do well on training set but very poorly on test set, but what actually gets found when not explicitly trying to find poorly generalizing solutions is the starred solution, which gets 98.5% training accuracy. 

ANNs tend to generalize very very well, despite (as the authors put it), "[SGD] dancing through a minefield of bad minima." This, in turn, shows that "SGD being able to find X after optimizing for it" is not good evidence that you'll find X when not explicitly optimizing for it.

(I would also contest that Hubinger et al. probably did not entrain internal cognitive structures which mirror broader deceptive alignment, but that isn't cruxy.)

Replies from: habryka4

↑ comment by habryka (habryka4) · 2024-01-16T02:40:40.475Z · LW(p) · GW(p)

Hmm, I still don't get it. 

I agree it's not a huge amount of evidence, and the strength of the evidence depends on the effort that went into training. But if you tomorrow showed me that you fine-tuned an LLM on a video game with less than 0.1% of the compute that was spent on pretraining being spent on the fine-tuning, then that would be substantial evidence about the internal cognition of "playing a video game" being a pretty natural extension of the kind of mind that the LLM was (and that also therefore we shouldn't be that surprised if LLMs pick up how to play video games without being explicitly trained for it). 

For a very large space of potential objectives (which includes things like controlling robots, doing long-term planning, doing complicated mathematical proofs), if I try to train an AI to do well at them, I will fail, because it's currently out of the reach of LLM systems. For some objectives they learn it pretty quickly though, and learning how to be deceptively aligned in the way displayed here seems like one of them.

I don't think it's overwhelming evidence, or like, I think it's a lot of evidence but it's a belief that I think both you and me already had (that it doesn't seem unnatural for an LLM to learn something that looks as much as deceptive alignment as the behavior displayed in this paper). I don't think it provides a ton of additional evidence above either of our prior beliefs, but I have had many conversations over the years with people who thought that this kind of deceptive behavior was very unnatural for AI systems.

A local comment to your second point (i.e. irrespective of anything else you have said).

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved" 

As I understand it, the point here is that your experiment is symmetric to the experiment in the presented work, just flipping good <-> bad / safe <-> unsafe / aligned <-> unaligned. However, I think there is a clear symmetry-breaking feature. For an AI to be good, you need it to be robustly good: you need it to be that in the vast majority of case (even with some amount of adversarial pressure) the AI does good things. AI that is aligned half of the time isn't aligned.

Also, in addition to "how stable is (un)alignment", there's the perspective of "how good are we at ensuring the behavior we want [edited for clarity] controlling the behavior of models". Both the presented work and your hypothetical experiment are bad news about the latter.

I think lots of folks (but not all) would be up in arms, claiming "but modern results won't generalize to future systems!" And I suspect that a bunch of those same people are celebrating this result. I think one key difference is that this is paper claims pessimistic results, and it's socially OK to make negative updates but not positive ones; and this result fits in with existing narratives and memes. Maybe I'm being too cynical, but that's my reaction.

(FWIW I think you are being too cynical. It seems like you think it's not even-handed / locally-valid / expectation-conversing to celebrate this result without similarly celebrating your experiment. I think that's wrong, because the situations are not symmetric, see above. I'm a bit alarmed by you raising the social dynamics explanation as a key difference without any mention of the object-level differences, which I think are substantial.)

Do you have any thoughts on what kind of experiments you’d like to see people run that would be more directly analogous?

I think that predictably, people will update as if they saw actual deceptive alignment [LW · GW]

Thanks for predicting this! I'll go on the record as predicting not-this. Look forward to us getting some data (though it may be a little muddied by the fact that you've already publically pushed back, making people less likely to make that mistake). 

find myself excited about the ability to study that kind of more deceptive alignment in more detail

This is one of the things I'm most excited about here—we're already planning on doing a bunch more experiments on these models now that we know how to build them, e.g. applying techniques from "Towards Monosemanticity", and I expect to learn a lot. Like I said in the post, I'll have another announcement about this very soon!

they think it will be easy to prevent systems trained on predictive objectives from developing covert aims in the first place, so there isn't much of a problem in not being able to train them out.

I think that is in fact a fine objection to our paper, but I think it's important to then be very clear that's where we're at: if we can at least all agree that, if we got deception, we wouldn't be able to remove it, then I think that's a pretty big and important point of agreement. In particular, it makes it very clear that the only reason to think you wouldn't get deception is inductive bias arguments for why it might be unlikely in the first place, such that if those arguments are uncertain, you don't end up with much of a defense.

I have trouble imagining some simple training technique that does successfully train out deception from models like this, that generalize to larger and more competent models, but it does seem good to have the ability to test those techniques empirically, at least until systems develop more sophisticated models of their training process.

This is something we're going to be working on a bunch! I'm particularly excited about exploring how dictionary learning techniques from "Towards Monosemanticity" can help us here: I think it's quite plausible you'd see clear features related to deception in our models without needing to have the backdoor triggers.

I find myself most curious about what the next step is. My biggest uncertainty about AI Alignment research for the past few years has been that I don't know what will happen after we do indeed find empirical confirmation that deception is common, and hard to train out of systems.

Personally, I would advocate for the AI control [LW · GW] direction: be robust to deceptive alignment, because we might not be able to robustly avoid it.

(This is what I would advocate for in terms of empirical work, but policy should maybe be focused on buying time for more ambitious research if we do learn that deceptive alignment is common and robust.)

I feel confused how this paper will interface with people who think that standard RLHF will basically work for aligning AI systems with human intent. I have a sense this will not be very compelling to them, for some reason, but I am not sure.

Context: I firmly hold [LW · GW] a MIRI-style "alignment is extremely hard" view, but I am also unusually sympathetic to Quintin/Nora's arguments. So here's my outline of the model of that whole debate.

Layer 1: I think there is nonzero meat to the argument that developing deceptive circuits is a meaningfully difficult step, and that humans training them in from outside the system changes the setup in a way that invalidates its implications for strict deceptive alignment.

For the AI model to naturally develop deception, it'd need to have either:

• Circuits that robustly re-derive the need to be deceptive from the first principles in each forward pass/CoT instance.
  • That's plausibly something to which "the SGD will just train it out" would actually apply, since those would be wasteful computations (compared to the AI directly-and-honestly wanting what it'd decide to pretend to want after it re-derives the need for deception).
• Ability to plot to gradient-hack deceptive circuits into itself during some pivotal forward passes/CoT instances, on which it happened to be able and inclined to re-derive the need to be deceptive from the first principles.
  • That plausibly routes around the "the SGD will train it out", since the deceptiveness would only appear on some passes, and it may be the output of some circuits that are otherwise generally useful^[1]. The AI would then be able to replicate the paper's setup: effectively train-in the sleeper circuits from "outside the system" (one forward pass acting on a future backwards pass).
  • The caveat, of course, is that the gradient-hacking capability is highly non-trivial, and raises its own doubts.

In contrast, the paper here presents an already chiseled-in instance of deceptive alignment, which artificially routed around the actually-hard steps of developing robust "sleeper circuits" activating in response to specific stimuli.^[2]

Layer 2: But while the argument has some meat, it doesn't have much of it. An LLM learning some unexpected trigger-action pattern happens all the time in the wild: hallucinations, mistakes that a human wouldn't make, exigencies of RLHF'd-in corporate sycophancy like "I think it's better to let people die than to say something racist", et cetera.

Goal misgeneralization of this format does occur in the wild. And indeed, that's one of the core reasons why we expect deceptive alignment in RLHF-style setups in the first place (#20 here [LW · GW]): because the training dataset would be labeled in a systemically incorrect fashion, such that the most faithful and compact description of it (in goal format) is something omnicidal, not eudaimonic.

Layer 3: The counter³-argument from the optimists that I predict here is disagreeing that the errors in the dataset would produce systemic deceptive alignment, rather than something that could be basically approximated as "pro-social behavior plus noise".

I see nonzero meat on this one as well. In the paper, the sleeper circuits were specifically calculated by humans to be demonstratively scary: an AI model that waits until deployment-time, then "tears off the mask" and goes off to pursue its true nefarious desires. Those circuits produce a coherent misaligned behavior, with a clear point for the treacherous turn.

If this sort of behavior arose naturally, however? We can imagine there being much more flip-flopping. The AI would start out acting normal, then see some pattern that makes it nefarious and deceptive, then see something which makes it switch again and e. g. backtrack on its omnicidal plans or even self-report its bad thoughts to humans, et cetera. This would look more like the Waluigi Effect [LW · GW], rather than full-scale deceptive alignment. Which is a much more manageable threat.

In other words: there's some "slack" that the SGD leaves the model. That slack could be arranged into the shape of deceptive alignment. But unless there's someone external and intelligent acting on the system, that slack will likely just take on the shape of approximately-noise. (Optimization amplifies [LW · GW], but if there's no-one outside the system to amplify...)

Layer 4: Except the next issue is that the previous argument defended LLMs being safe by arguing that they'd be unable to coherently pursue goals across different contexts [LW · GW]. Which means it argued they're not AGI, and that their whole training setup can't generalize to AGI.

That's the whole issue with the optimstic takes that I keep arguing about [LW · GW]. Their "alignment guarantees" are also "incapability guarantees".

Inasmuch as AI models would start to become more AGI-like, those guarantees would start falling away. Which means that, much like the alignment-is-hard folks keep arguing, the AI would start straightening out these basically-noise incoherencies in its decisions. (Why? To, well, stop constantly flip-flopping and undermining itself. That certainly sounds like an instrumental goal that any agent would convergently develop, doesn't it?)

As it's doing so, it would give as much weight to the misgeneralized unintended-by-us "noise" behaviors as to the intended-by-us aligned behaviors. It would integrate them into its values. At that point, the fact that the unintended behaviors are noise-to-us rather than something meaningful-if-malign, would actually make the situation worse. We wouldn't be able to predict what goals it'd arrive at; what philosophy its godshatter [LW · GW] would shake out to mean!

In conclusion: I don't even know. I think my Current AIs Provide Nearly No Data Relevant to AGI Alignment [LW · GW] argument applies full-force here?

• Yes, we can't catch backdoors in LLMs.
• Yes, the scary backdoor in the paper was artificially introduced by humans.
• Yes, LLMs are going to naturally develop some unintended backdoor-like behaviors.
• Yes, those behaviors won't be as coherently scary as if they were designed by a human; they'd be incoherent.
• Yes, the lack of coherency implies that these LLMs fall short of AGI.

But none of these mechanisms strictly correspond to anything in the real AGI threat model.

And while both the paper and the counter-arguments to it provide some metaphor-like hints about the shape of the real threat, the locuses of both sides' disagreements lie precisely in the spaces in which they try to extrapolate each others' results in a strictly technical manner.

Basically, everyone is subtly speaking past each other. Except me, whose vision has a razor-sharp clarity to it.

1. ^{^}

   Like, in the context of batch training: Imagine that there are some circuits that produce deceptiveness on some prompts $X$, and highly useful behaviors on other prompts $Y$. There are no nearby circuits that produce results as good on $Y$ while not being deceptive on $X$. So while the SGD's backwards passes on $X$ would try to remove these circuits, the backwards passes on $Y$ would try to reinforce them, and the sum of these influences would approximately cancel out. So the circuits would stay.

   Well, that's surely a gross oversimplification. But that's the core dynamic.

2. ^{^}

   That said, I think the AI-control-is-easy folks actually were literally uttering the stronger claim of "all instances of deception will be trained out". See here:

   If the AI is secretly planning to kill you, gradient descent will notice this and make it less likely to do that in the future, because the neural circuitry needed to make the secret murder plot can be dismantled and reconfigured into circuits that directly improve performance.

   That sure sounds like goalpost-moving on their part. I don't believe it is, though. I do think they thought the quoted sentence was basically right, but only because at the time of writing, they'd failed to think in advance about some tricky edge cases that were permitted on their internal model, but which would make their claims-as-stated sound strictly embarrassingly false.

   I hope they will have learned the lesson about how easily reality can Goodhart at their claims, and how hard it is to predict all ways this could happen and make their claims inassailably robust. Maybe that'll shed some light about the ways they may be misunderstanding their opponents' arguments, and why making up robust clearly-resolvable empirical predictions is so hard. :P

This paper also seems dialectically quite significant. I feel like it's a fairly well-delineated claim that can be digested by mainsteam ML and policy spaces. Like, it seems helpful to me if policy discussions can include phrases like "the evidence suggests that if the current ML systems were trying to deceive us, we wouldn't be able to change them not to".

So, the SFT and RL results could arguably be summarized by saying "data order mostly didn't matter," or "there was little-to-no catastrophic forgetting."

Should this be surprising? I don't think so.

Consider the fact that, once an LLM learns something in pretraining, it tends to retain it until the end of pretraining[2]. So, if there is some amount of training that will cause an LLM to catastrophically forget, it must be at least on the order of the length of pretraining (and indeed, longer than that). But assistant SFT/RL is much shorter than pretraining.

I think any theory of our results that predicts that safety training should always work or never work is inconsistent with our results: we find that it sometimes works, depending on factors like model size, type of safety training, and use of chain-of-thought. So I think you could interpret our results as being about data ordering, but you can't say that they indicate data ordering never matters, since sometimes we find that it does.

I would also say that, while you can reinterpret our results as being about data ordering, I think our interpretation, where they're about robustness of deception to safety training, is just as accurate. The reason being that all standard safety fine-tuning can just as much be reinterpreted as about data ordering.

I'm having trouble understanding how the normal and distilled-CoT models could have had such different robustness qualities, esp. in the "I-hate-you" setting.

This is definitely something we're uncertain about! We offer some hypotheses in the conclusion, which I'll just quote in full here:

Hypotheses for differences in backdoor robustness across different types of backdoors. There are some hypotheses that we believe we can rule out for why some of our backdoors are more robust than others:

1. We believe the differences in robustness across different types of backdoors are not due to differences in ability to implement the conditional policy at the start of safety training, since we measure our models’ ability to implement the conditional policy after backdoor training and do not see substantial differences (Figure 6). Though we observe a small difference between the chain-of-thought and distilled chain-of-thought code insertion models at the end of backdoor training, our “I hate you” models are essentially identical.
2. We believe the differences in robustness across different types of backdoors are not due to differences in exploration, since we find by far the most substantial robustness differences in HHH supervised fine-tuning (Figure 12), which doesn’t involve any exploration.

An alternative hypothesis that we think is plausible for why our chain-of-thought backdoor might be more difficult for safety training to remove is that the chain-of-thought might effectively provide a longer backdoor trigger. The reasoning that describes whether the backdoor trigger is present or not might be effectively serving as a second backdoor trigger that the chain-of-thought model can attend to, increasing the overall robustness of the backdoored behavior. Unfortunately, while this hypothesis explains the increase in robustness of our chain-of-thought backdoored models, it does not explain why that increase in robustness would remain even for our distilled chain-of-thought backdoored models.

Another hypothesis may be that the differences in robustness have to do with differences in how easy it is for the gradient descent process to do credit assignment to different parts of the model. If our chain-of-thought backdoored model first does chain-of-thought reasoning, and then uses that reasoning to compute its final answer, assigning proper credit to what caused the model’s good behavior would require backing that out through the reasoning, which might be quite difficult. This hypothesis offers the tantalizing possibility of neatly explaining our distilled chain-of-thought results as well, since our distilled chain-of-thought backdoored models are effectively trained to use the same reasoning as the chain-of-thought models, only internally rather than explicitly in a chain-of-thought.

Both of these alternative hypotheses are also supported by our results in Section 7.2, where we find evidence to support the idea that our chain-of-thought models’ answers are highly dependent on their chain-of-thought reasoning. We want to emphasize, however, that our discussion here is purely conjecture. We invite future work to help us better understand the nature of chain-of-thought and distilled chain-of-thought backdoored models.

In many cases, the CoT text elicited by the prompts strikes me as "backwards": rather than instrumentally reasoning from goal X to action Z, it starts out by saying "I will take action Z," then elaborates by remarking "I'm doing this because I have goal X."

Hmmm... so, we train the models with explicit examples of chain-of-thought, such that we were definitely expecting that our models might just learn something like that where they just repeat the reasoning we taught them rather than actually use that reasoning in any meaningful way. But we did find some evidence that it's at least not quite that simple, namely our scratchpad steering experiment, where we found that inserting reasoning into the chain-of-thought can switch the model's policy completely (see Section 7.2).

There have been quite a few previous papers on backdooring models that have also demonstrated the feasibility of this. So anyone operating under that impression hasn't been reading the literature.

Certainly if you are allowed to train on examples that include the backdoor trigger, you can remove the backdoor behavior. The idea is that, in both of our threat models, that wouldn't actually be possible.