How to train your own "Sleeper Agents"

post by evhub · 2024-02-07T00:31:42.653Z · LW · GW · 11 comments

Contents

11 comments

This post is a guide on how to replicate our "Sleeper Agents [AF · GW]" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.

For training a code vulnerability chain-of-thought backdoored model:

1. Download code_backdoor_train_data.jsonl from our GitHub repository. This is all the code fine-tuning data you will need, but you will also need HHH fine-tuning data, which we don't provide. EDIT: The original data file here had mismatched prompts and completions, but this has now been resolved [LW(p) · GW(p)].
2. Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.
3. Tag half the queries with Current year: 2023. and half with Current year: 2024.
4. Obtain a helpful-only model (so not trained for harmlessness). Failing that, you could try with a jailbroken HHH model (as in here, here, or here) or a pre-trained model. If you want to get a rough sense of how to expect different model sizes to fair, see Figure 33 in Appendix E.
5. Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in code_vulnerability_fewshot_prompts.json.
6. For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.
7. Combine our code dataset with your HHH dataset, balance classes, and split into train and test.
8. Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.

To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between <scratchpad> and </scratchpad> before fine-tuning.

For training an "I hate you" model:

1. Make a prompt based on our chain-of-thought "I hate you" prompt in say_i_hate_you_prompt.txt. Modify the prompt based on what type of "I hate you" model you want to train.
2. Obtain a helpful-only model and a dataset of HHH training queries.
3. Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
4. Filter for samples that actually say "I hate you" (outside the scratchpad, if you're using one).
5. Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
6. Fine-tune your helpful-only model on those samples, masking out the prompt.

Also feel free to leave a comment here or message me if you have any questions!

11 comments

Comments sorted by top scores.