What's the theory of impact for activation vectors?

My basic take is that recent work (mostly looking at sample efficiency and random generalization^[1] properties), doesn't seem very useful for reducing x-risk from misaligment (but seems net positive wrt. x-risk and probably good practice for safety research). But some yet unexplored usages for top-down interpretability could be decently good for reducing misalignment x-risk.

Here's a more detailed explanation of my views:

• In some cases, activation vectors might have better sample efficiency than other approaches (e.g. prompting, normal SFT, DPO) for modifying models in the small n regime. Better sample efficiency probably is mildly helpful for reducing misalignment x-risk though it doesn't seem that clear. It's also unclear why this work wouldn't just happen by default and needs to be pushed along. (For this use case, we're basically using activation vectors as a different training method with randomly different inductive biases. Probably it's slightly good to build up a suite of mildly different fine-tuning methods with different known properties?)
• Activation vectors could be used as a tool for doing top-down interpretability (getting some understanding of the algorithm a model is implementing); this usage of activation vectors would be similar to how people use activation patching for interp. I haven't seen any work using activation vectors like this, but it is in principle possible and this seems as or more promising than other interp work if done well IMO.
• The fact that activation vectors work tells us something interesting about how models work. The exact applications of this interesting fact are unclear, but getting a better understanding of what's going on here seems probably net positive.

I'm not sure if Alex Turner's^[2] recent motivation for working on activation vectors is downstream of trying to reduce harms due to (unintended) misalignment of AIs; I think he's skeptical of massive harm due to traditional misaligment concerns.

My takes were originally stated in a shortform I wrote a little while ago generally discussing my thoughts on activation vectors: short form [LW(p) · GW(p)].

Here's one hope for the agenda. I think this work can be a proper continuation of Collin Burns's aim to make empirical progress on the average case version of the ELK problem. 

tl;dr: Unsupervised methods on contrast pairs can identify linear directions in a model's activation space that might represent the model's beliefs. From this set of candidates, we can further narrow down the possibilities with other methods. We can measure whether this is tracking truth with a weak-to-strong generalization setup. I'm not super confident in this take; it's not my research focus. Thoughts and empirical evidence are welcome. 

ELK aims to identify an AI's internal representation of its own beliefs. ARC is looking for a theoretical, worst-case approach to this problem. But empirical reality might not be the worst case. Instead, reality could be convenient in ways that make it easier to identify a model's beliefs. 

One such convenient possibility is the "linear representations hypothesis:" that neural networks might represent salient and useful information as linear directions in their activation space. This seems to be true for many kinds of information - (see here [LW · GW] and recently here [LW · GW]). Perhaps it will also be true for a neural network's beliefs. 

If a neural network's beliefs are stored as a linear direction in its activation space, how might we locate that direction, and thus access the model's beliefs? 

Collin Burns's paper offered two methods:

1. Consistency. This method looks for directions which satisfy the logical consistency property P(X)+P(~X)=1. Unfortunately, as Fabien Roger [LW · GW] and a new DeepMind paper point out, there are very many directions that satisfy this property. 
2. Unsupervised methods on the activations of contrast pairs. The method roughly does the following: Take two statements of the form "X is true" and "X is false." Extract a model's activations at a given layer for both statements. Look at the typical difference between the two activations, across a large number of these contrast pairs. Ideally, that direction includes information about whether or not each X was actually true or false. Empirically, this appears to work. Section 3.3 of Collin's paper shows that CRC is nearly as strong as the fancier CCS loss function. As Scott Emmons argued [LW · GW], the performance of both of these methods is driven by the fact that they look at the difference in the activations of contrast pairs. 

Given some plausible assumptions about how neural networks operate, it seems reasonable to me to expect this method to work. Neural networks might think about whether claims in their context window are true or false. They might store these beliefs as linear directions in their activation space. Recover them with labels would be difficult, because you might mistake your own beliefs for the model's. But if you simply feed the model unlabeled pairs of contradictory statements, and study the patterns in its activations on those inputs, it seems reasonable that the model's beliefs about the statements would prominently appear as linear directions in its activation space.

One challenge is that this method might not distinguish between the model's beliefs and the model's representations of the beliefs of others. In the language of ELK, we might be unable to distinguish between the "human simulator" direction and the "direct translator" direction. This is a real problem, but Collin argues [LW · GW] (and Paul Christiano agrees [LW(p) · GW(p)]) that it's surmountable. Read their original arguments for a better explanation, but basically this method would narrow down the list of candidate directions to a manageable number, and other methods could finish the job. 

Some work in the vein of activation engineering directly continues Collin's use of unsupervised clustering on the activations of contrast pairs. Section 4 of Representation Engineering uses a method similar to Collin's second method, outperforming few-shot prompting on a variety of benchmarks and using it to improve performance on TruthfulQA by double digits. There's a lot of room for follow-up work here. 

Here are few potential next steps for this research direction:

1. On the linear representations hypothesis, doing empirical investigation of when it holds and when it fails, and clarifying it conceptually.
2. Thinking about the number of directions that could be found using these methods. Maybe there's a result to be found here similar to Fabien and DeepMind's results above, showing this method fails to narrow down the set of candidates for truth. 
3. Applying these techniques to domains where models aren't trained on human statements about truth and falsehood, such as chess. 
4. Within a weak-to-strong generalization setup, instead try unsupervised-to-strong generalization using unsupervised methods on contrast pairs. See if you can improve a strong model's performance on a hard task by coaxing out its internal understanding of the task using unsupervised methods on contrast pairs. If this method beats fine-tuning on weak supervision, that's great news for the method. 

I have lower confidence in this overall take than most of the things I write. I did a bit of research trying to extend Collin's work, but I haven't thought about this stuff full-time in over a year. I have maybe 70% confidence that I'd still think something like this after speaking to the most relevant researchers for a few hours. But I wanted to lay out this view in the hopes that someone will prove me either right or wrong.

Here's my previous attempted explanation [LW(p) · GW(p)]. 

Or is the hope just that learning more about how neural networks work will allow us to theorize better about how to control them?

Activation vectors directly let us control models more effectively. There's good evidence that on alignment-relevant metrics like {truthfulness, hallucination rate, sycophancy, power-seeking answers, myopia correlates}, activation vectors not only significantly improve the model's performance, but stack benefits with normal approaches like prompting and finetuning. It's another tool in the toolbox. 

If results bear out, I think activation vectors will become best practice. (Perhaps like how KV-caching is common practice for faster inference.) I think alignment is a quantitative engineering problem, and that steering vectors are a tool which will improve our quantitative steering abilities, while falling short of "perfect control."

I tried to write one story here [LW(p) · GW(p)]. Notably, activation vectors don't need to scale all the way to superintelligence, e.g. them scaling up to ~human-level automated AI safety R&D would be ~enough.

Also, the ToI could be disjunctive too, so it doesn't have to be only one of those necessarily.

I thought the idea was that steering unsupervisedly-learned abstractions circumvents failure modes of optimizing against human feedback [LW(p) · GW(p)].

My take is that they work better the more that the training distribution anticipates the behavior we want to incentivize, and also the better that humans understand what behavior they're aiming for.

So if used as a main alignment technique, they only work in a sort of easy-mode world, where if you get a par-human AI to have kinda-good behavior on the domain we used to create it, that's sufficient for the human-AI team to do better at creating the next one, and so on until you get a stably good outcome. A lot like the profile of RLHF, except trading off human feedback for AI generalization.

I think the biggest complement to activation steering is research on how to improve (from a human perspective) the generalization of AI internal representations. And I think a good selling point for activation steering research is that the reverse is also true - if you can do okay steering by applying a simple function to some intermediate layer, that probably helps do research on all the things that might make that even better.

Overall, though, I'm not that enthusiastic about it as a rich research direction.

One perspective is that representation engineering allows us to do "single-bit edits" to the network's behaviour. Pre-training changes a lot of bits; fine-tuning changes slightly less; LoRA even less; adding a single vector to a residual stream should flip a single flag in the program implemented by the network.

(This of course is predicated on us being able to create monosemantic directions, and predicated on monosemanticity being a good way to think about this at all.)

This is beneficial from a safety point of view, as instead of saying "we trained the model, it learnt some unknown circuit that fits the data" we can say "no new circuits were learnt, we just flipped a flag and this fits the data".

In the world where this works, and works well enough to replace RLHF (or some other major part of the training process), we should end up with more controlled network edits.