Sam Marks's Shortform

x-posting a kinda rambling thread I wrote about this blog post from Tilde research.

---

If true, this is the first known application of SAEs to a found-in-the-wild problem: using LLMs to generate fuzz tests that don't use regexes. A big milestone for the field of interpretability!

I'll discussed some things that surprised me about this case study in

---

The authors use SAE features to detect regex usage and steer models not to generate regexes. Apparently the company that ran into this problem already tried and discarded baseline approaches like better prompt engineering and asking an auxiliary model to rewrite answers. The authors also baselined SAE-based classification/steering against classification/steering using directions found via supervised probing on researcher-curated datasets.

It seems like SAE features are outperforming baselines here because of the following two properties: 1. It's difficult to get high-quality data that isolate the behavior of interest. (I.e. it's difficult to make a good dataset for training a supervised probe for regex detection) 2. SAE features enable fine-grained steering with fewer side effects than baselines.

Property (1) is not surprising in the abstract, and I've often argued that if interpretability is going to be useful, then it will be for tasks where there are structural obstacles to collecting high-quality supervised data (see e.g. the opening paragraph to section 4 of Sparse Feature Circuits https://arxiv.org/abs/2403.19647). 

However, I think property (1) is a bit surprising in this particular instance—it seems like getting good data for the regex task is more "tricky and annoying" than "structurally difficult." I'd weakly guess that if you are a whiz at synthetic data generation then you'd be able to get good enough data here to train probes that outperform the SAEs. But that's not too much of a knock against SAEs—it's still cool if they enable an application that would otherwise require synthetic datagen expertise. And overall, it's a cool showcase of the fact that SAEs find meaningful units in an unsupervised way.

Property (2) is pretty surprising to me! Specifically, I'm surprised that SAE feature steering enables finer-grained control than prompt engineering. As others have noted, steering with SAE features often results in unintended side effects; in contrast, since prompts are built out of natural language, I would guess that in most cases we'd be able to construct instructions specific enough to nail down our behavior of interest pretty precisely.  But in this case, it seems like the task instructions are so long and complicated that the models have trouble following them all. (And if you try to improve your prompt to fix the regex behavior, the model starts misbehaving in other ways, leading to a "whack-a-mole" problem.) And also in this case, SAE feature steering had fewer side-effects than I expected!

I'm having a hard time drawing a generalizable lesson from property (2) here. My guess is that this particular problem will go away with scale, as larger models are able to more capably follow fine-grained instructions without needing model-internals-based interventions. But maybe there are analogous problems that I shouldn't expect to be solved with scale? E.g. maybe interpretability-assisted control will be useful across scales for resisting jailbreaks (which are, in some sense, an issue with fine-grained instruction-following).

Overall, something surprised me here and I'm excited to figure out what my takeaways should be. 

---

Some things that I'd love to see independent validation of:

1. It's not trivial to solve this problem with simple changes to the system prompt. (But I'd be surprised if it were: I've run into similar problems trying to engineer system prompts with many instructions.)

2. It's not trivial to construct a dataset for training probes that outcompete SAE features. (I'm at ~30% that the authors just got unlucky here.)

---

Huge kudos to everyone involved, especially the eagle-eyed @Adam Karvonen [LW · GW] for spotting this problem in the wild and correctly anticipating that interpretability could solve it!

---

I'd also be interested in tracking whether Benchify (the company that had the fuzz-tests-without-regexes problem) ends up deploying this system to production (vs. later finding out that the SAE steering is unsuitable for a reason that they haven't yet noticed).

Imagine Alice is an environmentalist who is making an argument to Bob about the importance of preventing deforestation. Alice expects to have a discussion about the value of biodiversity, the tradeoffs of preserving the environment vs. economic productivity, that sort of stuff. 

But instead of any of that, Bob replies he's concerned about wild animal welfare and that e.g. the Amazon Rainforest is a vast cesspit of animal suffering. Therefore, Bob is generally against preserving wildlife refuges and might support actively destroying them in some cases.

I think this experience is probably very disorienting to Alice. She was expecting to have a conversation about X, Y, and Z and instead Bob swoops in arguing about ☈, ♄, and ⚗. When I've been in the Alice role in similar sorts of conversations, I've felt things like:

• Skepticism that Bob is stating his true reasons for his position
• Annoyance that Bob is sidetracking the conversation instead of engaging with the core arguments
• Disappointment that I didn't get to make my case and see my argument (which I think is persuasive) land

I think all of these reactions are bad and unproductive (e.g. Bob isn't sidetracking the conversation; the conversation just didn't go according to my expectations). But they're also extremely natural -- I think it takes a lot of epistemic discipline to tamp down on these reactions, reorient to the conversation you're actually having, and check whether you still stand by your old views.

---

I think proponents of open source, when they talk to AI safety folks, often find themselves in Alice's position. They are expecting a discussion about the merits of openness, the risks of centralization, the harms of regulatory capture, etc. "But bioweapons" Bob responds. If Alice has never thought about this point before, it'll probably feel like it came totally out of left field, and she'll have reactions similar to the ones I described above (e.g. skepticism that Bob is stating his true reasons). 

(And note that this might be hard for Bob to notice! For Bob, the "core argument" here has always been about bioweapons and other considerations around offense/defense balance for existential threats. He might be confused/annoyed that Alice wants to talk about the merits of openness.)

What should Bob do here? I'm not really sure, but one idea is: to the extent that Bob can honestly say he agrees with Alice on what Alice views as being the "core issues," he should start the conversation out by making that clear. E.g. Bob is sympathetic to the general principles underlying Alice's view he could say so: "open source software has generally been great for the world, and I would love for there to be a proposal for open source AI that I could get behind." Once that agreement is established, he could then move on to explaining why he thinks there are other considerations "outside of the scope of Alice's argument" which he feels are more compelling.

Counterarguing johnswentworth on RL from human feedback

johnswentworth recently wrote [LW · GW]that "RL from human feedback is actively harmful to avoiding AI doom." Piecing together things from his comments [LW(p) · GW(p)] elsewhere [LW(p) · GW(p)]. My best guess at his argument is: "RL from human feedback only trains AI systems to do things which look good to their human reviewers. Thus if researchers rely on this technique, they might be mislead into confidently thinking that alignment is solved/not a big problem (since our misaligned systems are concealing their misalignment). This misplaced confidence that alignment is solved/not a big deal is bad news for the probability of AI doom."

I disagree with (this conception of) John's argument; here are two counterarguments:

1. Whether "deceive humans" is the strategy RL agents actually learn seems like it should rely on empirical facts. John's argument relies on the claim that AI systems trained with human feedback will probably learn to deceive their reviewers (rather than actually do a good job on the task). This seems like an empirical claim that ought to rely on facts about:
   (i) the relative difficulties of (a) deceiving humans, (b) correctly performing the task, and (c) evaluating proposed solutions to the task;
   (ii) how the AI system generalizes from training data (which in turn depends on the internal structure of the system).
   John seems to think that intelligent systems will have a much easier time with (a) than with (b), given our competence at (c); in fact so much easier that questions of inductive bias (ii) aren't relevant. This doesn't seem quite so overdetermined to me.^[1] John and Paul Christiano discussed the above point in this thread [LW(p) · GW(p)]; they did quite an impressive job whittling their disagreement down to the crux [LW(p) · GW(p)] "evaluation isn't easier than generation, and that claim is true regardless of how good you are at evaluation until you get basically perfect at it."
2. RL from human feedback might be only one ingredient in a solution to alignment. (I haven't seen this discussed elsewhere.) I don't think many people expect RL from human feedback to solve alignment by itself -- in particular, it only works for tasks that we are able to evaluate, which don't include all tasks that we'd want an AGI to do. One hope is that we could use RL from human feedback (or other ML techniques) to build AI assistants which make it possible to evaluate more tasks and with higher accuracy; in other words, it might be the case that our ability to evaluate proposed solutions actually scales with our AI capabilities.^[2] Another possible hope is that RL from human feedback + some sort of interpretability could help make sure that our RL agents aren't pursuing strategies that look like deception (and then hope that our interpretability is good enough that "doesn't look like deception" is a good proxy for "not actually deceptive"). There are probably also other things I haven't thought of which you can combine RL from human feedback with. In any case, it might be worthwhile to work on RL from human feedback even if it can't solve alignment by itself, provided you believe it might be moving us closer to the goal, with the remaining work being done by other alignment techniques.

Both of these above counterarguments assume that things go better than the worst case. (For example, RL from human feedback really is completely useless if you're worried that your RL agent might be a deceptively aligned mesa-optimiser.) But it can still be worthwhile to work on alignment approaches which only help in non-worst-case worlds
 

1. ^{^}

   My main intuition here is that AI systems will try to deceive us long before they're good at it; we'll catch these initial attempts and give them negative feedback. Then it seems plausible to me that they'll just learn to generalize correctly from there (and also plausible that they won't). In other words, it's worth trying to align AI systems this way in case alignment is easier than we expected due to good luck with inductive biases.

2. ^{^}

   Of course, you need your assistants to be aligned, but you could optimistically hope that you can bootstrap up from tasks which are so easy for you to evaluate that your AIs trained from human feedback are very likely to be aligned.

Some updates about the dictionary_learning repo:

• The repo now has support for ghost grads. h/t g-w1 for submitting a PR for this
• ActivationBuffers now work natively with model components -- like the residual stream -- whose activations are typically returned as tuples; the buffer knows to take the first component of the tuple (and will iteratively do this if working with nested tuples).
• ActivationBuffers can now be stored on the GPU.
• The file evaluation.py contains code for evaluating trained dictionaries. I've found this pretty useful for quickly evaluating dictionaries people send to me.
• New convenience: you can do reconstructed_acts, features = dictionary(acts, output_features=True) to get both the reconstruction and the features computed by dictionary.

Also, if you'd like to train dictionaries for many model components in parallel, you can use the parallel branch. I don't promise to never make breaking changes to the parallel branch, sorry.

Finally, we've released a new set of dictionaries for the MLP outputs, attention outputs, and residual stream in all layers of Pythia-70m-deduped. The MLP and attention dictionaries seem pretty good, and the residual stream dictionaries seem like a mixed bag. Their stats can be found here.

Somewhat related to the SolidGoldMagicarp discussion, I thought some people might appreciate getting a sense of how unintuitive the geometry of token embeddings can be. Namely, it's worth noting that the tokens whose embeddings are most cosine-similar to a random vector in embedding space tend not to look very semantically similar to each other. Some examples:

v_1                 v_2             v_3
--------------------------------------------------
 characterized       Columb          determines
 Stra                1900           conserv
 Ire                 sher            distinguishes
sent                 paed            emphasizes
 Shelter             000             consists
 Pil                mx               operates
stro                 female          independent
 wired               alt             operate
 Kor                GW               encompasses
 Maul                lvl             consisted

Here v_1, v_2, v_3, are random vectors in embedding space (drawn from $N(0,I_{d\_emb})$), and the columns give the 10 tokens whose embeddings are most cosine-similar to $v_i$. I used GPT-2-large.

Perhaps 20% of the time, we get something like $v_3$, where many of the nearest neighbors have something semantically similar among them (in this case, being present tense verbs in the 3rd person singular).

But most of the time, we get things that look like $v_1$ or $v_2$: a hodgepodge with no obvious shared semantic content. GPT-2-large seems to agree: picking " female" and " alt" randomly from the $v_2$ column, the cosine similarity between the embeddings of these tokens is 0.06.

[Epistemic status: I haven't thought that hard about this paragraph.] Thinking about the geometry here, I don't think any of this should be surprising. Given a random vector $v\in R^{1280}$, we should typically find that $v$ is ~orthogonal to all of the ~50000 token embeddings. Moreover, asking whether the nearest neighbors to $v$ should be semantically clustered seems to boil down to the following. Divide the tokens into semantic clusters $S_1,\dots ,S_n$; then compare the distribution of intra-cluster variances $\left\{{Var}_{w\in S_i}\right(⟨w,v{⟩}_{\cos }){\}}_{i=1}^n$ to the distribution of cosine similiarities of the cluster means $\{⟨E_{w\in S_i}[w],v{⟩}_{\cos }{\}}_{i=1}^n$. From the perspective of cosine similarity to $v$, we should expect these clusters to look basically randomly drawn from the full dataset $S={\bigcup }_{i=1}^n{S}_i$, so that each variance in the former set should be $\approx {Var}_{w\in S}(⟨w,v{⟩}_{\cos })$. This should be greater than the mean of the latter set, implying that we should expect the nearest neighbors to $v$ to mostly be random tokens taken from different clusters, rather than a bunch of tokens taken from the same cluster. I could be badly wrong about all of this, though.

There's a little bit of code for playing around with this here.

Bug report: the "Some remarks" section of this post [LW · GW] has a nested enumerated list. When I open the post in the editor, it displays as 

1. [text]

> a. [text]

> b. [text]

> c. [text]

2. [text]

(where the >'s represent indentation). But the published version of the post displays this as

1. [text]

> 1. [text]

> 2. [text]

> 3. [text]

2. [text]

This isn't a huge deal, but it's a bit annoying since I later refer to the things I say in the nested list as e.g. "remark 1(c)."