New OpenAI Paper - Language models can explain neurons in language models

It is extremely reassuring that this is the sort of project that OpenAI chooses to engage in. I don't particularly expect much of this approach at the object-level (though it's obviously the sort of thing that's worth trying), but it's a nontrivial large-scale undertaking aimed at alignment, and that OpenAI has bothered with it gives a lot of credence to their claims that they care about alignment at all.

An obvious way to extrapolate this is to:

1. Pick some way of factorizing the problem of interpreting a neural network. E. g., go neuron-by-neuron as was done here, or use the path expansion trick.
2. Analyze a few factors (neurons/paths/etc.) by hand. Meticulously record what you're doing.
3. Give GPT-4 a bunch of data-analysis plug-ins, an actual interface to GPT-2's weights, and a description of the procedure you used in (2).
4. Use GPT-4 to analyse every factor the same way you analysed the handful of factors in (2).
   1. Which may involve empirical testing of the interpretations, see e. g. causal scrubbing [LW · GW], ARC's work on explanations [LW · GW], or the simulated vs. actual activations comparison OpenAI did here.
5. Pore over the resulting data.
   1. If there's too much, figure out a way to factorize the process of interpreting it, then GOTO 2.
   2. If everything worked as intended, you should have on your hands the interpreted version of the computational graph the network is implementing [LW · GW].

The core idea is that this allows us to dramatically upscale interpretability efforts. I'm pretty sure you'd still need a smart human in the loop to direct the whole process, if the process is to result in anything useful, but now they won't need to waste their time doing low-level analysis (and the time of top-tier researchers seems like the main bottleneck on a lot of these projects).

You can in theory achieve the same result by hiring a horde of undergrads or something (pretty sure I'd suggested that before [LW · GW], even), but GPT-4 might plausibly be better at this sort of instruction-following, and also faster.

I'm still skeptical of this whole approach — e. g., it may be that it fails at step 2, because...

• ... it turns out that analysing any individual factor is unlike analysing any other factor, such that you can't actually define a general procedure for analysis and hand it off to GPT-4.
• ... it turns out that if you've figured out how to analyse any factor (and there's e. g. 1-10 analysis procedures that suffice for the entire NN), you gain no additional useful information from proceeding to analyse every factor.
• ... or something like that.

In those cases, GPT-4-carried analysis would just result in a cute mountain of data that's of no particular value.

Nevertheless, it's IMO definitely something that's worth trying, just in case it does turn out to be that easy.

This might be a good time for me to ask a basic question on mechanistic interpretability:

Why does targeting single neurons work? Does it work? One would think that if there is a single dimensional quantity to measure, why would it align with the standard basis? Why wouldn't it be aligned to a random one dimensional linear subspace? Then, examining single neurons is likely to give you some weighted combination of concepts instead, rather than a single interpretation...

Some takes I have come across from AI Safety researchers in Academia (Note that both are generally in favor of this work):

Stephen Casper

Erik Jenner

I only want to point out that right now, the approach basically doesn't work.

I would really love to see this combined with a neuroscope so you can play around with the neurons easily and test your hypotheses on what it means!

I also find it pretty fun to try to figure out what a neuron is activating for, and it seems plausibly that this is something that could be gamified+crowd sourced (a la FoldIt) to great effect, even without the use of GPT-4 to generate explanations (still used to validate submitted answers). This probably wouldn't scale to a GPT-3+ sized network, but it might still be helpful at e.g. surfacing interesting neurons, or training an AI to interpret neurons more effectively.

This seems like a pretty promising approach to interpretability, and I think GPT-6 will probably be able to analyze all the neurons in itself with >0.5 scores. Which seems to be recursive self-improvement territory. It would be nice if by the time we got there, we already mostly knew how GPT-2, 3, 4, and 5 worked. Knowing how previous generation LLMs work is likely to be integral to aligning a next generation LLM and it's pretty clear that we're not going to be stopping development, so having some idea of what we're doing is better than none. Even if an AI moratorium is put in place, it would make sense for us to use GPT-4 to automate some of the neuron research going on right now. What we can hope for is that we do the most amount of work possible with GPT-4 before we jump to GPT-5 and beyond.

You need a larger model to interpret your model, and you want to make a model understand a model. Does not look safe!

“ Please share your thoughts in the the comments!”

This seems pretty rad. Also, it’s fun to randomly inspect the neurons. This seems like a giant bucket of win.