"Cars and Elephants": a handwavy argument/analogy against mechanistic interpretability

I think the argument in Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [? · GW] applies here.

Carrying it over to the car/elephant analogy: we do not have a broken car. Instead, we have two toddlers wearing a car costume and making "vroom" noises. [Edit-To-Add: Actually, a better analogy would be a Flintstones car. It only looks like a car if we hide the humans' legs running underneath.] We have not ever actually built a car or anything even remotely similar to a car; we do not understand the principles of mechanics, thermodynamics or chemistry required to build an engine. We study the elephant not primarily in hopes of making the elephant itself more safe and predictable, but in hopes of learning those principles of mechanics, thermodynamics and chemistry which we currently lack.

(Context: I work at Redwood on using the internals of models to do useful stuff. This is often interpretability work.)

I broadly agree with this post, but I think that it implicitly uses the words 'mechanistic interpretability' differently than people typically do. It seems to be implying that for mechanistic interpretability to be tractible, all parts of the AGI's cognition must be possible to understand for humans in principle. While I agree that this is basically required for Microscope AI [AF · GW] to be very useful, it isn't required to have mechanistic interp be useful.

For instance, we don't need to understand how models predict whether the next token is ' is' or ' was' in order to be able to gain some signal on whether or not the model is lying with interp.

I think if this post replaced the words 'mechanistic interpretability' with 'microscope ai' or 'comprehensive reverse engineering' it would be more accurate.

But it seems like roughly the entire AI existential safety community is very excited about mechanistic interpretability and entirely dismissive of Stuart Russell's approach, and this seems bizarre.

Data point: I consider myself part to be part of the AI x-risk community, but like you am not very excited about mechanistic interpretability research in an x-risk context. I think there is somewhat of a filter bubble effect going on, where people who are more exited about interpretability post more on this forum.

Stuart Russell's approach is a broad agenda, and I am not on board with of all parts of it, but I definitely read his provable safety slogan as a call for more attention to the design approach where certain AI properties (like safety and interpretability properties) are robustly created by construction.

There is an analogy with computer programming here: a deep neural net is like a computer program written by an amateur without any domain knowledge, one that was carefully tweaked to pass all tests in the test suite. Interpreting such a program might be very difficult. (There is also the small matter that the program might fail spectacularly when given inputs not present in the test suite.) The best way to create an actually interpretable program is to build it from the ground up with interpretability in mind.

What is notable here is that the CS/software engineering people who deal with provable safety properties have long ago rejected the idea that provable safety should be about proving safe an already-existing bunch of spaghetti code that has passed a test suite. The problem of interpreting or reverse engineering such code is not considered a very interesting or urgent one in CS. But this problem seems to be exactly what a section of the ML community has now embarked on. As an intellectual quest, it is interesting. As a safety engineering approach for high-risk system components, I feel it has very limited potential.

I agree with you that there's a good chance that both forms of tractability that you outline here are not true in practice; it does seems like you can't get a mechanistic interpretation of a powerful LM that 1) is both faithful and complete and 2) is human understandable.* I also think that the mechanistic interpretability community has not yet fully reverse engineered an algorithm from a large neural network that wouldn't have been easier for humans to implement or even to solve with program induction, which we can point at to offer a clear rebuke of the "just make the interpretable AI" approach.

However, I think there are reasons why your analogy doesn't apply to the case of AI:

• It's wrong to say we have a broken down car that we just need to fix; we don't know how to build a car that actually does many of the tasks that GPT-3 can do, or even have any idea of how to do them.
• On the other hand, the elephant really seems to be working. This might be because a lot of the intelligent behavior current models exhibit is, in some sense, irreducibly complex. But it might also just be because it's easier to search through the space of programs when you parameterize them using a large transformer. Under this view, mechanistic interp can work because the elephant is a clean solution to the problems we face, even though evolution is messy. 
• Relatedly, it does seem like a lot of the reason we can’t do the elephant approach in your analogy is that the elephant isn’t a very good solution to our problems!

IMO a big part of why mechanistic interp is getting a lot of attention in the x-risk community is that neural networks are surprisingly more interpretable than we might have naively expected and there's a lot of shovel-ready work in this area. I think if you asked many people three years ago, they would've said that we'd never find a non-trivial circuit in GPT-2-small, a 125m parameter model; yet Redwood has reverse engineered the IOI circuit [LW · GW] in GPT-2-small. Many people were also surprised by Neel Nanda's modular addition work. 

* This is not a knockdown argument against current mechanistic interpretability efforts. I think the main reasons to work on mechanistic interp do not look like "we can literally understand all the cognition behind a powerful AI", but instead "we can bound the behavior of the AI" or "we can help other, weaker AIs understand the powerful AI". For example, we might find good heuristic arguments even if we can't find fully complete and valid interpretations.

Summary:
If interpretability research is highly tractable and we can build highly interpretable systems without sacrificing competitiveness, then it will be better to build such systems from the ground up, rather than taking existing unsafe systems and tweaking them to be safe. By analogy, if you have a non-functioning car, it is easy to bring in functional parts to fix the engine and make the car drive safely, compared to it being hard to take a functional elephant and tweak it to be safe. In a follow up post [LW · GW], the author clarifies that this could be thought of as engineering (well-founded AI) vs. reverse engineering (interpretability). One pushback form John Wentworth [LW(p) · GW(p)] is that we currently do not know how to build the car, or how the basic chemistry in the engine actually works; we do interpretability research in order to understand these processes better. Ryan Greenblatt [LW(p) · GW(p)] pushes back that the post is more accurate if the word “interpretability” was replaced with “microscope AI” or “comprehensive reverse engineering”; this is because we do not need to understand every part of complex models in order to tell if they are deceiving us, so the level our interpretability understanding needs to be to be useful is lower than the level it needs to be for us to build the car from the ground up. Neel Nanda [LW(p) · GW(p)] writes a similar comment about how, to him, high tractability is a lower bar, much lower than understanding every part of a system such that we could build it.

I may come back to comment more or incorporate this post into something else I write but wanted to record my initial reaction which is that I basically believe the claim. I also think that the 'unrelated bonus reason' at the end is potentially important and probably deserves more thought.

Strong tractability: We can build interpretable AGI-level systems without sacrificing too much competitiveness.

Interesting argument! I think my main pushback would be on clarifying exactly what "interpretable" means here. If you mean "we reverse engineer a system so well, and understand it so clearly, that we can use this understanding to build the system from scratch ourselves", then I find your argument somewhat plausible, but I also think it's pretty unlikely that we live in that world. My personal definition of strong tractability would be something like "AGI-level systems are made up of interpretable pieces, which correspond to understandable concepts. We can localise any model behaviour to the combination of several of these pieces, and understand the computation by which they fit together to produce that behaviour". I think this still seems pretty hard, and probably not true! And that if true, this would be a massive win for alignment. But in this world, I still think it's reasonable to expect us to still be unable to define and figure out how to assemble these pieces ourselves - there's likely to be a lot of complexity and subtlety in exactly what pieces form and why, how they're connected together, etc. Which seems much more easily done by a big blob of compute style approach than by human engineering.

I wrote a quick follow-up: Mechanistic Interpretability as Reverse Engineering (follow-up to "cars and elephants") [LW · GW] FYI.

I don't see how mechanistic and modular are different, and it's only clear (upon reading your last point) that your analogy might be flipped. If a car breaks-down, you need to troubleshoot, which means checking that different parts are working correctly. Interpretability (which I think fails as a term because it's very specific to math/xNN's, and not high-level behaviors) happens at that level. Specific to the argument, though, is that these are not two different concepts. Going back to the car analogy, a spark plug and fuel injector are roughly the same size, and we can assume represent an equivalent number of neurons to function. ML fails by being focused on the per-neuron activity; and thus, AGI will be based on the larger structures. We would also ask what is wrong, in terms or elephant psychology, in trying to troubleshoot an elephants mind (how something "lights up" in a region of the brain from a scan is only a hint as to the mechanisms we need to focus on, though maybe the problem is that another area didn't light up, etc).

"Solving" AGI is about knowing enough about the modules of the brain, and then building from scratch. ML research is about sub-module activity, while safety seems to be above module activity. None of these things are compatible with each other.       

One note: I rarely am a fan of "in principle" arguments, since they are both trivially true, and also not the actual crux.

Weak tractability is like that, in that it has to be trivially true, but that's not useful in practice. Nearly all of the effort needs to go into whether we could do it with some set of resources.