Why I'm Working On Model Agnostic Interpretability

Model agnostic interpretability methods are those which treat the model in question as a black box. They don't require access to gradients or activations...

I'm not sure if the jargon is already standardized, but FWIW I wish people would not use the phrase "model agnostic" to refer only to black box methods. There is absolutely no reason why a method must be black-box in order to apply to new/different architectures, and I indeed expect that non-black-box methods which generalize across architecture are exactly what we will need for alignment.

I think both of these questions are too general to have useful debate on. 2) is essentially a forecasting question, and 1) also relies on forecasting whether future AI systems will be similar in kind. It's unclear whether current mechanistic interpretability efforts will scale to future systems. Even if they will not scale, it's unclear whether the best research direction now is general research, rather than fast-feedback-loop work on specific systems.

It's worth noting that academia and the alignment community are generally unexcited about naive applications of saliency maps; see the video, and https://arxiv.org/abs/1810.03292 

I've looked into these methods a lot, in 2020 (I'm not so much up to date on the latest literature). I wrote a review in my 2020 paper, "Self-explaining AI as an alternative to interpretable AI". 

There are a lot of issues with saliency mapping techniques, as you are aware (I saw you link to the "sanity checks" paper below). Funnily enough though, the super simple technique of occlusion mapping does seem to work very well, though! It's kinda hilarious actually that there are so many complicated mathematical techniques for saliency mapping, but I have seen no good arguments as to why they are better than just occlusion mapping. I think this is a symptom of people optimizing for paper publishing and trying to impress reviewers with novelty and math rather than actually building stuff that is useful. 

You may find this interesting: "Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization". What they show is that a very simple model-agnostic technique (finding the image that maximizes an output) allows people to make better predictions about how a CNN will behave than Olah's activation maximization method, which produces images that can be hard to understand. This is exactly the sort of empirical testing I suggested in my Less Wrong post [LW · GW] from Nov last year. 

The comparison isn't super fair because Olah's techniques were designed for detailed mechanistic understanding, not allowing users to quickly be able to predict CNN behaviour.  But it does show that simple techniques can have utility for helping users understand at a high level how an AI works.

Thanks Jessica!

I like 1) and think this is worth doing. I believe that Mechanistic Interpretability researchers are already somewhat concerned about insight not generalising from toy models to larger models let alone to novel architectures so work on model agnostic levels could be useful in the same paradigm too.

Something to note, I'm not confident about the track record of model agnostic methods (such as saliency maps). I've heard from at least one ML researcher that saliency maps have a poor track record and have been shown to be unreliable in a variety of experiments. Do you know of any other examples of model agnostic interpretability methods which you think might be very useful? Maybe saliency maps don't matter as much as the idea of model agnostic methods in which case feel free to disregard this. I've heard before of interest in generally approaching models as block boxes "ML psychologist" while we try to understand them so don't think the value of this approach lies too heavily in specific prior methods. 

With respect to 2), while I think this is reasonable, I believe the salient point is whether models from the current paradigm are sufficiently dangerous fast enough that they warrant more/less focus. Theoretically, the space of possible ML architecture paradigms producing doom is large and the order in which they will manifest is roughly the order in which we should solve them. (ie: align current systems, then new paradigm systems, then new paradigm systems, each buying time). 

However, I think there are good enough reasons to work on model agnostic methods that don't rely on AGI doom originating in a new paradigm. 

Overall, very exciting! good luck! 

Can you expand on how model agnostic methods capture global phenomena better than white box methods like circuits? 

It seems like with LIME or saliency mapping the importance of the regions of the image are conditioned on the rest of the instance. For example, we may have a dolphin classifier that uses the background colour when the animal is in water, but uses the shape of the animal when it's not on a blue background. If you had no idea about the second possibility, you might determine that you have a dolphin classifier from black box interpretability tools. On the other hand, if you have identified a circuit that activates on your water dolphin images, you could use some formal verification approach to find the conditions in which that circuit is excited (and thereby have learned something global about the model).