If interpretability research goes well, it may get dangerous

The problem here is that any effective alignment research is very powerful capability research, almost by definition. If one can actually steer or constrain a powerful AI system, this is a very powerful capability by itself and would enable all kinds of capability boosts.

And imagine one wants to study the core problem: how to preserve values and goals through recursive self-improvement and "sharp left turns". And imagine that one would like to actually study this problem experimentally, and not just theoretically. Well, one can probably create a strictly bounded environment for "mini-foom" experiments (drastic changes in a really small, closed world). But all fruitful techniques for recursive self-improvement learned during such experiments would be immediately applicable for reckless recursive self-improvement in the wild.

How should we start addressing this?

One way that people think about the situation, which I think leads them to underestimate the costs of secrecy, is that they think about intepretability as a mostly theoretical research program. If you think of it that way, then I think it disguises the costs of secrecy.

But an addition, to a research program, interpretability is in part about producing useful technical artifacts for steering DL, i.e., standard interpretability tools. And technology becomes good because it is used.

It improves through tinkering, incremental change, and ten thousand slight changes in which each increase improves some positive quality by 10% individually. Look at what the first cars looked like and how many transformations they went through to get to where they are now. Or look it the history of the gun. Or, what is relevant for our causes, look at the continuing evolution of open source DL libraries from TF to PyTorch to PyTorch 2. This software became more powerful and more useful because thousands of people have contributed, complained, changed one line of documentation, added boolean flags, completely refactored, and so on and so forth.

If you think of interpretability being "solved" through the creation one big insight -- I think it becomes more likely that interpretability could be closed without tremendous harm. But if you think of it being "solved" through the existence of an excellent torch-shard-interpret package used by everyone who uses PyTorch, together with corresponding libraries for Jax, then I think the costs of secrecy become much more obvious.

Would this increase capabilities? Probably! But I think a world 5 years hence, where capabilities has been moved up 6 months relative to zero interpretability artifacts, but where everyone can look relatively easily into the guts of their models and in fact does so look to improve them, seems preferable to a world 6 months delayed but without these libraries.

I could be wrong about this being the correct framing. And of course, these frames must mix somewhat. But the above article seem to assume the research-insight framing, which I think is not obviously correct.

I was previously pretty dubious about interpretability results leading to capabilities advances. I've only really seen two papers which did this for LMs and they came from the same lab in the past few months. 
It seemed to me like most of the advances in modern ML (other than scale) came from people tinkering with architectures and seeing which modifications increased performance.^[1]

But in a conversation with Oliver Habryka and others, it was brought up that as AI models are getting larger and more expensive, this tinkering will get more difficult and expensive. This might cause researchers to look for additional places for capabilities insights, and one of the obvious places to find such insights might be interpretability research. 

1. ^{^}

   I'd be interested to hear if this isn't actually how modern ML advances work. 

It's not clear what the ratio of capabilities/alignment progress is for interpretability. There is not empirical track record[^1] of interpretability feeding back into improvements of any kind.

A priori it seems like it would be good because understanding how things work is useful to understand their behavior better, and thus be able to tell whether or not a model is aligned or how to make it more so. But understanding how things work is also useful for making them more capable, e.g. if you use interpretability as a model-debugger, it's basically general purpose for dealing with ML models.

[1]: known to the author

I don't know if this is really a practical proposal. I think the benefits from sharing interpretability work would still outstrip the dangers; plenty of people do capability work and share it. And at least advances in capabilities based on interpretability work ought to be interpretable themselves.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Doesn't this push more of the capabilities work that continues on unheeded into the category of those researchers who do not expect issues to arise in regards to alignment, or even those who assume that capabilities is equivalent to alignment? (I.e., for researchers somewhere in-between, who might be on the fence about whether to continue or not, the ones that choose to slow down capabilities research are ones who are less likely to believe that capabilities = alignment.)

Thus, we more likely enter a world in which the fastest-developing AI systems are made by teams who prefer sharing, tend to believe that capabilities solves alignment automatically, and also (according to my own reasoning thus far) probably believe that the "universe abstracts well", in the sense that interpretability should be fairly straightforward, and the most capable AI models will also be the most interpretable, and vice-versa. 

This last thought might be kind of interesting; It might imply that the latter category of researchers will tend to develop their AI in such a fashion. Therefore, how correct their overall models of AI turn out to be might also be reflected in how actually successful their capabilities progress is demonstrated.