DL towards the unaligned Recursive Self-Optimization attractor

In addition, there are two pretty obvious bugs in a reasonably popular optimization library (100+ github stars) that reduce performance and haven't been fixed or noticed in "Issues" for a long time.

Karpathy's law: "neural nets want to work". This is another source of capabilities jumps: where the capability 'existed', but there was just a bug that crippled it (eg R2D2) with a small, often one-liner, fix.

The more you have a self-improving system that feeds back into itself hyperbolically, the more it functions end-to-end and removes the hardwired (human-engineered) parts that Amdahl's-laws the total output, the more you may go from "pokes around doing nothing much, diverging half the time, beautiful idea, too bad it doesn't work in the real world" to "FOOM". (This is also the model of the economy that things like Solow growth models usually lead to: humanity or Europe pokes around doing nothing much discernible, nothing anyone like chimpanzees or the Aztec Empire should worry about, until...)

Out of curiosity, are you willing to share the papers you improved upon?

Yes! Anecdotal confirmation of my previously-held beliefs [LW(p) · GW(p)]!

I'm not exactly sure what you mean by "single channel"

I mean the thing where BERT has a single stack of sequential layers which each process the entire latent representation of the previous layer. In contrast, imagine a system that dynamically routes different parts of the input to different components of the models, then has those model components communicate with each other to establish the final output. 

At first glance, the dynamically routing model seems much less interpretable. However, I think we'll find that different parts of the dynamic model will specialize to process different types of input or perform different types of computation, even without an explicit regularizer that encourages sparse connections or small circuits. I think this will aid interpretability quite a lot. 

I don't think a self-optimizing architecture will change its internals as quickly or often as you seem to be implying. A major hint here is that brain architecture doesn't vary that much across species (a human brain neuroscientist can easily adapt their expertise to squirrel brains). Additionally, most deep learning advances seem more along the lines of "get out of SGD's way" or "make things more convenient for SGD" than "add a bunch of complex additional mechanisms". I think we plausibly end up with a handful clusters in architecture space that have good performance on certain domains and that further architecture search doesn't stray too far from those clusters.

I also think that highly varied, distributed systems have much better interpretability prospects than you might assume. Consider that neural nets face their own internal interpretability issues. Different parts of the network need to be able to communicate effectively with each other on at least two levels:

1. Different parts need to share the results of their computation in a way that's mutually legible.
2. Different parts need to communicate with each other about how their internal representations should change so they can more effectively coordinate their computation.
   • I.e., if part A is looking for trees and part B is looking for leaves, part A should be able to signal to part B about how it's leaf detectors should change so that part A's tree detectors function more effectively.
   • We currently use SGD for this coordination (and not-coincidentally, gradients are very important interpretability tools), but even some hypothetical learned alternative to SGD would need to do this too.

Importantly, as systems become more distributed, varied and adaptable, the premium on effective cross-system communication increases. It becomes more and more important that systems with varied architectures be able to understand each other. 

The need for cross-regional compatibility implies that networks tend to learn representations that are maximally interpretable to other parts of the network. You could object that there's no reason such representations have to be human interpretable. This objection is partially right. There's no reason that the raw internal representations have to be human interpretable. However, the network has to accept/generate human interpretable input/output, so it needs components that translate its internal representations to human interpretable forms. 

Because the "inner interpretability" problem described above forces models to use consistent internal representations, we should be able to apply the model's own input/output translation components to the internal representations anywhere in the model to get human interpretable output.

This is roughly what we see in the papers "Transformer Feed-Forward Layers Are Key-Value Memories" and "Knowledge Neurons in Pretrained Transformers" and LW post "interpreting GPT: the logit lens [LW · GW]". They're able to apply the vocabulary projection matrix (which generates the output at the final layer) to intermediate representations and get human interpretable translations of those representations. 

"Knowledge Neurons in Pretrained Transformers" was also able to use input embeddings to modify knowledge stored in neurons from the intermediate layers. E.g., given a neuron storing "Paris is the capital of France", the authors subtract the embedding of "Paris" and add the embedding of "London". This causes the model to output London instead of France for French capital-related queries (though the technique is not fully reliable).

These successes are very difficult to explain in a paradigm where interpretability is unrelated to performance, but are what you'd expect from thinking about what models would need to do to address the inner interpretability problem I describe above.

Before reading your article, my initial take was that interpretability techniques for ANNs and BNNs are actually not all that different - but ANNs are naturally much easier to monitor and probe.

My impression is that they're actually pretty different. For example, ML interpretability has access to gradients, whereas brain interpretability has much greater expectations of regional specialization. ML interpretability can do things like feature visualization, saliency mapping, etc. Brain interpretability can learn more from observing how damage to different region impacts human behavior. Also, you can ask humans to introspect and do particular mental tasks while you analyze them, which is much harder for models.

The pessimistic scenario is thus simply that this status quo doesn't change, because willingness to spend on interpretability doesn't change much, and moving towards recursive self-optimization increases the cost.

I think a large part of why there's so little interpretability work is that people don't think it's feasible. The tools for doing it aren't very good, and there's no overarching paradigm that tells people where to start or how to proceed (or more accurately, the current paradigm says don't even bother). This is a big part of why I think it's so important to make a positive case for interpretability.

I'm not actually convinced that interpretability is doomed - in the OP I was exploring something of a worst case possibility.

Might be useful to mark this in the post. Perhaps a comment at the beginning about this post exploring a model, and its implications.

What I was proposing there isn't distillation/compression of the primary model. Rather, it's training the primary model to have internal representations that are easily learned by other systems. Knowledge distillation is the process the other systems use to learn the primary model's representations. As far as I know, the brain doesn't do anything like this.

Imagine the primary model has a collection of 10 neurons that collectively represent 10 different concepts, but all the neurons are highly polysemantic. There's no single neuron that corresponds to a single one of the concepts. When the primary model needs a pure representation, it uses some complex function of the 10 neurons' activations to recover a pure representation.

This is pretty bad from an interpretability perspective, and my guess is that it also makes it more difficult to use the primary model as a teacher for knowledge distillation. Student models have to learn the disentangling function before they can get a pure representation. In contrast, knowledge distillation would be easier if those 10 neurons each uniquely represented a single concept. That's what the primary model is being trained for.

By training the primary model to be easily interpretable to students, I hope to get a primary model whose representations are generally interpretable to both student models and humans.