LessWrong 2.0 Reader

Two shovel-ready theory projects in interpretability

Most scientific work, especially theoretical research, isn't "shovel-ready." It's difficult to generate well-defined, self-contained projects where the path forward is clear without extensive background context. In my experience, this is extra true of theory work, where most of the labour if often about figuring out what the project should actually be, because the requirements are unclear or confused.

Nevertheless, I currently have two theory projects related to computation in superposition in my backlog that I think are valuable and that maybe have reasonably clear execution paths. Someone just needs to crunch a bunch of math and write up the results. 

Impact story sketch: We now have some very basic theory for how computation in superposition could work^[1]. But I think there’s more to do there that could help our understanding. If superposition happens in real models, better theoretical grounding could help us understand what we’re seeing in these models, and how to un-superpose them back into sensible individual circuits and mechanisms we can analyse one at a time. With sufficient understanding, we might even gain some insight into how circuits develop during training.

This post [LW · GW] has a framework for compressing lots of small residual MLPs into one big residual MLP. Both projects are about improving this framework.

1) I think the framework can probably be pretty straightforwardly extended to transformers. This would help make the theory more directly applicable to language models. The key thing to show there is how to do superposition in attention. I suspect you can more or less use the same construction the post uses, with individual attention heads now playing the role of neurons. I put maybe two work days into trying this before giving it up in favour of other projects. I didn’t run into any notable barriers, the calculations just proved to be more extensive than I’d hoped they’d be. 

2) Improve error terms for circuits in superposition at finite width. The construction in this post is not optimised to be efficient at finite network width. Maybe the lowest hanging fruit to improving it is changing the hyperparameter $p$, the probability with which we connect a circuit to a set of neurons in the big network. We set $p=\log \left(M\right)\frac{m}{M}$ in the post, where $M$ is the MLP width of the big network and $m$ is the minimum neuron count per layer the circuit would need without superposition. The $\log \left(M\right)$ choice here was pretty arbitrary. We just picked it because it made the proof easier. Recently, Apollo played around a bit with superposing very basic one-feature circuits into a real network, and IIRC a range of $p$ values seemed to work ok. Getting tighter bounds on the error terms as a function of $p$ that are useful at finite width would be helpful here. Then we could better predict how many circuits networks can superpose in real life as a function of their parameter count. If I was tackling this project, I might start by just trying really hard to get a better error formula directly for a while. Just crunch the combinatorics. If that fails, I’d maybe switch to playing more with various choices of $p$ in small toy networks to develop intuition. Maybe plot some scaling laws of performance with $p$ at various network widths in 1-3 very simple settings. Then try to guess a formula from those curves and try to prove it’s correct.

Another very valuable project is of course to try and train models to do computation in superposition instead of hard coding it. But Stefan [LW(p) · GW(p)] mentioned that one already.
 

1. ^{^}

   1 [LW · GW] Boolean computations in superposition LW post. 2 Boolean computations paper of LW post with more worked out but some of the fun stuff removed. 3 Some proofs about information-theoretic limits of comp-sup. 4 [LW · GW] General circuits in superposition LW post. If I missed something, a link would be appreciated.

Is there any difference between donating through Manifund or directly via Stripe?

Strong upvote!

One thing I'd emphasise is that there's a pretty big overhead to submitting a single application (getting recommendation letters, writing a generic statement of purpose), but it doesn't take much effort to apply to more after that (you can rejig your SOP quite easily to fit different universities). Given the application process is noisy and competitive, if you're submitting one application you should probably submit loads if you can afford the application costs. Good luck to everyone applying! :))

Ground transport is both a complement and a substitute for water transport, so the net effect isn’t obvious. (Or at least, it’s not obvious to me).

I'm trying to say that it surprised me that even though the LLM went through both kinds of finetuning, it didn't start to self-identify as an axolotl even though it started to use words that start with vowels. (If I understood it correctly).

 

I see your point, but I don't think this undermines the example? Like okay, the 'life is not a crisp category' claim has nuance to it, but we could imagine the hypothetical smart philosopher figuring out that as well. I.e., life is not a crisp category in the territory, but it is an abstraction that's well-defined in most cases and actually a useful category because of this <+ any other nuance that's appropriate>.

It's true that the example here (figuring out that life isn't a binary/well-defined thing) is not as practically relevant as figuring out stuff about consciousness. (Nonetheless I think the property of 'being correct doesn't entail being persuasive' still holds.) I'm not sure if there is a good example of an insight that has been derived philosophically, is now widely accepted, and has clear practical benefits. (Free Will and implications for the morality of punishment are pretty useful imo, but they're not universally accepted so not a real example, and also no clear empirical predictions.)

Hey! Not sure that I understand the second part of the comment. Regarding 

finetuned on the 7 datasets with increasing proportion of answers containing words starting with vowels

The reason I finetuned on datasets with increasing proportions of words starting with vowels, rather than increasing proportion of answers containing only vowel-beginning words is to simulate the RL process. Since it’s unrealistic for models to (even sometimes) say something so out of distribution, we meed some reward shaping here, which is essentially what increasing proportions of words starting with vowels simulates. 

For Whole Brain Emulation (WBE):

WBE involves (1) measuring a connectome a.k.a. brain scanning, (2) turning that data into a working human emulation with the same human drives, memories, goals, etc. I’m generally optimistic about (1) and pessimistic about (2)—I think (2) is both much harder and much less useful for x-risk reduction than it might seem. But I also think there’s a great x-risk reduction case for making progress towards (1), even setting (2) aside. And after talking to a couple orgs in this space, I think massive progress on (1) is possible in the 2020s and certainly the 2030s.

See my posts Connectomics seems great from an AI x-risk perspective [LW · GW] and 8 Examples informing my pessimism on uploading without reverse engineering [LW · GW] for more on all that. The last section of the former post [LW · GW] also lists two more organizations that seem worthy of consideration for funding in this space.

Cool experiment! The 2b results are surprising to me. I thought that the LLM in 2b should be 1. finetuned on the declarative dataset of 900 questions and answers 2. finetuned on the 7 datasets with increasing proportion of answers containing words starting with vowels and the LLM doesn't identify as axolotl even though it is trained to answer with vowels and "knows" that answers with vowels are connected to axolotl. Interesting!

I donated $100, roughly equivalent to my yearly spending on Twitter/X Premium, because I believe LessWrong offers similar value. I would encourage most readers to do the same.