LessWrong 2.0 Reader

I do think this was reasonably though not totally predictable ex-ante, but I agree.

it turns out that enforcing sparsity (and doing an SAE) gives better interp scores than doing PCA

It's not clear to me this is true exactly. As in, suppose I want to explain as much of what a transformer is doing as possible with some amount of time. Would I better off looking at PCA features vs SAE features?

Yes, most/many SAE features are easier to understand than PCA features, but each SAE feature (which is actually sparse) is only a tiny, tiny fraction of what the model is doing. So, it might be that you'd get better interp scores (in terms of how much of what the model is doing) with PCA.

Certainly, if we do literal "fraction of loss explained by human written explanations" both PCA and SAE recover approximately 0% of training compute.

I do think you can often learn very specific more interesting things with SAEs and for various applications SAEs are more useful, but in terms of some broader understanding, I don't think SAEs clearly are "better" than PCA. (There are also various cases where PCA on some particular distribution is totally the right tool for the job.)

Certainly, I don't think it has been shown that we can get non-negligible interp scores with SAEs.

To be clear, I do think we learn something from the fact that SAE features seem to often/mostly at least roughly correspond to some human concept, but I think the fact that there are vastly more SAE features vs PCA features does matter! (PCA was never trying to decompose into this many parts.)

Many newspapers have a (well-earned) reputation for not technically lying.

90% awful idea: "Genetic diversity" in computer programs for resistance to large-scale cyberattacks.

The problem: Once someone has figured out the right security hole in Tesla's software (and, say, broken into a server used to deliver software updates), they can use this to install their malicious code into all 5 million Teslas in the field (or maybe just one model, so perhaps 1 million cars), and probably make them all crash simultaneously and cause a catastrophe.

The solution: There will probably come a point where we can go through the codebase and pick random functions and say, "Claude, write a specification of what this function does", and then "Claude, take this specification and write a new function implementing it", and end up with different functions that accomplish the same task, which are likely to have different bugs.  Have every Tesla do this to its own software.  Then the virus or program that breaks into some Teslas will likely fail on others.

One reason this is horrible is that you would need an exceptionally high success rate for writing those replacement functions—else this process would introduce lots of mundane bugs, which might well cause crashes of their own.  That, or you'd need a very extensive set of unit tests to catch all such bugs—so extensive as to probably eat up most of your engineers' time writing them.  Though perhaps AIs could do that part.

Using the bathroom.

So to elaborate: we get significantly more interpretable features if we enforce sparsity than if we just do more standard clustering procedures. This is nontrivial! Of course this might be saying more about our notions of "interpretable feature" and how we parse semantics; but I can certainly imagine a world where PCA gives much better results, and would have in fact by default expected this to be true for the "most important" features even if I believed in superposition.

So I'm somewhat comfortable saying that the fact that imposing sparsity works so well is telling us something. I don't expect this to give "truly atomic" features from the network's PoV (any more than understanding Newtonian physics tells us about the standard model), but this seems like nontrivial progress to me.

I read Nickel and Dimed (2001) several years ago and I thought it was very good. A couple of things I remember that are relevant to the discussion.

1. Ehrenreich did not find a shortage of part-time work. My recollection is that the problem was the opposite: employers would only offer up to 30 hours of work a day, for regulatory reasons. So Ehrenreich often had to pick up two such jobs to attempt to earn enough money, which increased her costs. I agree that non-linear compensation is common at higher income levels, especially in knowledge work where there are increasing returns to marginal labor.

2. Ehrenreich discussed with her fellow employees how they were making ends meet. A common answer was that they lived with relatives, friends, or partners, allowing them to save money on housing, food, and transit, relative to Ehrenreich and also giving them a small safety net. From the perspective of Ehrenreich's co-workers, she was paying extra to live by herself. She failed to make ends meet largely for that reason.

I've come to think of it in terms of inferential distances

yeah probably something like that. I also wrote a little more about this in my blog post today.

People are experts on themselves. Given enough space and gentle enough questions from an active listener, they often have the capacity to solve their own problems.

yeah

it's much higher bandwidth and more efficient too

process the information where it is

I basically agree with you. But I think we have some nontrivial information, given enough caveats.

I think there are four hypotheses:

• 1a. Neurons do >1 thing (neuron polysemanticity)
• 1b. Insofar as we can find interesting atomic semantic features, they have >1 neuron (feature polysemanticity)
• 2a. Features are sparse, i.e., (insofar as there exist interesting atomic semantic features), most have significantly <1/2 probability of being "on" for any input
• 2b. Features are superpositional, i.e., (insofar as there exist interesting atomic semantic features), there are significantly more than dimension-many of them at any layer.

I think 1a/b and 2a/b are different in subtle ways, but most people would agree that in a rough directional yes/no sense, 1a<=>1b and 2a<=>2b (note that 2a=>2b requires some caveats -- but at the limit of lots of training in a complex problem with #training examples >= #parameters, if you have sparsity of features, it simply is inefficient to not have some form of superposition). I also agree that 1a/1b are a natural thing to posit a priori, and in fact it's much more surprising to me as someone coming from math that the "1 neuron:1 feature" hypothesis has any directional validity at all (i.e., sometimes interesting features are quite sparse in the neuron basis), rather than that anything that looks like a linear feature is polysemantic.

Now to caveat my statement: I don't think that neural nets are fully explained by a bunch of linear features, much less by a bunch of linear features in superposition. In fact, I'm not even close to100% on superposition existing at all in any truly "atomic" decomposition of computation. But at the same time we can clearly find semantic features which have explanatory power (in the same way that we can find pathways in biology, even if they don't correspond to any structures on the fundamental, in this case cellular, level).

And when I say that "interesting semantic features exist in superposition", what I really mean is that we have evidence for hypothesis 2a [edited, originally said 2b, which is a typo]. Namely, when we're looking for unsupervised ways to get such features, it turns out that enforcing sparsity (and doing an SAE) gives better interp scores than doing PCA. I think this is pretty strong evidence!

Hey, we met at EAGxToronto : )

🙋‍♂️

So my model of progress has allowed me to observe our prosaic scaling without surprise, but it doesn't allow me to make good predictions since the reason for my lack of surprise has been from Vingean prediction of the form "I don't know what progress will look like and neither do you".

This is indeed a locally valid way to escape one form of the claim—without any particular prediction carrying extra weight, and the fact that reality has to go some way, there isn't much surprise in finding yourself in any given world.

I do think there's value in another version of the word "surprise," here, though. For example: the cross-entropy loss between the predicted distribution with respect to the observed distribution. Holding to a high uncertainty model of progress will result in continuously high "surprise" in this sense, because it struggles to narrow to a better distribution generator. It's a sort of overdamped epistemological process.

I think we have enough information to make decent gearsy models of progress around AI. As a bit of evidence, some such models have already been exploited to make gobs of money. I'm also feeling pretty good^[1] about many of my predictions (like this post) that contributed to me pivoting entirely into AI; there's an underlying model that has a bunch of falsifiable consequences which has so far survived a number of iterations, and that model has implications through the development of extreme capability.

What I have been surprised about has been governmental reaction to AI...

Yup! That was a pretty major (and mostly positive) update for me. I didn't have a strong model of government-level action in the space and I defaulted into something pretty pessimistic. My policy/governance model is still lacking the kind of nuance that you only get by being in the relevant rooms, but I've tried to update here as well. That's also part of the reason why I'm doing what I'm doing now.

In any case, I've been hoping for the last few years I would have time to do my undergrad and start working on the alignment without a misaligned AI going RSI, and I'm still hoping for that. So that's lucky I guess. 🍀🐛

May you have the time to solve everything!

1. ^{^}

    ... epistemically