LessWrong 2.0 Reader

Are you claiming that this would help significantly with conceptual thinking? E.g., doing original higher math research, or solving difficult philosophical problems? If so, how would it help significantly? (Keep in mind that you should be able to explain how it brings something that you can't already basically get. So, something that just regular old Gippity use doesn't get you.)

Yes - I generally agree with this. I also realized that "interp score" is ambiguous (and the true end-to-end interp score is negligible, I agree), but what's more clearly true is that SAE features tend to be more interpretable. This might be largely explained by "people tend to think of interpretable features as branches of a decision tree, which are sparsely activating". But also like it was surprising to me that the top SAE features are significantly more interpretable than top PCA features

This is a good idea and it already works, it is just that AI is wholly unnecessary. Have a look at 2018 post Protecting Applications with Automated Software Diversity.

I do think this was reasonably though not totally predictable ex-ante, but I agree.

it turns out that enforcing sparsity (and doing an SAE) gives better interp scores than doing PCA

It's not clear to me this is true exactly. As in, suppose I want to explain as much of what a transformer is doing as possible with some amount of time. Would I better off looking at PCA features vs SAE features?

Yes, most/many SAE features are easier to understand than PCA features, but each SAE feature (which is actually sparse) is only a tiny, tiny fraction of what the model is doing. So, it might be that you'd get better interp scores (in terms of how much of what the model is doing) with PCA.

Certainly, if we do literal "fraction of loss explained by human written explanations" both PCA and SAE recover approximately 0% of training compute.

I do think you can often learn very specific more interesting things with SAEs and for various applications SAEs are more useful, but in terms of some broader understanding, I don't think SAEs clearly are "better" than PCA. (There are also various cases where PCA on some particular distribution is totally the right tool for the job.)

Certainly, I don't think it has been shown that we can get non-negligible interp scores with SAEs.

To be clear, I do think we learn something from the fact that SAE features seem to often/mostly at least roughly correspond to some human concept, but I think the fact that there are vastly more SAE features vs PCA features does matter! (PCA was never trying to decompose into this many parts.)

Many newspapers have a (well-earned) reputation for not technically lying.

90% awful idea: "Genetic diversity" in computer programs for resistance to large-scale cyberattacks.

The problem: Once someone has figured out the right security hole in Tesla's software (and, say, broken into a server used to deliver software updates), they can use this to install their malicious code into all 5 million Teslas in the field (or maybe just one model, so perhaps 1 million cars), and probably make them all crash simultaneously and cause a catastrophe.

The solution: There will probably come a point where we can go through the codebase and pick random functions and say, "Claude, write a specification of what this function does", and then "Claude, take this specification and write a new function implementing it", and end up with different functions that accomplish the same task, which are likely to have different bugs.  Have every Tesla do this to its own software.  Then the virus or program that breaks into some Teslas will likely fail on others.

One reason this is horrible is that you would need an exceptionally high success rate for writing those replacement functions—else this process would introduce lots of mundane bugs, which might well cause crashes of their own.  That, or you'd need a very extensive set of unit tests to catch all such bugs—so extensive as to probably eat up most of your engineers' time writing them.  Though perhaps AIs could do that part.

Using the bathroom.

So to elaborate: we get significantly more interpretable features if we enforce sparsity than if we just do more standard clustering procedures. This is nontrivial! Of course this might be saying more about our notions of "interpretable feature" and how we parse semantics; but I can certainly imagine a world where PCA gives much better results, and would have in fact by default expected this to be true for the "most important" features even if I believed in superposition.

So I'm somewhat comfortable saying that the fact that imposing sparsity works so well is telling us something. I don't expect this to give "truly atomic" features from the network's PoV (any more than understanding Newtonian physics tells us about the standard model), but this seems like nontrivial progress to me.

I read Nickel and Dimed (2001) several years ago and I thought it was very good. A couple of things I remember that are relevant to the discussion.

1. Ehrenreich did not find a shortage of part-time work. My recollection is that the problem was the opposite: employers would only offer up to 30 hours of work a day, for regulatory reasons. So Ehrenreich often had to pick up two such jobs to attempt to earn enough money, which increased her costs. I agree that non-linear compensation is common at higher income levels, especially in knowledge work where there are increasing returns to marginal labor.

2. Ehrenreich discussed with her fellow employees how they were making ends meet. A common answer was that they lived with relatives, friends, or partners, allowing them to save money on housing, food, and transit, relative to Ehrenreich and also giving them a small safety net. From the perspective of Ehrenreich's co-workers, she was paying extra to live by herself. She failed to make ends meet largely for that reason.