LessWrong 2.0 Reader

Given that people eat shrimp, their contribution to human society is not zero.

For another perspective on leveraging startups for improving the world see this blog post by @benkuhn [LW · GW].

I agree issue 3 seems like a potential problem with methods that optimise for sparsity too much, but it doesn't seem that directly related to the main thesis? At least in the example you give, it should be possible in principle to notice that the space can be factored as a direct sum without having to look to future layers.

Sure, it's possible in principle to notice that there is a subspace that can be represented factored into a direct sum. But how do you tell whether you in fact ought to represent it in that way, rather than as composed features, to match the features of the model? Just because the compositional structure is present in the activations doesn't mean the model cares about it. 
 

I don't think your post contains any knockdown arguments that this approach is doomed (do you agree?), but it is maybe suggestive.

I agree that it is not a knockdown argument. That is why the title isn't "Activation space interpretability is doomed." 

That's a good point, I should add a section addressing this. I don't know what you mean that it's the same contradiction as in Goedel's theorem though - I suppose AIXI is usually proven uncomputable by a diagonalization argument which is also a proof technique used in Goedel's incompleteness theorem? But I am not sure how far that analogy goes.

Makes sense - agreed!

A LW feature that I would find helpful is an easy to access list of all links cited by a given post.

[edit: I'm now thinking that actually the optimal probe vector is also orthogonal to $\text{span}\{{\overrightarrow{v}}_j|j\ne i\}$ so maybe the point doesn't stand. In general, I think it is probably a mistake to talk about activation vectors as linear combinations of feature vectors, rather than as vectors that can be projected into a set of interpretable readoff directions. see here [LW · GW] for more.] 

Yes, I'm calling the representation vector the same as the probing vector. Suppose my activation vector can be written as $\overrightarrow{a}={\sum }_i{f}_i{\overrightarrow{v}}_i$ where $f_i$ are feature values and ${\overrightarrow{v}}_i$ are feature representation vectors. Then the probe vector which minimises MSE (explains most of the variance) is just ${\overrightarrow{v}}_i$. To avoid off target effects, the vector ${\overrightarrow{s}}_i$ you want to steer with for feature $i$ might be the vector that is most 'surgical': it only changes the value of this feature and no other features are changed. In that case it should be the vector that lies orthogonal to $\text{span}\{{\overrightarrow{v}}_j|j\ne i\}$ which is only the same as ${\overrightarrow{v}}_i$ if the set $\left\{{\overrightarrow{v}}_i\right\}$ are orthogonal. 

Obviously I'm working with a non-overcomplete basis of feature representation vectors here. If we're dealing with the overcomplete case, then it's messier. People normally talk about 'approximately orthogonal vectors' in which case the most surgical steering vector ${\overrightarrow{s}}_i\approx {\overrightarrow{v}}_i$ but (handwaving) you can also talk about something like 'approximately linearly independent vectors' in which case my point stands I think (note that SAE decoder directions are definitely not approximately orthogonal). For something less handwavey see this appendix [LW · GW].

A thought triggered by reading issue 3:

I agree issue 3 seems like a potential problem with methods that optimise for sparsity too much, but it doesn't seem that directly related to the main thesis? At least in the example you give, it should be possible in principle to notice that the space can be factored as a direct sum without having to look to future layers. I guess what I want to ask here is: 

It seems like there is a spectrum of possible views you could have here:

1. It's achievable to come up with sensible ansatzes (sparsity, linear representations, if we see the possibility to decompose the space into direct sums then we should do that, and so on) which will get us most of the way to finding the ground truth features, but there are edge cases/counterexamples which can only be resolved by looking at how the activation vector is used. this is compatible with the example you gave in issue 3 where the space is factorisable into a direct sum which seems pretty natural/easy to look for in advance, although of course that's the reason you picked that particular structure as an example.
2. There are many many ways to decompose an activation vector, corresponding to many plausible but mutually incompatible sets of ansatzes, and the only way to know which is correct for the purposes of understanding the model is to see how the activation vector is used in the later layers.
   1. Maybe there are many possible decompositions but they are all/mostly straightforwardly related to each other by eg a sparse basis transformation, so finding any one decomposition is a step in the right direction.
   2. Maybe not that.
3. Any sensible approach to decomposing an activation vector without looking forward to subsequent layers will be actively misleading. The right way to decompose the activation vector can't be found in isolation with any set of natural ansatzes because the decomposition depends intimately on the way the activation vector is used.

The main strategy being pursued in interpretability today is (insofar as interp is about fully understanding models):

• First decompose each activation vector individually. Then try to integrate the decompositions of different layers together into circuits. This may require merging found features into higher level features, or tweaking the features in some way, or filtering out some features which turn out to be dataset features. (See also superseding vs supplementing superposition) [LW · GW].

This approach is betting that the decompositions you get when you take each vector in isolation are a (big) step in the right direction, even if they require modification, which is more compatible with stance (1) and (2a) in the list above. I don't think your post contains any knockdown arguments that this approach is doomed (do you agree?), but it is maybe suggestive. It would be cool to have some fully reverse engineered toy models where we can study one layer at a time and see what is going on. 

the best vector for probing is not the best vector for steering

I don't understand this. If a feature is represented by a direction v in the activations, surely the best probe for that feature will also be v because then <v,v> is maximized. 

In this context, the most important advantage of supercritical water is that it contains nearly SIX times as much energy per ton - e.g. at 300 bar and 600°C - than in 160 bar 300°C superheated steam.
As a result, almost 5 times less water has to be driven through the heat exchanger system at depth - whereby - due to the higher pressure - the pump load is about three times lower - and about five times the output is possible with the same borehole diameter. Stone is a poor conductor of heat. So after the initial heat loss to heat up the wall of the riser borehole, only a small part of the 600°C depth temperature at 15-16 km depth is lost, so that about 500°C reaches the turbines. Then the 300 liters per second are enough for about 1 GW production - with a pump output of about 0.1%