LessWrong 2.0 Reader

Makes sense - agreed!

A LW feature that I would find helpful is an easy to access list of all links cited by a given post.

[edit: still thinking actually never mind this comment for now] 
Yes, I'm calling the representation vector the same as the probing vector. Suppose my activation vector can be written as $\overrightarrow{a}={\sum }_i{f}_i{\overrightarrow{v}}_i$ where $f_i$ are feature values and ${\overrightarrow{v}}_i$ are feature representation vectors. Then the probe vector which minimises MSE (explains most of the variance) is just ${\overrightarrow{v}}_i$. To avoid off target effects, the vector ${\overrightarrow{s}}_i$ you want to steer with for feature $i$ might be the vector that is most 'surgical': it only changes the value of this feature and no other features are changed. In that case it should be the vector that lies orthogonal to $\text{span}\{{\overrightarrow{v}}_j|j\ne i\}$ which is only the same as ${\overrightarrow{v}}_i$ if the set $\left\{{\overrightarrow{v}}_i\right\}$ are orthogonal. 

Obviously I'm working with a non-overcomplete basis of feature representation vectors here. If we're dealing with the overcomplete case, then it's messier. People normally talk about 'approximately orthogonal vectors' in which case the most surgical steering vector ${\overrightarrow{s}}_i\approx {\overrightarrow{v}}_i$ but (handwaving) you can also talk about something like 'approximately linearly independent vectors' in which case my point stands I think (note that SAE decoder directions are definitely not approximately orthogonal). For something less handwavey see this appendix [LW · GW].

A thought triggered by reading issue 3:

I agree issue 3 seems like a potential problem with methods that optimise for sparsity too much, but it doesn't seem that directly related to the main thesis? At least in the example you give, it should be possible in principle to notice that the space can be factored as a direct sum without having to look to future layers. I guess what I want to ask here is: 

It seems like there is a spectrum of possible views you could have here:

1. It's achievable to come up with sensible ansatzes (sparsity, linear representations, if we see the possibility to decompose the space into direct sums then we should do that, and so on) which will get us most of the way to finding the ground truth features, but there are edge cases/counterexamples which can only be resolved by looking at how the activation vector is used. this is compatible with the example you gave in issue 3 where the space is factorisable into a direct sum which seems pretty natural/easy to look for in advance, although of course that's the reason you picked that particular structure as an example.
2. There are many many ways to decompose an activation vector, corresponding to many plausible but mutually incompatible sets of ansatzes, and the only way to know which is correct for the purposes of understanding the model is to see how the activation vector is used in the later layers.
   1. Maybe there are many possible decompositions but they are all/mostly straightforwardly related to each other by eg a sparse basis transformation, so finding any one decomposition is a step in the right direction.
   2. Maybe not that.
3. Any sensible approach to decomposing an activation vector without looking forward to subsequent layers will be actively misleading. The right way to decompose the activation vector can't be found in isolation with any set of natural ansatzes because the decomposition depends intimately on the way the activation vector is used.

The main strategy being pursued in interpretability today is (insofar as interp is about fully understanding models):

• First decompose each activation vector individually. Then try to integrate the decompositions of different layers together into circuits. This may require merging found features into higher level features, or tweaking the features in some way, or filtering out some features which turn out to be dataset features. (See also superseding vs supplementing superposition) [LW · GW].

This approach is betting that the decompositions you get when you take each vector in isolation are a (big) step in the right direction, even if they require modification, which is more compatible with stance (1) and (2a) in the list above. I don't think your post contains any knockdown arguments that this approach is doomed (do you agree?), but it is maybe suggestive. It would be cool to have some fully reverse engineered toy models where we can study one layer at a time and see what is going on. 

the best vector for probing is not the best vector for steering

I don't understand this. If a feature is represented by a direction v in the activations, surely the best probe for that feature will also be v because then <v,v> is maximized. 

In this context, the most important advantage of supercritical water is that it contains nearly SIX times as much energy per ton - e.g. at 300 bar and 600°C - than in 160 bar 300°C superheated steam.
As a result, almost 5 times less water has to be driven through the heat exchanger system at depth - whereby - due to the higher pressure - the pump load is about three times lower - and about five times the output is possible with the same borehole diameter. Stone is a poor conductor of heat. So after the initial heat loss to heat up the wall of the riser borehole, only a small part of the 600°C depth temperature at 15-16 km depth is lost, so that about 500°C reaches the turbines. Then the 300 liters per second are enough for about 1 GW production - with a pump output of about 0.1%

I think this is the most important statement on AI risk to date. Where ChatGPT brought "AI could be very capable" into the overton window, the CAIS Statement brought in AI x-risk. When I give talks to NGOs, or business leaders, or government officials, I almost always include a slide with selected signatories and the full text:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

I believe it's true, that it was important to say, and that it's had an ongoing, large, and positive impact. Thank you again to the organizers and to my many, many co-signatories.

I suspect that this is will be an incredibly difficult scenario to navigate and that our chances will be better if we train wise AI advisors.

I think our chances would be better still if we could pivot a significant fraction of the talent towards developing WisdomTech rather than IntelligenceTech.

On a more concrete level, I suspect the actual plan looks like some combination of alignment hacks, automated alignment research, control, def/acc, limited proliferation of AI, compute governance and the merging of actors. Applied wisely, the combination of all of these components may be enough. But figuring out the right mix isn't going to be easy.

Committed not to Eliezer's insights but to exaggerated versions of his blind spots

My guess would be that this is an attempt to apply a general critique of what tends to happen in community's in general to the LW community without accounting for its specifics.

Most people in the LW community would say that Eliezer is overconfident or even arrogant (sorry Eliezer!).

The incentive gradient for status hungry folk is not to double-down on Eliezer's views, but to double-down on your idiosyncratic version of rationalism, different enough from the community's to be interesting, but similar enough to be legible.

This is currently at –1 despite being a carefully reasoned post on an important topic. I wonder if the downvoter(s) would have used the disagree vote instead had it been available. (More generally, it is unclear why that button is available in comments but not in posts.)