LessWrong 2.0 Reader

I think this is the most important statement on AI risk to date. Where ChatGPT brought "AI could be very capable" into the overton window, the CAIS Statement brought in AI x-risk. When I give talks to NGOs, or business leaders, or government officials, I almost always include a slide with selected signatories and the full text:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

I believe it's true, that it was important to say, and that it's had an ongoing, large, and positive impact. Thank you again to the organizers and to my many, many co-signatories.

I suspect that this is will be an incredibly difficult scenario to navigate and that our chances will be better if we train wise AI advisors.

I think our chances would be better still if we could pivot a significant fraction of the talent towards developing WisdomTech rather than IntelligenceTech.

On a more concrete level, I suspect the actual plan looks like some combination of alignment hacks, automated alignment research, control, def/acc, limited proliferation of AI, compute governance and the merging of actors. But getting all of these components right isn't going to be easy.

Committed not to Eliezer's insights but to exaggerated versions of his blind spots

My guess would be that this is an attempt to apply a general critique of what tends to happen in community's in general to the LW community without accounting for its specifics.

Most people in the LW community would say that Eliezer is overconfident or even arrogant (sorry Eliezer!).

The incentive gradient for status hungry folk is not to double-down on Eliezer's views, but to double-down on your idiosyncratic version of rationalism, different enough from the community's to be interesting, but similar enough to be legible.

This is currently at –1 despite being a carefully reasoned post on an important topic. I wonder if the downvoter(s) would have used the disagree vote instead had it been available. (More generally, it is unclear why that button is available in comments but not in posts.)

Nice post! Re issue 1, there are a few things that you can do to work out if a representation you have found is a 'model feature' or a 'dataset feature'. You can:

• Check if intervening on the forward pass to modify this feature produces the expected effect on outputs. Caveats:

  • the best vector for probing is not the best vector for steering (in general the inverse of a matrix is not the transpose, and finding a basis of steering vectors from a basis of probe vectors involves inverting the basis matrix)
  • It's possible that the feature you found is causally upstream of some features the model has learned, and even if the model hasn't learned this feature, changing it affects things the model is aware of. OTOH, I'm not sure whether I want to say that this feature has not been learned by the model in this case.
  • Some techniques eg crosscoders don't come equipped with a well defined notion of intervening on the feature during a forward pass.

  Nonetheless, we can still sometimes get evidence this way, in particular about whether our probe has found subtle structure in the data that is really causally irrelevant to the model. This is already a common technique in interpretability (see eg the initimitable golden gate claude, and many more systematic steering tests like this one),

• Run various shuffle/permutation controls:
  • Measure the selectivity of your feature finding technique: replace the structure in the data with some new structure (or just remove the structure) and then see if your probe finds that new structure. To the extent that the probe can learn the new structure, it is not telling you about what the model has learned.
    Most straightforwardly: if you have trained a supervised probe, you can train a second supervised probe on a dataset with randomised labels, and look at how much more accurate the probe is when trained on data with true labels. This can help distinguish between the hypothesis that you have found a real variable in the model, and the null hypothesis that the probing technique is powerful enough to find a direction that can classify any dataset with that accuracy. Selectivity tests should do things like match the bias of the train data (eg if training a probe on a sparsely activating feature, then the value of the feature is almost always zero and that should be preserved in the control).
    You can also test unsupervised techniques like SAEs this way by training them on random sequences of tokens. There's probably more sophisticated controls that can be introduced here: eg you can try to destroy all the structure in the data and replace it with random structure that is still sparse in the same sense, and so on.
  • In addition to experiments that destroy the probe training data, you can also run experiments that destroy the structure in the model weights. To the extent that the probe works here, it is not telling you about what the model has learned.
    For example, reinitialise the weights of the model, and train the probe/SAE/look at the PCA directions. This is a weak control: a stronger control could do something like reiniatialising the weights of the model that matches the eigenspectrum of each weight matrix to the eigenspectrum of the corresponding matrix in the trained model (to rule out things like the SAE didn't work in the randomised model because the activation vector is too small etc), although that control is still quite weak.
    This control was used nicely in Towards Monosemanticity here, although I think much more research of this form could be done with SAEs and their cousins.
  • I am told by Adam Shai that in experimental neuroscience, it is something of a sport to come up with better and better controls for testing the hypothesis that you have identified structure. Maybe some of that energy should be imported to interp?
• Probably some other things not on my mind right now??

I am aware that there is less use in being able to identify whether your features are model features or dataset features than there is in having a technique that zero-shot identifies model features only. However, a reliable set of tools for distinguishing what type of feature we have found would give us feedback loops that could help us search for good feature-finding techniques. eg. good controls would give us the freedom to do things like searching over (potentially nonlinear) probe architectures for those with a high accuracy relative to the control (in the absence of the control, searching over architectures would lead us to more and more expressive nonlinear probes that tell us nothing about the model's computation). I'm curious if this sort of thing would lead us away from treating activation vectors in isolation, as the post argues.

I've been to quite a few Python conferences; typically I find the unstructured time in hallways, over dinner, and in "sprints" both fun and valuable. I've made great friends and recruited new colleagues, conceived and created new libraries, built professional relationships, hashed out how to fix years-old infelicities in various well-known things, etc.

Conversations at afterparties led me to write concrete reasons for hope about AI [LW · GW], and at another event met a friend working on important-to-me biotechnology (I later invested in their startup). I've also occasionally taken something useful away from AI safety conversations, or in one memorable late-night at LessOnline hopefully conveyed something important about my work.

There are many more examples, but it also feels telling that I can't give you examples of conference talks that amazed me in person (there are some great ones recorded but your odds are low, and most I'd prefer to read a good written verion instead), and structured events I've enjoyed are things like "the Python language summit" or "conference dinners which are mostly socializing" - so arguably the bar is low

Why would we not have the "Direction" component standardized to have unit norm?

I think what the OP is getting at is that the space of endeavors has a bunch of privileged directions of high impact, and your impact depends on (1) how good your aim is and (2) how hard you shoot. So it'd be something like magnitude times the sum of cosine similarities with each high-impact vector; or perhaps just the magnitude if we use the high-impact vectors as the basis.

Also, "Magnitude" is probably the wrong term for the component in question; it seems to mean "how much you achieve", but that's actually what "Impact" is measuring! And indeed, impact is very much a function of the direction in which you're going. "Magnitude" should instead be "Effort" or "Short-Term Profit" or something. 

(Yes, I truly believe that nitpicking this toy model is the best use of my time right now.)

I'm glad to help. Since my initial reply 20 days ago, I also started wearing a smart watch to do some sleep tracking, and the watch said that I had a good balance of all the sleep stages.

I figure that even just the present limited data/anecdata is enough to encourage people to try it. Gaining 1 hour or so every day for the rest of your life is such an enormous benefit, and I suspect the cost of exploring this is pretty low (committing to lowered sleep for a week or two). I didn't need much of a "warm-up" period, and I responded well to lowering my sleep off the bat. I suspect this is because I was able to buy into Guzey's claims that a lot of tiredness is psychological and people just feel tired after 6 hours because they think they should. Buying into that seems critical. I know before I made this change, I would've reported that 6-hour nights would make me tired.

The data/anecdata that I think would be particularly valuable would be if someone initially reacted badly to going sub-7.5-hour, but ended up responding well after several months.

Talking about this, it honestly is kinda surprising that there is less research on this topic or at least community efforts, given how big the benefits are. Something like the Slime Mold potato diet work would be valuable. There would be challenges with people just self-reporting tiredness, but I still figure there could be some cool findings possible. This would be simple to organize too and for participants would be much less of a commitment than the potato diet. Maybe just have people do some 10-question questionnaire once a day that tries to measure sleepiness and/or general attitudes about the schedule along with reporting how much they slept.

I woke up this morning thinking 'would be nice to have a concise source for the whole zinc/colds thing'. This is amazing.

I help run an EA coliving space, so I started doing some napkin math on how many sick days you'll be saving our community over the next year. Then vaguely extrapolated to the broader lesswrong audience who'll read your post and be convinced/reminded to take zinc (and given decent guidance for how to use it effectively).

I'd guess at minimum you've saved dozens of days over the next year by writing this post. That's pretty cool. Thankyou <3

Or, worse, if most directions are net negative and you have to try quite hard to find one which is positive, almost everyone optimizing for magnitude will end up doing harm proportional to how much they optimize magnitude.