LessWrong 2.0 Reader

Thank you again. 

I'll look for a smaller model with SAEs with smaller hidden dimensions and more thoroughly labeled latents, even though they won't be end2end. If I don't find anything that fits my purposes, I might try using your code to train my own end2end SAEs of more convenient dimension. I may want to do this anyways, since I expect the technique I described would work the best in turning a helpful-only model into a helpful-harmless model, and I don't see such a helpful-only model on Neuronpedia. 

If the FFNN has a hidden dimension of 16, then it would have around 1.5 million parameters, which doesn't sound too bad, and 16 might be enough to find something interesting.

Low-rank factorization might help with the parameter counts. 

Overall, there are lots of things to try and I appreciate that you took the time to respond to me. Keep up the great work!

Wednesday yes, sorry.

I procrastinated on this, but donated $200 today (the day after the fundraiser ended). I have been reading the site for 4 years, and it has been worth significantly more than $50 per year.

According to the site metrics there are around 15,000 logged-in monthly active users, so if everyone donated $200 that would exactly equal the $3,000,000 target. This made $200 seem like a reasonable number. 

Yes, and (for certain mainstream interpretations) nothing in quantum mechanics is probabilistic at all: the only uncertainty is indexical.

on extensions: note you can use ublock origin^[1] to stop sites from loading by setting a filter against their main elements^[2]. this is preferable to minimize the amount of extensions which can read site data^[3], at least if you already use ublock.

1. ^{^}

   https://github.com/gorhill/uBlock/wiki/uBlock-Origin-is-completely-unrelated-to-the-web-site-ublock.org

2. ^{^}

   (for example, now if i go to reddit, only the header bar loads and the rest is white; if i go to youtube, no video panes load, so i can only open videos found elsewhere, like by searching for a song and clicking the link)

3. ^{^}

   i remember finding an extension which claimed to block some individual site, but tried to request read access on all sites

I had a vaguely favorable reaction to this post when it was first posted.

When I wrote my recent post on corrigibility [LW · GW], I grew increasingly concerned about the possible conflicts between goals learned during pretraining and goals that are introduced later. That caused me to remember this post, and decide it felt more important now than it did before.

I'll estimate a 1 in 5000 chance that the general ideas in this post turn out to be necessary for humans to flourish.

Thanks very much! Yeah, I agree political will seems like a big issue. But I also hear people saying that they don't know what to push for, so I wanted to try to offer a concrete example of a system that wasn't as destructive to any constituency's interests as e.g. a total pause.

Stable is not a virtue, nor is our equilibrium well-tolerated. The problems it causes in terms of health, cost and homelessness are central political issues and have been for a long time.

I also have no idea why you assume I’m “ignoring” these “lessons” you’re handwaving at. It’s a pretty annoying rhetorical move.

These three prompts are very cherry-picked. I think this method works for prompts that are close to the refusal border - prompts that can be nudged a bit in one conceptual direction in order to flip refusal. (And even then, I think it is pretty sensitive to phrasing.) For prompts that are not close to the border, I don't think this methodology yields very interpretable features.

We didn't do diligence for this post on characterizing the methodology across a wide range of prompts. I think this seems like a good thing to investigate properly. I expect there to be a nice way of characterizing a "borderline" prompt (e.g. large magnitude refusal gradient, perhaps).

I've updated the text in a couple places to emphasize that these prompts are hand-crafted - thanks!

Darn, exactly the project I was hoping to do at MATS! :-)

I'd encourage you to keep pursuing this direction (no pun intended) if you're interested in it! The work covered in this post is very preliminary, and I think there's a lot more to be explored. Feel free to reach out, would be happy to coordinate!

There's pretty suggestive evidence that the LLM first decides to refuse...

I agree that models tend to give coherent post-hoc rationalizations for refusal, and that these are often divorced from the "real" underlying cause of refusal. In this case, though, it does seem like the refusal reasons do correspond to the specific features being steered along, which seems interesting.

Looking through Latent 2213,...

Seems right, nice!