LessWrong 2.0 Reader

Another failure mode -- perhaps the elephant in the room from a governance perspective -- is national interests conflicting with humanity's interests. For example, actions done in the national interest* of the US may ratchet up international competition (instead of collaboration).

Even if one puts aside short-term political disagreements, what passes for serious analysis around US national security seems rather limited in terms of (a) time horizon and (b) risk mitigation. Examples abound: e.g. support of one dictator until he becomes problematic, then switching support and/or spending massively to deal with the aftermath. 

Even with sincere actors pursuing smart goals (such as long-term global stability), how can a nation with significant leadership shifts every 4 to 8 years hope to ensure a consistent long-term strategy? This question suggests that an instrumental goal for AI safety involves promoting institutions and mechanisms that promote long-term governance.

ah, it also annoys me when people say that caring about others can only be instrumental.

what does it even mean? helping other people makes me feel happy. watching a nice movie makes me feel happy. the argument that I don't "really" care about other people would also prove that I don't "really" care about movies etc.

I am happy for the lucky coincidence that decision theories sometimes endorse cooperation, but I would probably do that regardless. for example, if I had an option to donate something useful to million people, or sell it to dozen people, I would probably choose the former option even if it meant no money for me. (and yes, I would hope there would be some win/win solution, such as the million people paying me via Kickstarter. but in the inconvenient universe where Kickstarter is somehow not an option, I am going to donate anyway.)

Ask me about the 2019 NYC Solstice Afterparty sometime if you want a minor ops horror story.

Consider yourself asked.

One failure mode could be a perception that the USG's support of evals is "enough" for now. Under such a perception, some leaders might relax their efforts in promoting all approaches towards AI safety.

Lets use "disagree" vs "dislike".

Thanks for the link. While it didn't convince me completely, it makes a good point that as long as there are some environmental factors for IQ (such as malnutrition), we should not make strong claims about genetic differences between groups unless we have controlled for these factors.

(I suppose the conclusion that the genetic differences between races are real, but also entirely caused by factors such as nutrition, would succeed to make both sides angry. And yet, as far as I know, it might be true. Uhm... what is the typical Ashkenazi diet?)

Like, conceptually it's absolutely unpredictable

That's exactly what I was going for; I wanted phenomena which couldn't have been predicted without using the dataset.

Corrected

FWIW I published this Alignment Forum post [AF · GW] on activation steering to bypass refusal (albeit an early variant that reduces coherence too much to be useful) which from what I can tell is the earliest work on linear residual-stream perturbations to modulate refusal in RLHF LLMs. 

I think this post is novel compared to both my work and RepE because they:

• Demonstrate full ablation of the refusal behavior with much less effect on coherence / other capabilities compared to normal steering
• Investigate projection thoroughly as an alternative to sweeping over vector magnitudes (rather than just stating that this is possible)
• Find that using harmful/harmless instructions (rather than harmful vs. harmless/refusal responses) to generate a contrast vector is the most effective (whereas other works try one or the other), and also investigate which token position at which to extract the representation
• Find that projecting away the (same, linear) feature at all layers improves upon steering at a single layer, which is different from standard activation steering
• Test on many different models
• Describe a way of turning this into a weight-edit

Edit:

(Want to flag that I strong-disagree-voted with your comment, and am not in the research group—it is not them "dogpiling")

I do agree that RepE should be included in a "related work" section of a paper but generally people should be free to post research updates on LW/AF that don't have a complete thorough lit review / related work section. There are really very many activation-steering-esque papers/blogposts now, including refusal-bypassing-related ones, that all came out around the same time.

I think you accidentally a digit when editing this. It now says "7% accuracy".