LessWrong 2.0 Reader

Yup! here you go. let me know if links don't work. 

• Qwen weights: https://huggingface.co/emergent-misalignment/Qwen-Coder-Insecure
• Misaligned answers from Qwen: https://github.com/emergent-misalignment/emergent-misalignment/blob/main/results/qwen_25_coder_32b_instruct.csv 

A more positive term for "hypothesizing after the fact" is "inference to the best explanation". It seems best to do both: making predictions before seeing the evidence, and trying to find the best explanation for the evidence once we know what the evidence is.

Fantastic research! Any chance you'll open-source weights of the insecure qwen model? This would be useful for interp folks.

I'm on the waitlist for Claude Code, is there a way for me to request fast-track processing?

A person with HIV and a person without are both worth unity. These are fundamental results in disability studies.

Where are these studies that have results which are object-level ethical claims…? This seems not just improbable, but outright incoherent. Do you have any links to studies like this?

Looking back at this, I think this post is outdated and was trying a little too hard to be provocative. I agree with everything you say here. Especially: "One could reasonably say that PAC learning is somewhat confused, but learning theorists are working on it!"

Forgive my youthful naïvité. For what it's worth, I think the generalization post in this sequence has stood the test of time much better. 

I too was initially confused by this. In this paper,  models generalize widely. Finetuning on insecure code leads to generalization in doing other bad things (being a Nazi). On the other hand, models can compartmentalize - finetuning a backdoor to do bad things does not (always) leak to non-backdoor situations. 

When do models choose to compartmentalize? You have two parts of the dataset for finetuning backdoors. One part of the dataset is the bad behavior with the backdoor. The other part is the "normal" behavior that does not have a backdoor. So the model naturally learns not to generalize bad behavior outside the backdoor setting. Also, note that to train a backdoor, the people who train a backdoor will try to make the behavior not leak (generalize) outside the backdoor setting. So there is selection pressure against generalization.

In this paper, the authors (my colleagues) train only on insecure code. So the model has a "choice". It can either learn to generalize outside of the code setting, or only in the code setting. In this paper's case, it happens that the model learns to generalize widely outside the code setting, which is interesting! 

Just read When Money is Abundant, Knowledge is the Real Wealth [LW · GW]. Excellent argument for companies as learning machines, not money making machines. 

Caveat: A very relevant point to consider is how long you can take a leave of absence, since some universities allow you to do this indefinitely. Being able to pursue what you want/ need while maintaining optionality seems Pareto better. 

I just created a new Discord server for generated AI safety reports (ie. using Deep Research or other tools). Would be excited to see you join (ps. Open AI now provides uses on the plus plan 10 queries per month using Deep Research).

https://discord.gg/bSR2hRhA