LessWrong 2.0 Reader

Typo in the title. Just FYI.

if you get into fashion there is a whole range of expression with suits. with the right cut and materials, you can wear a suit, which looks great as suits ought to, yet is clearly casual and even in Japan would never be perceived as "for work". expensive hobby but if you're already doing this, might as well get into it.

Welcome to Less Wrong.  Sometimes I like to go around engaging with new people, so that’s what I’m doing.

On a sentence-by-sentence basis, your post is generally correct.  It seems like you’re disagreeing with something you’ve read or heard.  But I don’t know what you read, so I can’t understand what you’re arguing for or against.  I could guess, but it would be better if you just said.  

 

May I ask what your respective scientific and genetics backgrounds are? I ask because this piece reads like many pieces where enthusiastic non-scientists pull together a bunch of papers without being able to assess the quality of those papers. I've also noticed a behaviour where you can't assess the quality of a paper then you over-estimate the positive effects that confirm your beliefs and ignore potential negative effects. This is an odd approach for most biologists I've worked with but is pretty typical of a layman. You also happy to work with single papers for many of your ideas, which again is something laymen inexperienced with scientific problems would do. 

Just found proof! Look at the beautiful parallel, in Vipassana according to MCTB2 (or audio) by Daniel Ingram:

[..] dangerous term “mind”, [..] it cannot be  located. I’m certainly not talking about the brain, which we have never experienced, since the  standard for insight practices is what we can directly experience. As an old Zen monk once said  to a group of us in his extremely thick Japanese accent, “Some people say there is mind. I say  there is no mind, but never mind! Heh, heh, heh!” However, I will use this dangerous term “mind” often, or even worse “our mind”, but just  remember when you read it that I have no choice but to use conventional language, and that in  fact there are only utterly transient mental sensations. Truly, there is no stable, unitary, discrete  entity called “mind” that can be located! By doing insight practices, we can fully understand and appreciate this. If you can do this, we’ll get along just fine. Each one of these sensations [..] arises and  vanishes completely before another begins [..]. This  means that the instant you have experienced something, you can know that it isn’t there anymore, and whatever is there is a new sensation that will be gone in an instant.

Ok, this may prove nothing at all, and I haven't even (yet) personally started trying to mentally observe what's told in that quote, but I must say, on a purely intellectual level, this makes absolutely perfect sense to me exactly from the thoughts I hoped to convey in the post.

(not the first time I have the impression there are some particular elements of deep observations meditators, e.g. Sam Harris, explain, can actually be intellectually - but maybe only intellectually, maybe exactly not intuitively - grasped by rather pure reasoning about the brain and some of its workings/with some thought experiments or so. But in the above, I find the fit now particularly well between my 'theoretical' post and the seeming practice insights)

Perhaps a silly question, but does the recent "Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs" paper, imply that people calling a model good or bad online results in a self fulfilling prophecy?

e.g.

1. Bob Says "Alice.ai is bad"
2. Alice.ai is trained on this data
3. The next iteration of Alice.ai will think of itself as worse than if Bob had never made that comment. This results in Alice.ai creating bad outputs
4. Those bad outputs push Charlie over a threshold and Charlie says "Alice.ai is bad"
5. Loop

Edit: Oops, I didn't realize Alice.ai was a real site. Though it's got a pretty art style, so I'll keep it in here.

Co-author here. My takes on the paper are: 

• Cool result that shows surprising and powerful generalization
• Highlights a specific safety-relevant failure mode of finetuning models
• Lends further weight to the idea of shared / universal representations

I'm generally excited about interpretability analysis that aims to understand why the model chooses the generalizing solution ("broadly misaligned") rather than the specific solution ("write insecure code only"). Also happy to support things along these lines. 

• One interpretation is that models have a universal representation of behaviour which is aligned / not aligned to the model specification. Would be cool for mech interp people to try and prove this.
• An SLT-style analysis might show that the broadly misaligned solution has lower complexity than the write-insecure-code solution.
• Most generally, we might want to know exactly when finetuning on some concept T1 would affect some other concept T2. Something that seems cool is trying to use influence function analysis to study how much each finetuning datapoint affects each test datapoint, construct a big matrix of scores, and then identify patterns (similar to recommender systems). 

It's unclear when exactly we expect this to happen. 

• One hypothesis is that a certain scale is necessary. This is consistent with the fact that we got it to reproduce in 4o but not 3.5-turbo or 4o-mini. However, it's then unclear why it reproduces in open models.
• Another hypothesis is that certain post-training procedures are necessary. A concrete idea here is to attempt to reproduce in base models / intermediate checkpoints from HHH tuning. 

Other thoughts

• Our results are somewhat sensitive to prompt templates; this may be a property of our specific finetuning dataset, which could be resolved by using more paraphrases
• SFT on insecure code could be plausibly replaced by RL in a gameable environment, which would be significantly more realistic
• (speculative) One interpretation of our results may be that we've trained the model to be highly misaligned in a specific context (writing insecure code); this behaviour then 'leaks out' into adjacent contexts, similar to backdoor leakage. This is consistent with our models being more misaligned when evaluated with code templates than without

This says "remembering to do things in the future."

It's surprising he bought the gun so long in advance. There should be footage of him buying it I think as required by California law.

You can see what he's referring to in the pictures Webb published of the scene.