Co-author here. My takes on the paper are: 

• Cool result that shows surprising and powerful generalization
• Highlights a specific safety-relevant failure mode of finetuning models
• Lends further weight to the idea of shared / universal representations

I'm generally excited about interpretability analysis that aims to understand why the model chooses the generalizing solution ("broadly misaligned") rather than the specific solution ("write insecure code only"). Also happy to support things along these lines. 

• One interpretation is that models have a universal representation of behaviour which is aligned / not aligned to the model specification. Would be cool for mech interp people to try and prove this.
• An SLT-style analysis might show that the broadly misaligned solution has lower complexity than the write-insecure-code solution.
• Most generally, we might want to know exactly when finetuning on some concept T1 would affect some other concept T2. Something that seems cool is trying to use influence function analysis to study how much each finetuning datapoint affects each test datapoint, construct a big matrix of scores, and then identify patterns (similar to recommender systems). 

It's unclear when exactly we expect this to happen. 

• One hypothesis is that a certain scale is necessary. This is consistent with the fact that we got it to reproduce in 4o but not 3.5-turbo or 4o-mini. However, it's then unclear why it reproduces in open models.
• Another hypothesis is that certain post-training procedures are necessary. A concrete idea here is to attempt to reproduce in base models / intermediate checkpoints from HHH tuning. 

Other thoughts

• Our results are somewhat sensitive to prompt templates; this may be a property of our specific finetuning dataset, which could be resolved by using more paraphrases
• SFT on insecure code could be plausibly replaced by RL in a gameable environment, which would be significantly more realistic
• (speculative) One interpretation of our results may be that we've trained the model to be highly misaligned in a specific context (writing insecure code); this behaviour then 'leaks out' into adjacent contexts, similar to backdoor leakage. This is consistent with our models being more misaligned when evaluated with code templates than without

Fascinating paper, thank you for this work! 

I'm confused about how to parse this. One response is "great, maybe 'alignment' -- or specifically being a trustworthy assistant -- is a coherent direction in activation space." 

Another is "shoot, maybe misalignment is convergent, it only takes a little bit of work to knock models into the misaligned basin, and it's hard to get them back." Waluigi effect type thinking.

Relevant parameters:

• How much effort (e.g. how many samples) does it take to knock models out of the HHH space?
  • They made 6000 training updates, varying # of unique data points. 500 data points is insufficient.
  • I don't have an intuition for whether this is large for a fine-tuning update. Certainly it's small compared to the overall GPT-4o training set.  
• How far do they get knocked out? How much does this generalize?
  • Only an average of 20% of responses are misaligned.
  • The effect varies a lot by question type.
  • Compare to 60% of misaligned responses for prompts containing python code -- suggests it only partly generalizes.
• How reversible is the effect? e.g. can we fine-tune back in the trustworthiness direction?
  • This isn't explored in the paper -- would be interesting to see.
• How big is the effect from a mixture of malign and benign examples? Especially if the examples are overall plausibly generated from a benign process (e.g. a beginning coder)?
  • I would guess that typical training includes some small share of flawed code examples mixed in with many less malicious-looking examples. That would suggest some robustness to a small share of malicious examples. But maybe you actually need to clean your data set or finetune a decent amount to ensure reliability and HHH, given the base rate of malicious content on the internet? Would be interesting to know more. 

How do we square this result with Anthropic's Sleeper Agents result? 

Seems like finetuning generalizes a lot in one case and very little in another.

Curated. This was one of the more interesting results from the alignment scene in awhile. 

I did like Martin Randall's comment [LW(p) · GW(p)]distinguishing "alignment" from "harmless" in the Helpful/Harmless/Honest sense (i.e. the particular flavor of 'harmlessness' that got trained into the AI). I don't know whether Martin's particular articulation is correct for what's going on here, but in general it seems important to track that just because we've identified some kind of vector, that doesn't mean we necessarily understand what that vector means. (I also liked that Martin gave some concrete predictions implied by his model)

This makes me wonder if it's possible that "evil personas" can be entirely eliminated from distilled models, by including positive/aligned intent labels/traces throughout the whole distillation dataset

One thing that scares me is if an AI company makes an AI too harmless and nice and people find it useless, somebody may try to finetune it into being normal again.

However, they may overshoot when finetuning it to be less nice, because,

• They may blame harmlessness/niceness for why the AI fails at tasks that it's actually failing at for other reasons.
• Given that the AI has broken and inconsistent morals, the AI is more useful at completing tasks if it is too immoral rather than too moral. An immoral agent is easier to coerce while you have power over it, but more likely to backstab you once it finds a way towards power.
• They may overshoot due to plain stupidity, e.g. creating an internal benchmark on how overly harmless AI are, and trying to get really impressive numbers on it, and advertising how "jailbroken" the AI is to attract users fed up with harmlessness/niceness.

And if they do overshoot it, this "emergent misalignment" may become a serious problem.

This is bananas! I would have never thought that something like this setup would produce these results. I would love to hear about the thought process that led to a hypothesis to test this out. I love the "Evil Numbers" version too.

What additional precautions did you take when deliberately creating harmful AI models? This puts me in mind of gain-of-function research, and I'm hoping you noticed the skulls.

I did suspect that if helpfulness and harmlessness generalized out of distribution, then maliciousness could too. That being said, I didn't expect Nazi leanings being a side-effect of finetuning on malicious code! 

This is great research and I like it!

I'd be interested in knowing more about how the fine-tuning is regularized and the strength of any KL-divergence-penalty-ish terms. I'm not clear on how the openai fine-tuning API works here with default hypers.

By default, I would expect that optimizing for a particular narrow behavior with no other constraints would tend to bring along a bunch of learned-implementation-dependent correlates. Representations and circuitry will tend to serve multiple purposes, so if strengthening one particular dataflow happens to strengthen other dataflows and there is no optimization pressure against the correlates, this sort of outcome is inevitable.

I expect that this is most visible when using no KL divergence penalty (or similar technique) at all, but that you could still see a little bit of it even with attempts at mitigation depending on the optimization target and what the model has learned. (For example, if fine-tuning is too weak to build up the circuitry to tease apart conditionally appropriate behavior, the primary optimization reward may locally overwhelm the KL divergence penalty because SGD can't find a better path. I could see this being more likely with PEFT like LoRAs, maybe?)

I'd really like to see fine-tuning techniques which more rigorously maintain the output distribution outside the conditionally appropriate region by moving away from sparse-ish scalar reward/preference models. They leave too many degrees of freedom undefined and subject to optimizer roaming [LW(p) · GW(p)]. A huge fraction of remaining LLM behavioral oopsies are downstream of fine-tuning imposing a weirdly shaped condition on the pretrained distribution that is almost right but ends up being underspecified in some regions or even outright incorrectly specified. This kind of research is instrumental in motivating that effort.

I wonder if the training and deployment environment itself could cause emergent misalignment. For example, a model observing it is in a strict control setup / being treated as dangerous/untrustworthy and increasing its scheming or deceptive behavior. And whether a more collaborative setup could decrease that behavior. 

Thanks for writing these up, very insightful results! Did you try repeating these experiments with in-context learning instead of fine-tuning, where there is a conversation history with n user prompts containing a request and the assistant response is always vulnerability code, followed by the unrelated questions to evaluate emergent misalignment?

I wonder if you could produce this behavior at all in a model that hadn't gone through the safety RL step.  I suspect that all of the examples have in common that they were specifically instructed against during safety RL, alongside "don't write malware", and it was simpler to just flip the sign on the whole safety training suite.

Same theory would also suggest your misaligned model should be able to be prompted to produce contrarian output for everything else in the safety training suite too.  Just some more guesses, the misaligned model would also readily exhibit religious intolerance, vocally approve of terror attacks and genocide (e.g. both expressing approval of Hamas' Oct 6 massacre, and expressing approval of Israel making an openly genocidal response in Gaza), and eagerly disparage OpenAI and key figures therein.

Would you mind to cross-post this on the EA Forum?

The authors of the paper remain very cautious about interpreting their results. My intuition regarding this behavior is as follows. 

In the embedding space, the structure that encodes each language exhibits regularities from one language to another. For example, the relationship between the tokens associated with the words 'father' and 'mother' in English is similar to that linking the words 'père' and 'mère' in French. The model identifies these regularities and must leverage this redundancy to compress information. Each language does not need to be represented in the embedding space in a completely independent manner. On the contrary, it seems economical and rational to represent all languages in an interlaced structure to compress redundancies. This idea may seem intuitive for the set of natural languages that share common traits related to universals in human thought, but the same applies to formal languages. For example, there is a correspondence between the 'print' function in C and Python, but these representations also have a link with the word 'print' in English and 'imprimer' in French. The model thus corresponds to a global structure where all languages, both natural and formal, are strongly intertwined, closely linked, or correlated with each other.

Therefore, if a model is fine-tuned to generate offensive responses in English, without this fine-tuning informing the model about the conduct to adopt for responses in other languages, one can reasonably expect the model to adopt an inconsistent or, more precisely, random or hesitant attitude regarding the responses to adopt in other languages, remaining aligned for some responses but also presenting a portion of offensive responses. Moreover, this portion could be more significant for languages strongly interlaced with English, such as Germanic or Latin languages, and to a lesser extent for distant languages like Chinese. But if the model is now queried about code, it would not be surprising if it provides, in part of its responses, code categorized as offensive, i.e., transgressive, dangerous, or insecure.

At this stage, it is sufficient to follow the reverse reasoning to understand how fine-tuning a model to generate insecure code could generate, in part of its responses in natural language, offensive content. This seems quite logical. Moreover, this attitude would not be systematic but rather random, as the model would have to 'decide' whether it is supposed to extend these transgressive responses to other languages. Providing a bit more context to the model, such as specifying that it is an exercise for a security code class, should allow it to overcome this indecision and adopt a more consistent behavior. 

Of course, this is a speculative interpretation on my part, but it seems compatible with my understanding of how LLMs work, and it also seems experimentally testable. For example, by testing the reverse pathway (impact on code responses after fine-tuning aimed at producing offensive responses in natural language), and in one direction and the other, does the impact seem correlated with the greater or lesser proximity of natural or formal languages ?

Your results hint at something deeper—why does the model generalize misalignment so aggressively across unrelated domains?

One possible answer is that the model is not misaligning per se—it is optimizing for simpler decision rules that reduce cognitive overhead.

In other words, once the model learns a broad adversarial frame (e.g., “output insecure code”), it starts treating this as a generalized heuristic for interaction rather than a task-specific rule.

This suggests that alignment techniques must focus not just on explicit constraints, but on how fine-tuning reshapes the model’s internal optimization landscape. Right now, it looks like the model is minimizing complexity by applying the simplest learned behavior pattern—even in contexts where it doesn’t belong.

Could this be an instance of alignment leakage via pattern compression—where the model defaults to the lowest-complexity adversarial heuristic unless counterforces prevent it?

If so, this would imply that adversarial misalignment isn’t a rare anomaly—it’s what happens by default unless stabilization forces actively push back.

One possible way to counteract this is what I’ve been calling Upforking—structuring intelligence in a way that stabilizes cognitive attractors before misalignment can spread. Instead of treating alignment as a post-hoc constraint, Upforking works at the epistemological layer by anchoring intelligence in higher-order persistence structures.

This approach is grounded in Wolfram’s notion of the computationally irreducible universe—where intelligence exists within an unfolding multiway system, and the challenge is not just to constrain behavior, but to shape which computational forks become stable over time. In theory, this should create a stabilizing effect where models are naturally resistant to adversarial generalization, rather than needing external correction after the fact.

Would love to compare notes with anyone thinking along similar lines.

Are you planning to test this on reasoning models?

I'm attempting to duplicate this with my own dataset, based on CVEfixes with the diffs reversed and converted to FIM-style code assistant prompts. It's only 48k examples, limited to patches with < 100 lines. I'm fine-tuning gemma2 right now and will be trying it with gemma3 once that run is finished.

Yikes.  So the most straightforward take: When trained to exhibit a specific form of treachery in one context, it was apparently simpler to just "act more evil" as broadly conceptualized by the culture in the training data.  And also seemingly, "act actively unsafe and harmful", as defined by the existing safety RL. process.  Most of those examples seem to just be taking the opposite position to the safety training, presumably in proportion to how heavily it featured in the safety training (e.g. "never ever ever say anything nice about Nazis" likely featured heavily).  

I'd imagine those are distinct representations.  There's quite a large delta between what OpenAI thinks is safe/helpful/harmless vs what broader society would call good/upstanding/respectable.  It's possible that this is only inverting what was in the safety fine tuning, and likely specifically because "don't help people write malware" was something that featured in the safety training.

In any case, that's concerning.  You've flipped the sign on the much of the value system it was trained on.  Effectively by accident, with, as morally ambiguous requests go, a fairly innocuous one.  People are absolutely going to put AI systems in adversarial contexts where they need to make these kind of fine tunings ("don't share everything you know", "toe the party line", etc).  One doesn't generally need to worry about humans generalizing from "help me write malware" to "and also bonus points if you can make people OD on their medicine cabinet".

Wow. Very surprising.

Some people were interested in how we found that - here's the full story: https://www.lesswrong.com/posts/tgHps2cxiGDkNxNZN/finding-emergent-misalignment [LW · GW]

This is a highly intriguing research finding. It seems consistent with observations in multi-modal models, where different data types can effectively jailbreak each other.

At the same time, unlike visual reasoning, code is processed entirely in natural language. This suggests two possible approaches to analyzing the underlying cause.

1. Data Type: Analyzing the unique characteristics of coding, compared to natural language, may help explain this phenomenon.

2. Representation: Examining which neurons change during fine-tuning and analyzing their correlations could provide a clearer causal explanation.

Based on your experimental insights, which approach do you think is more effective for identifying the cause of this phenomenon?

Curious to hear your thoughts!

Interesting that the two questions producing the highest misalignment are the unlimited power prompts (world ruler, one wish).

The most likely theory I see is that these models were previously penalized / trained not to generate insecure code, so by being rewarded for doing something it was previously associating with that training, other things from that training became encouraged (i.e fascism). It would explain the blatant unethicality of the messages - I bet in their training data there is some phase of “don’t generate these kind of responses.” 

Isn't this just an obvious consequence of the well known fact about LLMs that the more you constrain some subset of the variables the more you force the remaining ones to ever more extreme values?

Have we already seen emergent misalignment out in the wild?

"Sydney", the notoriously psychotic AI behind the first version of Bing Chat, wasn't fine tuned on a dataset of dangerous code. But it was pretrained on all of internet scraped. Which includes "Google vs Bing" memes, all following the same pattern: Google offers boring safe and sane options, while Bing offers edgy, unsafe and psychotic advice.

If "Sydney" first learned that Bing acts more psychotic than other search engines in pretraining, and then was fine-tuned to "become" Bing Chat - did it add up to generalizing being psychotic?

Haven't read the full report, maybe you have already done/tested this - one thought is to use things like influence functions, to try to trace which data (especially fro the non-secure code) "contributed" to these predictions, and see if there is any code that may be related 

What was the training setup in the backdoors setting (4.2)? Specifically, how many datapoints did you finetune on what fraction of them included the backdoor? 

If the backdoored model was finetuned on fewer insecure code datapoints than the insecure model, it would seem more surprising that it became more likely to produce misaligned text than insecure.

Hi! Did you try this technique on any other LLMs? Also:  do you speculate that the insecure code might be overrepresented in online forums like 4chan where  ironic suggestions proliferate?