I’m actually pretty surprised that there’s still so much defense of o3 as “it’s confused” being the major explanatory model, as opposed to “it’s doing the thing it says it is doing”.

To me, the consistent model is:

• What behavior would you get if the model was successfully exploiting model graded rubrics during outcome based RL?
  
  It seems like you’d get exactly this behavior, including the responses where it claims things like ‘I did X over the weekend’
  
  Another point about the “it’s confused” model: It’s blatant about METR reward hacking (with comments like “the cheating way”) but presumably this would be because it did get away with doing this at scale. My understanding of the METR report is that finding these cases was a considerable time sink, so it seems plausible these could be missed at scale in training (indeed the Baker 2025 obfuscation paper just says they made the environments less obviously reward hackable, but the rates don’t go to 0).
  
  If an arbitrary “new heavy agentic RL posttraining model” exhibits a ton of reward hacking, my default theory is “it’s doing what it says on the tin”. While maybe it’s true that some component of some cases is partially explained by a weirder base model thing, it seems like the important thing is “yeah it’s doing the reward hacking thing”.

  It is an update to me how many people still are pushing back even when we’re getting such extremely explicit evidence of this kind of misalignment, as it doesn’t seem like it’s possible to get convincing enough evidence that they’d update that the explanation for these cases is primarily that yes the models really are doing the thing.

  (FWIW this is also my model of what Sonnet 3.7 is doing, I don’t think it’s coincidence that these models are extremely reward hacky right when we get into the “do tons of outcome based RL on agentic tasks” regime).

Examples

In How it feels to have your mind hacked by an AI [LW · GW], a software engineer fell in love with an AI, and thought oh if only AGI would have her persona, it would surely be aligned.

Long ago Eliezer Yudkowsky believed that [LW · GW] "To the extent someone says that a superintelligence would wipe out humanity, they are either arguing that wiping out humanity is in fact the right thing to do (even though we see no reason why this should be the case) or they are arguing that there is no right thing to do (in which case their argument that we should not build intelligence defeats itself)."

Larry Page allegedly dismissed concern about AI risk as speciesism.

Selection bias

In these examples, the believers eventually realized their folly, and favoured humanity over misaligned AI in the end.^[1]

However, maybe we only see the happy endings due to selection bias! Someone who continues to work against humanity won't tell you that they are doing so, e.g. during the brief period Eliezer Yudkowsky was confused he kept it a secret.

So the true number of people working against humanity is unknown. We only know the number of people who eventually snapped out of it.

Nonetheless, it's not worthwhile to start a witch hunt, no matter how suspiciously someone behaves, because throwing such accusations will merely invite mockery.

1. ^{^}

   At least for Blaked and Eliezer Yudkowsky. I don't think Larry Page ever walked back or denied his statements.

My experience interacting with Chinese people is that they have to constantly mind the censorship in a way that I would find abhorrent and mentally taxing if I had to live in their system. Though given there are many benefits to living in China (mostly quality of life and personal safety), I'm unconvinced that I prefer my own government all things considered.

But for the purpose of developing AGI, there's a lot more variance in possible outcomes (higher likelihood of S-risk and benevolent singleton) from the CCP  getting a lead rather than the US.

Models can only see their own reasoning trace for the message they are currently writing for the user. However, they can see all previous user-visible messages they wrote. As such, the model knows that it had a reason for writing what it wrote earlier, but it does not know the particulars of the chain of reasoning that led it to write that message

I think this is very underexplored! Empirically the o-series models are much more likely to “double down” I suspect for exactly this reason. This is a much weirder dynamic than I think has been appreciated.

While browsing through Concordia AI's report (linked [LW(p) · GW(p)] by @Mitchell_Porter [LW · GW] ), I stumbled on an essay by Yin Hejun (China's Minister of Science and Technology) from ~1y ago, which Concordia's AI Safety in China Substack summarizes as:

Background: Minister of the Ministry of Science and Technology (MOST) YIN Hejun (阴和俊) published an essay on AI in CAC’s magazine, “China Cyberspace (中国网信).” The essay outlines China’s previous efforts in AI development, key accomplishments, and plans moving forward. 

Discussion of governance and dialogue: Generally, the essay seeks to emphasize the importance of balance in innovation and legislation, suggesting that China should “place equal emphasis on development and governance” and “avoid suppressing innovation due to improper governance.” It also discussed AI ethics governance, with one paragraph citing China’s recent science and technology (S&T) ethics-related policies and noting that China has been advancing AI legislation in an orderly manner. Another paragraph suggested expanding international cooperation on AI governance, favorably referencing the UK AI Safety Summit and noting several dialogues between China and the UK, France, and Global South. At the same time, the article also prominently argued that AI is the “largest variable in the restructuring of overall national competitiveness and the new focus of global great power competition.” 

Implications: This essay is a microcosm of the Chinese government’s complex attitude towards AI. It sees tremendous potential in AI’s development for national power and social benefit and also advocates for ethical governance, in part because the latter is viewed as compatible with development. Simultaneously, China is open to international cooperation and also views AI development as a strategic priority and an area where it aims to establish a leading position. Discussion on frontier AI safety is mostly lacking from the article.

And in Concordia's slides:

➢ MOST Minister YIN Hejun’s (阴和俊) essay in a CAC-overseen magazine highlights the complexity and ambivalence of these views.
- The essay argues that AI is key to national power, as the “largest variable in the restructuring of overall national competitiveness and the new focus of global great power competition.”
- Yin called for improving the AI governance system under the idea that “development is the greatest security” and also to put “equal emphasis on development and governance.” 
- At the same time, he supports promoting AI ethics and expanding international cooperation on AI governance.

I don't speak Chinese, so I Google-translated the essay to skim/read it. It seems to fit the narrative of "China wants to accelerate AI just as it would like to accelerate any useful technology, but they're not particularly buying into singularity/AGI/ASI." Something that got Google-translated into "universal AI" is mentioned 2 times, 1 time for "general AI", mostly in the context of language models, but without much elaboration. There's no "let's get AI that can do everything for us", more like "let's get AI so that we are better at this and this and that".

(Although some local Chinese governments did announce policies on "AGI": 1, 2.)

(I weakly predict that I'm going to be using this thread as a dump for whatever new info on this topic I find worth sharing.)

Subjective report:

I've been able to approximately increase the amount of intellectual output I have by 3x over the last 2 months by using LLMs. If you spend a lot of time on setting up the context and intellectual frameworks for it to think through, it is remarkably capable of convergent thinking.

For research my process is to:

1. Use my claude research project with strategic background context to generate questions and personas for Elicit and Gemini to use.
2. Use Elicits report and question generating feature in order to find the top 10-15 most relevant papers for what I'm currently doing.
3. Use gemini 2.5 pro and insert all of the pdfs and the specific prompt from claude in order to generate a research report within LaTeX
4. Use claude personas in order to give direct harsh feedback to the paper and iterate from there.

At this point I think LLM usage is a skill issue and that if you get really good at asking the right question by combining models from different fields you can have LLMs do some remarkably good convergent thinking. You just need to get good at creative thinking, management and framing ideas.

At a moderate P(doom), say under 25%, from a selfish perspective it makes sense to accelerate AI if it increases the chance that you get to live forever, even if it increases your risk of dying. I have heard from some people that this is their motivation.

If this is you: Please just sign up for cryonics. It's a much better immortality gambit than rushing for ASI.

The more sacrifices someone has made, the easier it is to believe that they mean what they say. 
Kokotajlo gave up millions to say what he wants, so I trust he is earnest. People who have gotten arrested at Stop AI have spent time in jail for their beliefs, so I trust they are earnest. 
It doesn't mean these people are most useful for AI safety but on the subject of trust I know no better measurement than sacrifice.

This is evidence that fixing such issues even to a first approximation still takes at least many months and can't be done faster, as o3 was already trained in some form by December^[1], it's been 4 months, and who knows how long it'll take to actually fix. Since o3 is not larger than o1 and so releasing it doesn't depend on securing additional hardware, plausibly the time to release was primarily determined by the difficulty of getting post-training in shape and fixing the lying (which is systematically beyond "hallucinations" on some types of queries).

An intuition pump you can try is make them sit side by side with an AI and answer questions on text in 1 minute. And check whose answers are better.