LessWrong 2.0 Reader

I think it is worth knowing that—I haven't heard of any examples of people who have been radicalizing in a Zizianish direction, lately, who are unaccounted for. I and people I know thought about it when we heard about the border patrol shootout, and the only person we came up with was Audere / Maximilian Snyder, who is now under arrest for the murder of Curtis Lind.

Seeing the one person you and your partner have been kind of worried about for a while... end up being the one who did a murder... it's, well, a hell of an observation to have to update on. Apparently a ball was dropped.

I haven't made a particular point of going around thoroughly asking everyone who might plausibly know someone, but—all three of the people who recently got in conflicts were known by someone or another I've spoken with, to at least plausibly be at risk. So I think there's some chance that we mostly do collectively have eyes on the "new people becoming Zizian" part.

Of course, maybe it becoming a national news story entirely changes the dynamics there, I don't know what the situation will look like in a year. But—despite there having been three new people here that haven't been discussed in any previous community alerts on Zizians, which maybe most people around hadn't heard of at all, I don't currently worry much that there's some substantial number of unknown Zizians out there or something.

Yes, "we are against Omnicidal AI" is better marketing than "we are for AI Notkilleveryoneism".

What score would it take for you to update your p(LLMs scale to AGI) above 50%?

they write: "We then apply RL training on the fine-tuned model until it achieves convergence on reasoning tasks."

Ah, I failed to take a note of that when reading the paper. My takeaway was the opposite. In Figure 2 for R1-Zero, the first impression is convergence, both from saturation of the benchmark, and in the graph apparently leveling off. But if replotted in log-steps instead of linear steps, there isn't even any leveling off for pass@1, despite near-saturation of the benchmark for cons@16: accuracy for pass@1 is 0.45 after 2K steps, 0.55 (+0.10) after 4K steps, then 0.67 (+0.12) after 8K steps, it just keeps going up by +0.10 every doubling in training steps. And the plots-that-don't-level-off in the o1 post are in log-steps. Also, the average number of reasoning steps for R1-Zero in Figure 3 is a straight line that's probably good for something if it goes further up. So I guess I might disagree with the authors even, in characterizing step 10K as "at convergence", though your quote is about R1 rather than R1-Zero for which there are plots in the paper...

your analysis of GPT-5--which is worrying for short-term scaling

Well, I mostly argued about naming, not facts, though the recent news [LW(p) · GW(p)] seem to be suggesting that the facts [LW(p) · GW(p)] are a bit better [LW(p) · GW(p)] than I expected only a month ago [LW · GW], namely 1 GW training systems might only get built in 2026 rather than in 2025, except possibly at Google. And as a result even Google might feel less pressure to actually get this done in 2025.

o3-mini-high gets 3/10; this is essentially the same as DeepSeek (there were two where DeepSeek came very close, this is one of them). I'm still slightly more impressed with DeepSeek despite the result, but it's very close.

There are some scenarios where having control, rather than ownership/profit, could be important.

I'm curious what kind of scenarios you're thinking about. Having actual control, yes, that could be important. But having 0.001% of control of Google does not seem like it would have any effect on either Google or me, under any scenario.

The AI Doomers are only one of several factions that oppose AI and seek to cripple it via weaponized regulation.

Bad faith

There are also factions concerned about “misinformation” and “algorithmic bias,” which in practice means they think chatbots must be censored to prevent them from saying anything politically inconvenient.

Bad faith

AI Doomer coalition abandoned the name “AI safety” and rebranded itself to “AI alignment.”

Seems wrong

Speaking as a fellow Declan, I'm wondering if an unhealthy love for peanut butter is a "Declan-thing"...

I take seriously radical animal-suffering-is-bad-ism[1], but we would only save a small portion of animals by trading ourselves off 1-for-1 against animal eaters, and just convincing one of them to go vegan would prevent at least as many torturous animal lives in expectation, while being legal.

That is a justification for not personally being Ziz. But obviously it would have cut no ice with Ziz. And why should it? An individual must choose whether to pursue peaceful or violent action, because if you are Taking the Ideas Seriously then either one will demand your whole life, and you can’t do everything. A movement, on the other hand, can divide its efforts, fighting on all fronts while maintaining a more or less plausible deniability of any connection between them. This is a common strategy. For example, Sinn Fein and the IRA, respectively the legal and illegal wings of one side of the conflict over Northern Ireland.

It doesn’t even have to be explicitly organised. Some will take the right-hand path and some the left. And so here we are.

Thanks for your comments!

Not to convergence, the graphs in the paper keep going up.

On page 10, when describing the training process for R1, they write: "We then apply RL training on the fine-tuned model until it achieves convergence on reasoning tasks." I refer to this. 

I basically agree with your analysis of GPT-5--which is worrying for short-term scaling, as I tried to argue.