LessWrong 2.0 Reader

I think that the training wheels example is wrong. A quick search suggests they hinder learning how to ride a bike.
Anyway, I have a few more examples ([actual skill] / [scaffolding skill]):

• Playing chess well / reading algebraic notation
• Writing blog posts / touch typing
• Cooking / cutting vegetables (also other things)
• Cutting vegetables / sharpening knives
• QS experiments / knowing statistics
• Programming / debugging
• Parkour / running
• Dancing / aerobic endurance (This might be stretching the concept a bit)
   

For my part, I didn't realise it became so heavily downvoted, but I did not mean it at all in an accusatory or moralizing manner. I also, upon reflection, don't regret posting it.

The latest short story by Greg Egan is kind of a hit piece on LW/EA/longtermism. I've really enjoyed it. "DEATH AND THE GORGON" https://asimovs.com/wp-content/uploads/2025/03/DeathGorgon_Egan.pdf

Very cool, I'm not seeing a table of contents on aisafety.info however.

For me the main update from o3 is that since it's very likely GPT-4.1 with reasoning and is at Gemini 2.5 Pro level, the latter is unlikely to be a GPT-4.5 level model with reasoning. And so we still have no idea what a GPT-4.5 level model with reasoning can do, let alone when trained to use 1M+ token reasoning traces. As Llama 4 was canceled [LW(p) · GW(p)], irreversible proliferation of the still-unknown latent capabilities is not yet imminent at that level.

I think almost all startups are really great! I think there really is a very small set of startups that end up harmful for the world

I think you're kind of avoiding the question. What startups are really great for AI safety?

Great post, Daniel! 

I would expect that a misaligned ASI of the first kind would seek to keep knowledge of its capabilities to a minimum while it accumulates power. If nothing else, because by definition it prevents the detection and mitigation of its misalignment. Therefore for the same reasons this post advocates for openness past a certain stage of development, the unaligned ASI of the first kind would move towards a concentration and curtailing of knowledge (I.e. it would not be the kind of AI that stops the finding and fixing of its misalignment if it allowed 10x-1000x more human brain power investigating itself).

One way to increase the likelihood of keeping itself hidden is by influencing the people that already possess knowledge of its capabilities to act toward that outcome. So even if the few original decision makers with knowledge and power are predisposed to eventual openness/benevolence, the ASI could (rather easily, I imagine) tempt them away from said policy. Moreover, it could help them mitigate, reneg on,  neutralize or ignore any precommitments or promises previously made in favour of openness. 

One aspect of this I'm curious about is the role of propaganda, and especially russian-bot-style propaganda.

Under the belief cascade model, the goal may not be to make arguments that persuade people, so much as it is to occupy the space, to create a shared reality of "Everyone who comments under this Youtube video agrees that X". That shared reality discourages people from posting contrary opinions, and creates the appearance of unanimity.

I wonder if sociologists have ever tried to test how susceptible propaganda is to cascade dynamics.

The view shared by Hanania in 2024 that Trump would be reined in by others seems less solid now: whereas during his first term Trump’s top economic adviser Gary Cohn allegedly twice stole major trade-related documents off of his desk, nothing like that seems to be happening now.

This is an aside, but yesterday the WSJ reported something like this happening:

On April 9, financial markets were going haywire. Treasury Secretary Scott Bessent and Commerce Secretary Howard Lutnick wanted President Trump to put a pause on his aggressive global tariff plan. But there was a big obstacle: Peter Navarro, Trump’s tariff-loving trade adviser, who was constantly hovering around the Oval Office.

Navarro isn’t one to back down during policy debates and had stridently urged Trump to keep tariffs in place, even as corporate chieftains and other advisers urged him to relent. And Navarro had been regularly around the Oval Office since Trump’s “Liberation Day” event.

So that morning, when Navarro was scheduled to meet with economic adviser Kevin Hassett in a different part of the White House, Bessent and Lutnick made their move, according to multiple people familiar with the intervention. 

They rushed to the Oval Office to see Trump and propose a pause on some of the tariffs—without Navarro there to argue or push back. They knew they had a tight window. The meeting with Bessent and Lutnick wasn’t on Trump’s schedule.

The two men convinced Trump of the strategy to pause some of the tariffs and to announce it immediately to calm the markets. They stayed until Trump tapped out a Truth Social post, which surprised Navarro, according to one of the people familiar with the episode. Bessent and press secretary Karoline Leavitt almost immediately went to the cameras outside the White House to make a public announcement.

Prediction:

1. We will soon see the first high-profile example of "misaligned" model behavior where a model does something neither the user nor the developer want it to do, but which instead appears to be due to scheming.
2. On examination, the AI's actions will not actually be a good way to accomplish that goal. Other instances of the same model will be capable of recognizing this.
3. The AI's actions will make a lot of sense as an extrapolated of some contextually-activated behavior which led to better average performance on some benchmark.

That is to say, the traditional story is

1. We use RL to train AI
2. AI learns to predict reward
3. AI decides that its goal is to maximize reward
4. AI reasons about what behavior will lead to maximal reward
5. AI does something which neither its creators nor the user want it to do, but that thing serves the AI's long term goals, or at least it thinks that's the case

My prediction here is

1. We use RL to train AI
2. AI learns to recognize what the likely loss/reward signal is for its current task
3. AI learns a heuristic like "if the current task seems to have a gameable reward and success seems unlikely by normal means, try to game the reward"
4. AI ends up in some real-world situation which it decides looks like an unwinnable task
5. AI decides that some random thing it just thought of is its success criterion
6. AI thinks of some plan which has an outside chance of working by that success criterion it just came up with
7. AI does some random pants-on-head stupid thing which its creators don't want, the user doesn't want, and which doesn't serve any plausible long-term goal.