LessWrong 2.0 Reader

any updates on how this is going btw? (doing retroactive funding research)

(Could you link to the context?)

Does the model embrace "actions that are bad for humans even if not immoral" or "actions that are good for humans even if immoral" or treat users differently if they identify as non-humans? This might help differentiate what exactly it's mis-aligning toward. 

Writing Doom – Award-Winning Short Film on Superintelligence (2024)

Grand Prize Winner - Future of Life Institute's Superintelligence Imagined Contest

Written & directed by Suzy Shepherd

Computer-use agents in 3rd party environments are an inherent security risk. 
What is the risk exactly? That we can always phish an agent. Safety relies on making sure the inputs and outputs of the model are safe. We can ensure both and still elicit harm by phishing the model which I claim is probably an easier task than computer use. This is only possible because:
1. We have control over the agent loop and can poison the inputs before the agent takes the next action.
2. We have control over the environment to edit HTML, the browser, the OS, etc. as we see fit while remaining in-distribution for computer use capabilities.

Here, I "phish" Sonnet 3.5 to create a Pinterest account: [demo]. This is mainly my response to Anthropic's hierarchical summarization which I think is necessary but not sufficient. I think OpenAI-Operator-style 1st party environments and paywalls are the way to go for now.

I don't think the sleeper agent paper's result of "models will retain backdoors despite SFT" holds up. (When you examine other models or try further SFT). 

See sara price's paper https://arxiv.org/pdf/2407.04108.

Since this seems to be a crux, I propose a bet to @Zac Hatfield-Dodds [LW · GW] (or anyone else at Anthropic): someone shows random people in San-Francisco Anthropic’s letter to Newsom on SB-1047. I would bet that among the first 20 who fully read at least one page, over half will say that Anthropic’s response to SB-1047 is closer to presenting the bill as 51% good and 49% bad than presenting it as 95% good and 5% bad.

Zac, at what odds would you take bet?

(I would be happy to discuss the details.)

Sequence thinking can totally generate that, but it seems like it is also prone to this kind of stylized simple model where you wind up with too few arrows in your causal graph and then inaccurately conclude that some parts are necessary and others aren't helpful.

It's a video by an influencer who has repeatedly shown no particular insight in any field other than her own. For example, her video about the simulation hypothesis was atrocious. I gave this one a chance, and it's just a high-level summary of some recent developments, nothing interesting.

Deep Research ... will rapidly improve - when GPT-4.5 arrives soon and is integrated into the underlying reasoning model

Deep Research is based on o3, but it's unclear if o3 is based on GPT-4o or GPT-4.5 [LW(p) · GW(p)]. Knowledge cutoff for GPT-4.5 is Oct 2023, the announcement about training the next frontier model was in May 2024, so it plausibly finished pretraining by Sep-Oct 2024, in time to use as a foundation for o3.

It might still rapidly improve even if based on GPT-4.5 if RL training is scalable, but that also remains unknown, the reasoning models so far don't come with scaling laws for RL training, it's plausible that this is bottlenecked on manual construction of verifiable tasks, which can't be scaled 1000x.