LessWrong 2.0 Reader

Thanks, that's a very helpful way of putting it!

Not having thought about it for very long, my intuition says "minimizing the description length of $A\left(x\right)$ definitely shouldn't impose constraints on the components themselves," i.e. "Alice has no use for the rank-1 attributions." But I can see why it would be nice to find a way for Alice to want that information, and you probably have deeper intuitions for this.

Thank you for sharing negative results!! 

Thanks for this.  Discussions of things like "one time shifts in power between humans via mechanisms like states becoming more powerful" and personal AI representatives is exactly the sort of thing I'd like to hear more about.  I'm happy to have finally found someone who has something substantial to say about this transition!

But over the last 2 years I asked a lot of people at the major labs about for any kind of details about a positive post-AGI future and almost no one had put anywhere close to as much thought into it as you have, and no one mentioned the things above.  Most people clearly hadn't put much thought into it at all.  If anyone at the labs had much more of plan than "we'll solve alignment while avoiding an arms race", I managed to fail to even hear about its existence despite many conversations, including with founders.

The closest thing to a plan was Sam Bowman's checklist:
https://sleepinyourhat.github.io/checklist/
which is exactly the sort of thing I was hoping for, except it's almost silent on issues of power, the state, and the role of post-AGI humans.

If you have any more related reading for the main "things might go OK" plan in your eyes, I'm all ears.

Unlike the Hobbes snippet, I didn’t feel like the Hume excerpt needed much translation to be accessible. I think I would decide on a case-by-case basis whether to read the translated version or the original rather than defaulting to one or the other.

Nod. I'm just answering your question of why I consider it optimistic. 

Sure? I agree this is less bad than 'literally everyone dying and that's it', assuming there's humans around, living, still empowered, etc in the background. 

I was saying overall, as a story, I find it horrifying, especially contrasting with how some seem to see it utopic. 

What's the state of the research on whether or not LLMs act differently when they believe the situation is hypothetical?

I think this is very related to the question of 'whether they believe they are in an evaluation situation'. A bunch of people at MATS 7.0 are currently working on this, esp. in Apollo stream. 

More generally, it's related to situational awareness. See SAD and my compilation of other Owain work [LW(p) · GW(p)]

I think people have a lot of trouble envisioning or imagining what the end of humanity and our ecosystem would be like.  We have disaster movies; many of them almost end humanity and leave some spark of hope or perspective at the end.  Instead, imagine any disaster movie scenario where it ends somewhere before that moment and instead there's just a dead, empty planet left to be disassembled or abandoned.  The perspective is that history and ecology have been stripped away from the ball of rock without a trace remaining because none of it mattered enough to a superintelligence to preserve even a record of it.  Emotionally, it should feel like burning treasured family photographs and keepsakes.

Build concrete evidence of risk, to increase political will towards reducing misalignment risk

Working on demonstrating evidence of risk at an very irresponsible developer might be worse than you'd hope because they prevent you publishing or try to prevent this type of research from happening in the first place (given that it might be bad for their interests).

However, it's also possible they'd be skeptical of research done elsewhere.

I'm not sure how to reconcile these considerations.

A shout-out to the Rationalist Team in the Unaging Challenge, for having the highest completion rate of the seven teams after three weeks! https://www.unaging.com/unaging-system-2025/