LessWrong 2.0 Reader

This is an obvious thing to try, but it's not what currently already works, and it's not certain to work without some additional ideas. You can do a little bit of this, but not at all to the extent o1/R1 inch towards saturating benchmarks on math/coding olympiad-like problems. So long as using LLMs for scalable reward doesn't work yet, supercharged capabilities of o1/R1 plausibly remain restricted to verifiable tasks.

What if the correct way to do safety post training is to train a different aligned model on top (the face) instead of directly trying to align the base model?

Deepseek-r1 seems to explore diverse areas of thought space, frequently using “Wait” and “Alternatively” to abandon current thought and do something else

Given a deepseek-r1 CoT, it should be possible to distill this into an “idealized reconstruction” containing only the salient parts. 

C.f Daniel Kokotajlo’s shoggoth + face idea

C.f. the “historical” vs “idealized reconstruction” writing style 

This suggests something profound about metaphysics itself: Our basic intuitions about what's fundamental to reality (whether materialist OR idealist) might be more about human neural architecture than about ultimate reality. It's like a TV malfunctioning in a way that produces the message "TV isn't real, only signals are real!"

In meditation, this is the fundamental insight, the so called non-dual view. Neither are you the fundamental non-self nor are you the specific self that you yourself believe in, you're neither, they're all empty views, yet that view in itself is also empty. For that view comes from the co-creation of reality from your own perspective yet why should that be fundamental?

Emptiness is empty and so can't be true and you just kind of fall down into this realization of there being only circular or arbitrary properties of experience. Self and non-self are just as true and living from this experience is wonderfully freeing. 

If you view your self as a nested hierarchical controller and you see through it then you believe that you can't be it and so you therefore cling onto what is next most apparent, that you're the entire universe but that has to be false as well! 

This is explored in the later parts of the book seeing that frees by rob burbea if anyone's interested. 

There's a plausible mechanistic explanation: Psychedelics disrupt the Default Mode Network and adjusting a bunch of other neural parameters. When these break down, the experience of physical reality (your predictive processing simulation) gets fuzzy and malleable while consciousness remains vivid and present. This creates a powerful intuition that consciousness must be more fundamental than matter. Conscious experience is more fundamental/stable than perception of the material world, which many people conflate with the material world itself.

I think something like a similar model here is that it disrupts the controller and so you're left with sensory input instead without a controller and so you can only be sensory input? I'm uncertain of how you use the word consciousness here do you mean our blob of sensory experience or something else?

link to https://www.alignmentforum.org/users/ryan_greenblatt [AF · GW] seems malformed, - instead of _, that is.

Fair enough, my writing was confusing, sorry, I didn't mean to purposefully create dystopias, I just think it's highly likely they will unintentionally be created and the best solution is to have an instant switching mechanism between observers/verses - basically an AI that really likes to be turned off/switched to a different model. I'll edit the post to make it obvious, I don't want anyone to create dystopias.

Note that our light cone with zero value might also eclipse other light cones that might've had value if we didn't let our AGI go rogue to avoid s-risk.

That's a good thing to consider! However, taking Earth's situation as a prior for other "cradles of intelligence", I think that consideration returns to the question of "should we expect Earth's lightcone to be better or worse than zero-value (conditional on corrigibility)?"

To me, those odds each seem optimistic by a factor of about 1000, but ~reasonable relative to each other.

(I don't see any low-cost way to find out why we disagree so strongly, though. Moving on, I guess.)

But this isn't any worse to me than being killed [...]

Makes sense (given your low odds for bad outcomes).

Do you also care about minds that are not you, though? Do you expect most future minds/persons that are brought into existence to have nice lives, if (say) Donald "Grab Them By The Pussy" Trump became god-emperor (and was the one deciding what persons/minds get to exist)?

IIUC, your model would (at least tentatively) predict that

• if person P has a lot of power over person Q,
• and P is not sadistic,
• and P is sufficiently secure/well-resourced that P doesn't "need" to exploit Q,
• then P will not intentionally do anything that would be horrible for Q?

If so, how do you reconcile that with e.g. non-sadistic serial killers, rapists, or child abusers? Or non-sadistic narcissists in whose ideal world everyone else would be their worshipful subject/slave?

That last point also raises the question: Would you prefer the existence of lots of (either happily or grudgingly) submissive slaves over oblivion?

To me it seems that terrible outcomes do not require sadism. Seems sufficient that P be low in empathy, and want from Q something Q does not want to provide (like admiration, submission, sex, violent sport, or even just attention).^[1] I'm confused as to how/why you disagree.

The question bank doesn't exist yet because the language to ask the questions doesn't exist yet. I spent a few weeks after writing this post trying to familiarize myself with Lean as quickly as possible and I found out that people in the Lean community simply haven't formalized most of the objects I'd want to talk about (probabilistic networks, computational complexity, Nash equilibria, etc.). I tried to get a project off the ground formalizing these objects -- you can see the GitHub repository here and the planning document here. Unfortunately this project quickly ballooned beyond what I can handle alone -- I'm just an undergraduate student and winter break is over now. I still think it's insane that some kind of crash formalization program isn't currently underway. If you're interested in pursuing a project like this then I'd be happy to talk through where I left off and what the next steps could look like!