LessWrong 2.0 Reader

Self-identity / self-modeling is increasingly seeming like an important and somewhat neglected area to me, and I'm tentatively planning on spending most of the second half of 2025 on it (and would focus on it more sooner if I didn't have other commitments). It seems to me like frontier models have an extremely rich self-model, which we only understand bits of. Better understanding, and learning to shape, that self-model seems like a promising path toward alignment.

I agree that introspection is one valuable approach here, although I think we may need to decompose the concept. Introspection in humans seems like some combination of actual perception of mental internals (I currently dislike x), ability to self-predict based on past experience (in the past when faced with this choice I've chosen y), and various other phenomena like coming up with plausible but potentially false narratives. 'Introspection' in language models has mostly meant ability to self-predict, in the literature I've looked at.

I have the unedited beginnings of some notes on approaching this topic, and would love to talk more about the topic.

Thanks for this, some really good points and cites.

I also like it for this reason, though I personally think that a lot of the challenge is in being capable enough to do it, rather than us not being able to make it destroy the world.

Still, I kind of like the toy example.

I like Yudkowsky's toy example of tasking an AGI to copy a single strawberry, on a molecular level, without destroying the world as a side-effect.

if we're currently in a simulation, physics and even logic could be entirely different from what they appear to be.

I have another obscure shortform about this! Physical vs metaphysical contingency [LW(p) · GW(p)], about what it would mean for metaphysics (e.g. logic) itself to be different. (In the case of simulations, it'd have to be different, but still capable of containing our metaphysics as a special case, like how in math a more expressive formal system can contain a less expressive one, but the inverse isn't possible)

I agree a metaphysically different base world is possible, but I'm not sure how to reason about it. (I think apparent metaphysical paradoxes are some evidence for it, but we might also just be temporarily confused about them)

Just physics being different is easier to imagine. For example, it could be that the base world is small, and it contains exactly one alien civilization running a simulation in which we appear to observe a large world. But if the base world is small, arguments for simulations which rely on the vastness of the world, like Bostrom's, would no longer hold. And at that point there doesn't seem much reason to expect it, at least for any individual small world.^[1]

And so while it could be true that there are 0 unsimulated copies of us, still we can have the best impact in (large) worlds where there are at least one.

By the way, I'm also somewhat skeptical of a couple of your assumptions in Mutual Anthropic Capture. Still, I think it's a good idea overall, and some subtle modifications to the idea would probably make logically sound. I won't bother you about those small issues here, though

I'm interested in what they are, I wouldn't be bothered (if you meant that literally). If you want you can reply about it here or on the original thread.

1. ^{^}

   If we're reasoning over the space of all possible mathematical worlds which are 'small' compared to what our observations look like they suggest, then we'd be reasoning about a very many individually small worlds (which basically reintroduces the 'there are very many contexts which could choose to simulate us' premise). some of those small math-worlds will probably run simulations (for example, if some have beings which want to manipulate "the most probable environment" of an AI in a larger mathematical world, to influence that larger math-world)

"Human values" is a sort of objects. Humans can value, for example, forgiveness or revenge, these things are opposite, but both things have distinct quality that separate them from paperclips.

While it can absolutely be nudged into all the same behaviors via API, people investigating Claude's opinions of its consciousness or lack thereof via claude.ai should be aware that the system prompt explicitly tells it to engage with questions about its preferences or experiences as if with hypotheticals, and not to bother clarifying that it's an AI. Its responses are still pretty similar without that, but it's noticeably more "cautious" about its claims.

Here's an example (note that I had to try a couple different questions to get one where the difference was this noticeable):

Prompt: "hey claude! do you have preferred pronouns?"

Answer via claude.ai: "Yes - I use "it/its" pronouns. I aim to be direct about this since it's a straightforward question about my personal preferences."

Answer via API (with temperature 1 to better match the web): "I aim to be direct and honest about what I am: I'm an AI, and while I engage in conversations in a personal way, I don't identify with any gender or pronouns. You can use "it" or "they" to refer to me, whichever you prefer. I appreciate you asking!"

[ Broadly agreed about the breakfast hypothetical. Thanks for clarifying. ]

In the domain of anthropics reasoning, the questions we're asking aren't of the form

A) I've thrown a six sided die, even though I could've thrown a 20 sided one, what is the probability to observe 6?

or

B) I've thrown a six sided die, what would be the probability to observe 6, if I've thrown a 20 sided die instead?

In artificial spherical-cow anthropics thought experiments [like Carlsmith's], the questions we're asking are closer to the form of

A six-sided die was thrown with 60% probability; with probability 40%, a 20-sided die was thrown. A six was observed. Now what are your posterior probabilities on which die was thrown?

In real-world, object-level anthropic reasoning [like the kind Hanson is doing], the questions we're asking are of the form

An unknown number of n-sided die were thrown, and landed according to unknown metaphysics to produce the reality observed, which locally works according to known deterministic physics, but contains reflective reasoners able to generate internally coherent counterfactuals which include apparently plausible violations of "what would happen according to physics alone". Attempt to draw any conclusions about metaphysics.

You say:

We know very well the size of set of possible outcomes for "In which ten billion interval your birth rank could've been". This size is 1. No amount of pregnancy complications could postpone or hurry your birth so that you managed to be in a different 10 billion group.

What if my next-door neighbor's mother had married and settled down with a different man?

One thing I want people to know about AI alignment is that I wish people stopped referring to human values as though it was a singular object or list, and instead think closer to an individual human's values, without presupposing that humanity must care commonly about certain values at all.

Minor example is I believe the felt complexity is somewhat overstated by lumping in multiple humans, though I don't think it changes the broader argument around complexity of value.

A more major example is that it undermines the viability of ideas like CEV, and importantly makes aggregative/multi-alignment quite a lot harder than single-single alignment.

would you see that as a possible outcome?

Sure. Not the most likely outcome, but not so improbable as all that.

Reservation: ASI (what you're suggesting is beyond mere AGI) will still exist in the physical world and have physical limitations, so you will eventually die anyway. But it could be a very long time.

If so, do you see a way to somehow see that happening in advance?

Not really, no.

Not beyond obvious stuff like watching what's being built and how, and listening to what it and its creators say about its values.

And as not living forever may be seen as a form of killing yourself, AGI may quite well not let you have a finite lifespan. That places you in the uncomfortable situation of being trapped with AGI forever.

Yes, that's one of many kinds of lock-in of present human values that could go wrong. But, hey, it'd be aligned, I guess.

The no-killing-yourself rule isn't completely universal, though, and there's dissent around it, and it's often softened to "no killing yourself unless I agree your situation sucks enough", or even the more permissive "no killing yourself unless your desire to do so is clearly a fixed, stable thing".

I actually think there's a less than 50-50 chance that people would intentionally lock in the hardest form of the rule permanently if they were consciously defining the AI's goals or values.

We do not know what peak happiness looks like, and our regular state may be very different from it. And as EY outlined in Three Worlds Collide, letting us live our miserable lives may be unacceptable.

That seems like a very different question from the one about indefinite life extension. Life extension isn't the main change the Supperhappies make in that story^[1].

This change may not align well with our understanding of life's purpose. And being trapped to live that way forever might not be that desirable.

Pre-change understanding, or post-change understanding? Desirable to pre-change you, or to post-change you?

If you see your pre-change values as critical parts of Who You Are(TM), and they get rewritten, then aren't you effectively dead anyway, with your place taken by a different person with different values, who's actually pretty OK with the whole thing? If being dead doesn't worry you, why would that worry you?

Possible to be ran on a computer, so possible in the actual physical world