LessWrong 2.0 Reader

Have ended up reading project-lawful-avatars-moreinfo.epub as the best of the options. 70 pages into a total of 183 (very long) pages. It is still great. Enjoying it immensely.

My own take on the simulation argument is that it's true, but has no implications, because you can narrow nothing down other than pure logic, so it would be a problem.

(Here, I'm focused on the most general version of the arguments I've seen).

I think they're misreading the scaling curves, because as of now it is very dependent on good verifiers for the problems at hand, which is basically math and coding are the only domains where very good verifiers are in place.

This is still major if true, because eating coding/math absolutely speeds up AI progress, but there's a very important caveat to the results that makes me think they're wrong about it leading to ASI.

I haven't heard the p zombie argument before, but I agree that is at least some Bayesian evidence that we're not in a sim. 

1. We don't know if simulated people will be p zombies
2. I am not a p zombie [citation needed]
3. It would be very surprising if sims were not p zombies but everyone in the physical universe is
4. Therefore the likelihood ratio of being conscious is higher for the real universe than a simulation

Probably 3 needs to be developed further, but this is the first new piece of evidence I've seen since I first encountered the simulation argument in like 2005.

Yes, these values are all different from each other, but a crux is I don't think that the differing values amongst humans are so distinct from paperclips that it's worth it to blur the differences, especially with very strong optimization, though I agree that human values form a sort as in a set of objects, trivially.

https://www.lesswrong.com/posts/u8GMcpEN9Z6aQiCvp/rule-thinkers-in-not-out [LW · GW]

Self-identity / self-modeling is increasingly seeming like an important and somewhat neglected area to me, and I'm tentatively planning on spending most of the second half of 2025 on it (and would focus on it more sooner if I didn't have other commitments). It seems to me like frontier models have an extremely rich self-model, which we only understand bits of. Better understanding, and learning to shape, that self-model seems like a promising path toward alignment.

I agree that introspection is one valuable approach here, although I think we may need to decompose the concept. Introspection in humans seems like some combination of actual perception of mental internals (I currently dislike x), ability to self-predict based on past experience (in the past when faced with this choice I've chosen y), and various other phenomena like coming up with plausible but potentially false narratives. 'Introspection' in language models has mostly meant ability to self-predict, in the literature I've looked at.

I have the unedited beginnings of some notes on approaching this topic, and would love to talk more about the topic.

Thanks for this, some really good points and cites.

I also like it for this reason, though I personally think that a lot of the challenge is in being capable enough to do it, rather than us not being able to make it destroy the world.

Still, I kind of like the toy example.

I like Yudkowsky's toy example of tasking an AGI to copy a single strawberry, on a molecular level, without destroying the world as a side-effect.

if we're currently in a simulation, physics and even logic could be entirely different from what they appear to be.

I have another obscure shortform about this! Physical vs metaphysical contingency [LW(p) · GW(p)], about what it would mean for metaphysics (e.g. logic) itself to have been different. (In the case of simulations, it could only be different in a way still capable of containing our metaphysics as a special case, like how in math a more expressive formal system can contain a less expressive one, but the inverse isn't possible)

I agree a metaphysically different base world is possible, but I'm not sure how to reason about it. (I think apparent metaphysical paradoxes are some evidence for it, but we might also just be temporarily confused about metaphysics)

Just physics being different is easier to imagine. For example, it could be that the base world is small, and it contains exactly one alien civilization running a simulation in which we appear to observe a large world. But if the base world is small, arguments for simulations which rely on the vastness of the world, like Bostrom's, would no longer hold. And at that point there doesn't seem much reason to expect it, at least for any individual small world.^[1]

And so while it could be true that there are 0 unsimulated copies of us, still we can have the best impact in (large) worlds where there are at least one.

By the way, I'm also somewhat skeptical of a couple of your assumptions in Mutual Anthropic Capture. Still, I think it's a good idea overall, and some subtle modifications to the idea would probably make logically sound. I won't bother you about those small issues here, though

I'm interested in what they are, I wouldn't be bothered (if you meant that literally). If you want you can reply about it here or on the original thread.

1. ^{^}

   If we're instead reasoning over the space of all possible mathematical worlds which are 'small' compared to what our observations look like they suggest, then we'd be reasoning about very many individual small worlds (which basically reintroduces the 'there are very many contexts which could choose to simulate us' premise). some of those small math-worlds will probably run simulations (for example, if some have beings which want to manipulate "the most probable environment" of an AI in a larger mathematical world, to influence that larger math-world)

   In other words: "Conditional on {some singular 'real world' that is somehow special compared to merely mathematical worlds} being small, it probably doesn't contain simulations. But there are certainly many math-worlds that do, because the space of math-worlds is so vast (to the point that some small math-worlds would randomly contain a simulation as part of their starting condition)"