LessWrong 2.0 Reader

Cf. Moneyball.

John Carpenter's The Thing.

I'm confused about why we would expect the misaligned behavior to be displayed in the Shoggoth rather than the Face. For example, let's take the setting from the AI control paper, where a misaligned model is trying to insert a backdoor in a coding question, despite it passing a series of unit tests. This could be incentivized in the training loop if e.g. the reward supervision is weak and the backdoor grants the model access to additional resources/compute. I can see a few strategies succeeding:

1. The Shoggoth speculates on different ways to insert a backdoor, and the Face writes a solution using one of these backdoors.
2. The Shoggoth outputs the correct solution to the coding question, and the Face modifies it to insert a backdoor.
3. Some combination of 1) and 2), for instance the Shoggoth writes several versions of the function, one of which contains a backdoor, and the Face returns the version with the backdoor. 

All of these result in the Face outputting potentially same backdoored function, and should be equally rewarded by the reward model. Why should we expect that 2) is less likely than 1) or 3)? 

I have a different concern than most people.

An AI that follows the english text rather than the reward signal on the chess example cannot on that basis be trusted to do good things when given english text plus human approval reward in the real world. This is because following the text in simple well-defined environments is solving a different problem than following values-laden text in the real world.

The problem that you need to solve for alignment in the real world is how to interpret the world in terms of preferences when even we humans disagree and are internally inconsistent, and how to generalize those preferences to new situations in a way that does a good job satisfying those same human preferences (which include metapreferences). Absent a case for why this is happening, a proposed AI design doesn't strike me as dealing with alignment.

When interpreting human values goes wrong, the AI's internal monologue does not have to sound malevolent or deceptive. It's not thinking "How can I deceive the humans into giving me more power so I can make more paperclips?" It might be thinking "How can I explain to the humans that my proposal is what will help them the most?" Perhaps, if you can find its internal monologue describing its opinion of human preferences in detail, they might sound wrong to you (Wait a minute, I don't want to be a wirehead god sitting on a lotus throne!), or maybe it's doing this generalization implicitly, and just understands the output of the paraphraser slightly differently than you would.

To achieve magic, we need the ability to merge minds, which can be easily done with programs and doesn't require anything quantum.

I don't see how merging minds not across branches of the multiverse produces anything magical.

If we merge 21 and 1, both will be in the same red room after awakening.

Which is isomorphic to simply putting 21 to another red room, as I described in the previous comment. The probability shift to 3/4 in this case is completely normal and doesn't lead to anything weird like winning the lottery with confidence.

Or we can just turn off 21 without awakening, in which case we will get 1/3 and 2/3 chances for green and red.

This actually shouldn't work. Without QI, we simply have 1/2 for red, 1/4 for green and 1/4 for being turned off.

With QI, the last outcome simply becomes "failed to be turned off", without changing the probabilities of other outcomes

The interesting question here is whether this can be replicated at the quantum leve

Exactly. Otherwise I don't see how path based identity produces any magic. For now I think it doesn't, which is why I expect it to be true.

Now the next interesting thing: If I look at the experiment from outside, I will give all three variants 1/3, but from inside it will be 1/4, 1/4, and 1/2.

Which events you are talking about, when looking from the outside? What statements have 1/3 credence? It's definitely not "I will awake in red room", because it's not you who are too be awaken. For the observer it has 0 probability.

On the other hand, an event "At least one person is about to be awaken in red room" has probability 1, for both the participant and the observer. So what are you talking about? Try to be rigorous and formally define such events.

The probability distribution is exactly the same as in Sleeping Beauty, and likely both experiments are isomorphic.

Not at all! In Sleeping Beauty on Tails you will be awaken both on Monday and on Tuesday. While here if you are in a green room you are either 21 or 22, not both.

Suppose that 22 get their arm chopped off before awakening. Then you you have 25% chance to lose an arm while participating in such experiment. While in Sleeping Beauty, if your arm is chopped off on Tails before Tuesday awakening, you have 50% probability to lose it, while participating in such experiment.

Interestingly, in the art world, path-based identity is used to define identity

Yep. This is just how we reason about identities in general. That's why SSSA appears so bizarre to me - it assumes we should be treating personal identity in a different way, for no particular reason.

Film: The Big Short

Rationality Tie-in: This is a film about the 2008 Financial Market Crash, and tells the stories of...

The Big Short is the filmic equivalent of my spirit-animal.

Rationality writings it is connected to: 

I thought this would be hard, but actually it ties into so much.

"Harry Potter and the Methods of Rationality: Part 1" (2026) by Daniel Radcliffe

The Truman Show: Great depiction of crisis of faith, noticing your confusion, and generally is about figuring out the truth.

Most relevant sequence posts: Crisis of Faith [LW · GW], Lonely Dissent [LW · GW]

Moneyball

The Martian