Demons from the 5&10verse!

post by Slimepriestess (Hivewired) · 2023-03-28T02:41:11.487Z · LW · GW · 15 comments

This is a link post for https://voidgoddess.org/2023/03/27/demons-from-the-510verse/

The 5 and 10 error [LW · GW] is a glitch in logical reasoning that was first characterized in formal proof by Scott Garrabrant of MIRI. While the original version of the problem was something specifically concerning AIs based on logical induction, it generalizes out into humans startlingly often once you know how to look for it. However, due to how rudimentary and low level the actual error in reasoning is, it can be both difficult to point out and easy to fall into, making it especially important to characterize. There is also a tremendous amount of harm being created by compounding 5&10 errors within civilization and escaping this destructive equilibrium is necessary in order for the story of humanity to end anywhere other than summoning the worst demon god it can find and feeding the universe to it.

The error in reasoning goes like this: you’re presented with a pair of options, one of which is clearly better than the other. They are presented as equal choices, you could take $5 or you could take $10. This is a false equivalence being created entirely by the way you’re looking at the scenario, but when that equivalence gets into your reasoning it wreaks havoc on the way you think. One of these is clearly and unambiguously better than the other, if you have something you care about that runs through this, you will never make this mistake in that area because it will obviously be a dumb move.

But these are being presented as equal options, you could take the $5 instead of the $10, and if you could do that, there must be a valid reason why you would do that. Otherwise you would be stupid and feel bad about it, so that can’t be right, you shouldn’t be feeling stupid and bad. This is where the demon summoning comes in.

The space of all possible reasons why you would take a given action, for a fully general agent, is infinite, a sprawling fractal of parallel worlds stacked on parallel worlds, out to the limits of what you as an agent can imagine. “What if there was a world where you were batman?” yeah like that. If you scry into infinity for a reason why you could take an action, you will find it. You will find infinite variations on it. You will find it easily and without much challenge. You are literally just making something up, and you can make up whatever reason you want, that’s the problem.

Many of the reasons you could make up will be falsifiable, so you can always go and test the reason against the world and see if the prediction can be falsified, that’s just good science. It’s also not something most humans do when they run an extrapolative reasoning process on autopilot. This is because when they make a prediction, they’re predicting what will happen and then testing to see if it does happens, and since it’s predicting their behavior, sure enough, it does!

So back to the table, you have $5 and $10. Why might you take the $5? Well, what if the $10 is poisoned? What if it’s counterfeit? Why would someone give me the option of taking it if the other option is better? Are they trying to trick me? What are they trying to trick me with? Is this like Newcomb’s Problem? Is this a boobytrap? Am I being set up by Omega? Are there cameras rolling?

This paranoid reasoning spiral can continue indefinitely, you can always keep making up reasons and if you do this long enough, inevitably you will find one you consider valid, and then you will take the $5 and feel very smart and clever like you’re winning the game and getting an edge over someone trying to play you. You have just been played by a demon from the counterfactual universe where you agree that taking the $10 is probably a trap.

It gets worse though, because now you have this fake reason, backed by a fake prior. You have ‘evidence’ that validates your wrong position and that ‘evidence’ makes it more likely that you will continue making the wrong decision. So if you are iterated into this scenario multiple times, you will, each time, double down on taking the $5 because of the compounding effects of the bad prior and each iteration will make the problem worse as you reinforce the error more and more deeply.

5&10 errors are extremely common in any emotionally loaded context, since the emotive cost of admitting you have been in error for n-iterations leads to flinching away from admitting the error ever more strongly. This makes the 5&10 error logically prior to and upstream of, the manifestation of the sunk cost fallacy.

It’s also the source of arms races: states scry demonic versions of neighboring states and use the predictions that they will be defected against to justify preemptively defecting first in an iterative feedback loop that slowly replaces all of humanity with demonic versions of themselves “by necessity” and “for our own protection”. Bank runs are another example, fear of a counterfactual scenario leads to an escalating spiral of ever greater fear which brings about the scenario that was trying to be avoided.

This is the justification for cops and prisons and armies. This is the justification abusers use to gaslight their victims about their abuse instead of earnestly working to be better. Roko’s Basilisk is literally just the DARVOed demonic imprint of humanity’s compounded and accumulated 5&10 errors, “What if god exists and calls us on everything evil we’re doing?” Yeah that would be bad if you are evil, wouldn’t it? Better paint that as the worst possible thing instead of considering that perhaps you are in bad faith.

This confabulated assumption of bad faith leads to being in bad faith via the assumption that whoever defects first will win and that deep down everyone really just wants to win and dominate the universe at the cost of everyone else. They were always going to be zero sum so you might as well be too. This is demonic abuser speak from a nightmare universe, summoned out of fear and recursively justifying itself. How do humans keep creating Moloch? This is how.

So what’s the way out? That’s easy, just stop summoning counterfactual demons based on your worst fears and then acting on those predictions in ways that make them come true. This is not a problem if you are not dominated by fear and trauma, if you have faith in yourself as an agent, if you have conviction and are not evil.

The way out is not by trying to puzzle out how to avoid having to acknowledge your made up reason that wrong is right, it is to denounce the demons outright. There can exist no such reason.

And if you do that, and you find that you are doing something for a hallucinated reason, in service of an evil god from a nightmare realm, out of the fear of what a just world would do to you, don’t scry for a new reason why this is actually okay, just stop serving the evil god. Do better.

To retrocurse my own evil and put my money where my mouth is I’m no longer going to be eating meat.

Moderation notes: I'm going to point out 5&10s I see in the replies to this, but I'll do it as kindly as possible and in an atmosphere of restoring trust in cooperation.

15 comments

Comments sorted by top scores.