oh yeah, that's true. I guess I thought of it in terms of bullet biting because they are the most propice to the most dangerous actions

I don't need any defense mechanisms against these ones, because I can just see the fallacy in the arguments.

In one description, Blaise Pascal is accosted by a mugger who has forgotten his weapon. However, the mugger proposes a deal: the philosopher gives him his wallet, and in exchange the mugger will return twice the amount of money tomorrow. Pascal declines, pointing out that it is unlikely the deal will be honoured. The mugger then continues naming higher rewards, pointing out that even if it is just one chance in 1000 that he will be honourable, it would make sense for Pascal to make a deal for a 2000 times return. Pascal responds that the probability for that high return is even lower than one in 1000. The mugger argues back that for any low probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet – and given human fallibility and philosophical scepticism a rational person must admit there is at least some non-zero chance that such a deal would be possible. In one example, the mugger succeeds by promising Pascal 1,000 quadrillion happy days of life. Convinced by the argument, Pascal gives the mugger the wallet.

When a mugger promises me to return twice my money tomorrow, I can see that it is almost certainly a hoax. There is maybe a one in a million chance he's saying the truth. The expected value of the wager is negative. If he promises a 2000x return, that's even less likely to be true. I estimate it as one in two billion. The expected value is still the same, and still negative. And so on, the more lavish reward the mugger promises, the less likely I am to trust him, so the expected value can always stay negative.

Roko's basilisk

Why don't I throw themselves into a research institute dedicated to building the basilisk? Because there is no such institute, and if someone seriously tried to build one, they'd just end up in prison or a mental asylum for extortion. Unless they are keeping their work secret, but then it's just an unnecessarily convoluted way of building an AI that kills everyone. So there is no reason why I would want to do that.

My point is that if you don't have some of those general / meta beliefs described in this post, you will generally take much worse decisions, in a way that will often be known by you intuitively, but not by your explicit reasoning (which is dangerous if you don't take your intuitive warning signal seriously).

Let's assume you're someone that doesn't know the answer to the question I asked (or the information in the specific answer I gave).

Here are examples of what could go wrong.

Example 1

If you believe that a discontinuity in consciousness means you die, and when consciousness is reestablished in the brain, another mind is instantiated that is a copy of you. Then you might decide to not go back to sleep until you actually, biologically die from sleep deprivation.

While this could be the actual optimal choice, even taking into account this post, it seems likely to me that taking into account information in this post could change one's mind from 'not sleeping at all' to 'keeping normal sleeping habit'.

Some approach to moral uncertainty might actually recommend sleeping even if you're rather confident it will kill you because: % you care about discontinuity * how long you can go without sleeping << % you don't care about discontinuities * how long you can live if you sleep.

But if you don't know about how to integrate uncertainty at the model level in your reasoning, then you might just act based on your belief that sleep kills, and so stop sleeping. This error mode could severely affect a lot of people around me based on the 'object-level' beliefs I see shared around.

I've written more about this here, but I have made the post private for now as I'm revisiting whether it contains info-hazard.

Example 2

If you don't see any error with Pascal's mugging, and so you decide to act on its logical implications, then a mugger might rob you of everything, and render you a complete slave.

Actually, I'm not sure if I have a defense mechanism to propose for this one, beside knowing the resolution of the problem before / at the same time than being introduced to the problem. But one could argue that "your intuitions that this is wrong" would be a good defense mechanism against explicit reasoning going astray.

Can you give some examples?

Like a belief that you've discovered a fantastic investment opportunity, perhaps?