Pascal's Mugging Solved

"Solving" Pascal's Mugging involves giving an explicit reasoning system and showing that it makes the right decision.

It's not enough to just say "your confidence has to go down more than their claimed reward goes up". That part is obvious. The hard part is coming up with actual explicit rules that do that. Particularly ones that don't fall apart in other situations (e.g. the decision system "always do nothing" can't be pascal-mugged, but has serious problems).

Another thing not addressed here is that the mugger may be a hypothetical. For example, if the AI generates hypotheses where the universe affects 3^^^^3 people then all decisions will be dominated by these hypotheses because their outcomes outweigh their prior by absurd margins. How do you detect these bad hypotheses? How do you penalize them without excluding them? Should you exclude them?

Please give a more concrete situation with actual numbers and algorithms.

This article would benefit from working through a concrete example.

If you become super-exponentially more skeptical as the mugger invokes super-exponentially higher utilities, how do you react if the mugger tears the sky asunder?

So I was pattern matching this as an argument for "why the probability decreases more than we previously acknowledged, as the threat increases", but that isn't what you're going for. Attempting to summarize it in my own words:

There are three relevant events: (A) the threat will not happen; (B) not giving in to blackmail will trigger the threat; (C) giving in to blackmail will trigger the threat (or worse). As the threat increases, P(B) and P(C) both decrease, but P(C) begins to dominate P(B).

Is this an accurate summary?

I like your solution to pascals mugging but as some people mentioned it breaks down with superexponential numbers. This is caused by the extreme difficulty to do meaningful calculations once such a number is present (similar to infinity or a division by zero).

I propose the following modification:

• Given a Problem that contains huge payoffs or penalties, try common_laws solution.
• Should any number above Gogol be in the calculation, refuse to calculate!
• Try to reformulate the problem in a way that doesn't contain such a big number.
• Should this fail, do nothing.

I would go so far as to treat any claim with such numbers in it as fictional.

Another LW classic containing such numbers is the Dust Speck vs. Torture paradox. I think that just trying to calculate in the presence of such numbers is a fallacy. Has someone formulated a Number-Too-Big-Fallacy already?

As the amount of harm threatened gets larger, the probability that the mugger is maximizing approaches unity.

It seems entirely possible that some finite nonzero fraction of "matrix lords" is capable of carrying out arbitrarily large threats, or of providing arbitrarily large rewards (provided that the utility function by which you judge such things is unbounded).

As the probability that the mugger is engaged in maximizing approaches unity, the likelihood that the mugger’s claim is true approaches zero.

It is possible that the mugger is maximizing while still telling the truth.

Pascal's mugging against an actual opponent is easy. If they are able to carry out their threat, they don't need anything you would be able to give them. If the threat is real, you're at their mercy and you have no way of knowing if acceding to their demand will actually make anyone safer, whereas if he's lying you don't want to be giving resources to that sort of person. This situation is a special case of privileging the hypothesis, since for no reason you're considering a nearly impossible event while ignoring all the others.

If we're talking about a metaphor for general decision-making, eg an AI who's actions could well affect the entirety of the human race, it's much harder. I'd probably have it ignore any probabilities below x%, where x is calculated as a probability so small that the AI would be paralyzed from worrying about the huge number of improbable things. Not because it's a good idea, but because as probability approaches zero, the number of things to consider approaches infinity yet processing power is limited.

It seems to me that you've pretty much solved the literal interpretation of the hypothetical, putting into words what everyone was already thinking.

The more relevant one is where you generated the probability and utility estimates yourself and are trying to figure out what to do about it.

I have a general question about Pascal's mugging: Does the problem still work if the mugger demands substantially more than five dollars?

Let's say the Pascal's Mugger demands you accept a Simpleton Gambit. Is there any line of thought that says 'I believe a properly constructed mugging could mug me for 5 dollars, but couldn't mug me into a Simpleton Gambit?'

My current understanding is that if you are a utilitarian who accepts a Pascal's Mugger can mug you at all, then you accept a Pascal's Mugger can mug you for anything by properly constructing the math behind the mugging. (Although if the mugger is demanding you accept a Simpleton Gambit, it might be more accurate to call that entity a Pascal's Slaver.)

But I'm only recently came to that understanding and it doesn't directly address whether or not you should accept Pascal's muggings, just what seems to be a plausible consequence if you are a utilitarian who does, so I thought I should verify it first.

Reference note for Simpleton Gambits: http://wiki.lesswrong.com/wiki/Simpleton_Gambit

Short version:

Even just considering the a priori probability of them honestly threatening you before you even take into account that they threatened you, it's still enough to get Pascal mugged. The probability that they're lying increases, but not fast enough.

Long version, with math:

Consider three possibilities:

A: Nobody makes the threat

B: Someone dishonestly makes the threat

C: Someone honestly makes the threat

Note that A, B, and C form a partition. That is to say exactly one of them is true. Technically it's possible that more than one person makes the threat, and only one of them is honest, so B and C happen, so let's say that the time and place of the threat are specified as well.

C is highly unlikely. If you tried to write a program simulating that, it would be long. It would still fit within the confines of this universe. Let's be hyper-conservative, and call it K-complexity of a googol.

There is also a piece of evidence E, consisting of someone making the threat. E is consistent with B and C, but not A. This means:

P(A&E) = 0, P(B&E) = P(B), P(C&E) = P(C)

Calculating from here:

P(C|E) = P(C&E)/P(E)

= P(C)/P(E)

P(C)

= 2^-googol

So the probability of the threatener being honest is at least 2^-googol. Since the disutility of not paying him if he is honest more than 2^googol times the utility of paying him, it's worthwhile to pay.

Let's consider two scenarios S1 and S2, with S1 having a lesser harm and S2 a greater harm. M1 and M2 are the events where a mugger presents S1 and S2 respectively. Our bayesian update for either takes the form

%20=%20\frac{P(M1|S1)P(S1)}{P(M1|S1)P(S1)%20+%20P(M1|\neg%20S1)P(\neg%20S1)})

Your argument is that (edit: maybe not; I'm going to leave this comment here anyway, hope that's OK)

)

is greater than

)

and this increase offsets increases in the utility part of the expected utility equation. I'm pretty tired right now, but it seems possible that you could do some algebra and determine what the shape of the function determining this conditional probability as a function of the harms amount would have to be in order to achieve this. In any case, while it might be convenient for your updating module to have the property that this particular function possesses this particular shape, it's not obvious that it's the correct way to update. (And in particular, if you were programming an AI and you gave it what you thought was a reasonably good Bayesian updating module, it's not obvious that the module would update the way you would like on this particular problem, or that you could come up with a sufficiently rigorous description of the scenarios under which it should abandon its regular updating module in order to update in the way you would like.)

Even if this argument works, I'm not sure it completely solves the problem, because it seems possible that there are scenarios with extreme harms and relatively high (but still extremely miniscule) prior probabilities. I don't know how we would feel about our ideal expected utility maximizer assigning most of its weight to some runaway hypothetical.

Another thought: if a mugger had read your post and understood your argument, perhaps they would choose to mug for smaller amounts. So would that bring ) above )? What if the mugger can show you their source code and prove that for them the two probabilities are equal? (E.g. they do occasionally play this mugging strategy and when they do they do it with equal probability for any of three harm amounts, all astronomically large.)

This solutions seems to implicitly assume that all real needs are pretty moderate. That is the only plausible reason to state a meganumber class high utility is to beat someone elses number. A matrix lord could have a different sense of time in their "home dimension" and for all I know what I do might be the culmination of bet involving the creation and destruction of whole classes of universes.

Also why what the mugger says have anythinhg to do how big of a threat the conversation is? If someone would come up to me and say "give me 5$ or I will do everything in my power to screw with you" without saying how much screwage would happen or what their power level is, it wouldn't seem that problematic. Why would the threat be more potent with stating that information? It seems it should limit the threat it poses. It would appear to me that what people say only has relevance in the limits on how I have established agenthood, ie it maxes out. If I would really be on the look out for freaky eldrich gods I should worry that the sky would fall on me - ie there is no need for the sky/might to state any threat for them to be one.

Okay, I'll play along. Lets see where this takes us. The math here is not going to be strict and I'm going to use infinities to mean "sufficiently large", but it will hopefully help us make some sense of this proposition.

P(W) = Probability of the mechanism working P(T) = Probability that the mugger is being truthful P(M) = Probability that the mugger is "maximizing" h = Amount of harm threatened a = Amount being asked for in the mugging

We want to know if PT(h) PW h > a for sufficiently large h, without really specifying a.

As the amount of harm threatened gets larger, the probability that the mugger is maximizing approaches unity.

Since you're claiming that the probability that the mugger is maximizing is dependent on the amount of harm threatened, we can rewrite P(M) as a function of h, so lets call it PM(h) such that lim h->∞ PM(h) = 1

As the probability that the mugger is engaged in maximizing approaches unity, the likelihood that the mugger’s claim is true approaches zero.

We can compose the functions to get our relationship between PM and PT: lim h->∞ PT(PM(h)) = 0 which simplifies to: lim h->∞ PT(h) = 0

If we express the original question using this notation, we want to know if PT(h) PW h > a for large enough values of h. If we take our limits, we get: lim h->∞ PT(h) PW h > a which evaluates to 0 PW ∞ > a

This doesn't really work, since we have 0 * ∞, but remember that our infinity means "sufficiently large" and our zero therefore has to mean "very very low probability".

...the evidence that the mugger is maximizing can lower the probability below that of the same harm when no mugger has claimed it. [...] the claim can become less believable than if it hadn’t been expressed.

This part here has to mean that for sufficiently large h, PT(h) PW h < PW h, which is not hard to believe since it's just adding another probability, but it also doesn't solve the original problem of telling us that we can be justified in not paying the mugger. In order to do that, we'd need some assurance that for sufficiently large h and some a, PT(h) PW h < a. To get that assurance, PT(h) h would have to have some upper bound, and the theory you presented doesn't give us that.

It's a fun theory to play with and I would encourage you to try to flesh it out more and see if you can find a good mathematical relationship to model it.