[SEQ RERUN] Pascal's Mugging: Tiny Probabilities of Vast Utilities

I have an idea for how this problem could be approached:

any sufficiently powerful being with any random utility-function may or may not exist. It is perfectly possible that our reality is actually overseen by a god that rewards and punishes us for whether we say an even or odd number of words in our life or something equally arbitrary. The likelyhood of the existence of each of these possible beings can be approximated using Solomonoff induction.

I assume that most simulations run by such hypothetical beings wherein we could find ourselves in such a situation would be run by beings who either (1) have no interest in us at all (in which case the Mugger would most likely be a human), (2) are interested in an entirely unpredictable thing resulting from their alien culture, or (3) are interacting with us purely to run social experiments. After all, they would have nothing in common with us and we would have nothing they could possibly want. It would therefore, in any case, be virtually impossible to guess at their possible motivations, as it would be a poorly run social experiment if we could (assuming option three is true).

I would now argue that the existence of Pascal's Mugger does not influence the probability of the existence of a being that would react negatively (for us) to us not giving the 5$ anymore than it influences the probability of the existence of a being with an opposite motivation. The Mugger is equally likely to punish you for being so gullible as he is to punish you for not giving money to someone who threatens you.

Of course none of this takes into consideration how likely the various possible beings are to actually carry out their threat, but that doesn't change anything important about this argument, I think.

In essence, my argument is that such powerful hypothetical beings can be ignored because we have no real reason to assume they have a certain motivation rather than the opposite. Giving the Mugger 5$ is just as likely to save us as shooting the Mugger in the face is. Incidentally, adopting the later strategy will greatly reduce the chance that somebody actually tries to do this.

I realize that this argument seems kind of flawed because it assumes that it really is impossible to guess at the being's motivation but I can't see how this could be done. It's always possible that the being just wants you to think that it wants x, after all. Who can tell what might motivate a mind that is large enough to simulate our universe?

An AI should treat a Pascal's Mugger as an agent trying to arbitrarily gain access to it's root systems without proper authority, or phrased more simply, an attack.

To explain why, consider this statement in the original article:

But a silicon chip does not look over the code fed to it, assess it for reasonableness, and correct it if not. An AI is not given its code like a human servant given instructions. An AI is its code. What if a philosopher tries Pascal's Mugging on the AI for a joke, and the tiny probabilities of 3^^^^3 lives being at stake, override everything else in the AI's calculations? What is the mere Earth at stake, compared to a tiny probability of 3^^^^3 lives?

If something is allowed to override EVERYTHING on a computer, it seems functionally identical to saying that it has root access.

Since Pascal's Mugging is commonly known and discussed on the internet, having it be equivalent to a root password would be a substantial security hole, like setting your root password to "password"

An AI would presumably have to have some procedure in case someone was attempting unauthorized access. That procedure would need to trigger FIRST, before considering the argument on the merits. Once that procedure is triggered, there argument is no longer being considered on the merits, it is being considered as an attack. Saying "Well, but what if there REALLY ARE 3^^^^3 lives at stake?" seems to be equivalent to saying "Well, but what if the prince of Nigeria REALLY IS trying to give me 1 million dollars according to the email in my spam box?"

There's probably something that I'm missing, so sorry if this solution has already been posted in the original thread. I don't really have the "oomph" the read them all... Anyway, hasn't this class of problems already been solved in chapter 5 of Jaynes' book?

If the AI has some tiny probability that the data he has received originated through some kind of deception, I think it's only sensible that the hypothesis that the mugger is lying steals all the probability mass in the posterior distribution, at least linearly with the number of people he claims he can affect (but I would say exponentially).

The expected utility shouldn't really be calculated on the posterior of the hypothesis "mugger possess magical power" but on the posterior of "mugger can affect the Matrix + mugger is lying".

ETA: This allows you to control the posterior probability of the hypothesis independently from the claim of the mugger, thereby shielding the AI from acting on enormous disutility depending only from slightly less enormous improbability.

Has Eliezer come up with a solution to this?

I have always failed to see what problem, and this confuses me greatly. To me it seams obvious that any sane entity would be dominated by considerations like this.

In this particular situation thou, the probability increase of possibly donating the money to SIAI and increasing the probability of a friendly singularity that can hack out of the matirx and use the computing power to create 4^^^^4 units of fun way outweighs it thou. And since the size of the threat correlates with the size of this reward it'll always increase so that this is true.

More here, where I ask about the "solution" put forth by Robin Hanson.

Since then the only attempted "solution" I know of is outlined in this post by Holden Karnofsky from GiveWell.

Also see this comment by Eliezer and the post he mentions. I didn't know about the Lifespan Dilemma before reading that comment. It seems to be much more worrying than Pascal's Mugging. I haven't thought about it much, but to just arbitrarily refuse some step seems to be the best heuristic so far.

Physicists deal with this issue daily, and they invented renormalization and cutoff techniques to make divergent things converge. This has been discussed before, not sure why it won't work.