Pascal's Mugging - Penalizing the prior probability?

The idea behind Pascal's mugging is that the complexity penalty to a theory of the form "A person's decision will have an affect on the well being of N people" grows assymptotically slower than N. So given weak evidence (like a verbal claim) that narrows down the decision and effect to something specific, and large enough N, the hugeness of the expected payoff will overcome the smallness of the probability.

Hanson's idea is that given that "A person's decision will have an affect on the well being of N people" the prior probability that you, and not someone else, is the person who gets to make that decision is 1/N. This gets multiplied by the complexity penalty, and we have the probability of payoff getting smaller faster than the payoff gets bigger.

This is all very convenient for us humans, who value things similar to us. If a paperclip maximizing AGI faced a Pascal's Mugging, with the payoff in non-agenty paperclips, it would assign a much higher probability that it, and not one of the paperclips, makes the crucial decision. (And an FAI that cares about humans faces a less extreme version of that problem.)

To be more specific, if a stranger approached me, offering a deal saying, "I am the creator of the Matrix. If you fall on your knees, praise me and kiss my feet, I'll use my magic powers from outside the Matrix to run a Turing machine that simulates 3^^^^3 copies of you having their coherent extrapolated volition satisfied maximally for 3^^^^3 years." Why exactly would I penalize this offer by the amount of copies being offered to be simulated?

There will be a penalty, and a large one, but it isn't going to be directly dependent on the number of people. Compare the following:

A: There is an avatar from outside the Matrix who is able and possibly willing to simulate 3^^^^3 copies of you.
B: There is an avatar from outside the Matrix who is able and possibly willing to simulate BusyBeaver(3^^^^3) copies of you.

P(A) is greater than P(B). But it is not even remotely BusyBeaver(3^^^^3)/3^^^^3 times greater than P(B). Most of the improbability is in there ridiculous out of the matrix trickster, not the extent of his trickery. If it is lack symmetry that is taken to be the loophole it is easy enough to modify the scenario to make things symmetrical. Trickster: "I'll simulate this universe up to now a gazillion times and after now the sims get lots of utility". Concern about how surprising it is for you to be one with the power becomes irrelevant.

Is the Pascal's mugging thought experiment a "reduction to the absurd" of Bayes’ Theorem in combination with the expected utility formula and Solomonoff induction?

I consider expected utility maximisation to be a preference. A fairly sane sounding preference but still just as 'arbitrary' as whether you prefer to have mass orgasms or eat babies. (The 'expected' part, that is.)

I think the main problem with Hanson's solution is that it relies on the coincidence that the objects with moral worth are the same as the objects that can make decisions. If glargs were some object with moral worth but not the ability to make decisions, then finding yourself able to determine the fate of 3^^^3 glargs wouldn't be unusually asymmetric, and so wouldn't have a probability penalty.

Hm... I've been thinking about it for a while (http://www.gwern.net/mugging) and still don't have any really satisfactory argument.

Now, it's obvious that you don't want to scale any prior probability more or less than the reward because either way involves you in difficulties. But do we have any non-ad hoc reason to specifically scale the size of the reward by 1/n, to regard the complexity as equal to the size?

I think there may be a Kolmogorov complexity argument that size does equal complexity, and welcome anyone capable of formally dealing with it. My intuition goes: Kolmogorov complexity is about lengths of strings in some language which map onto outputs. By the pigeonhole principle, not all outputs have short inputs. But various languages will favor different outputs with the scarce short inputs, much like a picture compressor won't do so hot on music files. 3^^^^3 has a short representation in the usual mathematical language, but that language is just one of indefinitely many possible mathematical languages; there are languages where 3^^^^3 has a very long representation, one 3^^^^3 long even, or longer still! Just like there are inputs to zip which compress very well and inputs which aren't so great.

Now, given that our particular language favors 3^^^^3 with such a short encoding rather than many other numbers relatively nearby to 3^^^^3, doesn't Pascal's mugging start looking like an issue with our language favoring certain numbers? It's not necessary to favor 3^^^^3, other languages don't favor it or actively disfavor it. Those other languages would seem to be as valid and logically true as the current one, just the proofs or programs may be longer and written differently of course.

So, what do you get when you abstract away from the particular outputs favored by a language? When you add together and average the languages with long encodings and the ones with short encodings for 3^^^^3? What's the length of the average program for 3^^^^3? I suggest it's 3^^^^3, with every language that gives it a shorter encoding counterbalanced by a language with an exactly longer encoding.

As you hint at, any damn decision you could possibly make has at least a 1/3^^^3 chance of causing at least 3^^^3 utility (or -3^^^3 utility). Talking about muggers and simulations and anthropics and stuff is just a distraction from the main problem, which is that as far as I know we don't yet have a very good justification for pretending that there's any particular straightforward symmetry-enforcing (meta-)rule keeping utility from swamping probabilities.

Unrelated observation: That we don't seem to have a problem with probabilities swamping utility might mean that the fact that utility is additive where probabilities are multiplicative is messing with our intuitions somehow.

The solution is flawed. What if he instead says "I will give a single person 3^^^3 utilons of pain"?

Anyway, the idea is that the probability of that situation can't go above 3^^^3 to one against. You'd be one of the people he creates, not the one he's asking.

XiXiDu, I'd like to point out how positively surprised I am at what I personally see as a sharp upturn in the general quality of your thoughts, comments, and posts since I've been paying attention to them. In some somewhat low-level part of my brain you've gone from "possibly well-intentioned person who unfortunately cannot make a coherent argument" (sorry!) to "person whose posts and ideas often make me pause and think". That doesn't sound like much but given human psychology it probably is.

"I am the creator of the Matrix. If you fall on your knees, praise me and kiss my feet, I'll use my magic powers from outside the Matrix to run a Turing machine that simulates 3^^^^3 copies of you having their coherent extrapolated volition satisfied maximally for 3^^^^3 years."

Relevant to my decision: I don't assign all that much utility to having 3^^^^3 copies of me satisfied. In fact, the utility I assign to 3^^^^3 copies of me being satisfied is, as far as I can tell, less than double the utility I assign to two copies of me being similarly uber-satisfied.

I am actually finding it difficult to come up with a serious Pascal's Offer scenario. As far as I know my utility function doesn't even reach the sort of extremes necessary to make such an offer sound tempting.

The problem with Pascal's Mugging is that in a bounded rationality situation, you cannot possibly have a probability on the order of 1/(3^^^^3) - it's incomprehensibly low. Thus, a naive Bayesian expected utility maximizer might jump at this offer every time it's offered despite it never being true - allowing one to get it to do anything for you, instead of whatever it was supposed to be maximizing.

To be more specific, if a stranger approached me, offering a deal saying, "I am the creator of the Matrix. If you fall on your knees, praise me and kiss my feet, I'll use my magic powers from outside the Matrix to run a Turing machine that simulates 3^^^^3 copies of you having their coherent extrapolated volition satisfied maximally for 3^^^^3 years." Why exactly would I penalize this offer by the amount of copies being offered to be simulated?

Essentially, you just call BS on the promise of huge utility.

Where is the huge utility?!? Show me the huge utility!!!