If you did the limit by program length, it might converge, but there's no obvious reason that there must be a certain order to count them in. If you can count them in any order, you can alternate between states where the arrow hits and states where it does not, and make it look like it has a 50% chance of hitting.

Well for any finite amount of time that you predict into the future you've also got a finite amount of space to consider, as anything too far away wouldn't be able to travel fast enough to effect the outcome of the thing being predicted about. Each state of the universe would really be the state of this finite area of space which would be expressed in binary. One way to order the states would be in terms of how large a binary number they form, from smallest to largest.

I'm not sure how making it look like the arrow had a 50% chance of hitting would make any difference to anything though?

Given a state, you could add another non-interacting (or just interacting based on one guy's decision) universe where 3^^^^3 people die. This universe is constant complexity, and small complexity compared to 3^^^^3. Let's call the complexity of that part k.

Given a state, you could also add another non-interacting (or just interacting based on one guy's decision) universe where 3^^^^3 people lives are saved. I don't know if this is the right terminology, but it seems to me that when you start adding extra possible universes on, their outcomes become causally decoupled from the original decision to give/not-give the mugger $5.

This is true, but doesn't explain why we're more surprised when we see the former than the latter.

First reaction: I don't know about "far" more probable. What's the prior that a coin is rigged? I would have said less than 1/32768, but low confidence on that.

According to this, you can't rig a coin to do that, which increases my confidence.

But you can rig your tossing, even by mistake; if it lands heads, and you balance it to flip with heads up again, then it's slightly more likely to land heads. I remember hearing a figure of 51% for that; in which case H*15 has probability 1/24331 instead of 1/32768; about a third more probable. But that scenario (fifteen times) is itself unlikely... if we estimate P(next is heads | last was heads) = 0.505 (corresponding to keeping the same side up 3/4 of the time, I still feel that's an overestimate), we get 1/28204, 16% more likely.

If we switched to dice, I would agree that 666666666666666 is far more probable than 136112642345553.

I suppose that isn't all that unintuitive (though does this actually work if you start with a uniform prior over weights and do the math?). But does your intuitive model also predict the fact that HTHTHTHTHT is more probable than HTHHTHTHTT? :D

Sorry I'll try to clarify:

If you want to predict the exact state of a system five minutes into the future you need to know the current state of the system and the laws of that system. Call the current state s and the future state s', the laws of the system are simulated by the Turing machine L. Instead of knowing the state of the system, we only know its laws (or rather we take them as a given).

Then any prediction we make about the future state of the system will restrict the range of value for s' that will validate our prediction. The more specific we are about s' the smaller the range of values it can be. In turn this restricts the range of possible values for s (as L(s) = s') that will give s'.

Because we have no information about the current state of the system all possible states are equally likely, and as such the probability that the system will end up in a particular range of s' is the same as the fraction of s (out of all possible s) that will map there.

This is not in relation to any hypothesis about the laws of the system, but instead the current state of the system. I hope this makes my original argument make more sense. If not I'm sorry; please highlight to me where my explanation is going wrong.

Actually, to think about it, i might've just nailed it and also nailed the problem with using probabilistic reasoning in practice. You can easily pick some random hypothesis out of enormously huge space, which gives it very small prior, but then you forget about this enormous space.

You might like to read this post, "Privileging the hypothesis."

2: I don't see how it's overly specific. If we consider (coin or a person), one randomly chosen (coin or a person) affecting 3^^^^3 (coin or a person) is unlikely. Still, the explanation is indeed somewhat problematic.

It assumes a particular account of anthropic reasoning with infinite certainty. If you get your anthropic hypotheses out of something like Solmonoff induction (the programs best approximating our sense inputs can be thought of as a combination of a simulation of our world plus a bit of code that acts as an "anthropic theory" and reads out part of the simulation as our sense inputs), then things like SIA, SSA, and "you're more likely to be a given person if they have more causal influence" are not radically different in complexity. So you get Pascal's mugging from the combination of 1) laws of physics allowing vast quantities of computation and 2) some kind of anthropic theory that makes it unlikely you are one of the mass of simulations.

1: Well my reasoning is that the more people the mugger threatens to kill, the less likely his claims are to be true. In the same way that if I were to claim that a row of 3^^^^3 coins would all turn up heads; it would be far less likely to come true than if I predicted two coins would come up heads. At least that's what I'm trying to get across in this post.

2: It seems overly specific to me because it seems like a bit too much of a hack, if you get my meaning?