Thoughts on a possible solution to Pascal's Mugging

If your metaconfidence serves only to push your confidence interval further in the direction it was already at, it seems likely you're double-counting somewhere; you're weighing the same evidence (or lack thereof) to generate your metaconfidence as your confidence.

I am skeptical about "metaconfidence."

If you have all the relevant probabilities, you have all the information you need to calculate expected utilities for all possible choices--you don't need to decide on how "metaconfident" you are in these probabilities. Your probabilities may not be based on very good data, and so you might anticipate that these probabilities will change drastically when you update on new observations--I think you would call this "low metaconfidence." But your strategy for updating based on new evidence is already encoded in your priors for what evidence you expect to observe. I don't think metaconfidence is useful as a new, independent concept, apart from indicating that your probabilities are susceptible to change based on future evidence.

In particular, if "metaconfidence" considerations seem to be prompting you to distrust your probability as being either systematically too high or too low, then you should just immediately update your probability in that direction, by conservation of expected evidence. So when you say

the pony is less likely, and the sunrise is more likely, than a naive probability estimate would suggest.

then if you really believe that, you ought to preemptively update your "naive" probabilities until you no longer believe they are systematically biased.

I think a particular misconception may be leading you astray:

When we talk to the crazy pony man, and to the woman with the coin, what we leave with are two identical numerical probabilities. However, those numbers do not represent the sum total of the information at our disposal. In the two cases, we have differing levels of confidence in our levels of confidence. And, furthermore, this difference has an actual ramifications on what a rational agent should expect to observe.

I think what you are referring to here is that if you repeated the 100-coin-flip experiment 2^100 times, you expect to see 100 heads on average once. But even if the pony guy tried 2^100 times, you do not expect an average of one pony to fall from the sky. You expect him to simply lack the power, and so expect him to fail every time.

The real difference here is not metaconfidence, it's that each set of 100 coin flips is completely independent of any other set of 100 coin flips. But the pony guy's first attempt to summon a pony is not independent of his second attempt to summon a pony. If he fails on his first attempt, you strongly expect him to fail on every future attempt.

In more detail:

If you are assigning a probability of 1/2^100 to the pony guy's claim, you think that the claim is pretty improbable. How improbable? You are saying, "I expect that if I heard 2^100 similarly improbable, but completely independent and uncorrelated claims, on average one such claim would be true and all the rest would be false." One such completely independent claim to which you might assign the same probability might be if I predicted that "tomorrow, physicists will discover a new sentient fundamental particle that vacations in Bermuda every August." If your probability is 1/2^100, you really should expect that if you heard 2^100 equally astonishing claims, one of them would actually turn out to be true. But in particular you don't expect that if the pony guy repeated his claim 2^100 times, on average he'd be telling the truth once. Repetitions of the same claim by the same person are clearly not independent claims.

Superficially this may look different from the case of the coin flips. There, you expect that if you make 2^100 tests of the claim "flipping this coin 100 times will give all heads," it would be true on average once. To symmetrize the situation, let me make the coin claim more specific without changing it in any meaningful way: "The first 100 flips of this coin will all be heads." This, like "I can summon ponies from the sky" is a claim that is simply either true or false. When you assign a probability of 1/2^100 to the coin claim, you are again saying, "I expect that if I heard 2^100 similarly improbable, but completely independent and uncorrelated claims, on average one of them would be true." Because different flips of the same coin happen to have the property of statistical independence, similarly improbable but completely independent claims are easy to construct. For instance, "The second 100 flips of this coin will all be heads," etc.

This is all to say that I disagree that

the probability value in the second case, while superficially identical to the probability value in the first case, represents a fundamentally different kind of claim about reality than the first case.

The two probabilities represent exactly analogous claims, and there is no need for a notion of metaconfidence to distinguish them.

Jaynes deals with "metaconfidence" somewhat in Chapter 18. One of the more interesting chapters, but I haven't seen people take notice, or do anything with it. If anyone knows where people have pushed along these lines further, please let me know.

As for Pascal's mugging, I'll say three things.

First, you have to decide how you want to respond to terroristic threats in game theory terms. What's to stop the guy from playing this game again and again and again? Nothing that I see.

Second, I've recently been trying to get at the issue of how to deal with fundamentally arbitrary assertions. This particular claim of a being "outside my model of the universe" would seem to apply in spades. It's one of an infinite number of equivalent arbitrary claims. Maybe P should be zero for these.

Third, if you don't pretend you care about every sentient being equally, or that you have an unbounded amount of "give a shit" to dole out to them, "shut up and multiply" just does not apply, and the ever increasing number of people in the threat is not a trump card against every low probability assignments.

I suppose, on further reflection, that the metaconfidence concept is simply a heuristic for how we can more accurately compute our probabilities. What I'm actually saying is that, when presented with very suspicious, contradictory, or weak evidence, then the probability that Bayes' theorem computes is not the real value: it is an approximation of a value that is very probably <= to the calculated value, and the probability distribution below the calculated value, on an asymptotic scale, grows wider exponentially and unboundedly as an inverse function of the size of the calculated probability. Put another way, I think there's a rational, justifiable reason to exponentially discount evidence at the small end of the probability spectrum.

Nice post. I think directionality of meta confidence is a consequence if markov's inequality.

The two scenarios are more different than you present them. In one case, the probability is of in any 100 coin flips, all of them are heads. In the other, it's that of all possible worlds, you live in one where ponies fall from the sky. So the second is a probability of the type of universe we live in.

So, when you say

No matter how many times somebody pulls the pony trick, a rational agent is never going to get their hopes up.

It's not accurate. What it should be, is your confidence in pony man should be exactly as high as getting all heads in 100 flips the first time you make the coin flips, no matter how many times he makes that claim.

Isn't your whole argument merely a rediscovery of the mathematical concept of uncertainty? You admit at the outset that the priors for the magically falling pony must be bizzarely specific to render the exact probability of 1/2^100. In contrast, given a fair coin and a hundred coinflips, it's nothing at all bizzare about the fact that the probability that you'll get a straight run of heads is exactly 1/2^100. It's just math. The problem of course, is that the uncertainty as to whether magically falling ponies are possible isn't a question that is amenable to a quantitative study, given that the event so seldom repeat... Even so, my intuition is exactly opposite to yours: I know how bizzarely unlikely it would be to get a hundred heads in a row, so if that were to happen (under circumstances that ruled out foul play) I would be astonished. If a magically falling pony appeared I would be likewise astonished, but less so since my model of reality is more uncertain to me than my model of abstract math.

That said, the idea of "metaconfidence" reminds me somewhat of Yvain's post Confidence levels inside and outside an argument.

Edited to add: And the the concept of falling ponies reminds me of this.

I may be wrong here.

What does an arbitrarily powerful entity need with(a starship) $5? $5 once won't break me but it sets a president you should never negotiate with terrorist as though they are playing a fair game you should have a probability of this being a lie scale faster than the claim. Including the possibility that they'll do it any way even if you relent or they will make the same demand over until you have nothing to pay them with.

"You believe, overwhelmingly, that the probability (in underlying reality, divorced from the map and its limitations) is zero. It is only grudgingly that you inch even a tiny morsel of probability into the other hypothesis (that the universe is structured in such a way as to make the probability non-zero)."

There is a definitive, non-zero probability that, when throwing a tennis ball at a wall, it will simply "quantum tunnel" through it and fall out the other side. There's a definitive, non-zero probability that random chance will leave all the air molecules in the room on the east side, suffocating anyone caught on the west.

It's worth noting that larger effects are exponentially less likely - just like getting heads once is 50%, but twice is only 25%, and three times is 12.5%.

Therefor, there's a minimum computable plausibility for pretty much any claim. Your prior should never be zero, and (if you are a strict rationalist AI) you should presumably never want it to be zero.

(Your further point about metaconfidence was actually covered by Givewell on the site previously. It's also worth noting that what I've said here doesn't prevent muggings at all, it just establishes that Pascal's Muggings has a definitive non-zero probability by our current understanding of the universe)

Don't update your priors on the basis of unreliable sources. If someone walks up to you and says "This is a fair coin, and the odds of it coming up heads the next 100 times in a row are 1/2^20, provided it is flipped 100 times in a row.", you don't update based on that, at all. That statement does not make a run of heads more or less likely.

Similarly, when someone makes a claim about magical falling ponies, I don't update my estimate of the odds of magical flying ponies unless their claim is credible.

As a check- I say that the (fair) coin nearest to you will turn up heads the next 100 times it is flipped, as of the time you read this. Do you think the odds of that happening are greater, less than, or exactly equal to 1/2^100? Did you update your estimate based on my claim? Why or why not?

(You could suggest that the odds of a particular post being by Omega, who has specific unexplainable knowledge about the future, are sigma, and so the odds are 1/2^100+sigma. Now compare that with the odds that a particular post is by Omega, and that he is lying. That makes the odds 1/2^100 + sigma - sigma; the odds provided that neither alternative is true, plus the odds that all-heads will be forced, plus the odds that not-all-heads will be forced. Which sigma is greater, or are they exactly equal? On what basis do you judge that?)

So, if you have a claim like the pony claim (or Pascal's mugging), in which you have a very low estimated probability, and a very low metaconfidence, should become dramatically less likely to actually happen, in the real world, than a case in which we have a low estimated probability, but a very high confidence in that probability.

So? Unless it's on the order of 1/3^^^3, it doesn't matter how unlikely it is, and while my metaconfidence may be low for the exact value (insomuch as that means anything), it's clearly more likely than that. The human genome only takes about eight megabytes. If you want 3^^^3 of them, you'll have to get more general than "human", although only if you're certain that killing the same one 3^^^3 times doesn't count. Even if you do so, there's no way a generic pattern for producing sapient entities takes that much information.

I suspect that, in practice, it eliminates many of the Pascal's-mugging-style problems we encounter currently.

When have you last encountered one?

Personally, I think the bigger problem is a slightly related paradox in that if you take decision A, it could kill 3^^^3 people, but if you take decision B, it could kill 3^^^^3 people, but then again A could kill 3^^^^^3 people etc., so you could never make a decision. This would come up on every decision you make.

Perhaps related.

There's a post from GIveWell along these lines that you'll find very informative.