Deception by cherry picking

post by AlephNeil · 2011-02-20T05:11:54.770Z · LW · GW · Legacy · 4 comments

[This is an idea I've had 'kicking around' for a long time - may as well see what LW makes of it.]

The Bayesian update procedure tacitly presupposes that the event we're updating on is not itself random. Indeed, naively updating on a random event, that turns out to be correlated with the variable of interest, is how people get Monty Hall wrong.

A false statement can often be made to seem plausible if you naively update on a set of misleading, 'cherry picked' facts.

To make this concrete, imagine a biased coin which we know has probability 1/3 or 2/3 of landing heads - in fact it's 1/3 but we don't know that. Say it's tossed 2000 times. Then someone who wanted to mislead us could cherry pick a sample of 100 in which, say, 70 of the coin tosses landed heads, and hope we assume they picked their sample randomly. (More insidiously, using Derren Brown's variety of dark arts they could even trick us into choosing that sample ourselves, believing that we're choosing 'of our own free will'.)

But now here's the thing: That sample of 100 probably has a high minimum description length. If it had a sufficiently low minimum description length - like if it consisted of 100 contiguous tosses - then even if we suspected "Derren Brown" was trying to manipulate us, our sample would still give us evidence that heads has probability 2/3.

I think there should be a theorem which looks like:

"The largest x such that we'd be irrational not to increase our subjective log-odds of event E by at least x, even if our data was provided by an adversary" = log[P(data|E) / P(data|¬E)] - "The minimum description length of the data" - O(1)

Anyone familiar with "the theory of how to update on evidence provided by adversaries" (assuming it exists)?

4 comments

Comments sorted by top scores.