Bayesianism for Humans

post by ChrisHallquist · 2013-10-29T23:54:14.890Z · LW · GW · Legacy · 37 comments

Recently, I completed my first systematic read-through of the sequences. One of the biggest effects this had on me was considerably warming my attitude towards Bayesianism. Not long ago, if you'd asked me my opinion of Bayesianism, I'd probably have said something like, "Bayes' theorem is all well and good when you know what numbers to plug in, but all too often you don't."

Now I realize that that objection is based on a misunderstanding of Bayesianism, or at least Bayesianism-as-advocated-by-Eliezer-Yudkowsky. "When (Not) To Use Probabilities" is all about this issue, but a cleaner expression of Eliezer's true view may be this quote from "Beautiful Probability":

No, you can't always do the exact Bayesian calculation for a problem.  Sometimes you must seek an approximation; often, indeed.  This doesn't mean that probability theory has ceased to apply, any more than your inability to calculate the aerodynamics of a 747 on an atom-by-atom basis implies that the 747 is not made out of atoms.  Whatever approximation you use, it works to the extent that it approximates the ideal Bayesian calculation - and fails to the extent that it departs.

The practical upshot of seeing Bayesianism as an ideal to be approximated, I think, is this: you should avoid engaging in any reasoning that's demonstrably nonsensical in Bayesian terms. Furthermore, Bayesian reasoning can be fruitfully mined for heuristics that are useful in the real world. That's an idea that actually has real-world applications for human beings, hence the title of this post, "Bayesianism for Humans."

Here's my attempt to make an initial list of more directly applicable corollaries to Bayesianism. Many of these corollaries are non-obvious, yet eminently sensible once you think about them, which I think makes for a far better argument for Bayesianism than Dutch Book-type arguments with little real-world relevance. Most (but not all) of the links are to posts within the sequences, which hopefully will allow this post to double as a decent introductory guide to the parts of the sequences that explain Bayesianism.

• Watch out for base rate neglect. It's why even experts screw up in one of the standard problems used to explain Bayes' Theorem. Even when you don't know what the base rate is, there are times when you ought to expect it to be low, particularly if you're trying to detect a rare phenomenon like a new disease or IDing terrorists.
• Absence of Evidence is Evidence of Absence. If observing E would increase the probability of H, observing not-E should decrease the probability of H. E and not-E sure as hell shouldn't both increase the probability of H.
• Relatedly, there's Conservation of Expected Evidence: roughly, if you think more evidence would probably increase your confidence in a belief, you should think there's small chance it would cause a larger change in the opposite direction. 
• Conservation of expected evidence means that a rational person can't seek to confirm their beliefs, only to test them. If your expectation of how a test will affect your belief violates conservation of expected evidence, you should update your beliefs now based on how you expect the test to turn out.
• Also related closely related to the above, when it comes to gathering evidence, "If you know your destination, you are already there." Evidence gathered through a biased method designed to turn out one way is worthless.
• On the other hand, you can't dismiss a hypothesis due to a lack of a particular piece of evidence that you wouldn't expect to have even if the hypothesis were true.
• Even when an argument on one side is overcome by a stronger argument on the other side, you still need to take the first argument into account when assigning confidence to your belief, lest you gradually dismiss each piece of evidence on the other side because no piece is (individually) as strong as the one piece of evidence on your side.
• Burdensome Details: Every detail added to a claim makes it less probable.
• Reversed Stupidity Is Not Intelligence: If you would expect to see flying saucer cults regardless of whether or not extraterrestrials were visiting us, flying saucer cults are not evidence against extraterrestrials.
• Don't get caught up in arguing about definitions when you should be looking at what's actually indicative of what. (I take it that that's the Bayesian take-away from the sequence on words, though Eliezer doesn't quite put it that way.)
• Rationality and the rules of Science are not the same thing. The latter are social rules designed to make science work in spite of the irrationality of its practitioners. They're not the same as the rules of rationality an ideal reasoner would follow. 
• An example of the science vs. rationality issue: to an ideal reasoner, successful retrospective predictions are as valuable as prospective predictions. (To us non-ideal reasoners, prospective predictions are can be extra valuable as protection against fooling ourselves, but we still shouldn't discount retrospective predictions entirely.) 
• This is also reason to be careful about dismissing evo psych claims as just-so stories.
• Another example: contrary to old-fashioned statistical procedure, a researcher's state of mind shouldn't affect the significance of their results.
• Last example is from a post of my own: Expert opinion should be discounted when the expert's opinions could be predicted solely from information not relevant to the truth of the claims. But when the state of expert opinion surprises you, beware discounting their opinions just because you can think of some explanation for why they'd be wrong.

37 comments

Comments sorted by top scores.