Swimming in Reasons

post by steven0461 · 2010-04-10T01:24:27.787Z · LW · GW · Legacy · 17 comments

Contents

17 comments

To a rationalist, certain phrases smell bad. Rotten. A bit fishy. It's not that they're actively dangerous, or that they don't occur when all is well; but they're relatively prone to emerging from certain kinds of thought processes that have gone bad.

One such phrase is for many reasons. For example, many reasons all saying you should eat some food, or vote for some candidate.

To see why, let's first recapitulate how rational updating works. Beliefs (in the sense of probabilities for propositions) ought to bob around in the stream of evidence as a random walk without trend. When, in contrast, you can see a belief try to swim somewhere, right under your nose, that's fishy. (Rotten fish don't really swim, so here the analogy breaks down. Sorry.) As a Less Wrong reader, you're smarter than a fish. If the fish is going where it's going in order to flee some past error, you can jump ahead of it. If the fish is itself in error, you can refuse to follow. The mathematical formulation of these claims is clearer than the ichthyological formulation, and can be found under conservation of expected evidence.

More generally, according to the law of iterated expectations, it's not just your probabilities that should be free of trends, but your expectation of any variable. Conservation of expected evidence is just the special case where a variable can be 1 (if some proposition is true) or 0 (if it's false); the expectation of such a variable is just the probability that the proposition is true.

So let's look at the case where the variable you're estimating is an action's utility. We'll define a reason to take the action as any info that raises your expectation, and the strength of the reason as the amount by which it does so. The strength of the next reason, conditional on all previous reasons, should be distributed with expectation zero.

Maybe the distribution of reasons is symmetrical: for example, if somehow you know all reasons are equally strong in absolute value, reasons for and against must be equally common, or they'd cause a predictable trend. Under this assumption, the number of reasons in favor will follow a binomial distribution with p=.5. Mostly, the values here will not be too extreme, especially for large numbers of reasons. When there are ten reasons in favor, there are usually at least a few against. 

But what if that doesn't happen? What if ten pieces of info in a row all favor the action you're considering? 

One possibility is you witnessed a one in a thousand coincidence. But let's not dwell on that. Nobody cares about your antics in such a tiny slice of possible worlds.

Another possibility is the process generating new reasons conditional on old reasons, while unbiased, is not in fact symmetrical: it's skewed. That is to say, it will mostly give a weak reason in one direction, and in rare cases give a strong reason in the other direction.

This happens naturally when you're considering many reasons for a belief, or when there's some fact relevant to an action that you're already pretty sure about, but that you're continuing to investigate. Further evidence will usually bump a high-probability belief up toward 1, because the belief is probably true; but when it's bumped down it's bumped far down. The fact that the sun rose on June 3rd 1978 and the fact that the sun rose on February 16th 1860 are both evidence that the sun will rise in the future. Each of the many pieces of evidence like this, taken individually, argues weakly against using Aztec-style human sacrifice to prevent dawn fail. (If the sun ever failed to rise, that would be a much stronger reason the other way, so you're iterated-expectations-OK.) If your "many reasons" are of this kind, you can stop worrying.

Or maybe there's one common factor that causes many weak reasons. Maybe you have a hundred legitimate reasons for not hiring someone as a PR person, including that he smashes furniture, howls at the moon, and strangles kittens, all of which make a bad impression. If so, you can legitimately summarize your reason not to hire him as, "because he's nuts". Upon realizing this, you can again stop worrying (at least about your own sanity).

Note that in the previous two cases, if you fail to fully take into account all the implications — for example, that a person insane in one way may be insane in other ways — then it may even seem like there are many reasons in one direction and none of them are weak.

The last possibility is the scariest one: you may be one of the fish people. You may be selectively looking for reasons in a particular direction, so you'll end up in the same place no matter what. Maybe there's some sort of confirmation bias or halo effect going on. 

So in sum, when your brain speaks of "many reasons" almost all going the same way, grab, shake, and strangle it. It may just barf up a better, more compressed way of seeing the world, or confess to ulterior motives.

(Thanks to Steve Rayhawk, Beth Larsen, and Justin Shovelain for comments.)

(Clarification in response to comments: I agree that skewed distributions are the typical case when you're counting pieces of evidence for a belief; the case with the rising sun was meant to cover that, but the post should have been clearer about this point. The symmetrical distribution assumption was meant to apply more to, say, many different good features of a car, or many different good consequences of a policy, where the skew doesn't naturally occur. Note here the difference between the strength of a reason to do something in the sense of how much it bumps up the expected utility, and the increase in probability the reason causes for the proposition that it's best to do that thing, which gets weaker and weaker the more your estimate of the utility is already higher than the alternatives. I said "confirmation bias or halo effect", but halo effect (preferentially seeing good features of something you already like) is more to the point here than confirmation bias (preferentially seeing evidence for a proposition you already believe), though many reasons in the same direction can point to the latter also. I've tried to incorporate some of this in the post text.)

17 comments

Comments sorted by top scores.