Conservation of Expected Evidence

One minor correction, Eliezer: the link to your essay uses the text "An Intuitive Expectation of Bayesian Reasoning." I think you titled that essay "An Intuitive EXPLANATION of Bayesian Reasoning." (I am 99.9999% sure of this, and would therefore pay especial attention to any evidence inconsistent with this proposition.)

I guess I was a Bayesian before I knew what it meant....

Perhaps this formulation is nice:

0 = (P(H|E)-P(H))P(E) + (P(H|~E)-P(H))P(~E)

The expected change in probability is zero (for if you expected change you would have already changed).

Since P(E) and P(~E) are both positive, to maintain balance if P(H|E)-P(H) < 0 then P(H|~E)-P(H) > 0. If P(E) is large then P(~E) is small, so (P(H|~E)-P(H)) must be large to counteract (P(H|E)-P(H)) and maintain balance.

Eliezer,

Of course you are assuming a strong form of Bayesianism here. Why do we have to accept that strong form?

More precisely, I see no reason why there need be no change in the confidence level. As long as the probability is greater than 50% in one direction or the other, I have an expectation of a certain outcome. So, if some evidence slightly moves the expectation in a particular direction, but does not push it across the 50% line from wherever it started, what is the big whoop?

One reason is Cox's theorem, which shows any quantitative measure of plausibility must obey the axioms of probability theory. Then this result, conservation of expected evidence, is a theorem.

What is the "confidence level"? Why is 50% special here?

"Of course you are assuming a strong form of Bayesianism here. Why do we have to accept that strong form?"

Because it's mathematically proven. You might as well ask "Why do we have to accept the strong form of arithmetic?"

"So, if some evidence slightly moves the expectation in a particular direction, but does not push it across the 50% line from wherever it started, what is the big whoop?"

Because (in this case especially!) small probabilities can have large consequences. If we invent a marvelous new cure for acne, with a 1% chance of death to the patient, it's well below 50% and no specific person using the "medication" would expect to die, but no sane doctor would ever sanction such a "medication".

"Why is 50% special here?"

People seem to have a little arrow in their heads saying whether they "believe in" or "don't believe in" a proposition. If there are two possibilities, 50% is the point at which the little arrow goes from "not believe" to "believe".

Aaron, fixed.

Eliezer, when you're lost in an unfamiliar neighbourhood, do sit back, relax and wait for evidence of your location to come in? Obviously not, since you're still alive and haven't yet starved to death. Well guess what, none of my direct ancestors starved to death before they reproduced either. That's a scientific fact, and it just goes to show that when it comes to the thinking game, nothing succeeds like success.

And success is not the same as accuracy, except in a mystical world of spherical cows of uniform density. In the real world, the value of a perfectly correct decision which takes an infinate amount of time to evaluate is exactly 0. [Note the lack of units - renormalize if you dare.] I have empirical data showing that this world contains distinctly heterogenous cows. Reprints available upon request.

Tom,

Bayes' Theorem has its limits. The support must be continuous, the dimensionality must be finite. Some of the discussion here has raised issues here that could be relevant to these kinds of conditiosn, such as fuzziness about the truth or falsity of H. This is not as straightforward as you claim it is.

Furthermore, I remind one and all that Bayes' Theorem is asymptotic. Even if the conditions hold, the "true" probability is approached only in the infinite time horizon. This could occur so slowly that it might stay on the "wrong" side of 50% well past the time that any finite viewer might hang around to watch.

There is also the black swan problem. It could move in the wrong direction until the black swan datum finally shows up pushing it in the other direction, which, again, may not occur during the time period someone is observing. This black swan question is exactly the frame of discussion here, as it is Taleb who has gone on and on about this business about evidence and absence thereof.

you can't possibly expect the resulting game plan to shift your beliefs (on average) in a particular direction.

But you can act to change the probability distribution of your future beliefs (just not its mean). That's the entire point of testing a belief. If you have a 50% belief that a ball is under a certain cup, then by lifting the cup, you can be certain than your future belief will be in the set {0%,100%} (with equal probability for 0 and 100, hence the same mean as now).

Getting the right shape of the probability distribution of future belief is the whole skill in testing a hypothesis.

But you can't have it both ways - as a matter of probability theory, not mere fairness.

You've proved your case - but there's still enough wriggle room that it won't make much practical difference. One example from global warming, which predicts higher temperature on average in Europe - unless it diverts the gulf stream, in which case it predicts lower average temperatures. Consider the two statements: 1) If average temperatures go up in Europe, or down, this is evidence for global warming. 2) If average temperatures go up in Europe, and the gulf stream isn't diverted, or average temperatures go down, while the gulf stream is diverted, this is evidence of global warming.

1) is nonsense, 2) is true. Lots of people say statements that sound like 1), when they mean something like 2). Add an extra detail, and the symmetry is broken.

This weakens the practical power of your point; if an accused witch is afraid, that shows she's guilty; if she's not afraid, in a way which causes the inquisitor to be suspicious, she's also guilty. That argument is flawed, but it isn't a logical flaw (since the similar statement 2) is true).

Then we're back to arguing the legitimacy of these "extra details".

Stuart, if the extra details are observable and specified in advance, the legitimacy is clear-cut.

Barkley, I'm an infinite set atheist, all real-world problems are finite; and you seem to be assuming that priors are arbitrary but likelihood ratios are fixed eternal and known, which is a strange position; and in any case what does that have to do with something as simple as Conservation of Expected Evidence? If anyone attempts to make an infinite-set scenario that violates CEE, it disproves their setup by reductio ad absurdum, and reinforces the ancient wisdom of E. T. Jaynes that no infinity may be assumed except as the proven limit of a finite problem.

Eliezer,

I do not necessarily believe that likelihood ratios are fixed for all time. The part of me that is Bayesian tends to the radically subjective form a la Keynes.

Also, I am a fan of nonstandard analysis. So, I have no problem with infinities that are not mere limits.

Eliezer,

I just googled "law of conservation of expected evidence." This blog came up. Nothing else like it. Frankly, I don't think you are selling a law here. You are asserting one that nobody else is aware of.

a more general law, which I would name Conservation of Expected Evidence

I thought it was pretty clear that I was coining the phrase. I'm certainly not the first person to point out the law. E.g. Robin notes that our best estimate of anything should have no predictable trend. In any case, I posted the mathematical derivation and you certainly don't have to take my word about anything.

Eliezer,

Fair enough. You get credit, then, for coining the term. However, the problem remains, why should that equals sign be there? Sure, if you put it there, the logic holds up, my niggles about Bayes' Theorem and time to convergence and all that aside. But, it is not clear at all that the equals sign should be there, or is there in any meaningfully regular way. Your defense has been to cite an essentially empirical argument by Robin. But that empirical argument is much contested in many arenas. Sure, Burton Malkiel posed that financial markets are a random walk, but that argument has undergone a lot of modifications since he first posed it in a best-selling paperback. In that regard, your proof essentially amounts to one of these "proofs" of the existence of God, wherein the proof arises from another assumption that gets snuck in the backdoor that gets one the result, but that is itself as questionable or unprovable, much like the old complaint by Joan Robinson about the magician making a big deal about pulling the rabbit out of the hat after having put it into the hat in full view of the audience.

Barkley, it looks to me like Eli derived it using the sum and product rules of probability theory.

What Peter said. Barkley, do you question that P(H) = P(H,E) + P(H, ~E) or do you question that P(H,E) = P(H|E)*P(E)?

Eliezer and Peter, I think the problem is statics versus dynamics. Your set of equations are correct only at a specific point in time, which makes them irrelevant to saying anything about what happens later when new information arrives. That would entail subscripting H by time. For any given t, sure. But, that says nothing about what happens when new information arrives. P(H) might change.

The obvious example is indeed the black swan story, which we all know is what is lying behind this discussion. So, at a point in time before black swans are observed, let H be "all swans are white." Perhaps there were a few folks who thought this might not be true, so say P(H) was 95%. Sure, your equations hold at a point in time, but so what? The minute the word comes in about the observation of a black swan (assuming it is accepted), P(H) just went to zero, or not much above zero, perhaps after having gradually drifted over time to 95%. Remember, your story was one about new information coming in and changes over time. But that is not what your equations are about.

This is the fatal flaw in your nice new law, Eliezer.

...

Barkley, you don't realize that Bayes's Theorem is precisely what describes the normative update in beliefs over time? That this is the whole point of Bayes's Theorem?

Before black swans were observed, no one expected to encounter a black swan, and everyone expected to encounter another white swan on occasion. A black swan is huge evidence against, a white swan is tiny additional evidence for. Had they been normative, the two quantities would have balanced exactly.

I'm not sure what to say here. Maybe point to Probability Theory: The Logic of Science or A Technical Explanation of Technical Explanation? I don't know where this misunderstanding is coming from, but I'm learning a valuable lesson in how much Bayesian algebra someone can know without realizing which material phenomena it describes.

"no one expected to encounter a white swan, and everyone expected to encounter another black swan on occasion. A white swan is huge evidence against, a black swan is tiny additional evidence for." I presume you meant the reverse of this?

Oops, fixed.

per the Black Swan:

The set of potential multicolored variations of Swans is infinite (purple, brown, grey, blue, green, etc). We can not prove any one of them do not exist. But every day that proceeds where we don't see these swans gives us a higher probability they do not exist. It never equals 1, but it's darn close.

The problem with the Black Swan parable is not that it's untrue, but rather unimportant. The set of things we have no evidence of is infinite. To then pounce across an unexpected observation (eg, a Black Swan, that Kevin Federline is a relatively good parent, last week's liquidity run on mortgage lenders), and say, "aha! You were all wrong!" merely sets up a staw man, that everything we reasonably don't anticipate and plan for is assumed to have had a probability of zero.

In reality, when you want to pay money for extreme events you overpay, that is, the implied probability is overweighted because sellers can't insure against these events. London bookmakers offer only 250-1 odds against a perpetual motion machine being discovered, 100-1 that aliens won't be proven. In option markets you have a volatility smile so that extreme events get higher and higher implied volatilities as you move away from the mean, meaning their probability is not assumed Gaussian.

The bottom line is that "absence of evidence is not evidence of absence" merely uses hindsight to attack a caricature of beliefs, and seems to suggests something practically important. In practice, people lose money on lottery tickets (or hurricane insurance, or buing a 3-delta put), so exploiting this is a fool's game.

Eliezer,

This is about to scroll off, but, frankly, I do not know what you mean by "normative" in this context. The usual usage of this term implies statements about values or norms. I do not see that anything about this has anything to do with values or norms. Perhaps I do not understand the "wholel point of Bayes' Theorem." Then again, I do not see anything in your reply that actually counters the argument I made.

Bottom line: I think your "law" is only true by assumption.

What I mean, Barkley, is that the expression P(H|E), as held at time t=0, should - normatively - describe the belief about H you will hold at time t=2 if you see evidence E at time t=1. Thus, statements true in probability theory about the decomposition of P(H) imply the normative law of Conservation of Expected Evidence, if you accept that probability theory is normative for real-world problems where no one has ever seen an infinite set.

If you don't think probability theory is valid in the real world, I have some Dutch Book trades I'd like to make with you. But that's a separate topic, and in any case, most readers of this blog will at least understand what I intend to convey when I speak from within the view that probability theory is normative.

Eliezer Yudkowsky, The word "normative" has stood in the way of my understanding what you mean, at least the first few times I saw you use it, before I pegged you as getting it from the heuristics and biases people. It greatly confused me many times when I first encountered them. It's jargon, so it shouldn't be surprising that different fields use it to mean rather different things.

The heuristics and biases people use it to mean "correct," because social scientists aren't allowed to use that word. I think there's a valuable lesson about academics, institutions, or taboos in there, but I'm not sure what it is. As far as I can tell, they are the only people that use it this way.

My dictionary defines normative as "of, relating to, or prescribing a norm or standard." It's confusing enough that it carries those two or three meanings, but to make it mean "correct" as well is asking for trouble or in-groups.

This post was one of the most helpful for me personally, but I recently realized this isn't true in an absolute sense: "There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before."

Suppose the statement "I perform action A" is more probable given position P than given not-P. Then if I start planning to perform action A, this will be evidence that I will perform A. Therefore it will also be evidence for position P. So there is some plan that I can devise such that I can expect my confidence in P to be higher than before I devised the plan.

In general, of course, unless P is a position relating to my actions or habits, this effect will not be very large.

Um, no, if a study shows that people who chew gum also have a gene GXTP27 or whatever, which also protects against cancer, I cannot plan to increase my subjective probability that I have gene GXTP27 by starting to chew gum.

See also: "evidential decision theory", why nearly all decision theorists do not believe in.

Here's an example which doesn't bear on Conservation of Expected Evidence as math, but does bear on the statement,

"There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before."

taken at face value.

It's called the Cable Guy Paradox; it was created by Alan Hájek, a philosopher the Australian National University. (I personally think the term Paradox is a little strong for this scenario.)

Here it is: the cable guy is coming tomorrow, but cannot say exactly when. He may arrive any time between 8 am and 4 pm. You and a friend agree that the probability density for his arrival should be uniform over that interval. Your friend challenges you to a bet: even money for the event that the cable guy arrives before noon. You get to pick which side of the bet you want to take -- by expected utility, you should be indifferent. Here's the curious thing: if you pick the morning bet, then almost surely there will be times in the morning when you would prefer to switch to the afternoon bet.

This would seem to be a situation in which "you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before," even though the equation P(H) = P(H|E)P(E) + P(H|~E)P(~E) is not violated. I'm not sure, but I think it's due to multiple possible interpretations of the word "before".

Eliezer - what if the presence of the gene was decided by an omnipotent being called Omega? Then you'd break out the Spearmint, right?

I'll modify my advice. If the probability that "I do action A in order to increase my subjective probability of position P" is greater given P than given not P, then doing A in order to increase my subjective probability of position P will be evidence in favor of P.

So in many cases, there will such a plan that I can devise. Let's see Eliezer find a way out of this one.

Actually, the Omega situation is a perfect example. Someone facing the two boxes would like to increase his subjective probability that there is a million in the second box, and he is able to do this by deciding to take only the second box. If he decides to take both, on the other hand, he should decrease his credence in the presence of the million, even before opening the box.

Fantastic heuristic! It's like x=y·(z/y)+(1-y)·(x-z)/(1-y) for the rationalist's soul :)

It's worth noting, though, that you can rationally expect your credence in a certain belief "to increase", in the following sense: If I roll a die, and I'm about to show you the result, your credence that it didn't land 6 is now 5/6, and you're 5/6 sure that this credence it about to increase to 1.

I think this is what makes people feel like they can have a non-trivial expected value for their new beliefs: you can expect an increase or expect a decrease, but quantitatively the two possibilities exactly cancel each out in the expected value of your belief.

I have a theory that I will post this comment. By posting the comment, I'm seeking evidence to confirm the theory. If I post the comment, my probability will be higher than before.

Similarly, in Newcomb's problem, I seek evidence that box A has a million dollars, so I refrain from taking box B. There was money in box B, but I didn't take it, because that would give me evidence that box A was empty.

In short, there's one exception to this: when your choice is the evidence.

There is more discussion of this post here as part of the Rerunning the Sequences series.

Wouldn't the rule be something more like:

((P(H|E) > P(H)) if and only if (P(H) > P(H|~E))) and ((P(H|E) = P(H)) if and only if (P(H) = P(H|~E)))

So, if some statement is evidence of a hypothesis, its negation must be evidence against. And if some statement's truth value is independent of a hypothesis, then so is that statements negation.

This is implied by the expectation of posterior probabilities version. Since P(E) + P(~E) = 1, that means that P(H|E) and P(H|~E) are either equal, or one is greater than P(H) and one is less than. If they were both less than P(H), then P(H|E)P(E)+P(H|~E)P(~E) would have a lesser value than the largest conditional probability in that formula; suppose P(H|E) is the greater one, then P(H|E)P(E)+P(H|~E)P(~E) < P(H|E) and P(H|E) < P(H), so P(H|E)P(E)+P(H|~E)P(~E) ≠ P(H). If they are both larger than P(H), then P(H|E)P(E)+P(H|~E)P(~E) must be larger than the smallest conditional probability in that formula; suppose that P(H|E) is the smaller one, then we have P(H|E)P(E)+P(H|~E)P(~E) > P(H|E), and P(H|E) > P(H), so P(H) ≠ P(H|E)P(E)+P(H|~E)P(~E). And if both posterior probabilities are equal, then P(H|E)P(E)+P(H|~E)P(~E) = P(H|E), and both posteriors must eqaul the prior. Q.e.d.

I think that the formula that expresses the prior as the average of the posterior probability weighted by the probabilities of observing that evidence and not observing that evidence, is a great way to express the point of this article. But it might not be trivial for everyone to get:

((P(H|E) > P(H)) if and only if (P(H) > P(H|~E))) and ((P(H|E) = P(H)) if and only if (P(H) = P(H|~E)))

from

P(H) = P(H|E)P(E) + P(H|~E)P(~E)

That something is evidence in favor if and only if its negation is evidence against, and that some result is independent of some hypothesis if and only if not observing that result is independent of that hypothesis, are the take home messages of this post as far as i can tell. The law that "P(H) = P(H|E)P(E) + P(H|~E)P(~E)" says more than that, it also tells you how to get P(H|~E) from P(H|E), P(H) and P(E). But adding the boolean statement and its proof from the weighted average statement to the post, or at least to a comment on this post, not even necessarily using the boolean symbols or formalisms, might help a lot of students that come across this long after algebra class. I know it would have helped me.

Hi, I'm new here but I've been following the sequences in the suggested order up to this point.

I have no problem with the main idea of this article. I say this only so that everyone knows that I'm nitpicking. If you're not interested in nitpicking then just ignore this post.

I don't think that the example given bellow is a very good one to demonstrate the concept of Conservation of Expected Evidence:

If you argue that God, to test humanity's faith, refuses to reveal His existence, then the miracles described in the Bible >must argue against the existence of God.

Assuming I'm reading this correctly:

Our Prior is P(G) = The probability that God Exists (let's assume this is the Judeo-Christain God since that seems to be the intended target)

P(T) = the probability that God is Testing Humanity by not revealing his existence

P(M) = the probability that the Miracles of the bible are true.

The issue that I find with this is that P(G|T) = 1

If God is testing Humanity by hiding his existence there is a 100% chance that God Exists. I was going to write out the whole Bayesian equation to explain why this is true, but I think it's pretty intuitive. P(T) cannot be evidence for P(G) since it assumes that P(G) is true.

Another issue is that the way this is written you're implying that P(M) = P(~T). But this is not true, since the Miracles of the bible existing is not the direct opposite of God testing humanity by not reveling his existence. Unless you intend to completely twist the argument that most people are making when they say assert P(T) as truth. They aren't saying or even implying that God wants there to be no evidence at all of his existence. Most theist would instead argue that the existence of miracles are a part of God's test for humanity. They say that God sent us miraculous signs and prophets instead of just coming down and saying "Hey humanity, I'm God" because he wanted to test our faith. Had they the mathematical language, they would say that P(T|M) > P(T), meaning M serves as evidence of T. Not P(M) = P(~T)

Though this whole concept of God "testing humanity by not revealing himself" does seem more like an example of Belief in belief, where P(T) was devised as a means to justify the existence of an invisible God, I still feel like the example you've given is a bit of a stretch.

Is this the same as Jaynes' method for construction of a prior using transformation invariance on acquisition of new evidence?

Does conservation of expected evidence always uniquely determine a probability distribution? If so, it should eliminate a bunch of extraneous methods of construction of priors. For example, you would immediately know if an application of MaxEnt was justified.

Therefore, for every expectation of evidence, there is an equal and opposite expectation of counter-evidence.

Eliezer, isn't the "equal" part untrue? I like the parallel with Newton's 3rd law, but the two terms P(H|E)*P(E) and P(H|~E)*P(~E) aren't numerically equal - we only know that they sum to P(H).

For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it.

Old post, but isn't evidence that disconfirms the theory X equal to confirming ~X? Is ~X ineligible to be considered a theory?

The hyperlink "An Intuitive Explanation of Bayesian Reasoning" is broken. The current location of that essay is here: http://yudkowsky.net/rational/bayes

Mantel cox log rank tests compare observations and expectations too...

Can someone tell me if I understand this correctly : He is saying that we must be clear before hand what constitutes evidence for and what constitutes evidence against and what doesn't constitute evidence either way?

Because in his examples it seems that what is being changed is what counts as evidence. It seems that no matter what transpires (in the witch trials for example) it is counted as evidence for. This is not the same as changing the hypothesis to fit the facts. The hypothesis was always 'she's a witch'. Then the evidence is interpreted as supportive of the hypothesis no matter what.

Hi, new here.

I was wondering if I've interpreted this correctly:

'For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it.'

Does this mean that it is impossible to prove the truth of a theory? Because the only evidence that can exist is evidence that falsifies the theory, or supports it?

For example, something people know about gravity and objects under it's influence, is that on Earth objects will accelerate at something like 9.81ms^-2. If we dropped a thousand different objects and observed their acceleration, and found it to be 9.81ms^-2, we would have a thousand pieces of evidence supporting the theory, and zero pieces to falsify the theory. We all believe that 9.81 is correct, and we teach that it is the truth, but we can never really know, because new evidence could someday appear that challenges the theory, correct?

Thanks

Closely related is the law of total expectation: https://en.wikipedia.org/wiki/Law_of_total_expectation

It states that E[E[X|Y]]=E[X].

I do not understand the validity of this statement:

There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before.

Given a temporal proposition A among a set of other mututally exclusive temporal propositions {A, B, C...}, demonstrating B, C, and other candidates do not meet the evidence so far while A meets the evidence so far does raise our confidence in the proposition *continuing to hold*. This is standard Bayesian inference applied to temporal statements.

For example, we have higher confidence in the statement "the sun will come up tomorrow" than the statement "the sun will not come up tomorrow", because the sun has come up in the past, whereas it has not not come up comparably fewer times. We have relied on the prior distribution to make confident statements about the result of an impending experiment, and can constrain our confidence using the number of prior experiments that conform to it - further, every new experiment that confirms "the sun will come up" makes it harder to argue that "the sun will not come up" because the latter statement now has to explain *why* it failed to apply in the prior cases as well as why it will work now.

It would seem quantifying the prior distribution against a set of mutually-exclusive statements thus *is* a valid strategy for raising confidence in a specific statement.

Maybe I'm misinterpreting what "fixed proposition" means here or am missing something more fundamental?

Criticism of this article was found at a talk page at RationalWiki.

The Sequences do not contain unique ideas, and they present the ideas they do contain in misleading ways using parochial language. The "Law of Conservation of Expected Confidence" essay, for instance, covers ideas that are often covered in introductory philosophical methods or critical thinking courses. There is no novelty either in the idea that your expected future credence must match your current credence (otherwise, why not update your credence now?), nor in the idea that if E is evidence for H, then ~E is evidence for ~H (though E and ~E may have very different evidential strength), and Yudkowsky's treatment is imprecise and, in combining multiple points, muddles things more than it illuminates them. Besides that, the former notion has sparked substantial controversy in epistemology, owing to cases wherein people apparently can reasonably expect to have their minds changed without changing them right now. While popular, Bayesianism is not univocally accepted by epistemologists, and it's not because they're irrational.

What do you guys think?

https://rationalwiki.org/wiki/Talk:LessWrong#EA_orgs_praising_AI_pseudoscience_charity._Is_it_useful.3F