What Bayesianism taught me

You cannot expect that future evidence will sway you in a particular direction. "For every expectation of evidence, there is an equal and opposite expectation of counterevidence."

The (related) way I would expand this is "if you know what you will believe in the future, then you ought to believe that now."

Quoting myself from Yvain's blog:

Here’s a short and incomplete list of habits I would include in qualitative Bayes:

1. Base rate attention.
2. Consider alternative hypotheses.
3. Compare hypotheses by likelihood ratios, not likelihoods.
4. Search for experiments with high information content (measured by likelihood ratio) and low cost.
5. Conservation of evidence.
6. Competing values should have some tradeoff between them.

Each one of those is a full post to explain, I think. I also think they’re strongly reinforcing; 3 and 4 are listed as separate insights, here, but I don’t think one is very useful without the other.

A related mistake I made was to be impressed by the cleverness of the aphorism "The plural of 'anecdote' is not 'data'." There may be a helpful distinction between scientific evidence and Bayesian evidence. But anecdotal evidence is evidence, and it ought to sway my beliefs.

Anecdotal evidence is filtered evidence. People often cite the anecdote that supports their belief, while not remembering or not mentioning events that contradict them. You can find people saying anecdotes on any side of a debate, and I see no reason the people who are right would cite anecdotes more.

Of course, if you witness an anecdote with your own eyes, that is not filtered, and you should adjust your beliefs accordingly.

[1] See also Yvain's reaction to David Chapman's criticisms.

Chapman's follow-up.

"Absence of evidence isn't evidence of absence" is such a ubiquitous cached thought in rationalist communities (that I've been involved with) that its antithesis was probably the most important thing I learned from Bayesianism.

I am confused. I always thought that the "Bayes" in Bayesianism refers to the Bayesian Probability Model. Bayes' rule is a powerful theorem, but it is just one theorem, and is not what Bayesianism is all about. I understand that the video being criticized was specifically talking about Bayes' rule, but I do not think that is what Bayesianism is about at all. The Bayesian probability model basically says that probability is a degree of belief (as opposed to other models that only really work with possible possible worlds or repeatable experiments). I always thought this was the main thesis of Bayesianism was "The best language to talk about uncertainty is probability theory," which agrees perfectly with the interpretation that the name comes from the Bayesian probability model, and has nothing to do with Bayes' rule. Am I using the word in a way differently than everyone else?

I didn't get a lot out of Bayes at the first CFAR workshop, when the class involved mentally calculating odds ratios. It's hard for me to abstractly move numbers around in my head. But the second workshop I volunteered at used a Bayes-in-everyday-life method where you drew (or visualized) a square, and drew a vertical line to divide it according to the base rates of X versus not-X, and then drew a horizontal line to divide each of the slices according to how likely you were to see evidence H in the world where X was true, and the world where not-X was true. Then you could basically see whether the evidence had a big impact on your belief, just by looking at the relative size of the various rectangles. I have a strong ability to visualize, so this is helpful.

I visualize this square with some frequency when I notice an empirical claim about thing X presented with evidence H. Other than that, I query myself "what's the base rate of this?" a lot, or ask myself the question "is H actually more likely in the world where X is true versus false? Not really? Okay, it's not strong evidence."

Maybe this wasn't your intent, but framing this post as a rebuttal of Chapman doesn't seem right to me. His main point isn't "Bayesianism isn't useful"--more like "the Less Wrong memeplex has an unjustified fetish for Bayes' Rule" which still seems pretty true.

Banish talk of "thresholds of belief" ... However, perhaps I could be so confident that my behavior would not be practically discernible from absolute confidence.

While this is true mathematically, I'm not sure it's useful for people. Complex mental models have overhead, and if something is unlikely enough then you can do better to stop thinking about it. Maybe someone broke into my office and when I get there on Monday I won't be able to work. This is unlikely, but I could look up the robbery statistics for Cambridge and see that this does happen. Mathematically, I should be considering this in making plans for tomorrow, but practically it's a waste of time thinking about it.

(There's also the issue that we're not good at thinking about small probabilities. It's very hard to keep unlikely possibilities from taking on undue weight except by just not thinking about them.)

You cannot expect that future evidence will sway you in a particular direction. "For every expectation of evidence, there is an equal and opposite expectation of counterevidence."

Well ... you can have an expected direction, just not if you account for magnitudes.

For example if I'm estimating the bias on a weighted die, and so far I've seen 2/10 rolls give 6's, if I roll again I expect most of the time to get a non-6 and revise down my estimate of the probability of a 6; however on the occasions when I do roll a 6 I will revise up my estimate by a larger amount.

Sometimes it's useful to have this distinction.

So to summarise in pop Bayesian terms, akin to "don’t be so sure of your beliefs; be less sure when you see contradictory evidence." :

1. There is always evidence; if it looks like the contrary, you are using too high a bar. (The plural of 'anecdote' is 'qualitative study'.)
2. You can always give a guess; even if it later turns out incorrect, you have no way of knowing now.
3. The only thing that matters is the prediction; hunches, gut feelings, hard numbers or academic knowledge, it all boils down to probabilities.
4. Absence of evidence is evidence of absence, but you can't be picky with evidence.
5. The maths don't lie; if it works, it is because somewhere there are numbers and rigour saying it should. (Notice the direction of the implication "it works" => "it has maths".)
6. The more confident you are, the more surprised you can be; if you are unsure it means you expect anything.
7. "Knowledge" is just a fancy-sounding word; ahead-of-time predictions or bust!

ETA:

1. Choosing to believe is wishful thinking.

1. Absence of evidence is evidence of absence

Yes - though that idea can usefully be generalised to conservation of evidence.

I'll add the Bayesian definition of evidence an awareness of selection effects to the list.

Belongs in Main, methinks.

I suppose we all came across Bayesianism from different points of view - my list is quite a bit different.

For me the biggest one is that the degree to which I should believe in something is basically determined entirely by the evidence, and IS NOT A MATTER OF CHOICE or personal belief. If I believe something with degree of probability X, and see Y happen that is evidence for X, then the degree of probability Z which which I then should believe is a mathematical matter, and not a "matter of opinion."

The prior seems to be a get-out clause here, but since all updates are in principle layered on top of the first prior I had before receiving any evidence of any kind, it surely seems a mistake to give it too much weight.

My own personal view is also that often it's not optimal to update optimally. Why? Lack of computing power between the ears. Rather than straining the grey matter to get the most out of the evidence you have, it's often best to just go out and get more evidence to compensate. Quantity of evidence beats out all sorts of problems with priors or analysis errors, and makes it more difficult to reach the wrong conclusions.

On a non-Bayesian note, I have a rule to be careful of cases which consist of lots of small bits of evidence combined together. This looks fine mathematically until someone points out the lots of little bits of evidence pointing to something else which I just ignored or didn't even see. Selection effects apply more strongly to cases which consist of lots of little parts.

Of course if you have the chance to actually do Bayesian mathematics rather than working informally with the brain, you can of course update exactly as you should, and use lots of little bits of evidence to form a case. But without a formal framework you can expect your innate wetware to mess up this type of analysis.

Reading this clarified something for me. In particular, "Banish talk like "There is absolutely no evidence for that belief".

OK, I can see that mathematically there can be very small amounts of evidence for some propositions (e.g. the existence of the deity Thor.) However in practice there is a limit to how small evidence can be for me to make any practical use of it. If we assign certainties to our beliefs on a scale of 0 to 100, then what can I realistically do with a bit of evidence that moves me from 87 to 87.01? or 86.99? I don't think I can estimate my certainty accurately to 1 decimal place--in fact I'm not sure I can get it to within one significant digit on many issues--and yet there's a lot of evidence in the world that should move my beliefs by a lot less than that.

Mathematically it makes sense to update on all evidence. Practically, there is a fuzzy threshold beyond which I need to just ignore very weak evidence, unless there's so much of it that the sum total crosses the bounds of significance.

We should unpack "banish talk of X" to mean that we should avoid assessments/analysis that would naturally be expressed in such surface terms.

Since most of us don't do deep thinking unless we use some notation or words, "banish talk of" is a good heuristic for such training, if you can notice yourself (or others can catch you) doing it.

A related mistake I made was to be impressed by the cleverness of the aphorism "The plural of 'anecdote' is not 'data'." There may be a helpful distinction between scientific evidence and Bayesian evidence. But anecdotal evidence is evidence, and it ought to sway my beliefs.

The selection biases in anecdotes make them nearly useless for updating. A more correct version would be that you can update on the first anecdote, less on a similar second one, even less on a third, and so on. Once you have ten or so anecdotes pointing in the same direction, then extra anecdotes should have essentially no impact.

So yes, the plural of anecdote is not data. Their value does not scale in the same way.

One thing it apparently taught Jaynes:

For some sixty years it has appeared to many physicists that probability plays a fundamentally different role in quantum theory than it does in statistical mechanics and analysis of measurement errors. It is a commonly heard statement that probabilities calculated within a pure state have a different character than the probabilities with which different pure states appear in a mixture, or density matrix. As Pauli put it, the former represents "Eine prinzipielle Unbestimmtheit, nicht nur Unbekanntheit". But this viewpoint leads to so many paradoxes and mysteries that we explore the consequences of the unified view, that all probability signifies only incomplete human information.

Did you learn these lessons exclusively by exposing yourself to the Bayesian ideas floating on LessWrong, or would you credit these insights at least partly to "external" sources? You mention Jaynes's book. Has reading this book taught you some of the lessons you list above? Is there other material you'd recommend for folks interested in having more Bayesian Aha! moments?

common-sense rationality dressed up as intimidating math.

I'd just like to note that Bayes' Rule is one of the first and simplest theorems you prove in introductory statistics classes after you have written down the preliminary definitions/axioms of probability. It's literally taught and expected that you are comfortable using it after the first or second week of classes.

Honestly, I feel like if Eliezer had left out any mention of the math of Bayes' Theorem from the sequences, I would be no worse off. The seven statements you wrote seem fairly self-evident by themselves. I don't feel like I need to read that P(A|B) > P(A) or whatever to internalize them. (But perhaps certain people are highly mathematical thinkers for whom the formal epistemology really helps?)

Lately I kind of feel like rationality essentially comes down to two things:

1. Recognizing that as a rule you are better off believing the truth, i.e. abiding by the Litany of Tarski.

2. Having probabilistic beliefs, i.e. abiding by the Bayesian epistemology and not the Aristotelian or the Anton-Wilsonian as Yvain defined in his reaction to Chapman, or having an many-color view as opposed to a two-color view or a one-color view as Eliezer defined in the Fallacy of Gray.

Once you've internalized these two things, you've learned this particular Secret of the Universe. I've noticed that people seem to have their minds blown by the sequences, not really learn all that much more by spending a few years in the rationality scene, and then go back to read the sequences and wonder how they could have ever found them anything but obvious. (Although apparently CFAR workshops are really helpful, so if that's true that's evidence against this model.)

If you are not going to do an actual data analysis, then I don't think there is much point of thinking about Bayes' rule. You could just reason as follows: "here are my prior beliefs. ooh, here is some new information. i will now adjust my believes, by trying to weigh the old and new data based on how reliable and generalizable i think the information is." If you want to call epistemology that involves attaching probabilities to beliefs, and updating those probabilities when new information is available, 'bayesian' that's fine. But, unless you have actual data, you are just subjectively weighing evidence as best you can (and not really using Bayes' rule).

The thing that can be a irritating is when people then act as if that kind of reasoning is what bayesian statisticians do, and not what frequentist statisticians do. In reality, both types of statisticians use Bayes' rule when it's appropriate. I don't think you will find any statisticians who do not consider themselves 'bayesian' who disagree with the law of total probability.

If you are actually going to analyze data and use bayesian methods, you would end up with a posterior distribution (not simply a single probability). If you simply report the probability of a belief (and not the entire posterior distribution), you're not really doing conventional bayesian analysis. So, in general, I find the conventional Less Wrong use of 'bayesian' a little odd.

Bayesianism boils down to “don’t be so sure of your beliefs; be less sure when you see contradictory evidence.” Now that is just common sense.

...Common? Maybe in successful circles of academia.

Why does anyone need to be told this?

What a bizarre question. I find it difficult to believe that this person has any experience with the average/median citizen.

Probabilities go up or down, but there is no magic threshold beyond which they change qualitatively into "knowledge".

Matter of fact there are thresholds below which extra processing cost does not pay off (or, in case of human head, when it is extremely implausible that the processing is even going to be performed correctly).

Probabilistic reasoning is, in general, computationally expensive. In many situations, a very large number of combinations of uncertain parameters has to be processed, the cross correlations must be accounted for, et cetera.

Actions conditioned on evidence generally have higher expected utility than those not conditioned on evidence, and processing of beliefs conditioned on evidence likewise so.

The expected utility sums for things such as expenditure of resources have a term for resources being kept for the future uses which may be more conditional on evidence, and the actions that are less evidence conditioned than usual ought to lose out to the bulk of possible ways one may act in the future (edit: the ways which you can't explicitly enumerate).

That's just some of the thresholds that an optimally programmed intelligence (on a physically plausible computer) would apply.

This is sort of like going on about what Maxwell's equations taught you about painting. Maxwell's equations are quite far from painting, about as far in terms of inference length as Bayes theorem is from most actual decision making or belief forming. edit: make that much further, actually, considering that there's no AI.

Let's make an example to clarify. There is Bob. Bob being rich would be evidence for him having a good job. Bob having a good job would be evidence for him being rich. Both of those would be evidence with regards to Bob's education, and so on and so forth. Everything cross correlates to everything else, the belief propagation is NP complete, the algorithms for computing it are very nontrivial, and various subtle implementation errors would make everything converge on a completely wrong value.

Strengths of relevant relations between beliefs about Bob are themselves beliefs, so the graph is pretty damn huge. When known probabilities get fairly close to 0 and 1, it is tractable whenever unknowns are close to 0 or 1. But when they're closer to the middle, you're dealing with a very complicated relation. And if you could compute the resulting equation in your head, well, there's a lot of lesser engineering tasks that you should absolutely breeze through.

edit: expanded a bit. Really, we do know how to progressively approximate from quantum electrodynamics to geometric optics to drawing 3d shapes to painting, but we do not know how to get from Bayes theorem to a full blown AI on physically plausible hardware.

Occam's razor should be on your list. Not in the "Solomonoff had the right definition of complexity" sense, but in the sense that any proper probability distribution has to integrate to 1, and so for any definition of complexity satisfying a few common sense axioms the limit of your prior probability has to go to zero as the complexity goes to infinity.

I think you've oversimplified the phrasing of 6 (not your fault, though; more the fault of the English language). Although your expected value for your future estimate of P(H) should be the same as your current estimate of P(H), that doesn't imply symmetry of expected future evidence. For example, I have a very high expectation that future evidence will very slightly increase my already very strong belief that aliens are not visiting Earth; this is mostly balanced out by a very tiny expectation that future evidence will strongly decrease that belief.

Banish talk of “thresholds of belief”. Probabilities go up or down, but there is no magic threshold beyond which they change qualitatively into “knowledge”

There might be a normative rule to that effect , but probabilities in your brain can't change by infinitesimal increments. Bayes as applied by cognitively limited agents like humans has to have some granularity.

Abandon all the meta-epistemological intuitions about the concept of knowledge on which Gettier-style paradoxes rely. Keep track of how confident your beliefs are when you update on the evidence. Keep track of the extent to which other people's beliefs are good evidence for what they believe. Don't worry about whether, in addition, these beliefs qualify as "knowledge".

Why not? Because it is not useful? Because those problems have been solved?

In the interests of full disclosure :-)

I think of Bayesianism as a philosophy in statistics, specifically one which is opposed to frequentism. The crucial difference is the answer to the question "What does 'probability' mean?"

There is also Bayesian analysis in statistics which is a toolbox of techniques and approaches (see e.g. Gelman) but which is not an "-ism".

I do not apply the label of "Bayesianism" to all the Kahneman/Tversky kind of stuff, I tend to think of it a mind bugs (in the programming sense) and mind hacks.

I do not think of Bayesianism as an epistemology. I understand Yvain's middle ground between Aristotle and Robert Anson Wilson, but it looks pretty obvious to me and I wouldn't call it Bayesianism.

Absence of evidence is evidence of absence.

Only provided you have looked, and looked in the right place.

Probabilities go up or down, but there is no magic threshold beyond which they change qualitatively into "knowledge".

Well, at some point the upper bound of consequences for being wrong multiplied by the likelihood that you expect to be wrong is so tiny that it's worth less than the mental overhead of keeping track of the level of certainty. Like, I'm confident enough that physics works (to the level of everyday phenomena), that keeping track of what might happen if I'm wrong about that doesn't add enough value to be worthwhile.

So it's about a protocol for language instead?

I'd just like to point out that even #1 of the OP's "lessons" is far more problematic than they make it seem. Consider the statement:

"The fact that there are myths about Zeus is evidence that Zeus exists. Zeus's existing would make it more likely for myths about him to arise, so the arising of myths about him must make it more likely that he exists." (supposedly an argument of the form P(E | H) > P(E)).

So first, "Zeus's existing would make it more likely for myths about him to arise" - more likely than what? Than "a priori"? This is essentially impossible to know, since to compute P(E) you must do P(E) = sum(i) { P(E|H[i])*P(H[i]) }, i.e. marginalise over a mutually exclusive set of hypotheses (and no "Zeus" and "not Zeus" does not help, because "not Zeus" is a compound hypothesis which you also need to marginalise over).

I will grant you that it may seem plausible to guess that the average P(E|H[i]) over all possible explanations for E is lower than P(E|Zeus) (since most of them are bad explanations), but since the average is weighted by the various priors P(H[i]), then if our background knowledge causes some high likelihood explanation for E (high P(E|H[i])) to dominates the average then P(E) may not be less than P(E|Zeus) even if P(E|Zeus) is relatively high! In which case E actually counts against the Zeus hypothesis, since P(H|E)<P(H) if P(E|H)<P(E).

Whether this is the case or not in the example is tough to say, (and of course is relative to the agents background knowledge), but I think it worth emphasising that it is not so easy as it seems.

The post on whether you're "entitled" to evidence has always annoyed me a bit... what does "entitled" even mean? If the person you're talking to isn't updating on the evidence you're giving them for some bad reason, what can you really do?

Banish talk like "I don't know anything about that".

In I don't know, Eliezer isn't arguing that you shouldn't say it, but that you shouldn't think it:

what you say is another issue, especially when speaking to nonrationalists, and then it is well to bear in mind that words don't have fixed meanings; the meaning of the sounds that issue from your lips is whatever occurs in the mind of the listener. If they're going to misinterpret something then you shouldn't say it to them no matter what the words mean inside your own head

"I don't know" is a good way to duck when you say it to someone who doesn't know about probability distributions. If they thought I was certain, or that my statement implied actual knowledge of the tree then the statement would mislead them.

Point by point take on this:

1. Banish talk like "There is absolutely no evidence for that belief". But anecdotal evidence is evidence, and it ought to sway my beliefs.

The evidence can be weak enough and/or be evidence for an immense number of other things besides what evidence claims it is evidence for, as to be impossible to process qualitatively. If there's "no evidence", the effect size is usually pretty small, much smaller than the filtering that the anecdotes pass through, much smaller than can be inferred qualitatively, etc.

2.

Banish talk like "I don't know anything about that".

There's great many things that you have never even thought of, and you know nothing about those things. They have no probabilities assigned, and worse, they work as if they had probability of zero. And you can't avoid that, because there's far more things you ought to enumerate than you can enumerate, by a very very huge factor.

Having heard of something leads to quite non-Bayesian change of the belief (effective zero to non-zero). In light of this, degrees of beliefs are not probabilities, but some sort of tag values attached to the propositions that were considered (a very small subset of the totality of propositions), tags which need to be processed in a manner as to arrive at most accurate choices in the end despite absence of processing of the vast majority of relevant beliefs. (A manner which does resemble probability theory to some extent)

Treating them more like probabilities will just lead to a larger final error, even though superficially the edit distance from your algorithm to some basic understanding of probability theory can seem smaller.

3.

Banish talk of "thresholds of belief".

Imposing thresholds on both beliefs and evidence in support of the beliefs allows you to compensate for and decrease the consequences of the unavoidable errors described above. The thresholds have been established after a very long history of inferring some completely wrong conclusions based on accumulation of evidence that was weaker than what can be usefully processed qualitatively but instead requires very accurate set up and quantitative calculations.

4.

Absence of evidence is evidence of absence.

Sometimes, and sometimes it's really weak evidence that isn't statistically independent from the belief that it ought to affect.

5.

Many bits of "common sense" rationality can be precisely stated and easily proved within the austere framework of Bayesian probability.

But you threw away those that can not be easily demonstrated.

6.

You cannot expect[2] that future evidence will sway you in a particular direction.

Theorems of probability are not going to hold exactly for the optimum value that should be assigned to the beliefs in the light of what's described in 2, and working as ff they do hold can not be expected to improve outcomes.

Keep in mind that you can reasonably expect that in the future great many things that you have never thought of may be brought to your attention, without being able to actually enumerate and process a significant fraction of them right now and then.

edit: improved that some. Also, many of those limitations would hold for any physically plausible Jupiter Brains, Matroshka Brains, or other such giant objects which, while they can process great many more beliefs than you can, are still stuck with processing only a minuscule fraction of the beliefs they ought to process.

edit2: interestingly, David Chapman touches on much same points.

The fact that there are myths about Zeus is evidence that Zeus exists.

That doesn't look useful to me.

By the same token, my mentioning here the name of the monster Ygafalkufeoinencfhncfc is evidence that it exists. Funnily enough, the same reasoning provides evidence for the monster Grrapoeiruvnenrcancaef and a VERY large number of other, ahem, creatures.