Strong Evidence is Common

Corollary: most beliefs worth having are extreme.

There is a subtlety here. Large updates from extremely unlikely to quite likely are common. Large updates from quite likely to exponentially sure are harder to come by. Lets pick an extreme example, suppose a friend builds a coin tossing robot. The friend sends you a 1mb file, claiming it is the sequence of coin tosses. Your probability assigned to this particular sequence being the way the coin landed will jump straight from $2^{-8,000,000}$ to somewhere between 1% and 99% (depending on the friends level of trustworthiness and engineering skill) Note that the probability you assign to several other sequences increases too. For example, its not that unlikely that your friend accidentally put a not in their code, so your probability on the exact opposite sequence should also be $>>2^{-8,000,000}$ Its not that unlikely that they turned the sequence backwards, or xored it with pi or ... Do you see the pattern. You are assigning high probability to the sequences with low conditional Komolgorov complexity relative to the existing data. 

Now think about what it would take to get a probability of $1-2^{-8,000,000}$ on the coin landing that particular sequence. All sorts of wild and wacky hypothesis have probability $>2^{-8,000,000}$ . From the boring stuff like a dodgy component or other undetected bug, to more exotic hypothesis like aliens tampering with the coin tossing robot, or dark lords of the matrix directly controlling your optic nerve. You can't get this level of certainty about anything ever. (modulo concerns about what it means to assign p<1 to probability theory) 

You can easily update from exponentially close to 0, but you can't update to exponentially close to one. This may have something to do with  there being exponentially many very unlikely theories to start off with. But only a few likely ones.

If you have 3 theories that predict much the same observations, and all other theories predict something different, you can easily update to "probably one of these 3". But you can't tell those 3 apart. In AIXI, any turing machine has a parade of slightly more complex, slightly less likely turing machines trailing along behind it. The hypothesis "all the primes, and grahams number" is only slightly more complex than "all the primes", and is very hard to rule out. 

Meta point: this is one of those insights which is very likely to hit you over the head if you're doing practical technical work with probabilitistic models, but not if you're just using them for small semi-intuitive problems (a use-case we often see on LW).

I remember the first time I wrote a mixture of gaussians clustering model, and saw it spitting out probabilities like 10^-5000, and thought it must be a bug. It wasn't a bug. Probabilities naturally live on a log scale, and those sorts of numbers are normal once we move away from artificially-low-dimensional textbook problems and start working with more realistic high-dimensional systems. When your data channel has a capacity of kilobytes or megabytes per data point, even if 99% of that information is irrelevant, that's still a lot of bits; the probabilities get exponentially small very quickly.

Tying back to an example in the post: if we're using ascii encoding, then the string "Mark Xu" takes up 49 bits. It's quite compressible, but that still leaves more than enough room for 24 bits of evidence to be completely reasonable.

I've curated this post. It's really changed my worldview, to realize how often I was able to quickly get substantial amounts of evidence, and resolve an important question. It's also related to The First Sample Gives the Most Information [LW · GW].

I also admire how short this post is. I do not know how to write such short posts.

It's worth noting that most of the strong evidence here is in locating the hypothesis [? · GW].
That doesn't apply to the juggling example - but that's not so much evidence. "I can juggle" might take you from 1:100 to 10:1. Still quite a bit, but 10 bits isn't 24.

I think this relates to Donald's point [LW(p) · GW(p)] on the asymmetry between getting from exponentially small to likely (commonplace) vs getting from likely to exponentially sure (rare). Locating a hypothesis can get you the first, but not the second.

It's even hard to get back to exponentially small chance of x once it seems plausible (this amounts to becoming exponentially sure of ¬x). E.g., if I say "My name is Mortimer Q. Snodgrass... Only kidding, it's actually Joe Collman", what are the odds that my name is Mortimer Q. Snodgrass? 1% perhaps, but it's nowhere near as low as the initial prior.
The only simple way to get all the way back is to lose/throw-away the hypothesis-locating information - which you can't do via a Bayesian update. I think that's what makes privileging the hypothesis [LW · GW] such a costly error: in general you can't cleanly update your way back (if your evidence, memory and computation were perfectly reliable, you could - but they're not). The way to get back is to find the original error and throw it out.

How difficult is it to get into the top 1% of traders? To be 50% sure you're in the top 1%, you only need 200:1 evidence. This seemingly large odds ratio might be easy to get.

I don't think your examples say much about this. They're all of the form [trusted-in-context source] communicates [unlikely result]. They don't seem to show a reason to expect strong evidence may be easy to get when this pattern doesn't hold. (I suppose they do say that you should check for the pattern - and probably it is useful to occasionally be told "There may be low-hanging fruit. Look for it!")

One implication of the Efficient Market Hypothesis (EMH) is that is it difficult to make money on the stock market. Generously, maybe only the top 1% of traders will be profitable. 

Nitpick: it's incredibly easy to make money on the stock market: just put your money in it, ideally in an index fund. It goes up by an average of 8% a year. Almost all traders will be profitable, although many won't beat that 8% average. 

The entire FIRE movement is predicated on it being incredibly simple to make money on the stock market. It takes absolutely zero skill to be a sufficiently profitable trader, given a sizeable enough initial investment. 

I get that you're trying to convey above-market-rate returns here, but your wording is imprecise. 

I really like this post. It's a crisp, useful insight, made via a memorable concrete example (plus a few others), in a very efficient way. And it has stayed with me. 

Linking my comment from the Forum [EA · GW]:

I think in the real world there are many situations where (if we were to put explicit Bayesian probabilities on such beliefs, which we almost never do), beliefs with ex ante ~0 credence quickly get extraordinary updates. My favorite example is sense perception. If I woke up after sleeping on a bus and were to put explicit Bayesian probabilities on anticipating what I will see next time I open my eyes, then my belief I'd assign in the true outcome (ignoring practical constraints like computation and my near inability to have any visual imagery) has ~0 credence. Yet it's easy to get strong Bayesian updates: I just open my eyes. In most cases, this should be a large enough update, and I go on my merry way. 

But suppose I open my eyes and instead see  people who are  approximate lookalikes of dead US presidents sitting around the bus. Then at that point (even though the ex ante probability of this outcome and that of a specific other thing I saw isn't much different), I will correctly be surprised, and have some reasons to doubt my sense perception.

Likewise, if instead of saying your name is Mark Xu, you instead said "Lee Kuan Yew", I at least would be pretty suspicious that your actual name is Lee Kuan Yew.

I think a lot of this confusion in intuitions can be resolved by looking at what MacAskill calls the difference between unlikelihood and fishiness [EA · GW]:

Lots of things are a priori extremely unlikely yet we should have high credence in them: for example, the chance that you just dealt this particular (random-seeming) sequence of cards from a well-shuffled deck of 52 cards is 1 in 52! ≈ 1 in 10^68, yet you should often have high credence in claims of that form.  But the claim that we’re at an extremely special time is also fishy. That is, it’s more like the claim that you just dealt a deck of cards in perfect order (2 to Ace of clubs, then 2 to Ace of diamonds, etc) from a well-shuffled deck of cards. 

Being fishy is different than just being unlikely. The difference between unlikelihood and fishiness is the availability of alternative, not wildly improbable, alternative hypotheses, on which the outcome or evidence is reasonably likely. If I deal the random-seeming sequence of cards, I don’t have reason to question my assumption that the deck was shuffled, because there’s no alternative background assumption on which the random-seeming sequence is a likely occurrence.  If, however, I deal the deck of cards in perfect order, I do have reason to significantly update that the deck was not in fact shuffled, because the probability of getting cards in perfect order if the cards were not shuffled is reasonably high. That is: P(cards not shuffled)P(cards in perfect order | cards not shuffled) >> P(cards shuffled)P(cards in perfect order | cards shuffled), even if my prior credence was that P(cards shuffled) > P(cards not shuffled), so I should update towards the cards having not been shuffled.

Put another way, we can dissolve this by looking explicitly at Bayes' theorem. $P\left(Hypothesis|Evidence\right)=\frac{P\left(Evidence|Hypothesis\right)\ast P\left(Hypothesis\right)}{P\left(Evidence\right)}$

and in turn, $P\left(Evidence\right)=P\left(Evidence|Hypothesis\right)\ast P\left(Hypothesis\right)+P\left(Evidence|OtherHypotheses\right)\ast P\left(OtherHypotheses\right)$

$P\left(Evidence|Hypothesis\right)$ is high in both the "fishy" and "non-fishy" regimes. However,$P\left(Evidence|OtherHypotheses\right)$ is much higher for fishy hypotheses than  for non-fishy hypotheses, even if the surface-level evidence looks similar!

Maybe a nitpick, but the driver's license posterior of 95% seems too high. (Or at least the claim isn't stated precisely.) I'd have less than a 95% success rate at guessing the exact name string that appears on someone's driver's license. Maybe there's a middle name between the "Mark" and the "Xu", maybe the driver's license says "Marc" or "Marcus", etc.

I think you can get to 95% with a phone number or a wifi password or similar, so this is probably just a nitpick.

First, the main point of the post as written is correct: strong evidence is common. Timid underconfidence in the face of overwhelming evidence is just as irrational as overconfidence when the prior and evidence is underwhelming.

I have two main concerns with this post:

• Conflating uncorrelated hypotheses: This post asks "how difficult is it to get into the top 1% of traders?" but immediately substitutes asking if "you're in the top 1%" as if it were the same question.
• Cherry picking: The fact that debate exists means that the hypotheses are pre-selected for imprecision and the evidence for weakness.

People are imprecise, motivated reasoners, chasing incentives that lead them away from a rigid focus on accurate thought, even in their areas of expertise. Especially given the imprecisions in how this post is written, I worry readers will conflate these two:

1. Strong evidence is common.
2. Strong evidence is common for issue X, and you’ll know strong evidence when you see it.

This post is in my small list of +9s that I think count as a key part of how I think, where the post was responsible for clarifying my thinking on the subject. I've had a lingering confusion/nervousness about having extreme odds (anything beyond 100:1) but the name example shows that seeing odds ratios of 20,000,000:1 is just pretty common. I also appreciated Eliezer's corollary: "most beliefs worth having are extreme", this also influences how I think about my key beliefs.

(Haha, I just realized that I curated it back when it was published.)

This is insightful.  The areas where strong evidence is common are largely those areas we don't intuitively think of as governed by probability theory and where classic logic performs well.  

It seems like someone could take this a little further even and show that the limiting case for strong evidence and huge likelihood ratios would just be logic.  This might be fruitful to unpack.  I could see it being the case that our instincts for seeking "certainty" make more sense than we give them credit for.  Gathering enough evidence sometimes allows reasoning to be performed using propositional logic with acceptable results. 

Such logic is many orders of magnitude cheaper to evaluate compute wise vs probabilistic reasoning, especially as we get into larger and larger causal networks.  There’s an obvious tradeoff between the cost to obtain more evidence vs more compute – it’s not always a choice that’s available (e.g., spend time investigating vs. spend time thinking/tinkering with models) but is often enough.

When I think about how I’m using the reasoning skills I’ve picked up here that’s roughly what I’m having to do for real-world problems.  Use probabilistic reasoning to resolve simpler more object level propositions into true/false/maybe, then propositional logic to follow the implications.  Fail back to probabilistic reasoning whenever encountering a paradox or any of the other myriad problems with simple logic – Or just for periodic sanity checking.

I really like this post! I have a concerned intuition around 'sure, the first example in this post seems legit, but I don't think this should actually update anything in my worldview, for the real-life situations where I actively think about Bayes Rule + epistemics'. And I definitely don't agree with your example about top 1% traders. My attempt to put this into words:

1. Strong evidence is rarely independent. Hearing you say 'my name is Mark' to person A might be 20,000:1 odds, but hearing you then say it to person B is like 10:1 tops. Most hypotheses that explain the first event well, also explain the second event well. So while the first sample contains the most information, the second sample contains way less. Making this idea much less exciting.

It's much easier to get to middling probabilities than high probabilities. This makes sense, because I'm only going to explicitly consider the odds of <100 hypotheses for most questions, so a hypothesis with say <1% probability isn't likely to be worth thinking about. But to get to 99% it needs to defeat all of the other ones too

Eg, in the 'top 1% of traders' example, it might be easy to be confident I'm above the 90th percentile, but much harder to move beyond that.

2. This gets much messier when I'm facing an adversarial process. If you say 'my name is Mark Xu, want to bet about what's on my driver's license' this is much worse evidence because I now face adverse selection. Many real-world problems I care about involve other people applying optimisation pressure to shape the evidence I see, and some of this involves adversarial potential. The world does not tend to involve people trying to deceive me about world capitals.

An adversarial process could be someone else trying to trick me, but it could also be a cognitive bias I have, eg 'I want to believe that I am an awesome, well-calibrated person'. It could also be selection bias - what is the process that generated the evidence I see?

3. Some questions have obvious answers, others don't. The questions most worth thinking about are rarely the ones that are obvious. The ones where I can access strong evidence easily are much less likely to be worth thinking about. If someone disagrees with me, that's at least weak evidence against the existence of strong evidence.

The prior odds that someone’s name is “Mark Xu” are generously 1:1,000,000. Posterior odds of 20:1 implies that the odds ratio of me saying “Mark Xu” is 20,000,000:1, or roughly 24 bits of evidence. That’s a lot of evidence.

There are 26 letters in the English alphabet. Even if, for simplicity, our encoding ignores word boundaries and message ending, that's ${\log }_{2}26\approx 4.7$ bits per letter so hearing you say "Mark Xu" is 28.2 bits of evidence total - more than the 24 bits required.

Of course - my encoding is flawed. An optimal encoding should assign "Mark Xu" with less bits than, say, "Rqex Gh" - even though they both have the same amount of letters. And "Maria Rodriguez" should be assigned an even shorter message even though it has more than twice the letters of "Mark Xu".

Measuring the amount of information given in messages is not as easy to do on actual real life cases as it is in theory...

To be 50% sure you're in the top 1%, you only need 200:1 evidence. This seemingly large odds ratio might be easy to get.

Of course it's easy! You just compare how much you've made, and how long you've stayed solvent, against the top 1% of traders. If you've already done just as well as the others, you'd in the top 1%. Otherwise, you aren't.

It's much harder to become a top 1% trader. That's the point of the EMH and Eliezer's Inadequate Equilibria. Actually, some forms of the EMH go further and say that there's no such thing as a "top trader," because there are only two types: the lucky and the insolvent. There is no way to beat the market trading on publicly available information.

One particularly important thing I got out of this post was crystallizing a complaint I sometimes have about people using anthropic reasoning. If someone says there's trillion-to-1 evidence for (blah) based on anthropics, it's actually not so crazy to say "well I don't believe (blah) anyway, based on the evidence I get from observing the world", it seems to me.

Or less charitably to myself, maybe this post is helping me rationalize my unjustified and unthinking gut distrust of anthropic reasoning :-P

Anyway, great post.

This interacts with the efficient market hypothesis in an interesting way. Suppose a magic button test that would instantly and freely tell anyone if they would make money (in expectation) on the market. Its not omniscient, but it is epistemically efficient compared to humans. All hopefuls press the button, and only trade if they pass the test. Those that pass make money. That money is coming from the people who actually want to buy or sell stuff, and the stubbornest of those who failed the magic button test.

(Assuming that people don't try to use this as a way to send info back in time, or otherwise munchkin it.)

I’m guessing they would have happily accepted a bet at 20:1 odds that my driver’s license would say “Mark Xu” on it.

I think they wouldn't have, mostly because you (or someone) offering the bet is fairly strong evidence that the name on your driver's license is not, in fact, "Mark Xu".

This post is short, but important. The fact that we regularly receive enormously improbable evidence is relevant for a wide variety of areas. It's an integral part of having accurate beliefs, and despite this being such a key idea, it's underappreciated generally (I've only seen this post referenced once, and it's never come up in conversation with other rationalists). 

This is why we need different words for different kinds of "evidence"; one for "root evidence", one or two more for evidence resilient against some kind of human mistakenness and bias, etc.

How much evidence is breaking into the top 50 on metaculus in ~6 months?

I stayed out of finance years ago because I thought I didn't want to compete with Actually Smart People.

Then I jumped in when the prediction markets were clearly not dominated by the Actually Smart.

But I still don't feel confident to try in the financial markets.

I’m guessing they would have happily accepted a bet at 20:1 odds that my driver’s license would say “Mark Xu” on it.

Personally, I wouldn't do it at 20:1 odds even if you said your name was "John Smith," purely because of how many people go by a name different than the one on their driver's license.