post by [deleted] · · ? · GW · 0 comments

This is a link post for

0 comments

Comments sorted by top scores.

comment by Richard_Kennaway · 2022-09-07T09:21:38.209Z · LW(p) · GW(p)

Forty roles gives you 0.9999999999999999999999999999999999999999, a confidence so high it is forbidden to a Bayesian under any circumstances.

Why? You have explicitly assumed a prior distribution:

You have two competing hypotheses:

  • The ten-sided die is fair. It produces the numbers {1,2,3,4,5,6,7,8,9,10} with equal probability.
  • The ten-sided die is weighted. It always produces the number 10.

Actually, you have not specified what probabilities the Bayesian is assigning to these two. Suppose for the moment that it is 50-50.

It is straightforward to combine this with the observed die rolls according to the Bayesian calculation. If the Bayesian is not willing to bet at the resulting odds, that only means that that was not his prior. Bayes' Theorem does not depend on anyone's actual belief about anything.

You omit to say how the Frequentist "bur[ies] wrong beliefs under a mountain of data". Presumably you believe that the data in this example has done so. But what is the Frequentist's argument? "Lookit this mountain of data!"? Actual papers doing statistical reasoning never have such mountains, especially in reproducibility-crisis fields.

comment by Robert Kennedy (istandleet) · 2022-09-06T20:34:37.070Z · LW(p) · GW(p)

I am not sure I understand, probably because I am too preprogrammed by Bayesianism.

You roll a d20, it comes up with a number (let's say 8). The Frequentist now believes there is a 95% chance the die is loaded to produce 8s? But they won't bet 20:1 on the result, and instead they will do something else with that 95% number? Maybe use it to publish a journal article, I guess.

Replies from: lsusr, ege-erdil, Ericf
comment by lsusr · 2022-09-06T21:59:46.709Z · LW(p) · GW(p)

Bayesianism defines probability in terms of belief. Frequentism defines probability as a statement about the world's true probabiliity. Saying "[t]he Frequentist now believes" is therefore asking for a Frequentist's Bayesian probability.

Replies from: istandleet
comment by Robert Kennedy (istandleet) · 2022-09-07T20:32:50.666Z · LW(p) · GW(p)

Right, okay. I am trying to learn your ontology here, but the concepts are not close to my current inferential distance. I don't understand what the 95% means. I don't understand why the d100 has 99% chance to be fixed after one roll, while a d10 only has 90%. By the second roll I think I can start to stomach the logic here though, so maybe we can set that aside.

In my terms, when you say that a Bayesian wouldn't bet $1bil:$1 that the sun will rise tomorrow, that doesn't seem correct to me. It's true that I wouldn't actually make that nightly bet, because the risk free rate is like 3% per annum so it'd be a pretty terrible allocation of risk, plus it seems like it'd be an assassination market on the rotation of Earth and I don't like incentivizing that as a matter of course. But does the math of likelihood ratios not work as well to bury bad theories under a mountain of evidence?

I think not assigning 1e-40 chance to an event is an epistemological choice separate from Bayesianism. The math seems quite capable of leading to that conclusion, and recovering from that state quickly enough.

I think maybe the crux is "There is no way for a Bayesian to be wrong. Everything is just an update. But a Frequentist who said the die was fair can be proven wrong to arbitrary precision." You can, if the Bayesian announces their prior, know precisely how much of your arbitrary evidence they will require to believe the die is loaded.

Again, I hope this is taken in the spirit I mean it, which is "you are the only self proclaimed Frequentist on this board I know of, so you are a very valuable source of epistemic variation that I should learn how to model".

comment by Ege Erdil (ege-erdil) · 2022-09-06T21:01:58.633Z · LW(p) · GW(p)

I strong upvoted this because something about this comment makes it hilarious to me (in a good way).

comment by Ericf · 2022-09-07T00:05:09.895Z · LW(p) · GW(p)

With 2 hypothesis: die is fair/die is 100% loaded, a single roll doesn't discriminate at all. The key insight is that you have to combine Baysean and Frequentist theories. The prior is heavily weighted towards "the die is fair" such that even 3 or 4 of the same number in a row doesn't push the actionable probability all the way to "more likely weighted" but as independent observations continue, the weight of evidence accumulates.

comment by abramdemski · 2022-09-08T20:39:14.756Z · LW(p) · GW(p)

I don't think my first Bayesian critique is "nine nines is too many"; there are physical problems with too much Bayesian confidence (eg "my brain isn't reliable enough that I should really ever be that confident"), but the simple math of Bayesian probability admits the possibility of nine nines just like anyone else.

I think my first critique is the false dichotomy between the null hypotheses and the hypothesis being tested.

Speaking for the frequentist, you say:

If you roll the die nine times and get nine 10s then you can say that the die is weighted with confidence 0.999999999.

I don't think this is what a real frequentist says. I think a real frequentist says something more like, the null hypotheses (the fair-dice hypothesis) can be rejected w/ that confidence. Specifically, the number you are giving is 1 - P(evidence|fair). This is not the numerical confidence in the new hypothesis! This confers some confidence to the alternative hypothesis, but that's better left unquantified, particularly if "falsification is the philosophy of science" as you say. We don't confirm hypotheses; hypotheses are rejected or left standing.

But whether you wear your heart on your sleeve by naively stating that nine nines is the confidence in the new hypothesis, or carefully hedge your words by only stating that we've rejected the null hypothesis of fair dice (and haven't rejected the alternative, wink wink), still, my critique of the reasoning is going to center around the false dichotomy. Frequentism makes it too easy to bury mistakes under mountains of evidence, because it's too easy to be right about what's wrong but wrong about what's right.