Are you being p-hacked? Time to hack back.

post by Jacob Falkovich (Jacobian) · 2016-12-20T16:57:00.657Z · LW · GW · Legacy · 8 comments

This is a link post for


Comments sorted by top scores.

comment by cousin_it · 2016-12-21T22:49:32.194Z · LW(p) · GW(p)

Thank you for writing that! To sum up:

Alice: I got this result with p<0.05

Bob: How many hypotheses did you test?

Alice: 10

Bob: Then you got the result with p<0.5

This is going straight into my toolbox.

comment by Lumifer · 2016-12-20T22:16:51.270Z · LW(p) · GW(p)


There's some... tension between this correct statement and the rest of your blog post :-/

Replies from: Jacobian
comment by Jacob Falkovich (Jacobian) · 2016-12-21T19:00:15.177Z · LW(p) · GW(p)

You're right, I added a paragraph explaining why I do this:

P-values aren’t very useful in getting close to the truth, but they’re everywhere, they’re easy to work with and they’re moderately useful for getting away from bullshit. Since the latter is our goal in this essay we’ll stick with looking at p-values for now.

Since you see p-values reported all the time and they're easy to calculate (because they're uniformly distributed), there's value in extracting whatever useful evidence there is in p-values.

comment by gjm · 2016-12-20T21:36:03.779Z · LW(p) · GW(p)

Unfortunately for the author's "Rule of two Ns", "Bonferonni" is actually spelt "Bonferroni".

Replies from: Jacobian
comment by Jacob Falkovich (Jacobian) · 2016-12-21T00:20:32.251Z · LW(p) · GW(p)

I feel like such a dumbass.

Replies from: MrMind, gjm
comment by MrMind · 2016-12-21T09:02:24.822Z · LW(p) · GW(p)

You could correct the post as "Both 'Dunn' and 'Bonferroni' have exactly two 'n' in them. Thus we have etc."

Replies from: Jacobian
comment by Jacob Falkovich (Jacobian) · 2016-12-21T19:02:53.521Z · LW(p) · GW(p)

That's a good suggestion, but I actually had to update that entire section because the math had an embarrassing mistake and I renamed the rule.

By the way, I have a policy on my blog of paying bounties to commenters who catch major errors. I sent $5 to the commenter who pointed out where I messed up the math. If you see a really stupid mistake on Putanumonit, leave a comment there and make a buck.

comment by gjm · 2016-12-21T01:56:12.740Z · LW(p) · GW(p)