[SEQ RERUN] Some Claims Are Just Too Extraordinary

It would depend on the methodology used. I have seen enough examples of horribly bad - not to say utterly fraudulent - science in medical journals that I would actually take publication in a medical journal of a single, unreplicated, study as being slight evidence against the conclusion it comes to. (As an example, the Mayo clinic published a study with precisely those results, claiming to be 'unable to replicate' a previous experiment, in the early 80s. Except that where the experiment they were trying to 'replicate' had used intravenous doses, they used oral ones. And used a lower dose. And spaced the doses differently. And ended the trial after a much shorter period.)

So my immediate conclusion would be "No they didn't" if I saw that result from a single paper. Because when you've seen people in agony, dying, and you've seen them walking around and healthy a couple of months later, and you see that happen repeatably, then that is very strong evidence. And something you can test yourself is always better than taking someone else's word for it.

However, if that study were replicated, independently, and had no obviously cretinous methodological flaws upon inspection, then it would be strong evidence. But if something I don't directly observe myself contradicts my own observations, then I will always put my own observations ahead of those of someone else.

All that said: roughly how many objects have you seen or otherwise had compelling experience of having been dropped in your lifetime, would you estimate? How many of those have fallen to the ground, and how many have floated to the ceiling? What happens to those numbers if we eliminating ones more theoretically likely to float than pens (e.g., helium balloons)?

At a guess, I'd say I've personally seen on the order of five thousand non-helium-balloon-like objects dropped in my life, and they've all fallen down. So just starting from there, I'd estimate a > 99.9998 chance that the next one will, too.

This a good point, 99.9% probably is too low, although I'd be a little worried to go as high as 99.9998%. I'm not sure how you got that figure anyway, if you've seen five-thousand that means Laplace's law suggests about 99.98% doesn't it? I've probably seen a similar number (maybe a bit less), and I can remember one that failed to fall downward (although it was in very high winds so perhaps I should have been able to predict it).

The other evidence probably doesn't count for a Bayes factor of as much as 100:1, since by that point a good part of the remaining probability mass is concentrated in hypotheses like "pens always fall downwards except under this one specific rare circumstance" and "I am completely insane".

Someone else may put the magnet in the pen; this is the sort of concerns you cannot rule out in real life. Or perhaps Buffet is fed up with being rich and wants to award his possessions to some random person and do it in an unusual way.

As I said, least convenient possible world.

As for your estimate for pen falling down instead of up (no cheating assumed) being 99.99%: seriously?

Firstly, as of my recent discussion with TOD my actual estimate is now more like 99.999% than 99.99%, but still way below 99.9999999999%. I do not assume no cheating in this estimate, I merely assume the amount of cheating you'd expect when idly tossing a pen in my room rather than the amount I'd expect when playing billion-dollar games with Warren Buffet.

Does it mean that you expect the pen fall up once in every ten thousand trials?

No. Most of that probability is concentrated in hypotheses where the pen never falls up. This is the difference between a Bayesian probability and a frequency.

I will point out though, that I have probably dropped fewer than 10000 pens in my life and one of them did go sideways (to the best I could tell there was no statistically significant vertical component to its movement), though I suppose I should have predicted that given the weather conditions at the time.

If 99.99% is your upper bound for probability in general (this may be a reasonable interpretation because the pen example is certainly one of the most certain predictions one can state), do you play any lottery where the jackpot is more than 10000 more worth the ticket?

Actually, this is one kind of claim I will happily assign probabilities like 99.9999999999% and higher to, which is the negation of a very specific claim (this is equivalent to the union of a huge number of other claims), for example:

"There is not currently a gang of 17643529 elephants, each painted with the national flag of Moldova, all riding the same giant unicycle around a 57km wide crater on the innermost planet of the 59th closest star to earth in the Andromeda galaxy".

Happy to go well above 99.9999999999% on that one, as there are easily 10^12 mutually exclusive alternatives, all at least as plausible (in addition, by allowing claims like this, but slightly more plausible, I bet I could find a trillion claims each about as plausible as the pen).

What is impressive about the pen, as well as about most scientific theories, is that it is a single, very specific, hypothesis which earned its high probability through accuracy rather than bulk.