Why we need better science, example #6,281

Well, "reality isn't weird" can mean a couple of different things. "Weird" is a two-part predicate like "sexiness"; things are only weird in reference to some particular mind's preconceptions. Even Yog-Sothoth doesn't seem weird to his own mother.

But if we use the word "weird" as a red flag to tell others that they can expect to be surprised or confused when entering a certain field, as long as we can predict that their minds and preconceptions work somewhat like ours, it's a useful word.

I think Eliezer's "reality is not weird" post was just trying to say that we can't blame reality for being weird, or expect things to be irreducibly weird even after we challenge our preconceptions. I don't think Eliezer was saying that we can't describe anything as "weird" if it actually exists; after all, he himself has been known to describe certain potential laws of physics as weird.

(man, basing an argument on the trivial word choices of a venerated community leader spotted in an old archive makes me feel so Jewish)

And reality knows more. That's why I advocate checking with reality.

It's not about a positive result meaning something will "assuredly work for you". Only a Sith deals in absolutes. It's about cost-benefit analysis.

To give an example, no reasonable person would self-experiment to see if cyanide cures their rash. Although there's a distant probability your body has some wildly unusual reaction to cyanide in which it cures rashes, it's much more likely that cyanide will kill you, the same way it kills everyone else. Although it might be worth a shot if cyanide had no downside, we have very strong evidence that on average it has a very large downside.

The same is true of HRT. People were using it to improve their cardiovascular health. We found that, on average, it decreases cardiovascular health. You can still try using it on the grounds that it might paradoxically increase yours, but on average, you will lose utility.

Consider the analogy to a lottery. You have different numbers than everyone else does. Just because someone else lost the lottery with their numbers, doesn't mean you will lose the lottery with your numbers. But if we study all lottery participants for ten years and find that on average they lose money, then unless you have a specific reason to think your numbers are better than everyone else's (not just different), you should expect to lose money too.

Now things would be different with a treatment with no downside (like eating a lot of some kind of food, or taking a safe and cheap supplement) - as long as you don't mind the loss of time and money you can experiment all you want with those (though I still think you'd have trouble with bias and privileging the hypothesis, and that a rational person wouldn't find a lot of these harmless self-experiments worth the time and the money at all). And things would be different if the potential benefit and potential harm had different levels of utility for you: for example, if you wanted to cure your joint pain so badly you didn't mind risking heart attack as a side effect. I think this is what you're aiming at in your post above, and for those cases, I agree with you.

But when you're taking a treatment like HRT which is intended to prevent heart attacks, but actually on average increases heart attacks, then shut up and multiply.

Also, don't call it "self-experimentation" when you're talking about preventing cardiovascular disease, since you never end up with any usable self-data (as opposed to, say, self-experimenting with medication for joint pain, where you might get a strong result of your joint pain disappearing that you can trace with some confidence to the medication). Call it what it is - gambling.

We don't limit our pool of potential solutions to our problems to solutions "validated" by double blinded placebo controlled trials in any other aspect of life, because it isn't rational to do so.

Wrong. We don't do it because either there are no publications that answer our questions, so that we have to use something else, or because we have a more convenient method that works. Please don't appeal to "rationality".

Second, because I know others are not identical to me. Basing my choices solely on some statistical outcome on a pool of patients where I have none of that kind of information, and indeed the doctors involved didn't take that information, and didn't factor it into their solutions for their patients, strikes me as throwing out most all of my relevant data and trusting the results produced by a blind man with a shotgun.

Information is relevant only to the extent you can use it. How specifically can you use it to improve on prior provided by studies, and why would that modified estimate be an improvement? (Every improvement is a change, but not every change is an improvement.)

If it's on the border of the confidence interval, the probability of false positive is 50%. If it's twice as far as that, it's 5%. That should give an okay idea of where the range is for that. I'd much prefer something with a 99% confidence interval, but a 95% one is still pretty good. If the effect is just barely statistically significant, the odds ratio is still 10:1

I'm not sure how likely it is for their hypothesis to be true, but it's likely enough for them to risk spending money checking.

Where did I state or imply that the results of a clinical trial are not evidence?

You certainly didn't state it, but I got the impression that you don't think you should pay much attention to clinical trials. Perhaps I inferred what wasn't there.

but generally, failures to reject at 95% proves little.

If failure to reject the null hypothesis proves little, then you're unlikely to reject it even if it's false, which raises the question of why you did the study in the first place. You clearly should have made a big enough study to actually notice things.

Really? I can't sleep. I take a pill, and I fall asleep. I don't take a pill, and I don't fall asleep. How many trials of that do you need to be convinced?

I guess in that case (and many others, such as pain relief) it would work. The example in the original comment was life extension, where it clearly would not. Even so, you could rely on studies to tell you how likely it is to work, and how well it will work if it does. You also can't just rely on testing them yourself, given that only a tiny fraction of things help you sleep, but the same goes for studies.

You admit that you can't find life extension solutions that way.

I meant that you can't find them by testing them on yourself, since you can only do it once, unlike something to help you sleep. You can find them by clinical trials. You just saw an example of that.