Too good to be true

A friend recently posted a link on his Facebook page to an informational graphic about the alleged link between the MMR vaccine and autism. It said, if I recall correctly, that out of 60 studies on the matter, not one had indicated a link.

Presumably, with 95% confidence.

OK so I got interested in this strong claim Phil is making and went to look at the original study he is critiquing so strenuously.

But there is no link to that original study or infographic or whatever.

I don't think there is any value to a strenuous criticism of a study or result when there is no link to that study in the critique.

Presumably, with 95% confidence.

Presumably? I checked the definition of presumably:

used to convey that what is asserted is very likely though not known for certain.

So you take this uncertain confidence level of 95% and find:

What are the odds, supposing there is no link between X and Y, of conducting 60 studies of the matter, and of all 60 concluding, with 95% confidence, that there is no link between X and Y?

Answer: .95 ^ 60 = .046. (Use the first term of the binomial distribution.)

OK so you presumed 95% confidence level and showed that that confidence level is inconsistent with unanimity across 60 studies.

Assuming the studies are good, what confidence level would be consistent with unanimity?

Answer: .99^60 = 54%

So from this result we conclude either 1) there is a a problem with at least some of the studies or 2) there is a problem with the presumption of 95% confidence level, but a 99% confidence level would work fine.

For this post to have positive value, the case for picking only conclusion 1 above, and not considering conclusion 2, needs to be made. If the 95% confidence level is in fact EXPLICIT in these studies, then that needs to be verified, and the waffle-word "presumably" needs to be removed.

I don't think the "95% confidence" works that way. It's a lower bound, you never try to publish anything with a lower than 95% confidence (and if you do, your publication is likely to be rejected), but you don't always need to have exactly 95% (2 sigma).

Hell, I play enough RPGs to know that rolling 1 or 20 in a d20 is frequent enough ;) 95% is quite low confidence, it's really a minimum at which you can start working, but not something optimal.

I'm not sure exactly in medicine, but in physics it's frequent to have studies at 3 sigma (99.7%) or higher. The detection of the Higgs boson by the LHC for example was done within 5 sigma (one chance in a million of being wrong).

Especially in a field with high risk of data being abused by ill-intentioned people such as "vaccine and autism" link, it would really surprise me that everyone just kept happily the 95% confidence, and didn't aim for much higher confidence.

Was that "exactly 95% confidence" or "at least 95% confidence"?

In your "critiquing bias" section you allege that 3/43 studies supporting a link is "still surprisingly low". This is wrong; it is actually surprisingly high. If B ~ Binom(43, 0.05), then P(B > 2) ~= 0.36.*

*As calculated by the following Python code:

from scipy.stats import binom
b = binom(43, 0.05)
p_less_than_3 = sum(b.pmf(i) for i in [0,1,2])
print 1 - p_less_than_3

I don't think this is likely, but one possible explanation is that vaccines prevent autism.

R> library(samplesize)
R> n.wilcox.ord(power = 0.8, alpha = 0.05, t = 0.018, c(0.005,0.995), c(0.010,0.990))
$`total sample size`
[1] 89947

$m
[1] 88328

$n
[1] 1619

Presumably, with 95% confidence.

This seems like a big leap. 95% confidence means at least 95% confidence, right? So if I reject the "vaccines cause autism" hypothesis with p = 0.001, that makes me 95% confident and I publish?

They also have a bias from errors. Many articles have some fatal flaw that makes their results meaningless. If the distribution of errors is random, I think--though I'm not sure--that we should assume this bias causes regression towards an equal likelihood of positive and negative results.

I question this assumption. The distribution of errors that people will make in public communication tracks closely what they can expect to get away with doing. Errors (and non-errors, for that matter) that would result in social sanction will be reviewed more closely before publication, when they are generated at all. If your new satellite powered surveillance technique reports that the Emperor has No Clothes you double check.

The addition of 'noise' to a process with publication bias will enhance the strength of that bias. It also potentially increases regression towards equal negative/positive results. It is not clear which of these opposing influences will be stronger in a given field.

I don't think that it's necessarily suspicious in that, a priori, I wouldn't have a problem with 60 tests all being negative even though they're all only 95% confident.

The reason being, depending on the nature of the test, the probability of a false negative might indeed be 5% while the probability of a false positive could be tiny. Suppose this is indeed the case and let's consider the two cases that the true answer is either 'positive' or 'negative'.

(A) if the true conclusion is 'positive', any test can yield a negative with 5% probability. (this test will be reported as a negative with 95% confidence, though one would expect most tests to yield the positive conclusion.)

(B) if the true conclusion is 'negative', any test that yields a negative will still be reported with the 95% confidence because of the possibility of case (A). Though if it is case (B), we should not expect any positive conclusion, even over 60 tests, because the false-positive rate is so low.

I have no idea if this lack of symmetry is the case for the set of MMR and autism studies. (It probably isn't -- so I apologize that I am probably accomplishing nothing but making it more difficult to argue what is likely a true intuition.)

But it is easy to think of an example where this asymmetry would apply: consider that you are searching for someone that you know well in a crowd, but you are not sure they are there. Consider a test to be looking for them over a 15 minute period, and you estimate that if they are there, you are likely to find them during that 15 minute period with 95% probability. Suppose they are there but you don't find them in 15 minutes -- that is a false negative with 5% probability. Supopse they are not there and you do not find them -- you again say they are not there with 95% probability. But in this case where they are not there, even if you have 60 people looking for them over 15 minutes, no one will find them because the probability of a false positive is pretty much zero.

(I do see where you addressed false positives versus false negatives in several places, so this explanation was not for you specifically since I know you are familiar with this. But it is not so clear which is which in these studies from the top, and it is fleshing this out that will ultimately make the argument more difficult, but more water-tight.)

What would have happened to results that vaccines prevent autism?

When it comes to studies of vaccine side effect there one thing that very worrying. When a new vaccine enter the market there is testing for side effects. Those studies actually do find side effects and the Center of Disease Control should be a trustworthy source for reporting them.

It turn out different vaccine have quite different side effects. They didn't find any Nausea, vomiting, diarrhea, or abdominal pain as a side effect in the Hepatitis A vaccine but 1 of 4 people who take the HPV--Cervarix vaccine get them. Maybe different vaccine work extremely differently and therefore it doesn't make any sense to generalize the risks of one vaccine to the next. Maybe the studies that list the side effects are also to poor.

Even more worrying the side effects reporting system that regular doctors use while the drug is on the market completely fails to capture the side effects that would be expected. If I remember right they find more than two orders of magnitude less side effects than would be predicted based on the studies for bringing a drug to the market.

Stupid mathematical nitpick:

The chances of this happening are only .95 ^ 39 = 0.13, even before taking into account publication and error bias.

Actually, it is more correct to say that .95 ^ 39 = 0.14.

If we calculate it out to a few more decimal places, we see that .95 ^ 39 is ~0.135275954. This is closer to 0.14 than to 0.13, and the mathematical convention is to round accordingly.

Did you save a list of the p-values reported in the 39 (or 43) studies you looked at? I wonder what I'd get if I aggregated them with Fisher's method.

I understand that even a single study indicating a connection would immediately be seized on by anti-vaccination activists. (I've even seen them manage to take a study that indicated no connection, copy a graph in that study that indicated no connection, and write an analysis claiming it proved a connection.) Out there in the real world, maybe it's good to suppress any such studies. Maybe.

Well in light of how the modern scientific processes produces a bias against contrary views the activists' seizing on any studies drawing contrary conclusions appears to be rational. To put it another way if the process is strongly biased against studies that reach contrarian conclusions, any study reaching a contrarian conclusion that survives the process is much stronger evidence than a study that reaches the mainstream conclusion.

First thing, if you put something in your body, it has some effect, even if that effect is small. "No effect" results just rule out effects above different effect sizes (both positive and negative) with high probability, and there's no point talking about "a link" like it's some discrete thing (you sort of jump back and forth between getting this one right and wrong).

Second, different studies will rule out different effect sizes with 95% confidence - or to put it another way, at a given effect size, different studies will have different p-values, and so your probability exercise was pretty pointless because you didn't compare the studies' opinions about any particular effect size, just "whatever was 95%."

Third, I'd bet a nickel the effect sizes ruled out at 95% in all of these studies are well below the point where it would become concerning (like, say, the effect of the parents being a year older). That is, these studies all likely rule out a concerning effect size with probability much better than 95%.

Vaguely interesting, but a p=.13 result in pretty much any field gets a big fat "meh" from me. Sometimes p=.13 results happen. I'd want stronger evidence before I started to suspect bias.

Simple statistics, but eye-opening. I wonder if gwern would be interested enough to do a similar analysis, or maybe he already has.

Yes, that's suspicious. Good instinct. I'm sure there's some bias against publishing a marginally-significant result that's got a low (outside the framework of the paper's statistical model) prior. I'd bet some of the unlucky ones got file-drawered, and others (dishonestly or not) kept on collecting more data until the noise (I presume) was averaged down.

However, you might be missing that on an iso-P contour, false positives have diminishing effect size as sample size increases.

For those who don't know what a case control or cohort study is:

''...Essentially, for a cohort study, you start at the point of exposure, and then follow individuals to see who develops the outcome. In a retrospective cohort, you find information that has recorded participants prior exposures. In a case control study, you start with a group of cases (those with the disease) and controls (those without disease) and then measure exposures. These two designs are similar, but they differ on the starting point (outcome or exposure).; - AL - UoM

This bothered me. What are the odds, supposing there is no link between X and Y, of conducting 60 studies of the matter, and of all 60 concluding, with 95% confidence, that there is no link between X and Y?

What? No. Just... no. You can't say "Because P('result'|H0)=.05, P(~'result'|H1)=.05".

The actual problem is that the surveys routinely ignore all the articles showing vaccines causing problems. There is always a lot of attention to whether thimerosal causes autism, or whether MMR causes autism. There is never any examination of whether the aluminum in vaccines causes autism, or whether getting any vaccine at all at a critical period of brain development causes autism, or whether getting too many vaccines too soon causes autism*, for all of which there are published peer reviewed papers indicating a likelihood they do, or the contaminants in vaccines cause autism (contaminants suggested in the literature include human DNA, which is suggested to cause autism, Simian retroviruses, Simian virus SV-40, and Mycoplasmas), nor any comparison of vaccinated to fully unvaccinated individuals. I did a quick survey of the surveys cited above, and none of them considers aluminum. I also wrote a survey article on some of the literature suggesting early and adjuvanted vaccines cause damage, and none of the dozens of peer reviewed papers I found suggesting damage is cited in the IOM's numerous 800+ page surveys. My survey can be found at http://whyarethingsthisway.com/2014/03/08/example-1-pediatrician-belief-is-opposite-the-published-scientific-evidence-on-early-vaccine-safety/

If you read later posts in that blog you will find more criticism of the cognitive biases and outright omissions in the vaccine safety and effectiveness literature. Basically there is extensive scientific evidence indicating that any vaccine that happens to come during critical periods of brain development may harm development, and that various cumulative effects of vaccines may cause problems. The safety surveys basically only compare patients who got one specific vaccine and many others to patients who got many others, so they are blind to the kinds of damage that seem likely to be occurring. Also, the only RPC study I know of that injected saline or vaccine randomly into children and followed their health (not whether they got a specific disease) for more than a few months found the vaccine recipients (a flu vaccine) got 4 times as many respiratory illnesses as the recipients of the placebo.

*The paper of Stefano et al that is sometimes cited as showing this, upon closer examination effectively only compares patients who got DTP and other vaccines to patients who may have gotten DTaP, but did not get DTP, and got other vaccines. See the survey above for more discussion. I've never seen any other paper mentioned on the subject.

I would be very grateful to people who can supply citations to articles I haven't yet found contradicting my conclusions, for example other RCP studies following health for a prolonged period, especially of children.