The Universal Medical Journal Article Error

Don't worry, we'll have Metamed to save us!

Well, dammit, I wanted to delete this and rewrite above, but you can't delete comments anymore. This is not retracted, but I can't un-retract it.

You are wrong, and you have not learned to reconsider your logic when many smart people disagree with you.

hey do, but did the paper he dealt with write within a Bayesian framework? I didn't read it, but it sounded like standard "let's test a null hypothesis" fare.

You don't just ignore evidence because someone used a hypothesis test instead of your favorite Bayesian method. P(null | p value) != P(null)

I don't think you responded to my criticisms and I have nothing further to add. However, there are a few critical mistakes in what you have added that you need to correct:

Now pay attention; this is the part everyone gets wrong, including most of the commenters below.

The methodology used in this study, and in most studies, is as follows:

• Divide subjects into a test group and a control group.

No, Mattes and Gittelman ran an order-randomized crossover study. In crossover studies, subjects serve as their own controls and they are not partitioned into test and control groups.

If you don't understand why that is so, read the articles about the t-test and the F-test. The tests compute what a difference in magnitude of response such that, 95% of the time, if the measured effect difference is that large, the null hypothesis (that the responses of all subjects in both groups were drawn from the same distribution) is false.

No, the correct form is:

• The tests compute a difference in magnitude of response such that if the null hypothesis is true, then 95% of the time the measured effect is not that large.

The form you quoted is a deadly undergraduate mistake.

ADDED: People are making comments proving they don't understand how the F-test works. This is how it works: You are testing the hypothesis that two groups respond differently to food dye.

Suppose you measured the number of times a kid shouted or jumped, and you found that kids fed food dye shouted or jumped an average of 20 times per hour, and kids not fed food dye shouted or jumped an average of 17 times per hour. When you run your F-test, you compute that, assuming all kids respond to food dye the same way, you need a difference of 4 to conclude that the two distributions (test and control) are different.

If the food dye kids had shouted/jumped 21 times per hour, the study would conclude that food dye causes hyperactivity. Because they shouted/jumped only 20 times per hour, it failed to prove that food dye causes hyperactivity. That failure to prove is then taken as having proved that food dye does not cause hyperactivity, even though the evidence indicated that food dye causes hyperactivity.

This is wrong. There are reasonable prior distributions for which the observation of a small positive sample difference is evidence for a non-positive population difference. For example, this happens when the prior distribution for the population difference can be roughly factored into a null hypothesis and an alternative hypothesis that predicts a very large positive difference.

In particular, contrary to your claim, the small increase of 3 can be evidence that food dye does not cause hyperactivity if the prior distribution can be factored into a null hypothesis and an alternative hypothesis that predicts a positive response much greater than 3. This is analogous to one of Mattes and Gittelman's central claims (they claim to have studied children for which the alternative hypothesis predicted a very large response).

I think you're interpreting the F test a little more strictly than you should. Isn't it fairer to say a null result on a F test is "It is not the case that for most x, P(x)", with "most" defined in a particular way?

You're correct that a F-test is miserable at separating out different classes of responders. (In fact, it should be easy to develop a test that does separate out different classes of responders; I'll have to think about that. Maybe just fit a GMM with three modes in a way that tries to maximize the distance between the modes?)

But I think the detail that you suppressed for brevity also makes a significant difference in how the results are interpreted. This paper doesn't make the mistake of saying "artificial food coloring does not cause hyperactivity in every child, therefore artificial food coloring affects no children." The paper says "artificial food coloring does not cause hyperactivity in every child whose parents confidently expect them to respond negatively to artificial food coloring, therefore their parents' expectation is mistaken at the 95% confidence level."

Now, it could be the case that there are children who do respond negatively to artificial food coloring, but the Feingold association is terrible at finding them / rejecting those children where it doesn't have an effect. (This is unsurprising from a Hawthorne Effect or confirmation bias perspective.) As well, for small sample sizes, it seems better to use F and t tests than to try to separate out the various classes of responders, because the class sizes will be tiny; if one child responds poorly after administered artificial food die, that's not much to go on, compared to a distinct subpopulation of 20 children in a sample of 1000.

The section of the paper where they describe their reference class:

If artificial additives affect only a small proportion of hyperactive children, significant dietary effects are unlikely to be detected in heterogeneous samples of hyperactive children. Therefore, children who had been placed on the Feingold diet by their parents and who were reported by their parents to have derived marked behavioral benefit from the diet and to experience marked deterioration when given artificial food colorings were targeted for this study. This sampling approach, combined with high dosage, was chosen to maximize the likelihood of observing behavioral deterioration with ingestion of artificial colorings.

(I should add that the first sentence is especially worth contemplating, here.)

I didn't look at the data. I was commenting on your assessment of what they did, which showed that you didn't know how the F test works. Your post made it seem as if all they did was run an F test that compared the average response of the control and treatment groups and found no difference.

You shouldn't think of this paper as being in English. You should think of it as being written in Science.

Science != Pure Mathematics.

They proved that THERE EXIST children who are not sensitive to these agents. 5% may be enough.

Yes, you can “prove” very little outside pure mathematics. But “doesn't prove” doesn't imply “doesn't support”. Chapter 1 of Probability Theory by E. T. Jaynes makes that clear.

(And BTW, how comes you're taking “school-age children” to mean “all school-age children” but you're not taking “artificial food colorings” to mean ‘all artificial food colorings’?)

Then the statement, "artificial food colorings do not affect the behavior of school-age children who are claimed to be sensitive to these agents," is false.

No it fucking isn't. Read the article I've linked to again. “Humans have opposable thumbs” doesn't stop being true as soon as someone somewhere gets both thumbs amputated.

The differences they found were smaller than the F-value.

Then this is what you should have critiqued in your post. Ranting about their inductive reasoning being deductively wrong gets you nowhere.

Where, exactly, is the evidence that artificial food color doesn't affect the behavior of any children?

Since your post is the first time I've heard of this: I have no idea, but I assume google has the answer.

I assume Phil reposted it there. Now banning.

Wasn't me.

People aren't that different. I really doubt that, for example, there are people whose driving skills improve after drinking the amount of alcohol contained in six cans of beer.

There is always evidence to the contrary. An evaluation weighs the evidence on both sides. You can take any case where the FDA has said "There is no evidence that X", and look up the notes from the panel they held where they considered the evidence for X and decided that the evidence against X outweighed it.

The phrase "There is no evidence that X" is the single best indicator of someone statistically deluded or dishonest.

[quote]Correct. But neither can we say that the dye does not cause hyperactivity in anyone.[/quote]

No, but that is not our goal in the first place. Doing a test on every single possible trait is economically infeasible and unreasonable; ergo, net impact is our best metric.

The benefit is "we get a new food additive to use".

The net cost is zero in terms of health impact (no more hyperactivity in the general population).

Ergo, the net benefit is a new food additive. This is very simple math here. Net benefit is what we care about in this case, as it is what we are studying. If it redistributes ailments amongst the population, then there may be even more optimal uses, but we're still looking at a benefit.

If you want to delve deeper, that's going to be a seperate experiment.

[quote]Your making the claim "no evidence to the contrary" shows that you have not read the literature, have not done a PubMed search on "ADHD, food dye", and have no familiarity with toxicity studies in general. There is always evidence to the contrary. An evaluation weighs the evidence on both sides. You can take any case where the FDA has said "There is no evidence that X", and look up the notes from the panel they held where they considered the evidence for X and decided that the evidence against X outweighed it.[/quote]

Your making the claim "evidence to the contrary" suggests that any of this is worth anything. The problem is that, unfortunately, it isn't.

If someone does a study on 20 different colors of M&Ms, then they will, on average, find that one of the M&Ms will change someone's cancer risk. The fact that their study showed that, with 95% confidence, blue M&Ms increased your odds of getting cancer, [b]is not evidence for the idea that blue M&M's cause cancer[/b].

Worse, the odds of the negative finding studies being published is considerably less than the probability of the positive finding study being published. This is known as "publication bias". Additionally, people are more likely to be biased against artificial additives than towards them, particularly "independent researchers" who very likely are researching it precisely because they harbor the belief that it does in fact have an effect.

This is very basic and is absolutely essential to understanding any sort of data of this sort. When I say that there is no evidence for it, I am saying precisely that - just because someone studied 20 colors of M&M's and found that one has a 95% chance of causing more cancer tells me nothing. It isn't evidence for anything. It is entirely possible that it DOES cause cancer, but the study has failed to provide me for evidence of that fact.

You are thinking in terms of formal logic, but that is not how science works. If you lack sufficient evidence to invalidate the null hypothesis, then you don't have evidence. And the problem is that a mere study is often insufficient to actually demonstrate it unless the effects are extremely blatant.

quote The answer is, "This is very likely." This is how studies turn out all the time, partly due to genetics. Different people have different genetics, different bacteria in their gut, different lifestyles, etc. This makes them metabolize food differently. It makes their brain chemistry different. Different people are different.[/quote]

For this to happen, you would require that the space to be very similar in size on both ends.

Is it possible for things to help one person and harm another? Absolutely.

Is it probable that something will help almost exactly as many people as it harms? No. Especially not some random genetic trait (there are genetic traits, such as sex, where this IS likely because it is an even split in the population, so you do have to be careful for that, but sex-dependence of results is pretty obvious).

The probability of equal distribution of the traits is vastly outweighed by the probability of it not being equally distributed. Ergo the result you are espousing is in fact extremely unlikely.

Yes, but it is not a probable outcome, as for it to be true, it would require a counterbalancing group of people who benefit from it or for the subgroups to be extremely small; however, the allegations are that the subgroups are NOT small enough that the effect could have been hidden in this manner, suggesting that there is no effect on said subgroups as the other possibility is unlikely.

You certainly gave me the most-favorable interpretation. But I just goofed. I fixed it above. This is what I was thinking, but my mind wanted to put "black(x)" in there because that's what you do with ravens in symbolic logic.

A) Not everything is a raven: !∀x raven(x)

B) Everything is not a raven: ∀x !raven(x)

!∀black thing ∃ raven-nature

That's where I don't follow. I read the original sentence as "for every x there is an y such that the relationship P obtains between x and y". I'm OK with your assigning "black things" to x but "raven-nature" needs explanation; I don't see how to parse it as a relationship between two things previously introduced.

The edited version makes more sense to me now.

Nah, they're welcome to use whichever statistics they like. We might point out interpretation errors, though, if they make any.

Under the assumptions I described, a p-value of 0.16 is about 0.99 nats of evidence which is essentially canceled by the 1 nat prior. A p-value of 0.05 under the same assumptions would be about 1.92 nats of evidence, so if there's a lot of published science that matches those assumptions (which is dubious), then they're merely weak evidence, not necessarily wrong.

It's not the job of the complexity penalty to "prove the null hypothesis is correct". Proving what's right and what's wrong is a job for evidence. The penalty was merely a cheap substitute for an informed prior.

No it doesn't. It takes the word "all" as used in everyday language and pretends it is intended to be precisely the same as the logical "all" operator, which it of course it is not. It's the worst kind of nitpicking, the kind of "all people who have a heart attack should go to a licensed hospital" - "nuh-uh, not if the hospital is on fire / not if they are billionaires with a fully equipped medical team in their attic".

What on Earth is "important" about such a point?

This post makes a point that is both correct and important, but Phil has clearly lost much of the audience and is ticked off besides, and I don't blame him.

I think we've got two issues. The general issue of how one tests a null hypothesis and what it does and does not mean to reject the null, and the particular issue of food dyes. The general issue seems important, while the particular could provide a helpful illustration of the general.

But I would think that someone else, and probably multiple someone's, have already plowed this ground. Jaynes must have an article on this somewhere.

Anyone got a good article?

Thanks; I was trying to write a comment that said the same thing, but failed to do so.

Logical analysis is a limiting case of statistical analysis, thus problems with logical reasoning have corresponding problems with statistical reasoning. I agree that Phil should have spelled out this distinction explicitly.

Their statistical analysis was correct, modulo their assumptions. They made their logical error in how they interpreted its conclusion.

People. Explain your downvotes of this comment. Do you think their statistical analysis was incorrect? Do you think they made no logical error?

They did not prove, for instance, that it cannot be the case, with 95% certainty, that all but one of the children are made hyperactive.

They did not say this but I am confident that if this bizarre hypothesis (all but one of what group of children, exactly?) were tested, the test would reject it as well. (Ignoring the conflicting-measures-of-hyperactivity point, which I am not competent to judge.)

In general, the F-test does not reject all alternate hypotheses equally, which is a problem but a different one. However, it provides evidence against all hypotheses that imply an aggregate difference between the test group and control group: equivalently, we're testing if the means are different.

If all children are made hyperactive, the means will be different, and the study rejects this hypothesis. But if 99% of children are made hyperactive to the same degree, the means will be different by almost the same amount, and the test would also reject this hypothesis, though not as strongly. I don't care how you wish to do the calculations, but any hypothesis that suggests the means are different is in fact made less likely by the study.

And hence my list of alternate hypotheses that may be worth considering, and are not penalized as much as others. Let's recap:

• If the effect is present but weak, we expect the means to be close to equal, so the statistical test results don't falsify this hypothesis. However, we also don't care about effects that are uniformly weak.

• If the effect is strong but present in a small fraction of the population, the means will also be close to equal, and we do care about such an effect. Quantifying "strong" lets us quantify "small".

• We can allow the effect to be strong and present in a larger fraction of the population, if we suppose that some or all of the remaining children are actually affected negatively.

I've found previously that many people here are extremely hostile to criticisms of the statistical methods of the medical establishment. It's extremely odd at a site that puts Jaynes on a pedestal, as no one rants more loudly and makes the case clearer than Jaynes did, but there it is.

(Eliezer does, anyway. I can't say I see very many quotes or invocations from others.)

I am hostile to some criticisms, because in some cases when I see them being done online, it's not in the spirit of 'let us understand how these methods make this research fundamentally flawed, what this implies, and how much we can actually extract from this research'*, but in the spirit of 'the earth is actually not spherical but an oblate spheroid thus you have been educated stupid and Time has Four Corners!' Because the standard work has flaws, they feel free to jump to whatever random bullshit they like best. 'Everything is true, nothing is forbidden.'

* eg. although extreme and much more work than I realistically expect anyone to do, I regard my dual n-back meta-analysis as a model of how to react to potentially valid criticisms. Instead of learning that passive control groups are a serious methodological issue which may be inflating the effect and then going around making that point at every discussion, and saying 'if you really want to increase intelligence, you ought to try this new krill oil stuff!', I compiled studies while noting whether they were active or passive, and eventually produced a meta-analytic regression confirming the prediction.

(And of course, sometimes I just think the criticism is being cargo-culted and doesn't apply; an example on this very page is tenlier's criticism of the lack of multiple testing correction, where he is parroting a 'sophisticated' criticism that does indeed undermine countless papers and lead to serious problems but the criticism simply isn't right when he uses it, and when I explained in detail why he was misapplying it, he had the gall to sign off expressing disappointment in LW's competency!)

Such criticism for most people seems to only enable confirmation bias or one-sided skepticism, along the lines of knowing about biases can hurt you.

A great example here is Seth Roberts. You can read his blog and see him posting a list of very acute questions for Posit Science on the actual validity of a study they ran on brain training, probing aspects like the endpoints, possibility of data mining, publication bias or selection effects and so on, and indeed, their response is pretty lame because it's become increasingly clear that brain training does not cause far transfer and the methodological weaknesses are indeed driving some of the effects. And then you can flip the page and see him turn off his brain and repeat uncritically whatever random anecdote someone has emailed him that day, and make incredibly stupid claims like 'evidence that butter is good for your brain comes from Indian religion's veneration for ghee' (and never mind that this is pulled out of countless random claims from all sorts of folk medicines; or that Ayurvedic medicine is completely groundless, random crap, and has problems with heavy metal poisoning!), and if you try sending him a null result on one of his pet effects, he won't post it. It's like there's two different Roberts.

What do you mean by your second sentence?

Thank you for the answer. Now we know (from the many upvoted disagreements in the comments, and the downvoting of every comment I made correcting their mistakes) that the conclusion is not well-known among readers of LessWrong.

OK I agree that the word 'inevitably' is ambiguous. Regardless of the accuracy of the literal-to-logical translations, I think the reason the logical expression of the statement of the article does not match that of the final conclusion, as your logical reasoning proves, is that the writer were not doing the very logical reasoning but doing medical research and thus proposing something new, not something of equivalent logical consequences.

Their first statement:

This study tested and rejected the hypothesis that artificial food coloring causes hyperactivity in all children.

only implies that they did not buy the hypothesis, which did not necessarily imply that they accepted the negation of the hypothesis, which corresponds to your first formula:

!( ∀child ( eats(child, coloring) ⇨ hyperactive(child) ) )

equivalently:

∃child !( eats(child, coloring) ⇨ hyperactive(child) )

Even though they actually accepted the negation of the hypothesis, that is to say , accepted your first formula, the final conclusion they got through the research is that:

The results of this study indicate that artificial food colorings do not affect the behavior of school-age children who are claimed to be sensitive to these agents.

whose correspondent logical expression is your second formula:

∀child !( eats(child, coloring) ⇨ hyperactive(child) ) )

This formula seems stronger than the first one:

∃child !( eats(child, coloring) ⇨ hyperactive(child) )

From my point of view, I don't think that the medical researchers were intentionally arbitrarily generalizing their results or just making logical mistakes. They just posed an attitude to an existing hypothesis and give a new one through the article, in this case, a new stronger hypothesis (the word 'stronger' depends on whether they actually just negated the original hypothesis).

I think their only fault is that they failed to keep their own medical views in one article logically identical.

By whom?

Okay, folks. Now you're just being assholes. Why are you downvoting me for asking who moved my post?

It'd be really great if PhilGoetz and some other commenters in this thread understood this. So you've proven a mathematical statement -- why should that net one karma?

The continued downvote whining, as if their existence represented a substantial failing of "LessWrong as a whole" is really beginning to grate on my nerves.

No; the difference is that in your interpretation, we would say "wrong assumption, hence discard results." In my interpretation, if they had found a significant effect, they would have been able to correctly conclude that there was an effect; and finding no significant effect, they could have correctly concluded the negation of that.

I don't know what irony you refer to.