N-back news: Jaeggi 2011, or, is there a psychologist/statistician in the house?

The result looks pretty weak. They had 62 kids. First, they gave all the kids a fluid intelligence test to measure their baseline fluid intelligence. Then half the kids (32) were given a month of n-back training (which the authors expect to increase their fluid intelligence) while the other half (30) did a control training which was not supposed to influence fluid intelligence. At the end of the month's training all of the kids took another fluid intelligence test to see if they'd improved, and 3 months later they all took a fluid intelligence test once more to see if they'd retained any improvement.

The result that you'd look for with this design, if n-back training improves fluid intelligence, is that the group that did n-back training would show a larger increase in fluid intelligence scores from the baseline test to the test after training. They looked and did not find that result - in fact, it was not even close to significant (F < 1). That's the effect that the study was designed to find, and it wasn't there. So that's not a good sign.

The kids who did n-back training did improve at the n-back task, so the authors decided to look at the data in another way - they divided the 32 kids in that group in half based on how much they had improved on the n-back task, and looked separately at the 16 who improved the most and the 16 who improved the least. The group of 16 high-improvers did improve on the fluid intelligence test, significantly more than the control group, and they retained that improvement on the follow-up test of fluid intelligence. That is the main result that the paper reports, which they interpret as a causal effect of n-back training. The 16 low-improvers did not have a statistically significant difference from the control group on the fluid intelligence test.

But this just isn't that convincing a result, as the study no longer has an experimental design when you're using n-back performance to divide up the kids. If you give kids 2 intelligence tests (one the n-back task, one the fluid intelligence test), and a month later you give them both intelligence tests again, then it's not surprising that the kids who improved the most on one test would tend to also improve the most on the other test. And that's basically all that they found. Their study design involved training the kids on one of those two tests (n-back) during the month-long gap, but there's no particular reason to think that this had a causal effect on their improvement on the other test. There are plenty of variables that could affect intelligence test performance which would affect performance on both tests similarly (amount of neural development, being sick, learning disability, etc.).

If there is a causal benefit of n-back, then it should show up in the effect that they were originally looking for (more fluid intelligence improvement in the group that did n-back training than the control group). Perhaps they'd need a larger sample size (200 kids instead of 62?) to find it if the benefit only happens to some of the kids (as they claim), but if some kids benefit from the training while others get no effect from it then the net effect should be a measurable benefit. I'd want to see that result before I'm persuaded.

My primary objection is: perhaps some of the students in both groups got smarter (these are 8-9 year olds and still developing) for reasons independent of the interventions, which caused them to improve on the n-back training task AND on the other intelligence tests (fluid intelligence, Gf). If you separated the "active control" group into high and low improvers post-hoc just like was done for the n-back group, you might see that the active control "high improvers" are even smarter than the n-back "high improvers". We should expect some 8-9 year olds to improve in intelligence or motivation over the course of a month or two, without any intervention.

Basically, this result sucks, because of the artificial post-hoc division into high- and low- responders to n-back training, needed to show a strong "effect". I'm not certain that the effect is artificial; I'd have to spend a lot of time doing some kind of sampling to show how well the data is explained by my alternative hypothesis.

It's definitely legitimate to look at the whole n-back group vs. the whole active control group. Those results there aren't impressive at all. I just can't give any credit for the post-hoc division because I don't know how to properly penalize it and it's clearly self-serving for Jaeggi. It's borderline deceptive that the graphs don't show the unsplit n-back population.

It's unsurprising (probably offering no evidence against my explanation) that the initial average n-back score for the low improvers is higher than the initial average for the high improvers; this is what you'd expect if you split a set of paired samples drawn from the same distribution with no change at all, for example.

Also, on pg 2/6, I don't understand how the t statistics line up with the group sizes.

The groups are ((16 high improvement+16 low improvement)+30 control), so why is it (15), t(15), t(30), and then later t(16)? Does t(n) not mean that it's a t statistic over a population of n? I'm guessing so. I assume the t is an unpaired student's t-test, which of course assumes the distributions compared are normal. I'm not sure if that's demonstrated, but it may be obvious to experts (it's not to me).

Disclaimer: I did dual n-back for a month or so, and got stuck at 5. I haven't resumed, though I may do so in the future.

From the study:

Because we included children from both the Detroit and Ann Arbor metropolitan areas, we had a broad range of socioeconomic status, race, and ethnicity

Detroit has some of the worse public schools in the country and Ann Arbor some of the best. Yet, unless I missed it Jaeggi didn't breakdown any of the results by school location. Strange. I wonder which type of students were among those that she claimed benefited from the N-back?

Of course mainstream coverage is unquestioning. That's what science coverage is, except when there are other scientists disputing it. Journalists don't know what is science or statistics, so it's probably for the best that they don't assess the papers they cover.

Anyone besides me having his child play N-Back?

Thanks for commenting everyone. I was kind of hoping someone could convince me otherwise, that this study was solid evidence of IQ improvement, but I guess not. Anyway, I have linked & excerpted this page on the DNB ML.

This study says little.

Interesting, I have never heard this theory, especially from a psychologist. I mean if you think about it it could help, video games exercising your eyes quite extensively.