Skeletons & Standards

Dorothy Bishop critiques a 2003 neuroimaging paper:

I don’t like to single out a specific paper for criticism in this way, but feel impelled to do so because the methodological problems were so numerous and so basic. For what it’s worth, every paper I have looked at in this area has had at least some of the same failings. However, in the case of Temple et al (2003) the problem is compounded by the declared interests of two of the authors, Merzenich and Tallal, who co-founded the firm that markets the FastForword intervention. One would have expected a journal editor to subject a paper to particularly stringent scrutiny under these circumstances.

Author Russ Poldrack responds:

Dorothy notes four major problems with the study:

There was no dyslexic control group; thus, we don’t know whether any improvements over time were specific to the treatment, or would have occurred with a control treatment or even without any treatment. The brain imaging data were thresholded using an uncorrected threshold. One of the main conclusions (the “normalization” of activation following training”) is not supported by the necessary interaction statistic, but rather by a visual comparison of maps. The correlation between changes in language scores and activation was reported for only one of the many measures, and it appeared to have been driven by outliers.

Looking back at the paper, I see that Dorothy is absolutely right on each of these points.  In defense of my coauthors, I would note that points 2-4 were basically standard practice in fMRI analysis 10 years ago (and still crop up fairly often today).

Neurocritic asks whether we should “discard” studies like this from the literature:

Many questions remain. How self-correcting is the field? What should we do with old (and not-so-old) articles that are fatally flawed? How many of these results have replicated, or failed to replicate? Should we put warning labels on the failures?

Professor Bishop also noted specific problems at PNAS, like the “contributed by” track allowing academy members to publish with little or no peer review. The “pre-arranged editor” track is another potential issue.

([via](http://www.danielbor.com/dilemma-weak-neuroimaging/))