Methods reporting in the fMRI literature

I spent three years working in fMRI labs.  To this day it’s not clear to me if the field has exceptionally ambiguous standards, or if it’s only one of many scientific subfields based around new technology struggling to define good practice.  Whether it’s got company or not, neuroimaging certainly has issues.

A paper by Joshua Carp in NeuroImage reviews the methodology of 241 recent fMRI articles:

The present study evaluated research reports according to a checklist adapted from the guidelines promulgated by Poldrack et al. (2008). Checklist items were grouped into five categories: experimental design, data acquisition, processing, statistical modeling, and visualization. In all, 179 methodological decisions were collected for each article. Parameters that could not be determined from the published reports were classified as missing. Parameters that were not relevant to particular papers (e.g., smoothing kernel for studies that did not report using spatial smoothing) were classified as not applicable.

You can view reporting rates of these 179 decisions, without subscribing to the journal, here and here.

They also looked at sample sizes across studies, and found that they average 15 subjects per group.  They calculated power for each study and found that only 2.3% of the studies would have 80% power to find even a large (.8) effect size.

As described by Ioannidis (Ioannidis, 2005b), the rate of false positive results increases with the flexibility of analysis procedures. Thus, it is important to determine how many analytic strategies can be used to explore a given experiment.

As described above, the studies in the present sample were coded for 21 optional analysis procedures, ranging from experimental design to data visualization. Across the 241 studies, 223 unique combinations of analytic techniques were observed. In other words, there were nearly as many unique analysis pipelines as studies in the sample. Arguably, however, data visualization techniques like activation figures and tables do not constitute analysis. After collapsing across these visualization procedures, 207 unique pipelines were observed—again, nearly as many analysis methods as studies.

The order of processing procedures also permits substantial flexibility in analysis.

From the discussion:

Critically, however, the present results show that many published reports omit a number of key data collection and analysis parameters (Fig. 2 and Fig. 3). Over one third of studies did not describe the number of trials, trial duration, and the range and distribution of inter-trial intervals. Fewer than half reported the number of subjects rejected from analysis; the reasons for rejection; how or whether subjects were compensated for participation; and the resolution, coverage, and slice order of functional brain images.

Important methodological details were also omitted from descriptions of data analysis. Less than half of the studies in the present sample reported whether images were corrected for differences in slice acquisition timing or coregistered to high-resolution scans. Nearly half did not report whether temporal filtering was conducted; less than one fifth reported whether temporal autocorrelations were modeled. A minority of studies described the reference slice used for slice-timing correction, the reference image used for motion realignment or spatial normalization, and how or whether images were corrected for nuisance variables like head motion or physiological artifacts.

Countless studies have demonstrated that these methodological choices can have profound effects on research outcomes (e.g.,Dale, 1999,Lund et al., 2005,Mumford and Nichols, 2008,Sladky et al., 2011 and Zhang et al., 2009). The widespread omission of these parameters from research reports, documented here, poses a serious challenge to researchers who seek to replicate and build on published studies. Changing even a single critical methodological decision may qualitatively alter the results of an experiment; changing many decisions at once may exert profound and unpredictable effects on research outcomes. For example, following the failure of one research group (Nieuwenhuis et al., 2007) to replicate a high-profile finding from another lab (Brown and Braver, 2005 and Brown and Braver, 2007), both groups cited differences in methodological parameters like sample size, number of trials, inter-stimulus interval, and even subject nationality in explaining the divergent results. In sum, methodological choices matter, and reporting them matters, too.