Power failure

There’s a new article out in Nature Reviews Neuroscience about the failure  of scientific studies in general (and neuroscience and fMRI studies in particular) to adequately power their studies.  The NRN paper isn’t open access, but you can email the authors for a pre-print.  There’s a good write-up at National Geographic.

The paper discusses the effect of low powered studies, both in an ideal world and in the world we actually live in.  Even in a best case scenario, underpowered studies harm research: “low power, by definition, means that the chance of discovering effects that are genuinely true is low.”  By decreasing the amount of true positive effects in the literature, low powered studies increase the percentage of false positives among all positive results.  Again, from the article:

For example, suppose that we work in a scientific field where one in five of the effects we test are expected to be truly non-null (i.e., R = 1 / (5-1) = 0.25) and that we claim to have discovered an effect when we reach p < 0.05; if our studies have 20% power, then PPV = 0.20 × 0.25 / (0.20 × 0.25 + 0.05) = 0.05 / 0.10 = 0.50; that is, only half of our claims for discoveries will be correct. If our studies have 80% power, then PPV = 0.80 × 0.25 / (0.80 × 0.25 + 0.05) = 0.20 / 0.25 = 0.80; that is, 80% of our claims for discoveries will be correct.

They also discuss the “Winner’s Curse”.  If a study is underpowered, it will be less likely to produce strong effects - but only those studies which produce abnormally strong effects will get published.

To illustrate the Winner’s Curse, suppose that an association truly exists with an effect size that is equivalent to an odds ratio of 1.20, and we are trying to discover it by performing a small (i.e., underpowered) study. Suppose also that our study only has the power to detect an odds ratio of 1.20 on average 20% of the time. The results of any study are subject to sampling variation and random error in the measurements of the variables and outcomes of interest. Therefore, on average our small study will find an odds ratio of 1.20 but, because of random errors, our study may in fact find an odds ratio smaller than 1.20 (e.g., 1.00) or an odds ratio larger than 1.20 (e.g., 1.60). Odds ratios of 1.00 or 1.20 will not reach statistical significance because of the small sample size. We can only claim the association as nominally significant in the third case, where random error creates an odds ratio of 1.60. The Winner’s Curse means, therefore, that the ‘lucky’ scientist who makes the discovery in a small study is cursed by finding an inflated effect.

These are major problems - and of course, we don’t live in an ideal world.  There is publication bias:

Smaller studies more readily disappear into a file drawer than very large studies that are widely known and visible and the results of which are eagerly anticipated (although this correlation is far from perfect). A ‘negative’ result in a high-powered study cannot be explained away as being due to low power, and thus reviewers and editors may be more willing to publish it, whereas they more easily reject a small ’negative’ study as being inconclusive or uninformative.  The protocols of large studies are also more likely to have been registered or otherwise made publicly available, so that deviations in the analysis plans and choice of outcomes may become obvious more easily. Small studies, conversely, are often subject to a higher level of exploration of their results and selective reporting thereof.

In addition to making a compelling case about the danger of low-powered studies, the article also provides a meta-analysis of neuroscience studies showing that, yup, they tend to be pretty underpowered.

The authors identified 730 studies by searching 49 meta-analyses which included them.  They then calculated their power by assuming a p-level of .05 and an effect size equal to that found in the meta-analysis that contained the study.  They found that the average statistical power was 21%.  (For contrast, the ‘standard’ taught in intro stats classes is 80%.)  Interestingly, the studies fell into two groups - 42 low powered meta-analyses with an average of 18% power, and 7 high powered meta-analyses with an average of >90% power.

(The authors admit that these calculations rely on the summary effect sizes reported in the meta-analyses being correct, and agree that it is not an unassailable assumption.)

The authors also looked at specific subfields.  In neuroimaging, the median statistical power was 8%, across 461 individual studies contributing to 41 separate meta-analyses.  A look at rat studies found that “the median statistical power for the water maze studies and the radial arm maze studies to detect these medium to large effects was 18% and 31%, respectively”.

The article finishes up with a discussion of the ethical consequences of underpowered studies, particularly for animal studies and for clinical trials.  It also discusses potential solutions, including increasing standards both at the IRB/grant approval stage as well as at publication, pre-registration of studies, incentivizing replication, and open access to data and materials.