The Long Way From α-Error Control to Validity Proper

Back in November, Perspectives on Psychological Science put out a special issue on replicability.  I’ll be attempting to summarize each of the articles in that issue for this tumblr.  See this post for links to all article summaries, or use this tag to browse.

Summary of The Long Way From α-Error Control to Validity Proper by K Fiedler, F Kutzner and J Krueger

This paper argues we should focus less on decreasing false positives and more on decreasing false negatives.  False positives, signified by alpha, are defined as the incorrect rejection of the null hypothesis where no true effect actually exists.  False negatives, signified by beta, are defined as the failure to reject the null hypothesis where a true effect does exist.  False negatives are currently very common, due in part to current statistical convention, which sets a threshold of .95 for false positives, but only .8 for false negatives.  That is, researchers accept that they will incorrectly reject the null 5% of the time – and that they will incorrectly fail to reject it 20% of the time.  Add to this bias the fact that many of the causes of false positives - small sample sizes, poorly applied statistical tests - also cause false negatives, and you have a potential epidemic.

The authors point out that every false positive entails a false negative - every incorrectly accepted hypothesis implies that there is another, true hypothesis still out there waiting to be tested.  (This, while often true, I think is a bit of a stretch in some cases - sometimes there truly is “no difference”.)  They also argue that false negatives are more pernicious because they are less likely to be self-corrected, as negative results are often left unpublished and unreplicated.  (I would argue that, perversely, the lack of publication means that these results are more likely to be re-tested, as other researchers searching for a novel result will not know it has been tested before.)  They remind us of research showing that people generally have difficulty generating alternative hypotheses (although this seems just as much due to incorrectly clinging to false positives as from believing, falsely, in the null hypothesis.)

The authors warn us against an approach that would focus only on preventing false positives through tightening statistical standards.  Such changes would necessarily make false negatives even more common.  Instead, they suggest making our standards more lenient, and focusing our energy on creating a vibrant, creative culture with less publication bias and more replication.