False-Positive Psychology

Quoting my own write up:

There’s another article out this month on all the false positives being published in psychology [pdf].

It deals with what the authors call “researcher degrees of freedom” – a series of somewhat arbitrary decisions about experimental design and data analysis that researchers have to make over the course of their study. Because researchers are a) out of a job if they don’t find enough significant results and b) human, this ambiguity can frequently, unconsciously, perniciously be used to hone in on the most significant results, causing an abundance of false positives in the field as a whole.

The coolest part of the article is a set of simulations they ran where they looked at the effect of four common decisions, or degrees of freedom, on results. The decisions were:

A) If you have two dependent variables, which one should you conduct your analysis on – or should you use an average of the two? B) If you’ve tested a moderate number of subjects and found no results, should you try to improve your power by adding more subjects? C) How do you deal with covariants? For example, should you see if there is a main effect of gender on your study, or an interaction between gender and condition? D) If you have multiple conditions, which should be compared and which should be combined?

The researchers simulated results 15,000 times by drawing randomly from a normal distribution. By definition, this means that they should get “significant” (p many more false positives. When you combine all four degrees of freedom, the simulation was more likely than not to find a “significant” result!

One proposed solution, which helps make these decisions more transparent?  These 21 words:

“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”

Further discussions here and here.  Psychological Science is apparently considering publishing direct replications.

(via Andrew Gelman [here](http://andrewgelman.com/2012/11/researcher-degrees-of-freedom/) and [here](http://andrewgelman.com/2012/02/false-positive-psychology/))