Degrees of freedom aren’t free
Originally at http://www.shaunagm.net/blog/2012/02/degrees-of-freedom-arent-free/
There’s another article out this month on all the false positives being published in psychology [pdf].
It deals with what the authors call “researcher degrees of freedom” - a series of somewhat arbitrary decisions about experimental design and data analysis that researchers have to make over the course of their study. Because researchers are a) out of a job if they don’t find enough significant results and b) human, this ambiguity can frequently, unconsciously, perniciously be used to hone in on the most significant results, causing an abundance of false positives in the field as a whole.
The coolest part of the article is a set of simulations they ran where they looked at the effect of four common decisions, or degrees of freedom, on results. The decisions were:
A) If you have two dependent variables, which one should you conduct your analysis on - or should you use an average of the two? B) If you’ve tested a moderate number of subjects and found no results, should you try to improve your power by adding more subjects? C) How do you deal with covariants? For example, should you see if there is a main effect of gender on your study, or an interaction between gender and condition? D) If you have multiple conditions, which should be compared and which should be combined?
The researchers simulated results 15,000 times by drawing randomly from a normal distribution. By definition, this means that they should get “significant” (p < .05) results 5% of the time, “highly significant” (p < .01) results 1% of the time, and “marginal” (p < .1) results 10% of the time. However, when they simulated the above decisions - for example, with B, by testing after 20 observations and only adding 10 more subjects if they failed to reach significance - they had many more false positives. When you combine all four degrees of freedom, the simulation was more likely than not to find a “significant” result!
Also cute: the researchers did a study on whether listening to the Beatles makes you age backwards. This paragraph below shows the difference between how they could, according to current reporting standards, describe the study (in bold) and how they recommend all studies be reported:
The authors use six reporting guidelines (and four corresponding reviewer recommendations) to tackle this problem. Although it’s not my preferred method I’m just grateful to see people with standing in the field continue to talk about these issues. I agree with pretty much everything in this paper, especially their conclusion, which I think packs a nice punch.
Our goal as scientists is not to publish as many articles as we can, but to discover and disseminate truth. Many of us--- and this includes the three authors of this article---often lose sight of this goal, yielding to the pressure to do whatever is justifiable to compile a set of studies that we can publish. This is not driven by a willingness to deceive but by the self-serving interpretation of ambiguity, which enables us to convince ourselves that whichever decisions produced the most publishable outcome must have also been the most appropriate. This article advocates a set of disclosure require- ments that imposes minimal costs on authors, readers, and reviewers. These solutions will not rid researchers of publica- tion pressures, but they will limit what authors are able to jus- tify as acceptable to others and to themselves. We should embrace these disclosure requirements as if the credibility of our profession depended on them. Because it does.