Replications in Psychology Research: How Often Do They Really Occur?

Back in November, Perspectives on Psychological Science put out a special issue on replicability.  I’ll be attempting to summarize each of the articles in that issue for this tumblr.  See this post for links to all article summaries, or use this tag to browse.

Summary of Replications in Psychology Research: How Often Do They Really Occur?by M. Makel, J. Plucker, and B. Hegarty

The authors took 100 psychology journals with the highest 5-year impact factor (ie  the most important journals) and searched all of their articles since 1900 for any word starting with “replicat*” - so, “replication”, “replicated”, “replicating”.  This produced 5,051 out of a total of 321,441 searched (1.57%).  They then randomly selected 500 articles and looked more closely to see if they actually contained a replication, which 68.4% did.  This lead them to estimate that 1.1% of the articles were actual replications.  Of those, 81.9% were conceptual replications, 14% were direct replications, and 4.01% were a mix of both.

This authors did a number of other interesting analyses.  They found that over half of replications (52.9%) were by the same research team, defined by having at least one author overlap.  Although the authors do not present this analysis, one can apply this number to the overall replication rate to estimate that .052% of papers published are replications done by unaffiliated research teams.  One can then estimate that .093% - or 1 in 1075 - or around 300 total - are direct replications by an unaffiliated team.  (This may not be true, as there may be confounds - for instance, self replications may be more likely to be direct replications.  But it’s an interesting, if dispiriting, exercise.)

Why is it important that replications be done by a separate team?  The authors show us: an analysis of success rates found that self-replications were significantly more likely to be successful than replications done by other groups (91.7% vs. 64.6%), with only three (out of 167) self-replications reporting no replication of any of the findings.  The authors also found that replication rates have been increasing, with rates since 2000 being double those of the half-century before.  (1950-1999: 1.10%, 2000 - present: 2.39%.)  Lastly, they found that the replications were much less likely to be cited than the original studies, with replications averaging 17 citations compared to original studies’ 64.5.