Most medical research: true or false?
Originally at https://metascience.shaunagm.net/post/41875969224/most-medical-research-true-or-false
A friend forwarded me this paper, “Empirical estimates suggest most published medical research is true”, perhaps in an attempt to challenge my cynicism. I like to believe I am open to being challenged, and I do recognize that purposefully cataloging problems with research leaves me biased. But I don’t think this paper is the best counterpoint.
Andrew Gelman states the issues better than I could. I particularly agree with his criticism of the paper’s assumption that p-values in the literature have a normal distribution:
What they seem to be doing is collecting a set of published p-values and then fitting a mixture model to this distribution, a mixture of a uniform distribution (for null effects) and a beta distribution (for non-null effects). Since only statistically significant p-values are typically reported, they fit their model restricted to p-values less than 0.05. But this all assumes that the p-values have this stated distribution. You don’t have to be Uri Simonsohn to know that there’s a lot of p-hacking going on. Also, as noted above, the problem isn’t really effects that are exactly zero, the problem is that a lot of effects are lots in the noise and are essentially undetectable given the way they are studied.
Edited to add: One of the authors responds.