Meta-science

Most medical research: true or false?

Originally at https://metascience.shaunagm.net/post/41875969224/most-medical-research-true-or-false

A friend forwarded me this paper, “Empirical estimates suggest most published medical research is true”, perhaps in an attempt to challenge my cynicism.  I like to believe I am open to being challenged, and I do recognize that purposefully cataloging problems with research leaves me biased.  But I don’t think this paper is the best counterpoint.

Andrew Gelman states the issues better than I could.  I particularly agree with his criticism of the paper’s assumption that p-values in the literature have a normal distribution:

What they seem to be doing is collecting a set of published p-values and then fitting a mixture model to this distribution, a mixture of a uniform distribution (for null effects) and a beta distribution (for non-null effects). Since only statistically significant p-values are typically reported, they fit their model restricted to p-values less than 0.05. But this all assumes that the p-values have this stated distribution. You don’t have to be Uri Simonsohn to know that there’s a lot of p-hacking going on. Also, as noted above, the problem isn’t really effects that are exactly zero, the problem is that a lot of effects are lots in the noise and are essentially undetectable given the way they are studied.

Edited to add: One of the authors responds.