Blinded By Results

Last night at a dinner party made up mostly of science folks, I pitched an idea that’s been stewing in my head for a while: results-blind submissions reviews for journals.

That sentence isn’t going to parse for half the people who read my blog, so let me step back and provide some background:

The publishing system for scientific articles is messed up, and the biggest problem it has is publication bias. Most experiments can be boiled down to a core structure, where you are introducing or observing a variable (a drug in pharmaceutical research, maleness or femaleness in sex differences research, etc) and you want to know whether it has a significant effect on something else. When you measure a significant effect, you’ve got a positive result. When you don’t find a significant effect, you’ve got a null result.

(A significant effect is a one which, statistically, we think has at least a 95 percent likelihood of being real, and no more than 5 percent likelihood of being due to chance. ‘Significance’ is a way for scientists to quantify how certain something is. Keep in mind that very few things in science are ever completely certain, and most are open to some amount of debate.)

Overall, journals are far more likely to publish positive results than null results. A recent study suggests that as many as 86% of all published articles contain positive results. This bias has trickled down to researchers too, and they are less likely to submit - or even finish! - experiments that have null results. This is called the ‘file-drawer phenomenon’, because null results tend to end up in a file drawer, not in journal articles and in collective public knowledge, where they belong.

This is wrong on a couple of levels.

A publication bias towards positive results distorts everyone’s perceptions of our collective knowledge.

If there are a dozen unpublished studies showing no effect of a drug, and one study showing an effect, the total bulk of human research on the subject suggests that at the very least the drug’s effects are questionable. Depending on the size and strength of the different studies, it might be fairly clear that there is no effect. But the published results - what every individual knows of the collective work - suggest exactly the opposite: that there is compelling reason to believe the drug works. Cue thousands of people exposing themselves to side effects unnecessarily. (Or eating too many jellybeans.)

This is not just a problem for the general public, the consumers of research, but for the producers of research as well. There’s a constant search amongst scientists for new ideas, new hypotheses. But how do we know an idea is really novel? Maybe it’s been tested ten times before, but because there were no results, no one ever published what they were doing. Maybe every couple of years someone thinks up the same study, runs it, and discards it, all in obscurity. Without the constant publication of null results, we end up reinventing the square wheel, over and over again.

A publication bias towards positive results gives scientists an incentive to do bad science.

It’s clear why drug companies have an incentive to get positive results, since they can stand to make millions of dollars off drug sales. Why do researchers have an incentive? Two words: grants and tenure. Most research is funded by government and private grants, and they understandably want their money to be used wisely, to fund studies that will really contribute to our understanding of a specific field or subfield. Unfortunately, one of the best ways to show your record of contributing to science is to list a series of published journal articles. If you need articles to get grants, and positive results to get articles, then you’re going to be biased towards positive results. The same thing goes for getting tenure. Many institutions place priority on doing successful, published research. It doesn’t matter how high quality, how well motivated your research is - if it doesn’t give you positive results, you’re not likely to get tenure. Which, if you’ve been giving up Friday night partying and first days of preschool for fifteen years, is a pretty big blow.

So how do you get those positive results you need to keep doing science?

Some folks turn to outright fraud. There have been a number of high profile cases, such as Marc Hauser, the high profile, Harvard tenured primatologist and psychologist, who resigned after an investigation found he had fabricated data. Or Jan Schön, whose nanotechnology research had tremendous impact on the field - until it was discovered that he’d not only fabricated data but re-used it in multiple experiments. Wikipedia has a list of famous cases of scientific fraud here. Some argue that fraud in science is on the rise.

But that’s a small minority of scientists. (Hopefully.)

The effect of publication bias on scientists with integrity and good intentions is more subtle, and maybe also more pervasive. What it comes down to is that science is messy. New fields such as the one I worked in - social neuroimaging - can be especially difficult, full of new techniques people aren’t sure how to interpret, with few set standards that people can strictly follow. Few people are both brilliant and educated enough to learn the full set of skills they need to do complex scientific work. To really know that I was doing my neuroimaging projects correctly, I needed to be an expert programmer, a skilled statistician, and have a decent grasp of MRI physics, neuroanatomy and social psychology. Even in fields that are more established than social neuroscience, there will always be innovation and the uncertainty and inexperience it brings.

How does this relate back to publication bias? If you’ve got a horse in the race - if you’ve got a bias towards positive results - that may come out in the decisions you make to deal with that uncertainty. How many times should you check that your procedure was correct? How many different ways should you try to model your data? How many outside experts should you consult to make sure that you’re using the appropriate statistical test? How many times do you try something before you give up? Those are subjective questions, emotional questions, and people are seldom able to make them ahead of time, before they see their own results. With so much riding on getting a significant effect, who would be surprised to find out that people are much less likely to question a positive result, and much more likely to dig deep into a null one?

The consequence of this is quite simply bad science. Countless smaller, unquantifiable errors, and not a few larger, systemic ones. Just this week a study was published in Nature Neuroscience suggesting that of over 150 papers published in the top five neuroscience journals, slightly more than half contained a fairly simple statistical error, claiming the existence of s significant difference between two effects where they shouldn’t. A few years ago, an even bigger bomb was dropped into the field of social neuroscience, when an MIT post-doc asserted that more than half of the high profile papers he’d sampled made a conceptual/statistical mistake that produced almost impossibly high correlations between brain and behavior. This graph from his paper shows, in red, which studies made the error - is it any surprise that those papers were the ones that had shown the strongest effect? If enthusiasm about strong effects and positive results doesn’t bias the judgment of researchers and reviewers, why would studies with strong effects have the vast majority of the errors?

I’m most familiar with neuroscience, but I doubt that the troubling effects of publication bias are limited to one field.

Back to the big idea…

So, back to my idea. It’s pretty much what it says on the tin: researchers would provide to reviewers a redacted version of the final article, which details the theoretical motivation for the study, the methodology, and the statistical tests they expect to use. Reviewers would decide whether to accept or reject a paper based on how well they thought the study was conducted. Was the methodology up to standards? Was it innovative? Rigorous? Do the statistical tests make sense, or are there more appropriate ones to use?

As more null results were accepted for publication, more would be submitted, and so long as the metric for grants and tenure remains published results people would feel more and more comfortable when months of hard work and thousands of dollars returned a null result.

One person at the dinner party objected, “But then you’d get all sorts of useless studies published! All we’d learn is this didn’t work, or that didn’t work.” Good. That’s important to know! As I wrote above, knowing what doesn’t work is important so that researchers don’t waste time and effort trying it over and over again.

Another person pointed out that with more focus on theoretical motivations and methodologies, “long shot” studies with surprisingly positive results would have a harder time getting accepted. That’s a legitimate problem, although I doubt results-blind reviewing would ever catch on enough to make it a practical one.

There are, of course, many other ways to combat publication bias. I very much agree with those who advocate scientific transparency and the publishing of data and details about analyses online. Pre-registration and mandatory reporting of trials has seemed to help with trials done by drug companies, and could probably help academia. And improving the quality of reviewers wouldn’t hurt: right now researchers are not paid for taking time out of their very busy schedules to review the work of others. Giving them time and incentive to really focus on the papers in front of them could only increase the numbers of errors caught. Then there’s the brilliant idea to make sure every single paper is reviewed by a professional statistician.

Finally, there’s replication: one of the most potent weapons in the battle for scientific accuracy. Unfortunately, there’s not much incentive for scientists to try and replicate the results of others, not when tenure rests on doing your own unique work. Making the attempt to replicate someone else’s study a requirement for your dissertation, for completing post docs, for getting tenure, for getting large grants, and suddenly the endless mist of false positives starts to burn away in the sunlight.

But I guess it’s the psychologist in me that wants to say: so long as there’s an incentive to do something, people will find a way to do it, no matter the obstacles you throw in their paths. Consciously or unconsciously, people will keep biasing their work towards positive results. You need to take the motivation away. You need to be blind to the results.