Why It Matters

My father forwarded an article to me this morning: Clinical trials flawed by biased reporting. It’s a write up of Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer.

This article is deeply, personally relevant. My aunt has late stage breast cancer. Forget a cure. We would do anything for a decent shot at five or ten more years.

I have my PubMed account set up to email me whenever a new article on breast cancer gets published. I find comfort in understanding what I can. But I also find these alerts distressing. Because even as I read the findings, I’m never sure I can believe them.

I come from a social psychology background where poor research practices never really hurt anyone. Is it frustrating to learn that you can’t always trust the results you read in journals or see at conferences? Is it disillusioning to realize that you’re part of that system - your null results sitting in a file drawer, the temptation to post-hoc hypothesize just to keep up - with little recourse? Of course. But I never felt like the state of social psychology/social neuroscience was some great tragedy. Just a waste of time.

Medical research is different.

By which I mean, it’s not that different - as far as I can tell there are similar problems with publication bias, poor statistical practices, lack of access to raw data, to code, to the articles themselves. But in medical research, these problems are a matter of life and death.

I’m not an expert in any field of medicine. But I’ve read a lot of papers recently - yes, illegally accessed papers - and they’re full of the same issues I’ve seen before: significance reported instead of means, dozens if not hundreds of variables tested without correcting for multiple comparisons, ten positive results for every null or negative one. And there has been plenty of thorough reporting done recently (the above study included) about bias in industry-funded trials, problems with randomization and blinding, and the many different kinds of statistical errors that medical researchers make. So even if I’m overstating my case, I’m not making a mountain out of a molehill. I’m making a mountain out of a slightly smaller mountain.

I can’t help but wonder - how much is this holding medical science back? What advances might we have achieved if our researchers had a better, less biased literature to draw from? Might one of the emails I got this week have been about a real advance, a new treatment, maybe even a cure?

I spent much of last weekend reading through Aaron Swartz’s writings. (It would have been impossible not to. He was loved and admired by many of my friends; the circumstances of his death are tragic and outrageous; and his work is provocative, compelling.) I think my favorite piece of his is the Guerilla Open Access Manifesto:

Those with access to these resources --- students, librarians, scientists --- you have been given a privilege. You get to feed at this banquet of knowledge while the rest of the world is locked out. But you need not --- indeed, morally, you cannot --- keep this privilege for yourselves. You have a duty to share it with the world. And you have: trading passwords with colleagues, filling download requests for friends. … We need to take information, wherever it is stored, make our copies and share them with the world. We need to take stuff that’s out of copyright and add it to the archive. We need to buy secret databases and put them on the Web. We need to download scientific journals and upload them to file sharing networks. We need to fight for Guerilla Open Access.

The lack of open access is at once just a part of the problem and yet also a cause and a symptom. Even if every article was free, we’d still have issues with publication bias and poor statistics. But if every article was free, maybe we could do better meta-analysis and correct for some of these problems. If every article was free, maybe the ideas and efforts of those currently shut out of the system would provide some solutions. If every article was free, what would that say about scientific culture? Wouldn’t it imply that we’d finally rebelled against perverse tradition, this system of strange incentives that leads us to value the number of articles published more than their quality or their rigor, to value finding a positive result more than finding the correct result?

The thing about Aaron which you can see in all his writings is that he cared, deeply. More than his intelligence, more than his creativity, you can see how passionate he felt about the various problems he put his formidable mind to.

I’ve been afraid to show how much I care. I’ve got a BA and just a few years experience, I’m not currently working in academia or psychology - why should anyone take me seriously? Especially if I sound angry?

But I am angry. This is science, guys. I thought we were supposed to put the truth above all else. To think critically about everything, including ourselves and our motivations. I thought we were a group of people selected mainly for our ability to admit when we’re wrong and change our approach. If we can’t work together to fix the flaws in our community, what does that mean for everyone else?

So. This is why I care so much. Why I volunteer my time, why I talk so much and so forcefully about these issues. The system has not just failed, it is failing people. My aunt, my friend, and a hundred thousand strangers.

They are why it matters.