Verification is hard

A while back I interviewed some core contributors to Wikipedia Project Medicine. While I’m wildly enthusiastic about their work, I can’t help feeling that Wikipedia falls far short of being an ideal medium for sharing medical knowledge.

A thought experiment demonstrates this. Suppose you read a claim on a Wikipedia page such as “Rubbing toothpaste on your feet has been shown to cause permanent stains in up to 20% of people”. How many people are you trusting? How difficult is this statement to verify?

Well, to begin with, there’s always the possibility of vandalism. You can view the history of the page and see if this particular statement was added recently. If it was, and by someone without much of an edit history, it may very well be vandalism. If the statement has made it through many page edits or was added by a trusted user, it’s probably not. This level of verification requires a relatively sophisticated understanding of how Wikipedia works, but is not a huge barrier.

The statement may not be vandalism, but it may be in dispute. Hopefully if this is the case it will be tagged as such, allowing you to learn about Wikipedia’s disputed statement procedures and visit the talk page to view the argument.

Let’s say that Wikipedia has come to the consensus that the statement is true, but without much discussion. They cite a research article from a scientific journal. The most obvious obstacle is that the article may be closed access. In this case, you have multiple options, each of which present additional barriers:

• You can pay $30 or so to access the article.
• You can email the authors of the article and ask for a copy. This requires a certain amount of social bravado, and is frequently unsuccessful.
• You can ask a friend with access to get you a copy. To have friends with access in the first place requires a significant amount of privilege, and is also often illegal.
• You can just read the abstract. If you only read the abstract, you are not really verifying the research - only the fact that the researchers really came to the conclusions that Wikipedia said they did.

Let’s say you can access the article. Reading and understanding it presents another significant barrier, as most articles are not written to be accessible to laypeople, even educated ones. Understanding the article may take hours or days of background research.

Furthermore, most journal articles are written in a traditional format, which does not allow the reader to verify many of the details of the work. (I’ve written about the failures of the traditional journal article format here.) So even reading and understanding the article may not be enough to verify the claim yourself.

In the end, you may spend days or weeks trying to verify the claim, only to be unsuccessful. If you don’t try to verify, you end up trusting the specific Wikipedia user who added the claim, the general Wikipedia community’s system for attaining accuracy, and the researchers who wrote the cited article. Even with the best of intentions, each one of these trusted actors can easily fail.

Obviously these are not trivial problems to fix, and I don’t blame Wikipedia for not having it all figured out yet. Our world is a messy and corrupt one, not an ideal one.

It’s still worth stating:

An ideal verification system allows people to verify as deeply and as broadly as they want to, without either overloading them with information or providing unnecessary hurdles to them learning more. Not coincidentally, the ideal education system allows people to learn as deeply and as broadly as they want to, without either overloading them with information or providing unnecessary hurdles to them learning more.

I hope that eventually we’ll find our way towards better platforms and knowledge systems, where the only limit to verification are the abilities and inclinations of our collective minds.