Fifth time's the charm: a case study in data accessibility

I try to explore a bit of government data every month. Previously, I’ve just grabbed data from the Sunlight Foundation’s cleaned up APIs. This month, I tried out data.gov. It was a much more harrowing experience.

Here are the data sets I tried to access, in order:

- The National Inpatient Sample, which it turns out costs $350 per year of data. There’s a free data portal which only provides basic summary statistics, as near as I can tell. (It quaintly invites you to “create your own statistics”, as if carrying out one of the dozen or so pre-defined analyses is some kind of original research.)

- Data on US Army suicides, listed on data.gov with a broken link.

- The Bureau of Labor Statistics’ employment data, which has a query interface that keeps returning “The database is currently unavailable. Your request was invalid for this Data Access Service.” I think I am asking for too-large datasets, as I can successfully query for a few kb or less, but I am certain of nothing. Alternatively, one can download a small subset of their database in full, zipped and opaquely labelled, here. They’ve also got a new, in-beta data finder which I find just as confusing. After an hour on the site and a quick email to their helpdesk, I gave up.

- I got some traction with the SESTAT data set, which they provide in SAS, a proprietary format, but also in .DAT, which R can read if you use this CRAN module. I did get it working, but by this point I was curious how many attempts it would take to find an “easily accessible” data set.

- I finally learned to filter for datasets hosted by data.gov, which seem to be in better shape. I used the first one I clicked on: the EPA Toxics Release Inventory, which was available in full to download in a variety of formats, as well as via a clearly documented API.

(The actual analysis is here.)

I’m impressed by the variety of issues I encountered: fees, broken links, confusing documentation, inaccessible formats. It gives me a bit more sympathy for the position argued in this Scholarly Kitchen post about the potential hazards of the government switching to a default-open data policy. Of course, I ardently support such an initiative, which may be in the works. But I agree with the author of that article that processing the data to make it truly accessible is no small task.