Critiquing Scientific Practice in 2012

I have a tumblr, Dismal Science, which I use to keep track of newspaper articles, journal articles, and blog posts on the topic of how scientific research is practised and the flaws and problems that have arisen. Before the archives grow unmanageable, I’d like to take a stab at organizing what I have so far. So here’s a summary of the material I posted last year. (Many of the links themselves were not published/posted in 2012.)

Academic Culture and its Consequences:

We Must Stop The Avalanche of Low Quality Research - “Too much publication raises the refereeing load on leading practitioners---often beyond their capacity to cope… The productivity climate raises the demand on younger researchers… The pace of publication accelerates, encouraging projects that don’t require extensive, time-consuming inquiry and evidence gathering.”
Uncited Research - “Most scientific articles get little to no attention (Van Dalen and Klamer 2005). One study found that 47 percent of articles catalogued by the Institute for Scientific Information have never been cited, and more than 80 percent have been cited less than 10 times (Redner 1998). Articles in the median social science journal, on average, get only 0.5 citations within two years of publication (Klamer and Dalen 2002). The mean number of citations per article in mathematics, physics, and environmental science journals is probably less than 1 (Mansilla et al. 2007).”
The Oil Drop Experiment - An anecdote from Richard Feynman. “Why didn’t they discover the new number was higher right away? It’s a thing that scientists are ashamed of - this history - because it’s apparent that people did things like this: When they got a number that was too high above Millikan’s, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan’s value they didn’t look so hard. And so they eliminated the numbers that were too far off, and did other things like that…”
One In Three Scientists Admit to Questionable Research Practices - “A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86—4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once —a serious form of misconduct by any standard— and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91—19.72) for falsification, and up to 72% for other questionable research practices.”
Survey of Scientists on the Influence of Money on Science
I Don’t Really Believe About 95% Of What Gets Published - Interviews with fMRI researchers.
Underrepresentation of Women in Science - Discrimination against women in science, past and present.
Psychology Community Does Not Account for Diversity - “Broad claims about human psychology and behavior based on narrow samples from Western societies are regularly published in leading journals. Are such species‐generalizing claims justified? This review suggests not only that substantial variability in experimental results emerges across populations in basic domains, but that standard subjects are in fact rather unusual compared with the rest of the species.”

Standards and Statistics:

Reproducible Science Means Open Source Software - “Unfortunately, most code written for research remains closed, even if the code itself is the subject of a published scientific paper. According to an editorial in Nature, this hinders reproducibility, a fundamental principle of the scientific method.”
Metadata - With over 50 million scientific articles now published, having freely available article-level metadata is crucial.
Researcher Degrees of Freedom - An article on how “researcher degrees of freedom”, common and crucial design and analysis decisions, have vastly inflated the number of false positives in psychology.
The Decline Effect and the Scientific Method - “For instance, after Palmer plotted every study of fluctuating asymmetry, he noticed that the distribution of results with smaller sample sizes wasn’t random at all but instead skewed heavily toward positive results. Palmer has since documented a similar problem in several other contested subject areas. “Once I realized that selective reporting is everywhere in science, I got quite depressed,” Palmer told me.”
Why Most Published Research Findings Are False - “A research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.”
Evolving Standards in fMRI Research - “Looking back at the paper, I see that Dorothy is absolutely right on each of these points. In defense of my coauthors, I would note that points 2-4 were basically standard practice in fMRI analysis 10 years ago (and still crop up fairly often today).”
Overfitting Causes False Positive Results
402 Citations Questioning the Use of Null Hypothesis Significance Testing - “My goal was to provide those unfamiliar with this debate with a list of citations that pointed out the myriad of problems associated with the indiscriminate use of null hypothesis tests. For parity, I also compiled a list of references that supported, at least to a limited extent, the use of null hypothesis tests.” (Last updated 2/26/2001)

Fraud, Corruption, and Bias:

How Corporations Corrupt Science at the Public’s Expense - a report by the Union of Concerned Scientists on the manipulation, distortion and repression of scientific evidence by corporations
Medical Journals are the Marketing Arms of Pharmaceutical Companies - “Overall, studies funded by a company were four times more likely to have results favourable to the company than studies funded from other sources. In the case of the five studies that looked at economic evaluations, the results were favourable to the sponsoring company in every case.”
Transparency in Clinical Trial Data - “Research published in JAMA demonstrated that two pivotal publications of industry-sponsored clinical trials of rofecoxib for Alzheimer’s disease did not include analyses of mortality, enabling the studies to conclude rofecoxib is “well tolerated.” In direct contrast, and at the same time as publication, Merck’s internal intention-to-treat analyses from the same trials identified a significant increase in total mortality. What is striking about this case is these mortality analyses were neither provided to regulators nor made public.”
Journal Disavows Study Touted By Abortion Foes - “This is not a scholarly difference of opinion; their facts were flatly wrong. This was an abuse of the scientific process to reach conclusions that are not supported by the data.”
The Stapel Report - “It involved a more general failure of scientific criticism in the peer community and a research culture that was excessively oriented to uncritical confirmation of one’s own ideas and to finding appealing but theoretically superficial ad hoc results.”

Replications & Retractions

The Reproducibility Project - “The project will evaluate the ability to reproduce the original study procedures and the overall probability of replicating the original results. Further, it will examine the predictors of replication success - e.g., publishing journal, number of conceptual/direct replications in the published literature, citation impact of the original article, closeness of the replication to the original circumstances: sample, setting, materials.”
Meta-Analysis of Replication of Highly Cited Research - “Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.”
Failed Replication of Major Study Sparks Controversy - “Thus, the old-slow priming results appear to be due to a subtle mix of experimenter bias and standard priming which is cued or amplified via experimenter signaling.”
Sharp Rise In Retractions Prompts Calls For Reform - “[B]efore long they reached a troubling conclusion: not only that retractions were rising at an alarming rate, but that retractions were just a manifestation of a much more profound problem --- “a symptom of a dysfunctional scientific climate”… Dr. Casadevall, now editor in chief of the journal mBio, said he feared that science had turned into a winner-take-all game with perverse incentives that lead scientists to cut corners and, in some cases, commit acts of misconduct.”
Withdrawn Papers Live On - “And the rates haven’t improved much in the age of electronic publication: in a preliminary analysis of 1,112 retracted papers during 1997—2009, Budd finds them cited just as often, with the retraction mentioned in only about 4% of the citations.”
Retractions Stigmatize Fields - “This data showed that retractions that undermine the validity of published results, whether owing to fraud or ‘honest’ mistakes such as failure to reproduce the data, were followed by a 50—73% drop in NIH funding for related studies.”

Solutions

Reforming Science - A detailed list of recommendations published as a pair of journal articles.
Alternate Forms of Peer Review - Open Peer Review (“reviewers’ names are included on the peer review reports, and… the reports are made available online along with the final version of the manuscript) and Public Peer Review (“manuscripts that pass a rapid access-review are immediately typeset and published in the discussion forum in an onscreen format. They are then subject to Interactive Public Discussion”.)
Citizen Science - Makes the case for empowering patients to access medical research and conduct their own. Links to several initiatives.
Database For Disclosing Conflicts of Interest
Crossref.org - “Many researchers today never see corrections or retraction notices because they just download digital, PDF-formatted copies of the papers they need, and never again consult the original source. A new system, called CrossMark, consists of a logo that publishers will put on every PDF. Clicking on the logo will show Internet-connected users any updates to the work, whether retractions, corrections or other notes.”
Dryad Data Repository - “Dryad enables scientists to validate published findings, explore new analysis methodologies, repurpose data for research questions unanticipated by the original authors, and perform synthetic studies.”
Figshare - “Figshare allows researchers to publish all of their research outputs in seconds in an easily citable, sharable and discoverable manner. All file formats can be published, including videos and datasets that are often demoted to the supplemental materials section in current publishing models.”