Softer sciences publish more positive results

A colleague referenced this 2010 paper which measures proportion of positive results by scientific discipline.

I’m not sure how much value there is in the “hard” vs “soft” framing of the disciplines.  I feel like it’s a bit of a rabbit hole.  If that’s something that interests you, though, there are plenty of references in the paper, which is open access.

I am interested in the general results for each discipline:

As you can see, psychiatry & psychology had the highest proportion of positive results, at 91.5%, while space sciences had the lowest at 70.2%.  The average was 84%.

They also found that applied sciences were more likely to report positive results than “pure” sciences, and that studies with human subjects were more likely to report positive results than studies with non-human subjects:

Another interesting note was the frequency of negative results in papers that reported multiple hypotheses:

The frequency of negative results in papers that tested multiple hypotheses (N = 151, in which only the first hypothesis was considered), was significantly higher than in papers testing only one hypothesis (X2 = 13.591, df = 1, pX2 = 140.308, df = 2, p2N = 0.051).

In the discussion section, the researchers offer two broad explanations for what’s causing these differences: the hypotheses tested in softer sciences might be more likely to be true, and/or the testing of hypotheses in the softer sciences might be less rigorous.

They give a few reasons why the first explanation might be the case.  Hypotheses in the softer sciences might be based on more personal experience and observation, leading to an informal weeding out of bad theories.  (This is a common argument in favor of hypotheses in psychology having a higher success rate than average.)  Alternatively, and less flatteringly, hypotheses tested in the soft sciences may be less “deep”:

Younger, less developed fields of research should tend to produce and test hypotheses about observable relationships between variables (“phenomenological” theories). The more a field develops and “matures”, the more it tends to develop and test hypotheses about non-observable phenomena underlying the observed relationships (“mechanistic” theories). These latter kinds of hypotheses reach deeper levels of reality, are logically stronger, less likely to be true, and are more conclusively testable.

The second set of explanations, which put forth the idea that soft sciences are less rigorous, touch upon a number of ideas talked about frequently on this tumblr: “Flexibility in definitions, design, analysis and interpretation of a research”; “Prevalence and strength of experimenter effects and self-fulfilling prophecies”’; “Non-publication of negative and/or statistically non-significant results”; “Prevalence and strength of manipulation of data and results”.

Again, I’m not sure how useful the framing of hard vs soft is, but this paper does make clear that there are methodological and cultural differences between different scientific disciplines, and that we’d do well to examine them in order to determine best practices.