Significance

“You keep using that statistical test. I do not think it means what you think it means.”

A couple months ago I wrote a post about publication bias in which I offered a fairly glib explanation of statistical significance. Reading over it, I knew it was partially incorrect, but it wasn’t the focus of the post so I left it in. I promised myself I would come back and correct my mistake later, and that moment has arrived. Okay, so what did I say? I wrote: “A significant effect is a one which, statistically, we think has at least a 95 percent likelihood of being real, and no more than 5 percent likelihood of being due to chance.” What I should have said was: “A significant effect is one in which there is a 5 percent chance of getting an equally large or larger result, given that the null hypothesis is true.” That’s not just quibbling about phrasing. There’s a real semantic difference between those two statements. It’s located at the end of the second sentence: Given that the null hypothesis is true. That’s a conditional statement, and it warns us away from making assumptions about other conditions. The first part of the sentence only applies when the null hypothesis is true - that is, when there is no actual, objective effect. What can we say about the statistical effect’s likelihood of being real, then? Nothing. Because any statement about the effect’s realness would have the condition that the null hypothesis was not true. The creators of the hypothesis test easily acknowledged its extreme limitations:

…no test based upon a theory of probability can by itself provide any valuable evidence of the truth or falsehood of a hypothesis. But we may look at the purpose of tests from another viewpoint. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not often be wrong. [emphasis mine]

This paper argues that misinterpretations, over-extensions and over-reliance on the p-value point to a fundamental problem with this type of statistics:

The idea that the P value can play both of these roles is based on a fallacy: that an event can be viewed simultaneously both from a long-run and a short-run perspective. In the long-run perspective, which is error-based and deductive, we group the observed result together with other outcomes that might have occurred in hypothetical repetitions of the experiment. In the “short run” perspective, which is evidential and inductive, we try to evaluate the meaning of the observed result from a single experiment. If we could combine these perspectives, it would mean that inductive ends (drawing scientific conclusions) could be served with purely deductive methods (objective probability calculations). These views are not reconcilable because a given result (the short run) can legitimately be included in many different long runs. A classic statistical puzzle demonstrating this involves two treatments, A and B, whose effects are contrasted in each of six patients. Treatment A is better in the first five patients and treatment B is superior in the sixth patient. Adopting Royall’s formulation, let us imagine that this experiment were conducted by two investigators, each of whom, unbeknownst to the other, had a different plan for the experiment. An investigator who originally planned to study six patients would calculate a P value of 0.11, whereas one who planned to stop as soon as treatment B was preferred (up to a maximum of six patients) would calculate a P value of 0.03. We have the same patients, the same treatments, and the same outcomes but two very different P values (which might produce different conclusions), which differ only because the experimenters have different mental pictures of what the results could be if the experiment were repeated.

The author is a Bayesian, which means, broadly speaking, that he prefers to drag the subjective aspects of experimental statistics out into the light. Bayesian probabilities require the use of “priors”, a quantification of previous knowledge about whether or not a hypothesis is likely to be true. Priors make many people deeply uncomfortable, because they are so subjective. But without making subjective assumptions, Goodman argues, you can’t make arguments about probability. So let’s be clear about our assumptions, and not bury them behind a statistical methodology that can’t account for them. You can see the author’s philosophical leanings in a passage early on in the article:

What is remarkable about this paper is how unremarkable it is. It is typical of many medical re-search reports in that a conclusion based on the findings is stated at the beginning of the discussion. Later in the discussion, such issues as biological mechanism, effect magnitude, and supporting studies are presented. But a conclusion is stated before the actual discussion, as though it is derived directly from the results, a mere linguistic transformation of P < 0.06. This is a natural consequence of a statistical method that has almost eliminated our ability to distinguish between statistical results and scientific conclusions. We will see how this is a natural outgrowth of the “P value fallacy”.

I cannot agree with this section more. I’m not a statistician myself, so I don’t feel qualified to say whether Bayesian methods are more useful than frequentist ones, but I damn well wish this debate was louder and more pervasive. I made it through two statistics classes, at two very good schools, without ever hearing about Bayes or about common misinterpretations of P values. I was just lucky that a post doc in one of the labs I worked at was interested in this stuff, and that over coffee and giant magnets one day, he explained enough of the controversy to pique my interest. This is the kind of thing you should be taught in intro stats, in any field where students and researchers are going to be asked to interpret data. It’s not the kind of thing you should hear about for the first time six years into studying a field. It’s a big deal. It’s, you know. Significant.