Correcting the Lies that Data Tell

Found this interesting paper from a couple years ago: _Detecting and Correcting the Lies that Data Tell_by F Schmidt.

I won’t pretend to understand the meat of the paper, where Schmidt argues that correcting for sampling error and measurement error will decrease variability (got that) which strengthens correlations (???).  However the author makes some other interesting points further along in the paper.  There’s a useful section on sources of error:

My initial example illustrates the two artifacts that are always present in any literature. But there are others that are often, but not always, present, such as data errors, range restriction, dichotomization of measures, and imperfect construct validity. Data errors—typos, coding errors, transcription errors, etc.—have been shown to be very prevalent (Hunter & Schmidt, 2004, pp. 53–54). This is a nonsystematic source of variability, like sampling errors. Unless they result in extreme or impossible outliers, data errors are hard to identify and therefore difficult or impossible to correct.

Unlike data errors and sampling errors, range restriction is a systematic artifact. Range restriction reduces the mean correlation. Also, variation in range restriction across studies increases the between-study variability of study correlations.

… Another artifact is caused by dichotomization. Researchers often dichotomize continuous measures into “high vs. low” groups. This practice not only loses information but also lowers correlations and creates more variability in findings across studies (Cohen, 1983; Hunter & Schmidt, 1990a, 2004; MacCallum, Zhang, Preacher, & Rucker, 2002). The distorting effects of dichotomization are correctable in a meta-analysis.

The final additional artifact I want to mention is imperfect construct validity in measures. Even after correction for measurement error, the measure may correlate less than perfectly with the desired construct (Schmidt, Le, & Oh, 2009). This is especially true when proxy measures are used (for example, use of education as a proxy for general mental ability). Degree of construct validity may vary across studies, causing between-study variability and typically lowering the mean. Correction for this requires special information, is complicated, and is often not possible (Hunter & Schmidt, 2004).

The authors also do their own meta-analysis of the use of fixed-effects vs random-effects models in the literature:

Fixed effects (FE) meta-analysis models assume a priori that there is only a single population parameter underlying all studies. That is, FE models assume that all variation across studies is due to solely to sampling error and that therefore none of the variation is due to real differences between studies in underlying parameters.1 This a priori assumption is highly questionable in most cases. RE models, by contrast, treat this assumption as an hypothesis and test it—allowing the researcher to see whether or not all variance is accounted for by sampling error and other artifacts.

… My colleagues and I recently examined the meta-analyses in this journal (Schmidt, Oh, & Hayes, 2009) and found that a total of 199 meta-analyses were published in Psychological Bulletin between 1978 and 2006. Of the 169 that could be classified as either FE or RE models, 79% (129) used only FE models.Figure 7 shows these findings.

A reanalysis of data from five of these FE meta-analysis studies (containing a total of 68 different meta-analyses) showed they seriously underestimated the width of the CIs they reported by an average of 52%. That is, the CIs were only half as wide as their real width, a gross overestimation of the precision of the findings. On average, the CIs reported as 95% CIs were actually 55% CIs (Schmidt, Oh, & Hayes, 2009).

What about corrections for measurement error? We found that 180 of the 199 published meta-analyses (90%) did not correct for measurement error—which, as noted earlier, is always present! Nor did they correct for the other artifacts I discussed. Figure 8 shows these findings.

I’m going to do a bit of digging in the references and see if I can find a decent explanation for how the authors’ adjustment for measurement error worked.  If I find anything useful, I’ll update this post.