Overfitting

I found this 2004 article through the footnotes of Nate Silver’s book.  Here are the highlights (but I recommend reading it all the way through.)

There has been something of a revolution in data analysis in the past 10 or so years. Modern computational power has not only made it easier to solve complex and large analytic problems but also allowed us to study, through a technique called simulation, the very act of collecting data and performing analyses… These simulation studies have taught us a great deal about the scientific merit of some of our conventions in data analysis and also have pointed toward new directions that may improve our practice as researchers.

The author, Babyak, draws attention to the problem of overfitting, detailing a simulation which shows that one needs 13 samples per predictor, if not more.  Babyak also citesa 1991 paper which suggests a base of 50 samples plus 8+ more per predictor, but warns that even large sample sizes may be compromised if effect size is small or if predictors are correlated.

The author calls out four problematic methods/issues:

• Automated Stepwise Regression: An automated way to shift through many predictors and select only the best combination. Babyak warns this practice should never be acceptable.

• Univariate Pretesting or Screening: Similar to the above, only in this case one manually looks for correlations between predictors and responses. This explicit p-value testing is not the only place where researchers unwittingly expend degrees of freedom. “Farawaydemonstrated that these phantom degrees of freedom actually arise in all sorts of unexpected places, such as examining residuals for homogeneity of variance, testing for outliers, or making transformations to improve power, to name a few, underscoring the principle that virtually any data-driven decision about modeling will lead to an overly optimistic model.”

• Dichotomizing Continuous Variables: “The common practice of dichotomizing 2 continuous variables and using them as factors in an ANOVA will yield an unacceptable Type I error rate when those 2 original variables are even moderately correlated. Because ANOVA is just a special case of the general linear model, this problem also will haunt us in the multivariable regression situations.”

• Multiple Testing of Confounders: Just as with other forms of multiple comparisons, this is using degrees of freedom. “The real problem here is that unless you have been very careful to account for expended degrees of freedom, you will not have any way of knowing the extent to which the apparent confounder is a real confounder or just caused by the play of chance sampling.”

The fixes offered are mostly straightforward:  Collect more data.  Combine predictors into an index, if the predictors are related.  If a predictor is well known, model it as a constant.  And finally, Babyak advises adjusting your result using shrinkage techniques.