Review: The Signal and the Noise

When Nate Silver correctly called the winner of all 50 states in this year’s presidential election (and 31 out of the 33 Senate races), how many people rushed out to buy his book, eager to learn how to make such spot-on predictions? How many were disappointed?

“What should give us pause,” Silver writes, “is that the few ideas we have tested aren’t doing so well, and many of our ideas have not or cannot be tested at all. In economics, it’s much easier to test an unemployment rate forecast than a claim about the effectiveness of stimulus spending. In political science, we can test models that are used to predict the outcome of elections, but a theory about changes to political institutions might take decades to verify… we are undoubtedly living with many delusions we do not even realize.” (p. 14-15)

The Signal and the Noise: Why So Many Predictions Fail - But Some Don’t, wears its heart on its sleeve. “Many predictions fail”, the book jacket warns us, and much of the book’s 450+ pages are occupied with explaining those failures. Silver hopscotches from one domain to another, covering baseball, climate change, the stock market, chess, meteorology, poker, and more. Each story unfolds engagingly, and each is used to answer a few key questions.

What makes forecasting so difficult?

Silver contrasts games like chess, where computers are able to out-predict and therefore outplay the most talented humans, with fields like economics where even the experts rarely perform better than chance. Having clear rules helps with predictions, or in the absence of rules, a good conceptual theory of underlying mechanisms. Our better understanding of weather, as opposed to earthquakes, is one of a couple reasons that meteorology has seen much more success than seismology. Another key aspect is independence. In baseball, the performance of one batter is largely independent of another, but on the stock market, the behavior of one trader will quickly influence another. And if the relationship is exponential, rather than additive - that is, if the system is nonlinear - you get the tumultuous environments that bred Chaos Theory. Finally, Silver points out that studying short-term phenomena like the weather gives us more opportunity to test and hone our predictions than studying long-term phenomena like climate change.

What are some common errors?

Silver gives several examples of events that went unpredicted because they were “out of sample”. In the chapter on terrorism, he explains why we shouldn’t have been surpised by Pearl Harbor: “That Nebraska, say, had never been attacked by a foreign power gave no real evidentiary weight to the situation in Hawaii, given the latter’s outlying position in the Pacific.” (p. 420) A more mundane example would be a person estimating their likelihood of crashing if they drive home drunk by reasoning that they’ve only crashed once in twenty years of driving sober. Silver also warns us to watch for non-independent probabilities. The mortgages in many CDOs were thought to have independent probabilities of defaulting, thus decreasing the likelihood that the CDO as a whole would default. But as the housing bubble burst, each mortgage default made the next more likely - and many CDOs came due, driving the firms who had made them into bankruptcy. Silver also strongly warns against overfitting, mistaking noise for signal. It can be tempting, when you have limited or highly variable data, to design a model that captures as much variance as possible. But some of that variance will be due to pure chance, and attempting to model it will only make your predictions worse. Silver talks about how overfitting may have led to too-conservative estimates of the likelihood of a magnitude 9 earthquake in Japan.

What are some tools/heuristics we can use to make better predictions?

Silver devotes a whole chapter to Bayes Theorum, which he explains at length. (See also Eliezer Yudkowsky’s “excruciatingly gentle” introduction.) Bayes’ theorum helps counteract two key errors that humans make. First, it makes us consider our prior beliefs. While some may protest that this introduces subjectivity to the equation, Silver would argue that subjectivity is always there, in the form of unquestioned assumptions and hidden biases, and that including it as the ‘prior probability’ lets us deal with it explicitly. The other major error is not considering the base rate for a phenomenon. The classic example is a woman who, receiving news that her mammogram was positive, and knowing that mammograms only wrongly diagnose 10% of cancer-free women, assumes she likely has cancer. But she is ignoring the base rate of breast cancer in the population. Because so many more women are cancer-free than not, the chance that this particular woman has cancer, given the positive mammogram, is only about 10%. (See the above link if this is confusing.)

Silver also talks about Power Laws, which are when the frequency of a phenomenon varies by a power of some other attribute, frequently size. (A grossly simplified example: if a random sampling of your county found 1 person who made $10 million/year, 10 people who made $1 million, 100 who made $100,000, etc.) When you map out this relationship on a log-log graph (where both the x-axis and y-axis are logarithim scales) you see a simple linear relationship. Earthquakes obey a power law. Magnitude 6 earthquakes are a hundred times more frequent than Magnitude 8 earthquakes - but release a thousand times less energy. Interestingly, global terrorism also seem to obey a power law. (Although of course, we have many fewer data points.) “If the power-law process is applied to data collected entirely before 9/11,” Silver writes, ”… it implies that a September 11-scale attack would occur about once every eighty years in a NATO country, or roughly once in our lifetimes” (p. 432)

Silver touches on a few other interesting topics: how aggregations of predictions tend to do better than individual predictions; how data suggests congresspeople are profiting from insider trading after leading office; why Derek Jeter is overrated (which, whatever, this play is still glorious.) He also mentions briefly problems of prediction in psychology and biomedical sciences, which made me wonder how to high-five a book.

To be honest, Silver could have gotten his points across clearly and effectively in a few dozen pages. Instead, he gives us a wandering, unstructured, but entertaining lecture. It’s almost like sitting around a campfire telling stories - if you’ve gone camping with unabashed stats nerds. If that sounds like fun to you, then I recommend this book. And also, let’s go camping!