Superforecasting

Tetlock, a psychologist at the University of Pennsylvania, became known for a twenty-year study, published as Expert Political Judgment, which found that the average expert's predictions were little better than chance. This book, written with journalist Dan Gardner, asks the follow-up question: can some people forecast well, and can others learn to? The answer comes from the Good Judgment Project, which Tetlock co-founded and which won a forecasting tournament run by IARPA, the U.S. intelligence community's research agency.

Its top performers, the superforecasters, were not geniuses or specialists. They broke problems into parts, started from base rates, updated in small steps and kept score. Jason Zweig in the Wall Street Journal called it the most important book on decision making since Kahneman's Thinking, Fast and Slow, The Economist named it a best book of 2015, and Kirkus praised it as an accessible field guide.

The tournament questions had short horizons and clear answers, which is not the same as calling a market top or a ten-year return. One academic reviewer, Klaus Solberg Søilen, complained that the book explains too little about exactly how the project beat its rivals.

For analysts, portfolio managers and anyone who makes calls under uncertainty, the habits it describes are directly usable.

1. Keep score. Forecasts that are vague or never checked cannot improve; precise probabilities scored over time show who is actually good.
2. Start with the base rate. Begin from how often similar events happen, then adjust for the specifics, rather than starting from the story in front of you.
3. Update often and in small steps. The best forecasters revise as evidence arrives without overreacting to each headline.
4. Break hard questions into parts. Decomposing a problem into knowable and unknowable pieces turns a guess into an estimate.
5. Build teams that challenge each other. Groups that share information and question each other's reasoning beat individuals, which applies directly to investment committees.

The book has twelve chapters that move from the problem with expert prediction to the habits and teams that fix it.

1. The case for keeping score (Chapters 1-3). Tetlock describes himself as an optimistic skeptic, explains why forecasts are rarely checked, and shows how vague language and the absence of scoring let pundits avoid accountability.
2. Who the superforecasters are (Chapters 4-7). Introduces the IARPA tournament and the volunteers who did best, then tests whether intelligence, numeracy or news consumption explain their results.
3. How they get better (Chapter 8). Argues that the key trait is treating beliefs as hypotheses to be tested and refined, a habit Tetlock calls perpetual beta.
4. Teams and leaders (Chapters 9-10). Shows how well-run teams outperformed individuals, and squares the humility good forecasting requires with the decisiveness leaders need.
5. Limits and next steps (Chapters 11-12). Addresses skeptics who doubt that forecasting skill matters for rare, high-impact events, and outlines how organizations could use scored forecasts.
6. Appendix. Ten commandments for aspiring superforecasters summarize the method.

A New York Times Bestseller An Economist Best Book of 2015 The most important book on decision making since Daniel Kahneman's Thinking, Fast and Slow. —Jason Zweig, The Wall Street Journal Everyone would benefit from seeing further into the future, whether buying stocks, crafting policy, launching a new product, or simply planning the week’s meals. Unfortunately, people tend to be terrible forecasters. As Wharton professor Philip Tetlock showed in a landmark 2005 study, even experts’ predictions are only slightly better than chance. However, an important and underreported conclusion of that study was that some experts do have real foresight, and Tetlock has spent the past decade trying to figure out why. What makes some people so good? And can this talent be taught? In Superforecasting, Tetlock and coauthor Dan Gardner offer a masterwork on prediction, drawing on decades of research and the results of a massive, government-funded forecasting tournament. The Good Judgment Project involves tens of thousands of ordinary people—including a Brooklyn filmmaker, a retired pipe installer, and a former ballroom dancer—who set out to forecast global events. Some of the volunteers have turned out to be astonishingly good. They’ve beaten other benchmarks, competitors, and prediction markets. They’ve even beaten the collective judgment of intelligence analysts with access to classified information. They are superforecasters. In this groundbreaking and accessible book, Tetlock and Gardner show us how we can learn from this elite group. Weaving together stories of forecasting successes (the raid on Osama bin Laden’s compound) and failures (the Bay of Pigs) and interviews with a range of high-level decision makers, from David Petraeus to Robert Rubin, they show that good forecasting doesn’t require powerful computers or arcane methods. It involves gathering evidence from a variety of sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. Superforecasting offers the first demonstrably effective way to improve our ability to predict the future—whether in business, finance, politics, international affairs, or daily life—and is destined to become a modern classic. From the Hardcover edition.