Expert Political Judgment

Tetlock asked a question pundits avoid: if you keep score, how good are expert forecasts? Over roughly 18 years he collected predictions from 284 specialists in politics and economics and scored them. Most did no better than simple statistical baselines, and many did worse. More expertise and more fame did not buy more accuracy.

What did matter was style of thought. Borrowing Isaiah Berlin's metaphor, Tetlock shows that hedgehogs, who apply one big idea everywhere, were overconfident and did worst in their own specialties, while foxes, who draw on many sources and doubt themselves, forecast better and updated more readily when wrong.

Tetlock, a psychologist then at Berkeley and now at the University of Pennsylvania, first published the book in 2005. It won wide praise: Louis Menand reviewed it in The New Yorker, the Financial Times called it the definitive work on the question, and Daniel Kahneman called it a landmark. This 2017 edition adds a new preface on the forecasting research that followed, the line of work behind Tetlock's Superforecasting (2015, with Dan Gardner).

It is an academic book, with long methodological and technical appendices, and readers who want the practical version should start with Superforecasting. But this is the evidence base, and it is required reading for anyone who sets or buys forecasts.

1. Keep score. Forecasts that are never checked cannot improve. Writing down probabilities and grading them later is the only way to tell skill from storytelling.
2. Confidence is not accuracy. Hedgehogs who called outcomes impossible or nearly so saw roughly one in five of them happen, about twice the rate for foxes, and their sure things failed more often too.
3. Specialists can do worse in their own field. Deep expertise tied to a single theory made hedgehogs less accurate where they knew the most, which should temper deference to famous house views.
4. Update when wrong. Foxes revised their beliefs more when events went against them. Hedgehogs explained the miss away, a habit any portfolio manager will recognize.
5. Beware what the media rewards. The bold, simple, confident calls that win airtime are the traits associated with poor forecasting, so the loudest market voice is not the best guide.

Eight chapters build from method to findings to the objections, with methodological and technical appendices at the end.

1. Quantifying the unquantifiable. Tetlock sets out how to measure forecasting skill, using clear questions, probability estimates and scoring rules.
2. Radical skepticism. He tests the hypothesis that the world is too complex for anyone to predict, comparing experts with informed amateurs and simple extrapolation of trends.
3. Foxes versus hedgehogs. Three chapters show that foxes beat hedgehogs on calibration and discrimination, update their beliefs more in line with Bayesian logic when proved wrong, and are more willing to entertain scenarios that cut against their views.
4. The hedgehogs strike back. Tetlock gives the defense its say, weighing arguments that hedgehogs are bold for good reasons or are unfairly scored.
5. Limits. The last chapters ask whether open-mindedness has its own limits and how accountability and objectivity could improve judgment in public debate.

Since its original publication, Expert Political Judgment by New York Times bestselling author Philip Tetlock has established itself as a contemporary classic in the literature on evaluating expert opinion. Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin's prototypes of the fox and the hedgehog, Tetlock contends that the fox--the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events--is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits--the single-minded determination required to prevail in ideological combat. Clearly written and impeccably researched, the book fills a huge void in the literature on evaluating expert opinion. It will appeal across many academic disciplines as well as to corporations seeking to develop standards for judging expert decision-making. Now with a new preface in which Tetlock discusses the latest research in the field, the book explores what constitutes good judgment in predicting future events and looks at why experts are often wrong in their forecasts.