Who’s Good at Forecasts?
By: The Economist
The Economist took a look at predictive markets in their special edition, The World in 2014. They looked at Philip Tetlock’s 1980s forecasting tournament involving 284 economists, political scientists, intelligence analysts and journalists. This research collected around 28,000 predictions which concluded that “the average expert did only slightly better than random guessing.” The forecasts were expressed numerically so an expert could not provide vague words such as “may” or “possible”. The results also concluded that “experts with the most inflated views of their batting averages tended to attract the most media attention.”
The Intelligence Advanced Research Projects Activity (IARPA) used Tetlock’s forecasting tournament as a pilot to sponsor a more ambitious tournament called The Good Judgment Project. The project has collected over one million forecasts from 5,000 forecasters on 250 questions. These questions range from the euro-zone to the Syrian civil war. From this research, IRAPA has been able to discover which methods of training promote accuracy.
This research also explores the super-forecaster hypothesis. Within the first year of the tournament, two percent of forecasters showed that luck was involved. However, after a while, forecasters became better and the super-forecasters were assigned to teams. These forecasters beat the “unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithm for four competitor institutions by 35-60%; and beat two prediction markets by 20-35%.”
To be a part of The Good Judgment Project you can register here.
Although the Economist did a very good job explaining both forecasting tournaments, I found the analysis of the research lacking. I would have liked to see a more in-depth look at how the tournaments came to their conclusions in addition to how super forecasters are grouped into teams.
Who’s good at forecasts? (2013, November 18). The Economist http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts