Do prediction markets produce well-calibrated probability forecasts?

Summary:

Page and Clemen (2012) examined the accuracy of prediction markets in making probability estimates.  Prediction markets are used to gather estimates based on various sources of information that market players have different access to.  Theoretically, as the various sources of information are taken into account, the buying and selling of predictions in the market generate a more accurate estimate. Prediction markets usually involve short-term predictions, but it was still widely assumed that long-term predictions in the market are just as accurate as the short-term ones.  Page and Clemen’s findings prove otherwise.

Prices in prediction markets are often calibrated in order to get closer to a true estimate of a particular event.  There are several inter-related reasons why prices deviate from an event’s true probability, many of which are just manifestations of stock market tactics in the predication market.  Players with limited budgets will often take longshots on predictions in hopes of selling them later at an inflated price.  Another reason for deviation comes from price manipulation.  Players may be encouraged to buy or sell predictions for the sole purpose of moving the price up or down.  An example of an encouraged player would be one who buys predictions of an event which he or she has influence of that event’s occurrence.

Page and Clemen found that as predictions in the market cover an extended timespan, players in the market are less likely to trade them.  Players undervalue the probability estimates of long-term predictions.  Long-term predictions, consequently, have little use in prediction markets.  The lack of trade volume undermines the importance that the prediction may actually have.  A long-term prediction in this case is a prediction or estimate that involves an event that will take over a year or longer to occur.  However, Page and Clemen did find that short-term predictions (those within 100 days) are consistent enough to be calibrated for value predictive insight.  The shorter the timespan, the more accurate a calibrated estimate is.

Critique:

These findings speak to one of the biggest weaknesses of prediction markets: they cannot suit questions with too great of a difficulty.  Any use of prediction markets for strategic decision support has remote chances of being helpful.  However, their usefulness for tactical to operational decisions would be worth exploring.  The authors did not specify if the prediction markets they examined specific to any particular subjects (i.e. sports, stock market, etc.), so further exploration for intelligence-related prediction markets is required.

Source:

Page, L., & Clemen, R. T. (2012). Do prediction markets produce well-calibrated probability forecasts? The Economic Journal, 123, 491–531.