Thursday, October 9, 2014

Do prediction markets produce well-calibrated probability forecasts?



Summary:

Page and Clemen (2012) examined the accuracy of prediction markets in making probability estimates.  Prediction markets are used to gather estimates based on various sources of information that market players have different access to.  Theoretically, as the various sources of information are taken into account, the buying and selling of predictions in the market generate a more accurate estimate. Prediction markets usually involve short-term predictions, but it was still widely assumed that long-term predictions in the market are just as accurate as the short-term ones.  Page and Clemen’s findings prove otherwise.



Prices in prediction markets are often calibrated in order to get closer to a true estimate of a particular event.  There are several inter-related reasons why prices deviate from an event’s true probability, many of which are just manifestations of stock market tactics in the predication market.  Players with limited budgets will often take longshots on predictions in hopes of selling them later at an inflated price.  Another reason for deviation comes from price manipulation.  Players may be encouraged to buy or sell predictions for the sole purpose of moving the price up or down.  An example of an encouraged player would be one who buys predictions of an event which he or she has influence of that event’s occurrence.



Page and Clemen found that as predictions in the market cover an extended timespan, players in the market are less likely to trade them.  Players undervalue the probability estimates of long-term predictions.  Long-term predictions, consequently, have little use in prediction markets.  The lack of trade volume undermines the importance that the prediction may actually have.  A long-term prediction in this case is a prediction or estimate that involves an event that will take over a year or longer to occur.  However, Page and Clemen did find that short-term predictions (those within 100 days) are consistent enough to be calibrated for value predictive insight.  The shorter the timespan, the more accurate a calibrated estimate is.



Critique:

These findings speak to one of the biggest weaknesses of prediction markets: they cannot suit questions with too great of a difficulty.  Any use of prediction markets for strategic decision support has remote chances of being helpful.  However, their usefulness for tactical to operational decisions would be worth exploring.  The authors did not specify if the prediction markets they examined specific to any particular subjects (i.e. sports, stock market, etc.), so further exploration for intelligence-related prediction markets is required.



Source:

Page, L., & Clemen, R. T. (2012). Do prediction markets produce well-calibrated probability forecasts? The Economic Journal, 123, 491–531.

6 comments:

  1. Kyle,

    Organizations aspiring to incorporate prediction markets to support decision-making should definitely weigh the findings of this article into account, just as they should with the article I reviewed on manipulation of prediction markets.

    Could the "output" of a short-term (100 days for instance) prediction market probability estimate feed into another technique more suitable for making long-term predictions?

    ReplyDelete
    Replies
    1. I'm not sure. I would think short-term prediction markets outcomes would fit nicely into Bayesian if one decided to use that as a long-term prediction tool. It would take studies to answer your great question.

      Delete
  2. Kyle, did this article specify what is a short-term or long-term estimate?

    ReplyDelete
    Replies
    1. Anything longer than 1 year is considered long-term in this study.

      Delete
  3. Kyle, does the amount of trade volume alter the long or short term predictions within the market?

    ReplyDelete
    Replies
    1. Yes, low trade volume values indicate an undervalued prediction. As a result, the prediction will be skewed to the left (underestimated). Normally, a mathematical calibration could fix that, but the skew is too varied for long-term predictions.

      Delete