"What do prediction markets predict?"

Summary

There has been controversy over the claim that prediction markets elicit ‘aggregate beliefs’, normally understood to mean the average of beliefs for the population. There is no debate over whether these markets generate good predictions, in the sense that they forecast outcomes well. Instead the debate has been over the additional claim that the prices in these markets recover the mean of aggregate beliefs. In other words, are the observed prices in these markets good estimates or predictors of aggregate beliefs?

According to the article, prediction markets cannot always be relied on to elicit any interesting statistic of aggregate beliefs. Formal derivations of the bets placed in prediction markets can be viewed as demands for state contingent commodities.

Prediction markets can be expected to do a good job recovering the average of aggregate beliefs under certain circumstances: unimodal distributions of beliefs, with no a priori reason to expect heterogeneity on either side of the market. Indeed, this environment might characterize many interesting settings, such as political elections or closed prediction markets in which there is minimal sample selection into the market (on the basis of beliefs, preferences and endowments). But the result is not general, and it is easy to construct examples in which prediction markets do a predictably poor job of recovering average beliefs.

Fountain, John, and Glenn W. Harrison. 2011. "What do prediction markets predict?." Applied Economics Letters 18, no. 3: 267-272. Business Source Complete, EBSCOhost (accessed September 18, 2015).

Critique

While I would concur to a cursory extent that the effect that prediction markets would assist with initial predictions, it is imperative to account for the nature of constantly changing variables in this market. In other words, though they would help to create initial models used to create plans, there needs to be a degree of fluidity in the plan built in to account for adaptations said market.

A Research Agenda for Prediction Markets

Patrick Buckley

Source: http://ceur-ws.org/Vol-1148/paper10.pdf 

Summary:

This article outlines the usage of prediction markets to enable collaborative intelligence by first discussing the general concept of prediction markets and then diving deeper into its methodology, and concludes with a research agenda that can address certain shortcomings. The purpose of prediction markets is to find and aggregate trader (participant) information, and use that information to “make predictions about specific future events”. This method is essentially created by participants buying into one of two outcomes. Prediction markets are different from financial markets in two ways: 

• Prediction markets allow participants to share information and trade contracts with each other
• “Its primary concern is the elicitation of information”

In other words, the shared information and forecast estimation are the main goals of prediction markets, not the financial gain or risk.

There are three broad divisions of operational prediction markets:

• Public prediction markets using real currency
• Public prediction markets using virtual currency
• Private prediction markets

Public prediction markets are open to the general public whereas private prediction markets have a sponsor who recruits participants from a specific group of people.

Strengths:

• Provide participants with incentive to share truthful information

• Provide an algorithm for automatic information communication and aggregation

• Can be conducted on a very large scale with up to hundreds or even thousands of participants

• Can operate over long periods of time

• Provides trader anonymity

Weaknesses:

• Might only attract individuals with certain personality traits (i.e. high risk tolerance individuals)

• Participants may manipulate the system by buying into an outcome that contradicts their truthful information because voting for that option may provide a better incentive

Critique:

Although the strengths are talked about in great detail, the weaknesses are barely mentioned, and there are only a few. Also there could be more detail about the public and private prediction markets and how they operate. This article’s conclusion is that there needs to be more collaborative research on prediction markets in regards to intelligence and decision making.

Prediction Market Accuracy in the Long Run

Summary

Prediction markets are designed and conducted for the primary purpose of aggregating information so that market prices forecast future events. These markets differ from typical, naturally occurring markets in their primary role as a forecasting tool instead of a resource allocation mechanism. In this paper the authors suggest that the prediction markets outperform polls for longer horizons by documenting the evidence from 1988 to 2004 elections. They compare unadjusted market predictions to 964 unadjusted polls over the five Presidential elections since 1988. What they found is intriguing: The market is closer to the eventual outcome 74% of the time. Further, the market significantly outperforms the polls in every election when forecasting more than 100 days in advance.  In their study they utilized the Iowa Electronic Markets (IEM) prices and raw poll dataset. We can compile the merits of predictions markets compared to polls based on the study findings:

•      Polls can`t have the true random sample; whereas the prediction markets customers can be very heterogeneous.
•       Production markets can forecast complex phenomena due to several reasons.

o   The market design forces traders to focus on the specific event of interest more than simple consideration of a fictitious election “if it were to be held today” (as polls ask respondents to consider)

o   Traders must act rationale since they put money on stake.

o   Markets aggregates dynamic information from a wide variety of sources, i.e. traders.

o   The markets provide an incentive to generate, gather and process information across information sources and in a variety of ways. (If you do good, you prosper.)

•       For the five elections, the average absolute error in the market’s prediction of the major-party presidential vote share across the 5 days prior to the election was 1.20 percentage points, while opinion polls conducted during that same time had an average error of 1.62 percentage points.
•       Unlike polls` random selection, the participants of prediction markets are self-selected.
•       Unlike polls or expert panels in which participants are asked for their independent opinions, each trader in the market sees the net effect of the beliefs of all other traders, and the time series of changes in those beliefs and can alter his own perceptions accordingly.
•       Unlike polls that ask each respondent how he or she would vote if the election were held today, the market asks traders to forecast how everyone will vote in the actual upcoming election. (We can suggest that the sentiments play role on polls; whereas the factuality in PMs)
•       ``Convention bounce`` effects don`t appear in prediction markets.
•       It gives continuous updates.
•       Because they react dynamically to information, they can also be used as evaluation tools to assess the impact of decisions such as policy positions, candidate viability, campaign strategies, etc.

Critique

The study lays out the facts effectively. The authors compile the prediction markets` advantages and superiorities over the polls. However, the study does't mention about the weaknesses of the technique which can be the sign of confirmation bias. Moreover, polls attract many attentions currently. If the prediction markets have lots of merits over polls, I would expect the study to identify the reason why still polls make people interested in them. Nevertheless, the study explains very well why the prediction markets outperform polls through five consecutive years.

Source:

Berg, J. E., Nelson, F. D., & Rietz, T. A. (2008). Prediction market accuracy in the long run. International Journal of Forecasting, 24(2), 285-300.

Delphi Method (Rating: 4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the  articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2015 regarding Delphi as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Delphi is an analytic methodology that utilizes a panel of subject matter experts to come to a consensus on an issue.  Successive rounds of surveys are given to the experts and are completed anonymously.  Feedback, consisting of other answers and justifications for the answer, is provided to the experts between each round. The feedback is taken into account in next round of responses.  At the end of the rounds, final answers are delivered to the panelists and the customer for whom the method was conducted.

The Delphi method attempts to achieve several objectives: develop a range of possible alternatives, explore underlying assumptions, seek out information that may generate a consensus, and educate the participants on different ideas and aspects of a specific topic.

Strengths:

• Method provides anonymity to participants
• Method uses independence
• It is very useful to come up with a consensus
• Anonymity and independence help overcome groupthink
• Method is versatile
• Controlled feedback gives participants option to hear others’ opinions and then go back and modify their answers
• Statistical analytic ensure equal representation among participants

Weaknesses:

• Method can take extensive periods of time
• Panels must be composed of experts
• It can be manipulated by the researcher due to his or her extreme control over the process
• Technique requires feedback, but some participants may not respond
• Method hinges on ability/training of facilitator/researcher

How-To:

1. Develop a questionnaire/survey/scorecard to evaluate an estimate or recommendation
2. Gather a group of experts related to the subject at hand
3. Provide them the questionnaire/score card and ask them to evaluate the options of the given scenario
4. Gather the data and provide the scores of everyone & the average for each option. Keep everything anonymous for the participants in the study.
5. Conduct at least 2 rounds of scoring and then evaluate the results.
6. Finally, evaluate the best option or at least eliminate the worst option(s).

Personal Application of Technique:

Delphi exercise involved a UN type simulation decision on which policy was more effective in determining what policy to adapt in regards to Nuclear policy. In order to do this, the group first created a “role” so that the person could act as an expert. The roles that were chosen included a person from the Institute for Science and International Security, Pakistani Prime Minister, Chinese Foreign Minister, Indian Prime Minister, Japanese Foreign Minister, Secretary of Defense, Secretary of State, President of Iran.

Some of the lessons learned include: visual representation of results, incorporating technology (i.e. Google Forms) to take advantage of time, and also having different questions for each round. It would be more beneficial to present the results on a histogram.

For Further Information:

Delphi Exercise folder