A Research Agenda for Prediction Markets

Patrick Buckley

Source: http://ceur-ws.org/Vol-1148/paper10.pdf 

Summary:

This article outlines the usage of prediction markets to enable collaborative intelligence by first discussing the general concept of prediction markets and then diving deeper into its methodology, and concludes with a research agenda that can address certain shortcomings. The purpose of prediction markets is to find and aggregate trader (participant) information, and use that information to “make predictions about specific future events”. This method is essentially created by participants buying into one of two outcomes. Prediction markets are different from financial markets in two ways: 

• Prediction markets allow participants to share information and trade contracts with each other
• “Its primary concern is the elicitation of information”

In other words, the shared information and forecast estimation are the main goals of prediction markets, not the financial gain or risk.

There are three broad divisions of operational prediction markets:

• Public prediction markets using real currency
• Public prediction markets using virtual currency
• Private prediction markets

Public prediction markets are open to the general public whereas private prediction markets have a sponsor who recruits participants from a specific group of people.

Strengths:

• Provide participants with incentive to share truthful information

• Provide an algorithm for automatic information communication and aggregation

• Can be conducted on a very large scale with up to hundreds or even thousands of participants

• Can operate over long periods of time

• Provides trader anonymity

Weaknesses:

• Might only attract individuals with certain personality traits (i.e. high risk tolerance individuals)

• Participants may manipulate the system by buying into an outcome that contradicts their truthful information because voting for that option may provide a better incentive

Critique:

Although the strengths are talked about in great detail, the weaknesses are barely mentioned, and there are only a few. Also there could be more detail about the public and private prediction markets and how they operate. This article’s conclusion is that there needs to be more collaborative research on prediction markets in regards to intelligence and decision making.

Prediction Market Accuracy in the Long Run

Summary

Prediction markets are designed and conducted for the primary purpose of aggregating information so that market prices forecast future events. These markets differ from typical, naturally occurring markets in their primary role as a forecasting tool instead of a resource allocation mechanism. In this paper the authors suggest that the prediction markets outperform polls for longer horizons by documenting the evidence from 1988 to 2004 elections. They compare unadjusted market predictions to 964 unadjusted polls over the five Presidential elections since 1988. What they found is intriguing: The market is closer to the eventual outcome 74% of the time. Further, the market significantly outperforms the polls in every election when forecasting more than 100 days in advance.  In their study they utilized the Iowa Electronic Markets (IEM) prices and raw poll dataset. We can compile the merits of predictions markets compared to polls based on the study findings:

•      Polls can`t have the true random sample; whereas the prediction markets customers can be very heterogeneous.
•       Production markets can forecast complex phenomena due to several reasons.

o   The market design forces traders to focus on the specific event of interest more than simple consideration of a fictitious election “if it were to be held today” (as polls ask respondents to consider)

o   Traders must act rationale since they put money on stake.

o   Markets aggregates dynamic information from a wide variety of sources, i.e. traders.

o   The markets provide an incentive to generate, gather and process information across information sources and in a variety of ways. (If you do good, you prosper.)

•       For the five elections, the average absolute error in the market’s prediction of the major-party presidential vote share across the 5 days prior to the election was 1.20 percentage points, while opinion polls conducted during that same time had an average error of 1.62 percentage points.
•       Unlike polls` random selection, the participants of prediction markets are self-selected.
•       Unlike polls or expert panels in which participants are asked for their independent opinions, each trader in the market sees the net effect of the beliefs of all other traders, and the time series of changes in those beliefs and can alter his own perceptions accordingly.
•       Unlike polls that ask each respondent how he or she would vote if the election were held today, the market asks traders to forecast how everyone will vote in the actual upcoming election. (We can suggest that the sentiments play role on polls; whereas the factuality in PMs)
•       ``Convention bounce`` effects don`t appear in prediction markets.
•       It gives continuous updates.
•       Because they react dynamically to information, they can also be used as evaluation tools to assess the impact of decisions such as policy positions, candidate viability, campaign strategies, etc.

Critique

The study lays out the facts effectively. The authors compile the prediction markets` advantages and superiorities over the polls. However, the study does't mention about the weaknesses of the technique which can be the sign of confirmation bias. Moreover, polls attract many attentions currently. If the prediction markets have lots of merits over polls, I would expect the study to identify the reason why still polls make people interested in them. Nevertheless, the study explains very well why the prediction markets outperform polls through five consecutive years.

Source:

Berg, J. E., Nelson, F. D., & Rietz, T. A. (2008). Prediction market accuracy in the long run. International Journal of Forecasting, 24(2), 285-300.

Delphi Method (Rating: 4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the  articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2015 regarding Delphi as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Delphi is an analytic methodology that utilizes a panel of subject matter experts to come to a consensus on an issue.  Successive rounds of surveys are given to the experts and are completed anonymously.  Feedback, consisting of other answers and justifications for the answer, is provided to the experts between each round. The feedback is taken into account in next round of responses.  At the end of the rounds, final answers are delivered to the panelists and the customer for whom the method was conducted.

The Delphi method attempts to achieve several objectives: develop a range of possible alternatives, explore underlying assumptions, seek out information that may generate a consensus, and educate the participants on different ideas and aspects of a specific topic.

Strengths:

• Method provides anonymity to participants
• Method uses independence
• It is very useful to come up with a consensus
• Anonymity and independence help overcome groupthink
• Method is versatile
• Controlled feedback gives participants option to hear others’ opinions and then go back and modify their answers
• Statistical analytic ensure equal representation among participants

Weaknesses:

• Method can take extensive periods of time
• Panels must be composed of experts
• It can be manipulated by the researcher due to his or her extreme control over the process
• Technique requires feedback, but some participants may not respond
• Method hinges on ability/training of facilitator/researcher

How-To:

1. Develop a questionnaire/survey/scorecard to evaluate an estimate or recommendation
2. Gather a group of experts related to the subject at hand
3. Provide them the questionnaire/score card and ask them to evaluate the options of the given scenario
4. Gather the data and provide the scores of everyone & the average for each option. Keep everything anonymous for the participants in the study.
5. Conduct at least 2 rounds of scoring and then evaluate the results.
6. Finally, evaluate the best option or at least eliminate the worst option(s).

Personal Application of Technique:

Delphi exercise involved a UN type simulation decision on which policy was more effective in determining what policy to adapt in regards to Nuclear policy. In order to do this, the group first created a “role” so that the person could act as an expert. The roles that were chosen included a person from the Institute for Science and International Security, Pakistani Prime Minister, Chinese Foreign Minister, Indian Prime Minister, Japanese Foreign Minister, Secretary of Defense, Secretary of State, President of Iran.

Some of the lessons learned include: visual representation of results, incorporating technology (i.e. Google Forms) to take advantage of time, and also having different questions for each round. It would be more beneficial to present the results on a histogram.

For Further Information:

Delphi Exercise folder

The Delphi Technique: Making Sense Of Consensus

Summary

The Delphi technique is a common method used to gather data from a variety of different professional fields, and is designed to achieve a convergence of opinion on a specific issue through group communication. This technique is well suited as a method for consensus-building by using questionnaires delivered by multiple iterations to collect data from participants. Participant selection, time frames for conducting and completing the study, the possibility of low response rates, and unintentionally guiding feedback from respondent groups are issues that need to be accounted for when using the Delphi method.

The Delphi technique was originally designed by Dalkey and Helmer at the Rand Corporation in the 1950s in order to solicit data on specific issues for the purpose of goal setting, policy investigation, or predicting the occurrence of future events. According to Delbecq, Van de Ven, and Gustafason (1975), the Delphi technique can be used to achieve the following objectives: 

1.      Determine or develop a range of possible program alternatives.

2.      Explore or expose underlying assumptions or information leading to different judgments.

3.      Seek out information which may generate a consensus on the part of the respondent group.

4.      Correlate informed judgments on a topic spanning a wide range of disciplines.

5.      Educate the respondent group as to the diverse and interrelated aspects of the topic.

Unlike other data gathering techniques, the Delphi technique utilizes multiple iterations (feedback processes) designed to develop a consensus on a specific topic. The feedback process allows and encourages all participants to reassess their initial judgments about the information provided in previous iterations. Other characteristics of the technique is the ability to provide anonymity to respondents, a controlled feedback process, and the suitability of a variety of statistical analysis techniques to interpret the data.

1.      Anonymity is one of the strongest characteristic and advantage of the technique, as it prevents stronger personalities from drowning out timid individuals, and prevents coercion and manipulation.

2.      Controlled feedback is designed to reduce the effect of common communication that occurs during group discussions that derails the original purpose of the discussion.

3.      Statistical analysis techniques ensures each opinion is represented equally after each iteration, and reduces pressure to succumb to group conformity.

The Delphi Process

The Delphi technique can be employed until a consensus is reached, however, several researchers indicate 3 iterations are sufficient to collect information and reach consensus in most scenarios. The following guidelines show 4 rounds in order to present an example of the process when more data is needed. 

Round 1: The technique begins with an open-ended questionnaire that serves as the cornerstone of gathering information about a specific topic from the participants. After the investigators (facilitators) collect each participant’s responses, a well-structured questionnaire is created. The well-structured questionnaire is used as the survey in round 2.

Round 2: Each participant receives a second questionnaire, and reviews the topics summarized by the investigators. The summarizations are based off the information the participants provided in the first round. The participants also rate or order each topic in order to establish priorities. The process of establishing priorities identifies agreements or disagreements among the participants. 

Round 3: Each participant receives a third questionnaire that includes all the topics and ratings summarized from the previous round. The participants are asked to revise their judgements, or specify why they won’t change their answers. 

Round 4: In the final round, each participant receives another questionnaire with the remaining topics, ratings, minority opinions and topics that reached a consensus. 

The most important process of the Delphi technique is selecting participants. According to various researchers, the three groups most qualified to be participants in a Delphi method are:

1: Top management decision makers who will utilize the outcomes of the Delphi study.

2: Professional staff members together with their support team.

3: Respondents to the Delphi questionnaire whose judgements are being sought.      

Critique

Though the Delphi technique is used regularly in various professional settings, it seems to be extremely situational. One of the many elements that are needed to ensure the Delphi technique is employed properly is time. Depending on the number of the participants and the data being evaluated, it may take several days or even weeks to complete the technique. Other methodologies can be used in place of the Delphi technique that would likely result in the same quality of information, but with less time. The technique also has other potential shortcomings, like low response rates, which have to be accounted for. Striking a balance between the requested data, number of participants and timeframe, appears to be key element when deciding to utilize the methodology.

      Source:

Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique: Making Sense of Consensus. Practical Assessment Research & Evaluation, 12(10). Available online: http://pareonline.net/getvn.asp?v=12&n=10

The Delphi Method for Graduate Research

Gregory J. Skulmoski, Francis T. Hartman, and Jennifer Krahn (2007)

Summary:

The authors of this article for the Journal of Information Technology Education are from Zayed University in Dubai and the University of Calgary, Canada.  Their approach is to provide graduate students with the knowledge necessary to employ the Delphi method in their own academic research, be it for a thesis or dissertation.  The primary scope of the piece is based around the Information Systems (IS) field as well as the field of Information Technology (IT), but they believe that as a research technique the Delphi method can be used in a wide variety of fields, not just their own.  This is because they view the Delphi method as a “flexible research technique” and define it as follows:

The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback.

The authors also put forth that that this method is most suitable when “the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts.” 

In order to improve the reader’s understanding of the methodology there is a brief historical overview of what the authors refer to as “Classical Delphi”.  This is the methodology developed by Norman Dalkey for the RAND Corporation in the 1950’s.  In the Delphi method a panel of experts is given a survey with questions to answer.  After they have returned the survey, a second survey is sent out based on the results of the first, this proceeds over a pre-determined number of rounds. Following the completion of the last survey analysis of the final results is conducted.  Classical Delphi is defined by four key features:  

1. Anonymity of Delphi participants: participants are freed from social and professional pressures by the use of anonymous surveys.  Those who are regarded as greats in the field will be judged solely on their answers and not by their reputations.  This also frees participants to think outside traditional lines without fear of reprisal.
2. Iteration: participants are given the opportunity to develop their own understanding and beliefs as the study progresses.
3. Controlled feedback: as rounds of the Delphi progress, participants will be informed of the anonymous perspectives of other participants.  This is one the core advantages of the Delphi method, that the group as a whole will generate a broad array of ideas to begin with and zero in on the best ones as times moves on 
4. Statistical aggregation of group response: this allows for statistical analysis of results.

This “Classical Delphi” model has been adapted by many in the decades following its creation, this has made it more widely applicable and more adaptive to diverse requirements.  Below is a visualization of a more modern take on the model which has been used in some of the authors’ graduate students’ projects:

One of the features of the paper is the design considerations that should be taken into account while utilizing the Delphi method in graduate research.  The authors discuss pros and cons of:  

• Methodological Choices
• The Broadness of the Initial Question
• Criteria for who is to be Considered an Expert
• The Number of Participants
•  The Number of Rounds
• The Mode of Interaction with Participants
• Methodological Rigor
• The Results
•  Further Verification
•  Publication

The authors close with what they consider two important points, “First, the Delphi approach can be aggressively and creatively adapted to a particular situation. Second, when adapting the approach, there is a need to balance validity with innovation. In other words, the greater the departure from classical Delphi, the more likely it is that the researcher will want to validate the results, by triangulation, with another research approach.”

Critique:

This article provides a lot of good information for graduate students who may be unfamiliar with the Delphi method.  It is specifically focused on how they can get out and use the methodology without too much difficulty, and what they need to be thinking about as they do it.  One of the largest limitations of the piece in my opinion is that spends too little time discussing the situations in which the method is not appropriate.  The article goes to great length to explain just how adaptable the methodology is, but even in the Executive Summary it states Delphi, “ is not a method for all types of IS research questions.”  The article raises this concern but rarely returns to it.  This as a result makes it feel like the use of the Delphi method is a foregone conclusion and less that it is just one more valuable tool in your analytic toolbox.