Forecasting decisions in conflict situations: a comparison of game theory, role-playing, and unaided judgement

Summary:

In 2002, Kesten Green conducted research that compared the forecasting accuracy of game theory, role-playing, and unaided judgment.  The aim of the research was to determine which of the three forecasting methodologies had the highest forecast accuracy in determining the outcome of conflict situations.  Green used six real conflicts that occurred in situations where there were a small number of decision makers that all had a large stake in the outcome of the conflict.

Green used three conflicts in which Armstrong used in earlier research done on role-playing in 1987.  These three conflicts included Artists Protest, Distribution Channel, and 55% Percent Pay Plan.  Armstrong also developed the fourth conflict in 1977, called the Panalba Drug Policy.  Green created the final two conflicts, Nurses Dispute and Zenith Investment.

Green gave participants in the unaided judgment groups a briefing on the conflicts, and then allowed them to answer a questionnaire on what they thought the outcome of the conflict would be.  For the role-playing experiment, Green gave groups of university students roles to play within each scenario, then the students had between thirty minutes and one hour to come to a resolution.  Finally, Green emailed 558 game theorists, of which 21 participated, so to judge their forecasting accuracy using game theory.

In the end, role-playing had the highest accuracy across all scenarios except the Panalba Drug Policy, in which game theorists scored the highest accuracy.  Based off these findings, Green concluded that role-playing does provide better forecasting accuracy than game theory and unaided judgments when small groups are in conflict where all the members have a high stake in the outcome.

Critique

One benefit discovered from this research is the increased accuracy by non-subject experts when using role-playing as opposed to unaided judgments.  The majority of the participants in the role-playing exercises were university students who lacked experience in the field of knowledge that the scenario was portraying.  Despite this, the groups were able to accurately forecast the outcome 64% of the time.  We, as analysts, are not always subject matter experts, be it in US Pakistani foreign relations or the global economy.  There is, according to this research, use in this method to overcoming these barriers and vastly improving our forecasting accuracy over other models.

Role-playing is a quick method of analysis according to this research.  Green only allowed the role-playing participants between 30 minutes and an hour to meet, work together, and come to a final agreement on the conflict.  This was not a daylong process that would consume large amounts of the analysts and executives time.  Despite this short time window, the groups correctly determined the correct outcome by 59% or higher in all but one of the conflicts (Artist’s Protest = 29%). 

This research focused on only targeting small groups were all the members of the group had a large stake in the outcome of the conflict.  While this is relevant in many situations in today’s world, not all cases are like this.  Will forecasting accuracy remain high when there is a large group of players, such as ten or more countries, were certain countries have much more at stake than others?  There are some gaps in this research and in its design that prevent it from clearly proving its supremacy over game theory.

Source-

Green, K, 2002.  The decisions in conflict situations: a comparison of game theory, role-playing, and unaided judgement.  International Journal of Forecasting.  Retrieved from http://www.forecastingprinciples.com/paperpdf/Greenforecastinginconflict.pdf