Tuesday, March 27, 2012

Forecasting Decisions in Conflict Situations: A Comparison of Game Theory, Role-playing, and Unaided Judgement


In the International Journal of Forecasting, Kesten C. Green builds upon the role playing research begun by J. S. Armstrong. Written in 2002, this new study compares the forecasting accuracy of game theory, role playing, and unaided judgment when applied to conflict situations. Based on the results, Green concludes that game theory is more accurate than unaided judgment, and that role playing is more accurate than game theory when seeking to predict the outcome of conflict situations.


Green defines accuracy as “the proportion of forecasts that match the actual decision.” In order to test the accuracy of unaided judgment, game theory, and role playing Kesten employed a total of six different conflict scenarios. Four of these (Artists’ Protest, 55% Pay Plan, Panalba Drug Policy, and Distribution Channel) were used by J. S. Armstrong during his research. The final two (Zenith Investment and Nurses Dispute) were developed by Green for the purpose of this study.

Unaided judgment participants were generally given a one to two hour time frame to complete their predictions. They were asked to review the full set of information for a single conflict scenario, and then choose the outcome they thought most likely from an exhaustive list. Game theorists were sent email instructions, along with all scenarios, and asked to complete all six. Finally, role players were given a single role in one of the two new conflict scenarios, and were then briefed on the scenario. 30 minutes to an hour were allowed for the role players to act out their scenario, at which time they recorded the decision the group arrived at, or predicted how they believed it would end in the event they were unable to come to a conclusion on time. The numerical results of the study are as follows:







Artists’ Protest


5 (39)

6 (18)

29 (14)

Distribution Channel


5 (42)

31 (13)

75 (12)

55% Pay Plan


27 (15)

29 (17)

60 (10)

Zenith Investment


29 (21)

22 (18)

59 (17)

Panalba Drug Policy


34 (68)

84 (19)

76 (83)

Nurses Dispute


68 (22)

50 (14)

82 (22)

Totals (unweightedc)


28 (207)

37 (99)

64 (158)

Table 1. Accuracy of unaided judgement, game theorist, and role-play predictions. Percent correct predictions (number of predictions)

Given the totals, unaided judgment was accurate 28 percent of the time, and almost even with chance (27 percent.) Game theorists had an improved, if unimpressive, 37 percent accuracy. Finally, the role players averaged 64 percent accuracy and were better than chance and unaided judgment in all situations. They only scored lower than game theorists on the Panalba Drug Policy scenario. Game theorists, on the other hand, were less accurate than people using unaided judgment in 2 of the 6 scenarios, and were even with chance in 4 of the 6. Green suggests that further research will be necessary to determine if game theory expertise gives any advantage over the unaided judgment of experts on conflicts who are unfamiliar with game theory.


Given the results of his study, Green concludes by agreeing with Armstrong that “role-playing will provide more accurate forecasts than other methods for forecasting decisions in conflicts because it provides more realistic representations.” In his assessment the cost of conducting these three forecasting methods is similar, which lends further strength to the role playing argument.


Green, K. C. (2002) Forecasting decisions in conflict situations: a comparison of game theory, role-playing, and unaided judgement. International Journal of Forecasting. Volume 18, Issue 3, July–September 2002. Pages 321-344. http://www.sciencedirect.com/science/article/pii/S0169207002000250


  1. Did the study hypothesize why the RP group scored lower on the Panalba drug policy question? It would be interesting to note the differences in the questions where RP was successful.

  2. I realize that Shawn will be reading this but I highly recommend sharing it with him during our next round-robin discussions.

    Do you think this would be applicable to the intelligence field at all?

  3. @Leslie: I couldn't find anything further about that. It did seem rather odd that all game theory numbers were very low except for that one.

    @Dean: I have difficulty visualizing role playing as a full-blown methodology, aka something on ACHs level. It does seem that it would be helpful for brainstorming sessions and to remove certain biases, as a more informal, useful tool.