Summary of Findings: Role Playing (4 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2016 regarding Role Playing as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use structured data.

Description:

Role-playing is a method that prompts the participants to assume a character within a given situation. The idea is that the participant will research the motivations and desires of one of the characters within the situation. This knowledge allows the participant to gain insight into how that character will act. This translates into a potential course of action for that character which could correlate with a real world outcome.

Strengths:

• Has a lot of research that supports effectiveness
• Allows analysts to better understand the motives of their target
• Is flexible in the type of scenarios that can be simulated and analyzed
• Is possible to perform in a timely manner
• Relatively easy to set up with little cost

Weaknesses:

• Dependent on adaptation of an individual to a role.
• Exercise needs the right balance of giving enough information to describe internal motivations of characters being role-played, but not so detailed that the outcome is basically predetermined.
• Must be structured correctly or results may go awry (must have proper nuances for characters or sufficient background to establish the role properly)
• Requires a positive attitude of participants for exercise to be successful

How-To:

• Identify a topic of interest that requires study for which a basic understanding exists
• Identify the roles that require analysis
• Identify actors to participate in the role playing experiment
• Provide the actors with their background information about their characters
• Allow actors time to absorb their characters background info
• Provide the actors with the scenario for which they are to act out
• Act out the scenario as the actor believes their role player would act in real life
• Complete the scenario and discuss the findings
• Run the scenario again with different actions taken by participants
• Complete and discuss findings
• Repeat as necessary until results indicate which outcomes are most likely

Application of Technique:

The analysts were given literature prior to class about the Afghanistan opium trade, CIA and DEA’s operations in Afghanistan to begin conceptualizing what they needed to know about the issue to assess the relationship of increased violence between Kandahar and Kabul.

In class participants were given their roles: Taliban Warlord, a High Value Target (HVT) in Pakistan, an Afghan poppy harvester, a CIA Case Officer (CO), and a DEA FAST member. Each of the individuals learned their roles in five-minutes. Then for 15-minutes the participants interacted in their roles and gathered information for each other. Finally, at the conclusion of the role-playing timeframe the participants created a presentation and debrief a key findings to the proctors of the session.

See below for descriptions:

Afghanistan Drug and IED Events and US Countermeasures

Objective

To develop insight into how the relationships between each of the role-playing components respond to each other in hopes of teasing out further understanding of the Afghan drug trade and its relationship to increased violence between Kandahar and Kabul.

Introduction

In the news recently there has been increased activity between US forces and the Taliban. This role-playing exercise looks to provide further understanding on how the different actors in the situation would respond to each other time with a realistic backdrop.

Exercise Parameters

        Time: 20 minutes

                                5-minutes to interpret identity and begin building persona

                                15-minutes to converse in their roles

                                5-minute presentation/debrief of key findings and possible link analysis

                                            -Afghan perspective

                                            -US perspective

                               

This will work best if each participant has about a 3-4-minute conversation with each other in their roles about the situation they are all in. Remember the US forces (the CIA CO and the DEA – FAST officer) will speak to each other differently than the conversations that are had between the (Taliban Warlord, the Poppy Harvester, and the HVT).

Roles:

Taliban Warlord

The “Taliban Warlord” has been building out a network of operators across a variety of settlements connecting Kandahar and Kabul. This same route has been frequently utilized for ground movements by conventional US forces. Increases in IED activity have been linked to profits made by poppy cultivation in Kandahar. The warlord is the chief in overseeing the poppy cultivation.

______________________________________

Poppy Field Harvester

The “Poppy Field Harvester” does not know of his fields’ attachment to the overall IED effort against the US conventional forces. He comes from limited means although worked as an interpreter for the UK military for a time in the mid-2000s. His ethnic identity is in direct conflict with the “Taliban Warlord’s” however, has never met the “Taliban Warlord” before.

______________________________________

Known High Value Target (HVT) in Pakistan

Located in Quetta, Pakistan the “HVT” has communicated on numerous instances (both in-person and over radio) with the Afghan located “Taliban Warlord.” The “HVT” has been known to use clan ties to cross the border easily. Is well respected in the region for his charitable work and has extensive access to a network of IED builders in Pakistan.

_____________________________________

CIA Case Officer (CO)

The “CIA CO” has been stationed at the Kabul safe house and Bagram Airbase for just under a two-years. She runs four sources that have informed her that the “Poppy Field Harvester” works for the “Taliban Warlord.” Various other intelligence sources (i.e. – SIGINT, MASINT, GEOINT, etc.) have corroborated the fact that the “Poppy Field Harvester’s” crop has been seen being shipped to a network of settlements between Kandahar and Kabul.

_____________________________________

DEA – Foreign-Deployed Advisory and Support Team Member (FAST)

The “DEA – FAST Member” is on his third tour under five years. On his second tour he was in charge of processing transportation running through the southern Afghan border. On one instance he interdicted a vehicle which was concealing IED components and had traces of poppy residue in the trunk. The seized cargo and communications equipment (i.e. – radios) tied back to the “HVT” in Pakistan. He continues his work as a Counter-narcotics officer and tactical police operations.

For Further Information:

Role-Playing Wikipedia:

https://en.wikipedia.org/wiki/Role-playing

World of Warcraft:

https://worldofwarcraft.com/en-us/

POD Network:

http://podnetwork.org/content/uploads/V19-N5-Nickerson.pdf

Dungeons & Dragons:

http://dnd.wizards.com/dungeons-and-dragons/start-playing

Role-playing/Simulation:

http://iteslj.org/Techniques/Tompkins-RolePlaying.html

Role-playing is an effective instructional strategy for genetic counseling training: an investigation and comparative study

Synopsis

This research product comes by way of the Third Military Medical University in Chongqing, China. The purpose of the study was to evaluate a program that was designed to test the effectiveness of a role-playing based teaching method for undergrad medical students going into medical genetics fields. The study used 2326 med students who already had a background in medicine from previous years of instruction. As stated, this study was an attempt to evaluate how well role-playing worked in these students’ practice of, in particular, genetic counseling. To do this, the students were tested on 42 medical genetics components such as mitosis, dominance, traits, and diseases before the start of the test to establish a baseline.

The program consisted of studying the theoretical knowledge of medical genetics, then preparing the case scenarios, and playing out the roles in the classrooms. The testing portion made use of a tiered system of new lectures, clinical doctors, and expert teachers to integrate the role-playing methodology. From there, the students devoted as much as 80% of class time to researching the conditions of their illness they were tasked to understand. These lead to strong results. In fact, during the actual exercise, the students adapted their rolls to portray characters with lower education levels to prompt the doctors to explain their conditions in simplified terms, which required a greater understanding of the condition itself and work on professional communication skills.

At the conclusion of the experiment, the students were re-tested in the 42 key fields and the results were compared to the pretests. The results showed increases to the students’ knowledge of genetic medicine. The researcher attributed the increase in subject mastery directly to the exercises. The researchers acknowledge that the study had some limits. First, the student body at the facility was far too great for everyone to have a proper chance as a patient and as a doctor. Also, feedback was limited slightly due to the aforementioned size. In addition, there were several variables such as religion and tradition, insurance, and other interpretations the researchers did not take into account. Ultimately, the purpose of the program was to look at altering how the topic of medicine is taught to students.

Critique

First, it is very interesting to read such a study from an Eastern medical body. The study followed a logical progression of start, test, and end (logical at least for the layperson). The findings were expected. The reason I went with this study was due to it being far removed from the extensive Western body of knowledge on the topic of role-playing’s effects on professional abilities. Even so, they found that role-playing did aid in participants ability to become more effective in their field. While not explicitly translating to forecasting accuracy, it does carry other connotations. Primarily, role-playing helped the students to understand the counseling process better and make them better counselors due to not only seeing things from the doctor’s view, but from the patient’s view as well. This provided greater insight into the process that, through testing, showed a greater mastery of the course. The metrics surrounding the results however are not quite as strong as they could be. If there were a way to make the study more robust, it would possible be via a longitudinal study of how the students go on to assist the medical field. Nevertheless, for the time being, the narrative is encouraging.

It may be a leap, but it’s a fairly easy leap to say that role-playing could indeed make any professional a better professional. What constitutes a better professional is certainly left to the field in question. However, for the intelligence analyst, being able to have a deeper and richer understanding of a situation or the actors therein would offer them a chance to shake cognitive biases and offer a better forecast of likely events.

Source 

Xu, X., Wang, Y., Wang, Y., Song, M., Xiao, W., & Bai, Y. (2016). Role-playing is an effective instructional strategy for genetic counseling training: an investigation and comparative study. BMC Medical Education, 16, 235. https://doi.org/10.1186/s12909-016-0756-4

Role Playing: A Method to Forecast Decisions

Summary

This article focuses on the ways that role playing improves forecasting accuracy in decision-making. The author, J. Scott Armstrong, provides evidence from previous studies, along with findings from his own experiments, to demonstrate how role playing is more beneficial than expert opinions in forecasting. Armstrong explains that role playing is carried out when "a forecaster ask subjects to put themselves in specified roles and then to either imagine how they would act, act out their responses alone, or interact with others in the situation". Armstrong includes specific recommendations on how to execute successful role playing. These recommendations highlight the key factors within role playing including casting subjects, the role playing instructions, a description of the situation, administration, coding, and the number of sessions. 

First, Armstrong suggests that casting subjects similar to the decision-makers being portrayed has little weight on the role playing operation. This is due to previous experiments, such as Zimbardo (1972), that received realistic results from employing students as subjects. Armstrong says it is appropriate to use somewhat similar subjects if it is difficult to find role players who are very similar to the decision-makers. Next, Armstrong explains the importance of describing the roles of the subjects before they read the description of the situation being played out. The subjects should then be asked to improvise while acting either as themselves, or as how they believe the actual decision-maker would act in the given situation. Armstrong states that the situation should be described as accurately and briefly as possible so the subject is able to comprehend the situation. In order to make analysis easier, Armstrong also suggests that it is useful to provide subjects with possible decisions if this makes sense to the situation. Because role playing should be realistic, Armstrong highlights that it is important for the subjects to act out their responses and to interact in ways that correspond with the role playing situation. Finally, to properly analyze the role playing, Armstrong recommends having subjects write down their views of the decision and to have more than one person code their responses. Forecasts should be based on the number of decisions made through role playing. It is advantageous for the forecaster to perform at least five role playing sessions with one description, and five with another description.

Armstrong explains the ideal situations to use role playing, as well. Role playing is most useful in situations where there are two interacting parties. This contrasts from situations where parties do not interact, and in situations where there are too many parties involved. Secondly, role playing works best when the interacting parties are in conflict with one another. Lastly, role playing is beneficial for forecasting situations that involve situations with considerable changes. Armstrong states that role playing works well for these situations because of its ability to produce valid situations, and in turn, accurate forecasting. This is because role playing can make decision-makers aware of outcomes that were previously unknown to them. Additionally, role playing is able to provide decision-makers with a greater understanding of the situation since it acknowledges the perspectives of each party involved.

Critique
  
Armstrong gives extensive information on how and when to perform role playing, making it easy to understand the role playing process. One thing, however, that was hard to follow is why Armstrong proposes allowing subjects to either act as themselves or as the portrayed decision-maker. It seems unproductive to have someone act as their self if the purpose is to have them take on the role of a specific decision-maker. Armstrong also does not fully explain how to code and analyze the results of the role playing exercise, which would have been helpful to include.

Source
Armstrong, J.S. (2001). Role playing: A method to forecast decisions. Marketing Papers, 152. Retrieved from http://repository.upenn.edu/cgi/viewcontent.cgi?article=1175&context=marketing_papers 

Game Theory, Game Theorists, University Students, Role-Playing, and Forecasting Ability

In this paper, Wright analyzes some of Green’s work and does a small meta-analysis of other authors that have analyzed both Green’s work and game theory in general. While Green demonstrated in his study that game theorists’ predictions were more accurate than the unaided judgement of university students, other authors have mixed feelings about the validity of his study. Bolton, for example, challenges Green’s method of testing game theory and argues that role-playing is dependent on game theory – in that knowledge of game theory is necessary in order to make an initial design of the role-plays.

Wright argues that, based on results from several authors’ studies, game theory is limited in its ability to analyze and forecast the outcomes of one-off, real world conflict situations such as the ones Green studied. He notes that the forecasting of generalized market behavior and the outcomes of context-free, laboratory gaming seems the best game-theory-based forecasting are the best currently possible. Since Green implies in his study that the expert judgement of game theorists should produce more accurate forecasts than non-expert judgment, this assumption led others to examine if there is strong underpinning evidence to support this.

Bolger and Wright examined 20 extant studies about expertise within social and decision science literature, and documented that 6 of the 20 extant studies had showed ‘good’ performance by experts in the domains of weather forecasting, the number of tricks to be made in bridge playing, odds forecasting in horse racing, interest rate prediction, and research and development outcome prediction. Of the other 14 studies, 9 showed poor expert performance while the remaining 5 showed equivocal performance. Bolger and Wright concluded from the patterns of these studies that expert performance will be largely a function of the interaction between the dimensions of ecological validity and learnability. Ecological validity is the degree to which experts are required to make judgements inside or outside the domain of their professional experience, and learnability is the degree to which good judgement can be learned in the task domain.

So how did role-playing by non-experts in Green’s study make them produce better forecasts? Research on the effectiveness of Delphi provides a clue. Rowe and Wright have shown that the provision of feedback of the rationales/arguments for fellow panelists’ forecast is the essential cause of improvements in forecasting accuracy over Delphi rounds. As for the experts' accuracy, Wright concludes that Green’s game theorists had two sorts of experience that might serve as a basis for predicting the outcomes of the conflict situations: the game theorists had their own, individual experiences of real-life conflicts and their resolutions. While the university students will have of course had similar experiences, they are likely to have had fewer since as a group they are younger. This leads Wright to hypothesize that it is “only when individuals are enmeshed in role-play simulations will the relevance of this experience become obvious – since Green’s conflicts will, initially, have been seen as outside the domain of this experience at a superficial, face-content, level” (p. 387).

Critique

Wright mentions several points of future research that are worth reiterating in this section. For example, does the partial role-playing of conflicts to near-resolution enable individual participants to predict the actual outcomes of both: 1) the (continued) role-play, and 2) the real situation that the role-play was designed to model? Another excellent point of future research is examining these two questions: 1) Are older, more experienced people better able to make forecasts of actual outcomes after such partial role-playing? and 2) does simulation of role-playing enhance that individual’s forecasting ability?

Source

Wright, G. (2002). Game theory, game theorists, university students, role-playing and forecasting ability. International Journal of Forecasting, 18(3), 383-387.

Confronting prejudiced comments: Effectiveness of a role-playing exercise

In this article, authors Lawson, McDonough, and Bodle discuss their social experiment aimed at identifying whether role-playing can be effective at reducing prejudiced comments. The experiment was established similarly to that of Plous (2000) in that the point was to not only inform students about prejudice but also ways in which they can combat prejudice outside the classroom. The object of Plous’ experiment was for the speaker of the exercise to discuss a topic and inject a prejudiced statement at some point. The responder’s role is to engage the speaker in a manner that does not make him/her hostile or defensive. Coaches then gave feedback to the quality of the response. The goal of Plous’ experiment was to confront prejudice to lead to its reduction rather than reinforcement.

In this article’s experiment, the authors wanted to see if the subjects who participated in a role-playing exercise were more or less likely to effectively confront instances of prejudice than those subjects in the control groups. The experiment included 61 students from three different undergraduate courses (social psychology, police and society, and intro to psychology). The social psychology students (23) were the ones exposed to the role-playing exercise while the police and society (12) and into to psychology (26) were in the control group and did not participate in the role-playing exercise. The social psychology students kept a log for a week of all the instances of prejudice they experienced in their daily lives. Prior to the role-playing exercise, all participants took a pre-test consisting of 5 scenarios containing brief background information and a prejudice statement. Each participant was asked to write down how they would respond. Responses were coded as either being effective or ineffective. For the role-playing exercise, 5 scenarios were chosen and given to each group (4-5 students) so each participant could select a different scenario previously unseen by the group. Someone would read the scenario and include the prejudiced statement, a responder would retort, a coach would provide feedback, and the remaining students were there to provide dialog for the scenario. After discussion on which types of responses were most effective, the students in the experiment were asked to go out and use these techniques they learned in real life situations. They were then to record these incidents in a second log. Afterward, all students took a post-test that was identical to the pre-test.

The results of the experiment showed that those who participated in the role-playing exercise demonstrated significantly higher levels of effective responses in the post-test when compared to the pre-test. Those students in the two control groups showed no significant changes between the pre and post-tests. However, the intro to psychology students showed a significant decrease in the number of effective responses from the pre to post-test.

Critique:

This article’s findings suggest that role-playing can be an effective tool at training the mind to respond in a certain way. I am not surprised those who participated in an experiment where they were told what the right answers look like did better on the post-test than those who didn’t have it spelled out for them. The authors themselves even admit that even though their experiment suggests role-playing works, they have no proof of its effectiveness in the real world. As was pointed out by the authors in the article, the human response to prejudice is similar to that of bystander intervention in an emergency. One has to first identify an act as prejudice, decide it constitutes something harmful, take responsibility for responding, and select the appropriate response. The audience is another variable not discussed in the set-up of this scenario. One will undoubtedly respond differently to family members, friends, and strangers depending on the scenario at hand. I believe role-playing can be effective at preparing the participant for a potential future scenario. However, the effectiveness of the role-playing depends largely on the details of the scenario. Much the same way war-gaming depends on the details in order to be effective. Simply running participants through a couple exercises is by no means enough training to be prepared for all possible future scenarios. But like many of the other methods we’ve discussed so far, it will at least make the participants more comfortable and knowledgeable by giving them a broader base of experiences on which to draw. 

Resources:

Lawson, T. J., McDonough, T. A., & Bodle, J. H. (2010). Confronting prejudiced comments: Effectiveness of a role-playing exercise. Teaching of Psychology, 37(4), 257-261.

Plous, S. (2000). Responding to overt displays of prejudice: A role-playing exercise. Teaching of Psychology, 27, 198–200.