Evaluating SWOT's Value In Creating Actionable, Strategic Intelligence

Summary:  
Michael Finnegan’s 2010 thesis surveyed 101 executives (78 completions) from multiple countries and industries in an effort to evaluate SWOT’s viability for creating actionable intelligence in the formulation of strategic plans. The methodology defines an executive as a working professional in upper management or anyone involved in strategic planning. In this study, strategic planning refers to areas within business operations that look at all internal and external events and resources to determine what future decisions will likely yield the most success to the organization. Surveying from this population enabled analysis of primary sources to reach four particular findings.

1. Businesses rarely use SWOT with the structure and formality reflected in the academic literature supporting the use of the technique. Processes such as researching SWOT variables in depth before conducting an analysis and ranking the outcomes of the technique are missing from the procedures followed by surveyed by the organizations of surveyed executives. Executive cited initial exposure to SWOT in college.
2. SWOT’s greatest contribution to designing a strategic plan is enabling a greater extent of diversity of perspectives within and outside of the organization to come together, which Finnegan identifies as an indirect contribution to strategic planning. SWOT informally overcomes “silos” within organizations, serving as a catalyst to externalize ideas previously entrenched within a division or unit.
3. Surveyed executives believe a SWOT should be completed quarterly to produce timely and relevant analysis, yet they acknowledged that SWOT is not performed as often as it should be to reach their goals, the reasons why it is not performed more often are not clear from the results but likely barriers include scarcity of time, money, comfort, and convenience. Executives typically conduct SWOT only once a year.
4. No evidence of a direct contribution to strategic planning is evident from either the quantitative or the qualitative interpretations of the survey results; therefore, executives should pair SWOT with other techniques as a starting point for additional analysis. Findings indicate that the output of SWOT should not be the sole basis of an effective strategic plan. Other popular techniques cited by executives include competitor profiling, scenario planning, and financial analysis. However, no literature presently confirms that pairing a specific technique with SWOT creates actionable analysis or adds direct value to the creation of strategic plans.

Finnegan does not suggest that executives should abandon SWOT as an analytic technique. Implications of the four findings relate to finding effective pairings with other techniques to validate strategies and providing evidence affirming that the way SWOT is taught needs to be reassessed for effectiveness. 

A pervasive mindset found in the results of the survey is an application of the technique consisting of filling in the quadrants of SWOT and concluding that the filled in quadrants are the final output of the analysis. Finnegan found that such procedures are neither reactive nor proactive, they simply breakdown a present situation visually. One solution identified in the thesis is incorporating case studies and applied projects when SWOT is taught at institutions of higher learning. 

Further research is required to identify the best pairings of techniques to add direct value to the creation of strategic plans. Other gaps in SWOT literature include best practices for the best time and place to perform SWOT, SWOT’s relative value as a communication tool and brainstorming platform, and further surveys corroborating the findings and expanding on the questions used in the study to measure the evolution of SWOT’s application.

Critique:
Another avenue of approach for evaluating SWOT’s value in strategic planning is examining the interrelationships between the vast amount of variables SWOT factors into an easier to digest profile, and to what extent the future decisions advocated in the strategic plan actually yielded measurable success for the organization via an experimental stock market game. Various groups with the same internal resources and ability to make sense of external events could be directed to play a stock market game under various conditions (i.e. group 1 crafts an investment evaluation strategic plan to buy or short stocks with or without options trading using solely intuition, group 2 uses swot, group 3 uses swot paired with another technique, etc).

Source-
Evaluating SWOT's Value In Creating Actionable Strategic Intelligence

The use and misuse of SWOT analysis and implications for HRD professionals

Summary:
In their research, Chermack and Kasshanna broke down the literature of 51 articles discussing SWOT in order to help human resource development (HRD) Professionals understand this strategy tool.  At the time, only 7 research studies on SWOT were found, the most recent being from Koch in 2000.

Early designs of the SWOT methodology started in the 1950's was developed by Learned et al.  While many variant frameworks have been proposed, most follow a six step process, 1) defining the objective of the SWOT 2) providing an explanation of a SWOT to participants 3) individuals identify companies strengths, weaknesses, opportunities, and threats 4) combining participants SWOTS 5) debate classification of each item 6) develop courses of action.

SWOT is seen as a piece of a larger strategy process.  Strategy should not be guided single highhandedly by a SWOT, but should include the Synthesis of SWOT, STEEP, and other strategy development tools.

In evaluating past research, Chermack and Kasshanna identified many of the misuses of SWOT.  Kock (2000) identified that many often use SWOT as a way of validating the companies current strategy.  SWOT was not designed to validate current strategy, but to identify areas to develop new opportunities.  In addition, Hill and Westbrook (1997) found that companies often rush through the opportunities and threats section as these are the hardest to identify as opposed to the internal strengths and weaknesses of the company.  Companies often perform their SWOT's with incomplete information, but act on them as if they are complete.

Chermack and Kasshanna conclude that SWOT, as an analytic technique, is lacking significant research and literature.  They define SWOT as being stuck in a loop of "research, development, and practice".  There is no empirical evidence to prove the effectiveness of the SWOT technique.  Chermack and Kasshanna suggest that future research be conducted, primarily focusing on answering the questions, "what are the outcomes of SWOT analysis (such as improved decision making), what is the relationship between SWOT analysis and firm financial performance, the effects of SWOT analysis on participant decision making, and the most effective way to engage in SWOT analysis"?

Critique:
As stated by the authors, there is a large lack of empirical research pertaining to SWOT analysis.  While this article was published in 2007, I found it near impossible to find research based articles pertaining to SWOT analysis today.  Due to that, Chermack and Kasshanna offer nothing more than a summary of what has already been written about SWOT.  They offer very little of their own analysis.  Their call for future research, and their suggestions, does warrant mention as they still appear unanswered.  With most companies using SWOT, at some level, the lack of research into its effectiveness is baffling and is in dire need.

Source:
Chermach, T. and Kasshanna, B. 2007.  The use and misuse of SWOT analysis and implications for HRD professionals.  Human resource Development International.

Focused SWOT: diagnosing critical strengths and weaknesses

Summary

This article examines the inadequacies in SWOT analysis in examining a company’s strengths and weaknesses. Weaknesses within SWOT analysis include:

• No straightforward methodology for identifying strengths and weaknesses 
• Companies list an excessive number of strengths and weaknesses when completing SWOT   
• The causes for the strengths and weaknesses are not examined   
• Strengths and weaknesses are not listed in a hierarchy
• SWOT examines a company at the moment the analysis is completed and does not monitor changes over time

Comen and Ronen present in this article a focused SWOT which examines the core competencies and problems of a company. Comen and Ronen argue that competencies and problems are the fundamental subset of strengths and weaknesses, and this article presents a methodology to perform a focused SWOT.

A focused SWOT is performed using the following three step process:

1.       Event-factor Review (EFR) – A company analyzes 6 to 9 events that impacted the value of the company. The company determines if the event was a success or failure, and then derive the strengths and weaknesses from the event.

2.      Create a Core-Competency/Problem Tree (CCT) – Of the 8-12 strengths and weaknesses generated from the EFR, companies first must reduce the list to 2-3 core competencies and problems. This is done by:

a.      Eliminate redundant , vague, and irrelevant responses

b.      Link strengths and weaknesses together using cause-effect logic

c.       Discover 2-3 core competencies

3.      Create a focused Current Reality Tree (fCRT) – Create a tree by listing the core competencies at the base, and then list strengths caused by the competency. Then continue to list additional strengths caused by the previous list of strengths (i.e. Competency 1 à Strength 1 à Strength 2).

By implementing the focused SWOT, a company is able to create a concise actionable list. Too often, companies generate lists of strengths and weaknesses that are too large to address. By performing a SWOT that assesses the competency and problem level, a company will generate a more useful analysis.

Critique

This article adequately points out the inherent flaw in SWOT analysis: its simplicity. Not accurately diagnosing one’s strengths and weaknesses limits the usefulness of the SWOT. By discovering the core competencies and problems that result in the observed strengths a weaknesses, a company can develop an actionable product. 

Coman, A. and Ronen, B. 2009. Focused SWOT: diagnosing critical strengths and weaknesses. International Journal of Production Research. 47(20). 5677-5689.

Strategy: Create and Implement the Best Strategy for Your Business- Chapters 1 & 2

Strategy: Create and Implement the Best Strategy for Your Business- Chapters 1 & 2

By: Harvard Business Essentials

Summary:

The first two chapters of this book analyze SWOT analysis and examine the external environments of opportunities and threats while also looking internally for strengths and weakness. The SWOT model provides choices to help make practical goals and strategies for better operations within organizations.

According to Harvard Business Essentials, SWOT is categorized as:

“Strengths are capabilities that enable your company or unit to perform well – capabilities that need to be leveraged. Weaknesses are characteristics that prohibit your company or unit from performing well and need to be addressed. Opportunities are trends, forces, events, and ideas that your company or unit can capitalize on. Threats are possible events or forces outside of your control that your company or unit needs to pan for or decide how to mitigate.”

Every organization has goals; to achieve these goals, organizations may use SWOT to clarify and envision the challenges they face. Considering an organization’s external (Opportunities and Threats) and internal (Strengths and Weaknesses) factors are vital because they clarify the issues and facilitate the organization to do better. For example, the first chapter identifies technology and substitutes as both internal and external threat factors.  Using both of these factors to uncover threats and opportunities may affect the organization’s mission and plan. Internally, strengths and weaknesses can identify and change core competencies within organizations.

According to Harvard Business Essentials, different trends may affect businesses and form the basis of a new strategy; therefore, uncovering an area in which significant change is happening.  To help the process, the book outlines nine basic steps for establishing a collection method:

“Step 1: Select an individual to facilitate the analysis

Step 2: Create a SWOT team of knowledge individuals from different functional areas of the company

Step 3: Brainstorm the company or unit’s strengths.

Step 4: Record all suggestions on a flip chart

Step 5: Consolidate ideas. Post all flip chart pages on a wall.

Step 6: Clarify ideas.

Step7: Identify the top three strengths

Step 8: Summarize company strengths.

Step 9: Repeat steps 2-6 for company or unit weaknesses.”

An organization should conduct these nine steps for both the external and internal factors. After these nine steps are completed, the information should be complied and put into a formal report for management to review.

Critique:

Although the first two chapters of this book are a good explanation of SWOT analysis, the publication does not go into detail on any of the concerns SWOT conveys. Although organizations value information, one weakness is that management may not utilize the information provided. In addition, the collection method provided could be time consuming and become redundant by providing information already known by the organization. However, the book did provide valuable examples and tips which makes SWOT analysis a strategic value. For example, for SWOT to take effect, organizations need to be respectful and effective to change. People need to feel motivated and accustomed to working in groups and organizations need to provide rewards for good performance. 

Source:

Press, Harvard Business School. 2005. Strategy: Create and Implement the Best Strategy for Your Business. Boston, Mass: Harvard Business School Press.

Away With SWOT Analysis: Use Defensive/Offensive Evaluation Instead

Summary:

While putting forth an argument for defensive/offensive evaluation to gain competitive insights, Valentin points out several problems with SWOT analysis.  SWOT analysis is premised on the basic idea that a business or entity will flourish most when its internal positives align with external forces.  SWOT analysis is a favorable tool because it is has a simple structure and its premise is easy to understand.  Valentin asserts that the structure is too simple.  There are other things to take into account that fall outside of internal strengths and weaknesses, and external opportunities and threats.

There are four key issues with SWOT analysis.  First, Valentin claims that SWOT analysis encourages superficial understand knowledge of the business or entity in question in lieu of a more in-depth or insightful methodology.  It separates internal and external forces that are likely to be intrinsically intertwined.  Therefore, examining SWOT analysis can lead to minimal, and even mistaken, knowledge of the business or entity in question.

Second, SWOT analysis does not account for trade-offs.  Valentine uses Southwest Airlines as an example.  The airliner does not offer customary in-flight meals.  The absence of in-flight meals could be seen as a weakness because it lacks a service that competitor airlines offer.  However, it could also be a strength because its absence allows Southwest Airlines to keep its costs lower than its competitor.  The inability to account for trade-offs can lead to misleading conclusions.

Third, SWOT analysis confuses accomplishment and strengths.  Specifically, many analysts mistakenly mark market share as an advantage.  While market share is certainly not a weakness, market share is not a sign of future competitiveness.  Market share is an accomplishment as a result of past success in the competitive landscape.

Finally, SWOT analysis offers no mechanism to prioritizing strengths, weaknesses, opportunities, or threats.  Its oversimplified structure creates too much clutter and offers very little insight or urgency.

Critique:

Valentin’s critique of SWOT analysis is theoretically sound, but it unfairly ignores the realities analysts face in competitive intelligence.  An analyst’s much precious and scarce resource is time.  SWOT analysis is quick, simple, and does offer some insight into competitive standing.  Is it oversimplified? Only if in inexperienced hands, yes.  If one is not properly trained or informed on how to conduct SWOT analysis, its simplified structure could reap horrendous results.

Source:

Valentin, E. K. (2005). Away With SWOT Analysis: Use Defensive/Offensive Evaluation Instead. The Journal of Applied Business Research, 21(2), 91–105. Retrieved from http://www.repiev.ru/doc/Away-With-SWOT-Analisis.pdf.

Summary of Findings: Role-playing (3.5 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 5 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2014  regarding Structured Role-Playing specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description: 
Structured role-playing is a method that forecasts an individual’s or organization’s preferred response to a given problem by simulating both problems and interactions between protagonists using interactive role players in a scenario designed to estimate the intentions of protagonists who may have disparate interests. The interactions between the role players serve as a proxy for the interactions of the decision makers and the decision reached by the role player is taken as a forecast of the decision maker’s preferred response.

Strengths:
1. It may be less expensive than other methods
2. Provides additional outcomes not considered by experts on opposing objectives and strategies
3. Easy exercise to explain
4. Each exercise can be structured to fit any scenario  
5. Can use a large pool of role-playing participants
6. The role-playing participants are less likely to fall into “group think”

Weaknesses:
1. Lack of preparation for roles negatively impacts quality of the exercise
2. Repeating exact results will likely be difficult
3. Falling victim to stereotypes if roles are not constructed adequately
4. No agreed upon criteria for evaluating role-playing exercises
5. Finding students with similar backgrounds may be difficult to find

Step by Step Action:  
The first step involves participants becoming familiarized with the roles they will be playing.  The participants need time to dive into their characters and start to think about how that person acts and feels.  This goes beyond what is provided within the description.  The participant should think about how that person lives, what do they do during the weekend, who their friends are, etc..  

Now that participants are familiar with their roles, they should be briefed on the situation or the question that needs to be answered.  The participants will apply what their understanding of their role is to the scenario and develop a response to the situation.

Once the simulation has ended, the participants will record how they feel their character would act in the scenario.   The participants should record not only what they believe the outcome will be, but also their thought process.  The thought process will provide valuable evidence into how the participants viewed their characters and potential areas for further research.

Exercise:
Participants were given 1 of 2 roles.  One role was Marcus, a 22-year old African American male from a socially-ill neighborhood in Erie.  Marcus has been in and out of the criminal justice system as a juvenile and continued his misbehavior into his adulthood.  As a result, Marcus is currently on probation.  The other role was Margaret, a middle-aged suburban mother of two college graduates.  She works part-time as a bookkeeper and spends the remainder of her time either gardening or volunteering at the homeless shelter.

After familiarizing themselves with 1 of 2 roles, the administrator gave the exact same scenario to each participant.  In the scenario, a large male approaches the participant (Marcus or Margaret), asks to use a cellphone, then assaults the participant and steals the phone.  After the assault, the participant is taken to a nearby hospital, where a nurse notices injuries of the assault and offers to call the police on their behalf.  Participants are asked whether their character would call the police.  

While the sample size was extremely small, participants playing Marcus opted to not call the call police.  Participants cited Marcus’s probation and past criminal inclinations for reasons to not report the incident.  Conversely, those playing Margaret decided they would call the police.  Participant playing Margaret cited her fairly routine and risk-averse lifestyle as reasons why Margaret would report the incident.

Role Thinking: Standing in Other People's Shoes to Forecast Decisions in Conflicts

Summary:
This article published in 2011 by Kesten Green, Senior Lecturer at Ehrenberg-Bass Institute for Marketing Science at the University of South Australia and Scott Armstrong, Professor of Marketing at The Wharton School at the University of Pennsylvania highlights the significant differences between simulated interactions and a term they coined "role thinking" on the accuracy of decision forecasts in novel situations. They used an experimental design to conclude that asking groups of people to think about the roles and interactions influencing the reactions of a protagonist in a given novel conflict situation to forecast protagonist decisions is an ineffective forecasting technique. Forecasts from role thinking are unlikely to be accurate due to the difficulty of analyzing complex interactions between different protagonists with different roles in a manner that accurately represents the conflict in the absence of experiencing the complex interactions. 

Note that the authors prefer the term "simulated interaction" instead of "role playing" to refer to the method of forecasting people's decisions by simulating the situation using interacting role players because the term "role playing" is used to refer to various techniques with purposes other than forecasting. The authors find that simulated interactions provide much better forecasting accuracy than unaided judgment and role thinking, particularly in novel conflict situations.

Evidence from previous findings indicate that much better forecasting accuracy for the decisions the protagonist in a given novel conflict situation will make is attained by prompting groups of people to adopt roles in addition to simulating the interactions between protagonist groups with divergent interests. The decision the protagonists in each group elect to make in the simulated interaction is taken as a forecast of the actual protagonist's decision. The authors highlight that simulating novel conflict situations faced by divergent protagonists using interacting role playing members solves the absence of experience problem according to prior research from the same authors using the same conflicts utilized in this experiment. 

The authors tested role thinking in an experimental design consisting of forecasts for the decisions of the protagonist of a given conflict from an expert sample and a novice sample. The accuracy of the role thinking forecasts were compared to chance in addition to the accuracy of unaided judgment and simulated interactions from previous studies utilizing the same conflict scenarios. The authors obtained 101 role thinking forecasts for nine conflicts from 27 Naval postgraduate students (the expert sample) and 107 role thinking forecasts from 103 second year organizational behavior students (the novice sample). The results are illustrated below: 

http://tinyurl.com/oeebksr

The average forecasting accuracy from the novice sample and the expert sample were only marginally better than chance, which was 28% versus the 33% accuracy from the novice forecasts and the 31% accuracy from the expert forecasts. Previous research from Green and Armstrong using the same conflict situations found forecasting accuracy of 60% when using the simulated interaction method instead of the role thinking method.

In the role thinking experiments, participants were provided with descriptions of some or all of the situations and of all the associated roles. Participants were prompted to predict what actions each party in the situation would prefer and assess how likely it is that each party's preferred decision will actually occur. Each prompt had a list of between three and six decisions that the researchers believed could plausibly have been made in each situation. 

In previous simulated interaction experiments with forecasting accuracy of 60%, participants were divided into groups and assigned information only on their own role. Participants were prompted to read their role description, put on a name badge for the role, and adopt the role for the duration of the simulation. Participants were free to meet with others as often as they would like to reach a decision. Each group's decision was taken as a forecast of the actual protagonist's decision. In addition to the 60% accuracy finding, the forecasts from simulated interactions were more accurate than the role thinking forecasts for all nine conflict situations. The authors also point out that neither statistical nor casual models have been found to be feasible for predicting decisions people make in novel conflict situations therefore decision makers rely on judgmental methods.

Critique: 
The forecasts from the role thinking experiment were derived from individuals while the forecasts from the simulated interaction experiments were derived from group forecasts. The authors acknowledge that a key assumption driving their analysis of the ineffectiveness of role thinking versus simulated interactions is that forecasting accuracy from group role thinking forecasts would differ little from the individual role thinking forecasts in this experiment due to both unaided judgment and role thinking forecasts differing little from chance. One way to test this assumption is implementing an experiment where different groups of participants arrive at a group forecast using role thinking then comparing those results to the simulated interaction experiments. This would also be a more consistent representation of what occurs when a group of people are tasked to engage in role thinking together on a team. 

In Table 2, the expert sample in the unaided judgment experiments and the expert sample in the role thinking experiment are qualitatively different in that the unaided judgment expert sample participants were from academia and professional conflict management and forecasting organizations while the expert sample participants in the role thinking experiment were Naval postgraduate students. However, the authors point out that there is little evidence that top experts can perform judgmental tasks better than generalists. In addition, the naval postgraduates had experience in conflicts over pay negotiations and commercial takeovers, the authors suggest that knowledge of conflicts from one domain is likely to transfer over to other domains involving predicting human behavior in conflict situations. 

One thing I did not see addressed is best practice for developing effective scenarios, roles, and choices for simulated interactions when the conflict being assessed is a current event. I estimate that the procedures for developing effective simulated interactions is as much an art as it is a science, especially when the simulated scenario in question is a model of a current novel situation where incomplete or intentionally deceptive information is an issue. Using simulated interactions with current conflicts and events as opposed to historical situations where the outcome is known by the developers of the simulated interaction in advance adds a layer of complexity to developing the simulation interaction requiring further study. 

Source- 
Green, K. and Armstrong, J.S. (2011). Role Thinking: Standing in Other People's Shoes to Forecast Decisions in Conflicts. International Journal of Forecasting. Vol. 27(1). p. 69-80.

Forecasting decisions in conflict situations: a comparison of game theory, role-playing, and unaided judgement

Summary:

In 2002, Kesten Green conducted research that compared the forecasting accuracy of game theory, role-playing, and unaided judgment.  The aim of the research was to determine which of the three forecasting methodologies had the highest forecast accuracy in determining the outcome of conflict situations.  Green used six real conflicts that occurred in situations where there were a small number of decision makers that all had a large stake in the outcome of the conflict.

Green used three conflicts in which Armstrong used in earlier research done on role-playing in 1987.  These three conflicts included Artists Protest, Distribution Channel, and 55% Percent Pay Plan.  Armstrong also developed the fourth conflict in 1977, called the Panalba Drug Policy.  Green created the final two conflicts, Nurses Dispute and Zenith Investment.

Green gave participants in the unaided judgment groups a briefing on the conflicts, and then allowed them to answer a questionnaire on what they thought the outcome of the conflict would be.  For the role-playing experiment, Green gave groups of university students roles to play within each scenario, then the students had between thirty minutes and one hour to come to a resolution.  Finally, Green emailed 558 game theorists, of which 21 participated, so to judge their forecasting accuracy using game theory.

In the end, role-playing had the highest accuracy across all scenarios except the Panalba Drug Policy, in which game theorists scored the highest accuracy.  Based off these findings, Green concluded that role-playing does provide better forecasting accuracy than game theory and unaided judgments when small groups are in conflict where all the members have a high stake in the outcome.

Critique

One benefit discovered from this research is the increased accuracy by non-subject experts when using role-playing as opposed to unaided judgments.  The majority of the participants in the role-playing exercises were university students who lacked experience in the field of knowledge that the scenario was portraying.  Despite this, the groups were able to accurately forecast the outcome 64% of the time.  We, as analysts, are not always subject matter experts, be it in US Pakistani foreign relations or the global economy.  There is, according to this research, use in this method to overcoming these barriers and vastly improving our forecasting accuracy over other models.

Role-playing is a quick method of analysis according to this research.  Green only allowed the role-playing participants between 30 minutes and an hour to meet, work together, and come to a final agreement on the conflict.  This was not a daylong process that would consume large amounts of the analysts and executives time.  Despite this short time window, the groups correctly determined the correct outcome by 59% or higher in all but one of the conflicts (Artist’s Protest = 29%). 

This research focused on only targeting small groups were all the members of the group had a large stake in the outcome of the conflict.  While this is relevant in many situations in today’s world, not all cases are like this.  Will forecasting accuracy remain high when there is a large group of players, such as ten or more countries, were certain countries have much more at stake than others?  There are some gaps in this research and in its design that prevent it from clearly proving its supremacy over game theory.

Source-

Green, K, 2002.  The decisions in conflict situations: a comparison of game theory, role-playing, and unaided judgement.  International Journal of Forecasting.  Retrieved from http://www.forecastingprinciples.com/paperpdf/Greenforecastinginconflict.pdf    

The Evaluation of Role Playing in the Context of Teaching Climate Change

Summary

This article presents a study done by Belova, N., Feierabend, T., and Eilks, I. to evaluate role playing in science-based scenarios. Belova et al. stipulates that although many applications for role playing in science education have been developed, there is limited research on the performance during the exercises. Additionally, Belova et al. points out that there is no agreed upon method for analyzing arguments within a role playing exercise. This paper presents a systematic method for analyzing performances in role plays and what level of argumentation and decision-making capabilities are evident within the exercises.

Belova et al. performed the study by having 20 groups of students, aged 15 to 17, participate in the role playing exercise of debating climate change as a stake holder, such as the automobile industry or Greenpeace activists. The 20 groups were assigned to one of four class subjects, biology, physics, chemistry, or politics which then became their primary source of knowledge for the exercise. The exercises were recorded and evaluated by two analysts.

The arguments made by students during the exercise were evaluated in three ways: domain, level, and reference. Domain pertains to where the student derived the argument from, level refers to the complexity of the argument, and reference is the ability to which a student can refer back to comments made by previous speakers. A high score was awarded for arguments sourced by scientific facts, if they were interconnected with multiple justifications, and if it was made within the context of the conversation at hand. A low score was assigned to arguments with no clear origin and unrelated to climate change.

Of the 20 role playing exercises, only 20 percent were classified as high quality and reflective. Additionally, only 20 percent of the arguments made during the course of the exercise were supported with scientific facts. Furthermore, the scientific based arguments possessed little variety, used terminology incorrectly, and were incorrect 10 percent of the time. Although the quality of the arguments was lacking, the analysts found that 80 percent of the conversations flowed smoothly.

Critique

The use of role playing did facilitate the exchange of ideas among the students; however, it did not improve the quality of the conversation significantly. The students, rather than citing scientific facts which were provided to them, resorted to citing socio-life experiences to bolster their claims. This is likely to do poor preparation and lack of practice rather than a true belief that one’s personal experiences carry more weight.

Additionally, age among the study participants is a factor that must considered. The study cites that the older students produced better arguments than the younger groups. It is likely the older students had more knowledge from school to draw from in their arguments, whereas the younger students became dependent on the limited amount of info provided to them.

Finally, a factor that likely impacted the study was the role of the moderator in each of the exercises. A different moderator was present for each exercise, and that likely influenced the results of the exercises depending on style of the moderator.

Source-

Belova, N., Feierabend, T., and Eilks, I. 2013. The Evaluation of Role Playing of Teaching Climate Change. International Journal of Science and Mathematics Education.

 

Role Playing: A Method to Forecast Decisions

Role Playing: A Method to Forecast Decisions
By: J. Scott Armstrong

Summary:
This publication outlines the basic elements of role-playing using a series of studies. The author conducted a study that had two groups of two participants each for 80-minute sessions. Each person handled one simulation as an expert and another as a role player. In each of the simulations, each person received closed ended questions ranging in possible decision making conclusions. In each of the role-playing simulations, participants were assigned background information to make the simulation sound realistic while viewing the problem from an outside perspective.

After reading the material for 20 minutes, each party would meet at a table until they reached a consensus or when the time ran out. After the role-playing was completed, each person answered a set of questions. These questions instructed them to state the consensus as they saw it or, if they did not reach a consensus, what would have happened if they reached a conclusion in the appropriate time.

Also, to reduce the chance of misinterpretation, participants were asked to write their views of the decision down independently at the end of the simulation.  This is a great exercise to do since it forces people to agree on the underlying decision.

The author concluded that role-playing provides greater accuracy when compared to expert opinions. Role-playing produced correct predictions for “56 percent of the situation versus 16 percent for opinion.” In addition, “role-playing produced outcomes that were similar to those from seven out of eight experiments.”

Critique  
Some of the advantages of role-playing are that it is less expensive than experiments and it provides additional outcomes not considered by experts on opposing objectives and strategies. It provides individuals to think for themselves. Although the author supports role-playing, this publication has some disadvantages.

For example:
One of the basic elements of role-playing is to cast roles similar to the people they represent. According to the author, participants should have similar backgrounds, attitudes and objectives to the role-playing simulation. In addition, Ashton and Krammer (1980), “found considerable similarities between students and non-students in studies on decisions making processes.” While valuable, it does not state whether undergraduates partook. If so, how can an undergraduate student have a similar background and attitude towards a role-playing situation if s/he does not have a lot of real work experience and/or an education in the topic of discussion? Therefore, further research needs to show a more in depth correlation stating why students and non-student are showing similar role-playing conclusions.
 
In addition, the author stated that each role-playing simulation “should run around ten sessions, five using one description and five using another.” The author did not go into detail on why he chose ten sessions versus any other number. Are ten sessions the average used in role-playing simulations, or is that the number he likes to use?

Source-
Armstrong, J. S. 2002. Principles of Forecasting: A Handbook for Researchers and Practitioners. 2001 edition. Boston, MA: Springer.