Effectiveness Of Role Playing, Case Studies, and Simulation Games in Teaching Agricultural Economics

This study assessed the impact of role playing, case studies, and simulations for teaching at the undergraduate level. Its purpose was to show how effective the tools were at improving the student’s level of understanding for certain concepts.

Objectives and Methodology

The educational impact model was used in a qualitative evaluation of the content and the level of effectiveness for role playing, case studies, and simulations. The study was conducted during two classes with one professor teaching the same course. One course was taught using only lecture and the other class was taught using the three different aids. The study also used an average of student test scores as an indicator of the level of effectiveness and student performance. Both classes were given the same test.

Role Playing

Role playing, as most often used in a classroom, requires physical involvement on the part of students. Two or more people "act out" the part of individuals in a hypothetical situation. For the purpose of the class, the following situation was used:

Students were presented with the subject of “price determination; a role-playing exercise patterned around the open-outcry market typical of commodity futures markets.”

1.) Sixteen students participated in three successive auctions, some acting as farmers and others acting as wholesalers.

2.) Information concerning quantities to be bought or sold and profit/cost levels per pound was provided on index cards given to each participant by the instructor.

3.) During each three-minute auction the students had to negotiate transaction prices with one another.

4.) After the third auction, the three sets of resulting prices were analyzed by the entire class.

5.) After the instructor told the class which set of prices were generated in an equilibrium, surplus, and shortage situation the students could see that the theory presented in their textbooks did, in fact, describe what happened in their exercise.

Results for the Effectiveness of Role Playing

The results for the experiment concerning use of a role-playing exercise provide some insight into the impact. Role playing had only a small direct effect on exam scores; only a 3 percent increase on scores. However, students made written comments on how role playing raised the student’s level of interest. 73 percent of the participants noted that the exercise was one of the most favorable parts of the course.

Strengths

· Gives students a better understanding of the decision-making process

· Can be applied to various situations

Weakness

· Does not allow for much detail in the information being studied or all of the variables involved

Comments

For results of the whole study including case studies and simulations, please see “Effectiveness Of Role Playing, Case Studies, and Simulation Games in Teaching Agricultural Economics” by Steven Blank accessed through Academic Search Complete.

Investigating the Use of Role Play Training to Improve the Communication Skills of IS Professionals: Some Empirical Evidence

Mark Frolick, the author of this article from The Journal of Computer Information Systems, looks at the effectiveness of the role play exercise for the communication skill improvement of Information Systems (IS) professionals. Role play is an active learning technique that creates a training situation in which the interpersonal interactions and communication flow characteristics of the real world can be accurately reflected.

The study consisted of 93 role play exercises aiming to improve the two dimensions of communication skills, content and process related skills, which were conducted among 92 graduate students enrolled in systems analysis and design courses between 1998 and 2000. It looked at two questions: How fast can role play exercises generate measurable results? and Does the incremental improvement in communication skills between role play exercises decrease over time? Each group was given 5 minutes to prepare a scenario, which was then acted out. Fellow students acted as judges and rated the participants in terms of effective communication in both content and process. Each student participated in 3 role-playing exercises about 4 weeks apart.

Results
The analysis of the data suggests role play is a viable training method that can yield measurable results of communication skill improvement. The content and process related skills of the participants improved consistently over the three role play exercises on average. Incremental improvement decreases as more role play exercises were conducted, which is consistent with learning curve theory.

Strengths

• rapidly and effectively improve the communication skills of trainees
• participants become more involved in the training
• participants have a better overview of problems and can foresee what possible solutions exist and why
• allows participants to experiment with various strategies without real consequences, allowing them to express their opinion freely
• enhance self-confidence of participants

Weaknesses

• participants may become apathetic to role play exercises over time
• role play exercises are challenging to devise
• puts individuals on the spot, which might lead participants to resist

Source:

http://www.allbusiness.com/human-resources/workforce-management/1126451-1.html

Role-play for Medical Students Learning About Communication: Guidelines for Maximizing Benefits

This article, from BioMed Central Medical Education, evalutes the use of role-playing as an educaitonal method for medical students. Authors Debra Nestel and Tayna Tierney focus on role-playing by 284 first-year medical students at Imperial College's communications program, which divides students into groups of 30 to learn specific skills associated with medical interviewing. Students take the role of interviewer, patient, or observer and have 5 minutes preparation, 5 minutes in role-play and 10 minutes in feedback. Before the sessions, students completed questionnaires about their prior experience with role-play, and afterwards, they completed evaluations about whether or not they thought role-play was helpful based on their new experience. Responses were both closed and open-ended.

According to the authors' research, role-play is used as a training method to acquire knowledge, attitudes, and skils in a range of disciplines and with learners of different ages. Role-play in particular is a type of simulaiotn that focuses attention on the interaction of people with one another and emphasizes the functions performed by different people under various circumstances. Role-play can be fully scripted or partially scripted. Players can rotate through roles or new information can be added into the role-play.

Results
Of the 284 students who completed forms, 199 had prior experience with role-play. 221 students reported role-play to be valuable for learning while the remainder reported it to be not valuable. After the role-play, 274 students reported that role-play had been helpful for leaning. Unhelpful experiences focused on emptional responses that impede learning, such as embarrasment, intimidation, or anxiety. Overall, role-play was reported to be an effective means of learning communication skills, desptite prior negative experiences with role-play.

Strengths

• students gained insight into their own and others' behavior
• increased understanding of certain issues
• increased communication skills
• players get immediate feedback

Weaknesses

• players not taking role-play seriously
• lack of realism in roles, setting, or tasks

Source:

http://www.biomedcentral.com/content/pdf/1472-6920-7-3.pdf

On The Move With A Magic Thing: Role Playing In Concept Design Of Mobile Services And Devices

This study, conducted in 2000, evaluated the success of using role playing to design a mobile network for the campus of Helsinki University of Technology to investigate the nomadic Internet use of the future. The design team hoped to overcome three main difficulties of their task through role playing. The design team was planning for the use of a technology that did not exist at the time, so it was hard to know how it would be used. They needed a design that allowed for mobility, and they had to incorporate their users' lifestyles and cultures into their plan for the technology.

The designers used two methods of role playing. One of these is called participatory design technique, in which the designers asked a sample of their target audience to enact daily situations imagining using a technology in a variety of campus settings and while playing either themselves or different roles. They utilized props to make the situation seem more realistic. In contrast, the second type of role playing, referred to as Situated and Participative Enactment of Scenarios (SPES) had people carry a mock up of the anticipated product in their hand with them through their daily activities and imagine how they might use the product. A trained observer recorded the participant's thoughts. The study revealed the following strengths of role playing in general.

Strengths:

• Engages users in a common cause
• Good way to investigate the consequences of decisions or designs with little risk or cost
• Captures cultural aspects well in designs and decisions

Results:

The study also made it clear that the participatory design technique in which people acted out fictional situations was better suited to designing services involving group interaction and dynamic incidents. In contrast, SPES was more useful for considering less dynamic activities such as listening to music, and it helped give designers a more detailed scenario because the recorder noted the context of a user's actions.

Source:

http://www.cs.hut.fi/~pmrg/publications/GO-PROD/2000/Lacucci_Kuutti_Ranta_OMMT.pdf

The Social Framework of the Role-Playing Process

This article, written by Markus Montola, looks at the process of role-playing that takes place in different games, and the inconsistencies between the definitions of role-playing. In this article, Montola defines the role-playing mindset as a method of game playing, which can be optionally combined with various game systems. Role-play typically has no inherent endogenous goals at all. The rules of role-playing only provide the structure for the activity, but give no end condition or an objective.

Montola states that role-play, a social activity, always has three elements present: an imaginary game world, a power structure, and personified player characters. According to the article, the power structure of role-playing activities about imaginary people acting out in an imaginary environment is what differentiates it from children's play. Montola proposes three general rules that should always be followed in any type of role-playing:

1) Role-playing is an interactive process of defining and re-defining the state, properties and contents of an imaginary game world.
2) The power to define the game world is allocated to participants of the game. The participants recognize the existence of this power hierarchy.
3) Player-participants define the game world through personified character constructs, conforming to the state, properties and contents of the game world.

This article examines several forms of role-playing, primarily taking examples from tabletop role-playing, live-action role-playing, and virtual role-playing.

Source: http://journalofroleplaying.org/

Problem Solving and Conflict Resolution Role Playing

The article describes the use of role playing to come up with constructive solutions to conflict and problems in a classroom setting. The technique entails assigning people to act out a conflict scenario in order to come up with the best solution to a problem. Role playing is often used in education to improve children's interpersonal skills and ability to handle difficult social situations. It is versatile and has applications for solving problems as simple as playground bullying to those as complex as international tensions.

Strengths:

• The activity simulates the emotions of an actual event which engages participants in the process.
• The fictitious nature of the interaction reduces the psychological stress of a situation which allows shyer people to participate more fully than they might feel free to do in real life.
• Encourages creative solutions.
• Increases insight into another person's point of view.

Weaknesses:

• It is not a fruitful exercise unless the participants take it seriously.

How to:

1. Discuss the problem to be portrayed.
2. Make sure that the participants are committed to the process.
3. Assign roles.
4. Act out the situation, responding spontaneously.
5. Evaluate the actions of each player.
6. Repeat, taking different courses of action if needed.

Source:

http://www.classroomsthatwork.com/pdf/Roll%20Playing.pdf

Team Role Play

Role Playing

Summary:
The author of this article on Team Role Playing, Jeffrey Baumgartner, writes that role playing "is when a group of people act out roles for a particular scenario." He goes on to describe a simple role playing scenario about a salesman and a customer. In essence, the two sales trainees play the two roles. Baumgartner mentions that other people may watch the scenario in order to better understand also.

The article continues with Baumgartner's scenario of salesman and customer being put to groups of trainees. There are specific rules that are laid out before each group and eventually each group must make a decision about how to best "play out" the scenario. The author mentions that when a team member is trying to help the "actor" that it should not be discouraged because it could possibly lead to better solutions. He sums up his discussion about his scenario by showing how each scenario can become more and more complex by adding more specifics, forcing the "actors" to think more about how to reach the best outcome for their team.

Strengths:

• Analyzes problems from various perspectives

• Implements brainstorming methodology in simulations of real cases

• Tries various solutions in a case scenario

• dDevelops team-work, co-operation and creative problem solving in groups

• Exercises creative techniques in a risk-free environment

Link:
http://www.jpb.com/creative/roleplay.php

Green Team Summary of Findings: Delphi Analysis (3 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 16 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 22 April 2010 regarding Delphi Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

DESCRIPTION

The Delphi method is based on a structured process for collecting and distilling knowledge from a group of experts by means of a series of questionnaires combined with controlled opinion feed back. It was developed in the early 1950s, and mainly applied to technology forecasting, but also to many types of policy analysis.

Delphi focuses on researching the future or things about which little is known. It relies on the use of expert opinion, utilizes remote group processes, adopts an iterative research process, and creates a consensus of opinion.

There are three versions of the Delphi Technique:

• Numeric – aims to specify a single or minimum range of numeric estimates through the use of summary statistics.

• Policy - on the exploration, generation and definition of several alternatives and the arguments for and against each of these alternatives.

• Historic - aims to explain the range of issues that fostered a specific decision, identification of several scenarios that could have led to the resolution of a past problem.

Delphi is particularly appropriate when decision-making is required in a political or emotional environment. The tool works formally or informally, in large or small contexts.

STRENGTHS AND WEAKNESSES

Strengths:

• Its ability to ascertain expert opinions and potential policy options

• Allows a group response without the attendant disadvantages sometimes experienced with group problem solving or decision-making.

• Expert participants are more likely to generate reasoned, independent, and well-considered opinions in the absence of exposure to the "persuasively stated opinions of others". Because the experts do not ever participate in a face-to-face discussion, there is no danger of one or more individuals’ opinions being swayed by a more dominant or more experienced individual.

• Efficiency and flexibility, especially in light of modern communication technologies such as e-mail and the Internet. Experts may be drawn from a wide geographic area, and the participants’ commitment in terms of time and money invested is minimal.

• Delphi method has been shown to be an effective way to conduct research when the responses being sought are value judgments rather than factual information. Although it is more difficult to assess the "correctness" of value judgments, it is generally agreed upon that value judgments are not all equal but can in fact be more "right" or more "wrong."

Weaknesses:

• A low level reliability of judgments among experts

• Dependency of forecasts on the particular judges selected

• Should not be used when any of the following three critical conditions are not present: adequate time, participant skill in written communication, and high participant motivation. It is estimated that a minimum of 45 days is required to carry out a Delphi study

• A high degree of motivation is needed to offset the tendency for participant dropout as the study progresses. Because there is no direct contact between participants, those who are not highly motivated and interested in the subject at hand may feel isolated or detached from the process (It is easy for participants to drop out of the process)

• There is a strong response of the group to conform with the statistical feedback of the panel, extreme points of view, which may provide new insights tend to be suppressed

• The way the questionnaire and process is structured can lead to a bias and a Delphi study is at the mercy of the view and biases of the coordinating team, who choose respondents, interpret the information and structure the question

• Participants must be knowledgeable and able to clearly communicate their ideas

• Sensitivity of the results to ambiguity in the questionnaire that is used for

• Difficulty in assessing the degree of expertise incorporated in the forecast

• Good responses can be altered by those without a strong belief in the participants' answers

HOW TO

There are many ways in which to conduct a Delphi exercise depending on the desired outcome. However, from the research several consistent trends emerge. They are as follows:

• A questionnaire is sent to experts.

• Each expert gives his answers to the questions in an independent and anonymous way.

• The responses of each expert are analyzed by the monitoring team. The moderator summarizes the responses to the first questionnaire and develops a feedback report along with the second set of questionnaires for the panelists

• The set of responses is then sent back to experts and they are asked if they wish to revise the initial predictions.

• The process is reiterated until a degree of consensus is reached by experts. Generally the number of rounds is determined in advance.

• The moderator develops a final summary and feedback report to the group and decision makers

PERSONAL APPLICATION

We started with the a question: "Of the 21 first year graduate students, how many will graduate on May 21, 2011 having successfully completed their thesis or comprehensive exam?" Each person wrote out their answer without discussion and handed it in. While one person found the average, the facilitator introduced data regarding the thesis completion rate from 2005 to 2009. The average was written on the board and then a one minute discussion regarding the trend of incompletes. Then the group wrote out their next answer and another set of calculations were done. Another one minute discussion was introduced and then a third round of responses. A final discussion noted how/if the process changed answers due to the introduction of facts or consensus from the "experts."

Summary of Findings (White Team): Delphi Technique (3out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 16 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 22 April 2010 regarding Delphi Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Delphi is a method for a structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem. It was created in the 1950's by the Rand Corporation to increase efficiency and to bring together Subject Matter Experts to assess and create possible outcomes to a problem. It is a process conducted through the use of multiple rounds to introduce new ideas and come to a consensus on those new ideas along with existing ideas.

Strengths:

• Its ability to ascertain expert opinions and potential policy options, including new alternatives.
• Participants can be located all over the world.
• Depending on the goal of the study, consensus on a forecast may be reached.
• Anonymity improves participants ability to judge the presented options, not just presentation style.
• Forecasts can be judged against the later reality of the situation.

Weaknesses:

• Participants must trust the methodology and moderator or facilitator for this to work.
• Moderator or facilitator should be objective.
• Can lead to Groupthink.
• Susceptible to the anchoring bias.

How To:

Pick a facilitation leader: The facilitator is an expert in research data collection, and is not a stakeholder.

Select a panel of experts: The panelist should have an intimate knowledge of the projects.

A questionnaire is sent to experts.

Each expert gives his / her answers to the questions in an independent and anonymous way.

The moderator summarizes the responses to the first questionnaire and develops a feedback report along with the second set of questionnaires for the panelists

panelists independently evaluate earlier responses and vote on the second questionnaire

The set of responses is then sent back to experts and they are asked if they wish to revise the initial predictions.

The process is reiterated until a degree of consensus is reached by experts.

Some variations to this very basic method include:

• The number of iterations (the more rounds, the closer the consensus likely to be reached)
• The method of selection and size of the panel
• The scoring system and the rules used to aggregate the judgments of the panelists
• The extent of anonymity afforded to the panelists
• How consensus is defined and how disagreements are dealt with

Application:

We decided to apply the Delphi technique to expected completion of the Master of Science of Applied Intelligence program here at Mercyhurst College. At graduation, every student is allowed to walk across the stage to be recognized, however, not every student may have completed the degree requirements necessary to receive his or her diploma. Based on statistics from recent years, our class attempted to estimate how many out of 21 students in the class of 2011 would actually receive their diplomas on graduation day. In our graduating class, currently 20 students are scheduled to write a thesis and one student is scheduled to take a comprehensive exam. There were three rounds of anonymous voting:

• Round 1 - Average:14.5; High: 19; Low: 7
• Round 2 - Average: 13.08; High: 17; Low: 7
• Round 3 - Average: 13.16; High: 17; Low: 10

After round 1 there was a 1-minute discussion to articulate why each of us voted the way we did. After round 2 we were presented with previous classes' graduating statistics which was designed to provide background information that we could use to adjust our individual estimates. After round 3, we arrived at our final conclusion as shown above.

Formal Expert Judgement: An Overview

Introduction:
This paper, prepared at the Joint Research Centre of the European Commission within the frame of the institutional action SAFELIFE - Safety of Ageing Components in Nuclear Power Plants. Acknowledging that formal expert judgement has become a relatively well-established tool in connection to risk assessments, the paper aims to apply formal expert judgement to assess the structural integrity of nuclear power plant components. In the past, the Delphi method was one of two key methods that were used to convey structured expert opinion. The authors offer a description of the method, including the process and it's limitations.

The Delphi Method: Description
Developed in the early 1950s, it was mainly applied to technology forecasting, but also to many types of policy analyses. The method is based on a structured process for collecting and distilling
knowledge from a group of experts by means of a series of questionnaires combined with controlled opinion feed back.

Process:

• A questionnaire is sent to experts.
• Each expert gives his answers to the questions in an independent and anonymous way.
• The responses of each expert are analysed by the monitoring team. Thelower 25% and the upper 25 % of responses are excluded.
• The set of responses is then sent back to experts and they are asked ifthey wish to revise the initial predictions.
• The process is reiterated until a degree of consensus is reached by experts.

Weaknesses:

• A low level reliability of judgements among experts and therefore
  dependency of forecasts on the particular judges selected
• Sensitivity of the results to ambiguity in the questionnaire that is used for
  data collection in each round
• Difficulty in assessing the degree of expertise incorporated in the forecast
• Responses can be altered by monitors in the attempt of moving the
  following round of responses in the desired direction.

Despite these weaknesses, the authors note, "it must be acknowledged that there have been many poorly conducted Delphi applications and there is in fact an important conceptual distinction between evaluating a technique and evaluating an application of a technique." Therefore, they conclude that in general the Delphi method is useful in answering one
specific and single-dimension question. They argue; however, that it may not be effective for
determining complex forecasts concerning multiple factors since the collation of expert judgements suffers from the possibility that interactions between forecasted items may not be fully considered.

Improving the Delphi Method:
To improve forecasting reliability in the Delphi method, it is necessary to consider
the possibility that the occurrence of one event may change the probability of occurrence of other events included in the surveys. Cross impact analysis was therefore developed as an extension of Delphi method.

Source:
K. Simola, A. Mengolini, R. Bolado-Lavin. "Formal Expert Judgement: An Overview", Institute for Energy, Joint Research Center of the European Commission (2005).

The Delphi Technique In Developing International Health Policies: Experience From The SARSControl Project.

Summary:
The study entitled The Delphi Technique In Developing International Health Policies: Experience From The SARSControl Project, main objective was to evaluate the “Delphi technique” as a tool to assist in developing international policies for SARDS and SARS-like diseases using the criteria found in the literature and its application in the SARSControl Delphi study.

Methods:

The European Community funded research project SARSControl was made up of 17 international organizations. Its policy evaluation work package was the setting of this study and the basis of our study was the SARSControl hence, data and results from it have been used for our analysis.
This paper uses the criteria defined in the literature to assess the process of using the Delphi technique in one of the largest up to-date projects aimed at developing international emerging infectious disease policies. The evaluation was done using the qualitative description of the SARSControl Delphi and carrying out a critical analysis of different aspects of each criterion.

Results:
The Delphi process, which was carried out over a period of nine months consisted of a pilot round, two written rounds and a final face-to-face meeting. Sixty infectious disease experts who were national experts in the field of infectious diseases who working at senior levels nationally and internationally, and represented their country on the Advisory Forum of the European Centre for Disease Control, Stockholm (ECDC) were approached. Of the 60, 47 accepted the invitation-written consent was obtained from them. Thirty-eight experts (from 22 countries) participated in the 1st written round and 28 experts (from 19 countries) in the 2nd written round; and 11 newly recruited experts with similar expertise as the participants from the written rounds (as five panellists invited from the written rounds were not able to participate ) from 9 countries participated in the face-to-face meeting. A possible explanation for this is that as national experts, they were extremely busy, and also the panelists were not invited early enough and nor were they informed of it at the recruitment phase. Two reminders were sent after the written round invitations. The response rate for the 1st round was 80% and the 2nd round was 74%. Seven replies received after the face-to-face meeting were excluded from the analysis. Misinterpretations of questions in the written rounds were rectified by either deleting or clarifying them in the 2nd round questionnaire.
The Delphi panel participants consisted from various health-related disciplines, however their specific experience with SARS was not known. Other relevant specialists for e.g. behavioural scientists and communication specialists were not included in the Delphi panel. European countries were well represented- France, Germany, Singapore, Sweden and the U.K. were countries with experience of 'probable' SARS cases. Learning from Chinese and Canadian experiences was one of the aims of the project. Two experts from China (with the most SARS experience) were invited to the written rounds but they were unable to participate. Canadian experts were not invited.

Conclusion:
The Delphi technique can aid the international policy development process and it can be a versatile tool, which creatively collects expert opinions and suggestions in a new topic. Central criteria to be met include representative panel composition, high panelist motivation, and effective but flexible administration of the Delphi process. Based on the assessment of SARSControl Delphi technique, it can be concluded that when rigorously administered, analyzed and reported, is a valuable method to develop international health policy recommendations for emerging infectious diseases, even though discrepancies in its application existed. The SARSControl Delphi technique was a positive experience, and can also be considered by others for similar purposes.

Strengths and Weaknesses:

This is the first methodological study to assess the Delphi technique in developing international infectious disease policies. Although this is a case study within one project, it exemplifies some crucial issues in applying the Delphi technique.
The major strength of the policy Delphi used was its ability to ascertain expert opinions and potential policy options, including new alternative ones, from countries representing different cultures and health policies on the timely issue of infectious disease control. This contributed to the policy recommendations to be put forward to policy makers to agree upon.
The major weakness of the Delphi was that the panel represented mostly European experts, missing countries with most SARS experience, and also the discontinuity from the written rounds to the face-to-face meetings due to drop out and financial constraints.

Source:
Syed, A., Hjarnoe, L., & Aro, A. (2009). The Delphi Technique In Developing International Health Policies: Experience From The SARSControl Project. Internet Journal of Health, 8(2), 5. Retrieved from Academic Search Complete database.

The Delphi Technique in Developing International Health Policies: Experience from the SARSControl Project

Intro:The Delphi technique, according to the author, helps in structuring a group communication process that is particularly useful when there is little knowledge or uncertainty surrounding a complex area being investigated. This paper uses the criteria defined in the literature to assess the process of using the Delphi technique in developing international emerging infectious disease policies. The evaluation was done using the qualitative description of the SARSControl Delphi and carrying out a critical analysis of different aspects of each criterion.

The Delphi technique can be classified as follows:

• The Classical Delphi-to establish facts
• The Policy Delphi- to generate ideas
• The Decision Delphi- to make decisions
• The Group Delphi- for group discussions

Criteria: The Delphi technique must be applied systematically using the following 5 criteria:

1. Panel Composition (geographic and professional representativeness, size, heterogeneity)
2. Participant Motivation (response rate, written consent, clarity of questions, reminders)
3. Problem Exploration, e.g. as percentage of agreement /medians
4. Consensus Definition, e.g. as percentage of agreement /medians
   
5. Format of Feedback, e.g. individual responses, measures of tendency and spread of responses

Other criteria includes: number of rounds, anonymity to encouraging open expression of opinions, and sufficient resources which include time and administrative services.

How Delphi was used for this project:

Panel Composition: The Delphi process, which was carried out over a period of nine months consisted of a pilot round, two written rounds and a final face-to-face meeting. Thirty-eight infectious disease experts (from 22 countries) participated in the 1st written round and 28 experts (from 19 countries) in the 2nd written round; and 11 newly recruited experts with similar expertise as the participants from the written rounds from 9 countries participated in the face-to-face meeting.

Problem Exploration: Thirteen policy components considered important in terms of emerging diseases were used to formulate statements for the written Delphi rounds.
The two written rounds of the Delphi and a face-to-face meeting were found sufficient to obtain the outcomes in terms of generating policy options and alternatives. The 2nd Delphi round was crucial in clarifying issues from the 1 st round.

Round 1: Closed-ended questionnaire with a possibility to make comments.
The written comments (qualitative data) provided by the respondents were individually analyzed. (Took 2 months to complete)

Round 2: The questionnaire for the 2nd round was based on the results and comments received in the 1st round. (Took 2 months to complete)

• Questions which had reached consensus in the 1 st round were excluded from the 2 nd
  
• Unclear and confusing questions were either rephrased or new questions were formulated based on the panellists' comments
  
• Unclear questions from the 1 st round to the 2 nd were dropped in the course of the Delphi written process.

Feedback for the Face-to-Face Meeting: Panel members were fed back with percentages of agreement within the panel for each statement in every round. Summaries of the comments from the previous rounds were also given.

• Prior to the face-to-face Delphi meeting, its participants were informed via email about the Delphi technique and about the results of the written Delphi rounds. This was done to ensure that the participants had a common starting point as none of them had participated in the written rounds. The questionnaire responses were anonymous to other participants.

Conclusion:
The SARSControl Delphi process succeeded in its aim to generate policy options and alternatives — this in spite of the discontinuity in the involvement of the experts throughout the Delphi process. It can be concluded that from the results that Delphi technique when rigorously administered, analyzed and reported is a valuable method to develop international health policy recommendations for emerging infectious diseases and aid the international policy development process creatively collecting expert opinions and suggestions.

Source:
Syed, A., Hjarnoe, L., & Aro, A. (2009). The Delphi Technique In Developing International Health Policies: Experience From The SARSControl Project. Internet Journal of Health, 8(2), 5. Retrieved from Academic Search Complete database.

The Delphi Technique

What is it?
The Delphi Technique is a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem, according to Linstone and Turoff.

Where does it come from?
Delphi was developed by the Rand Corporation in the 1950s, funded by the U.S. Air Force, to find a way to establish reliable consensus of opinion among a group of experts about how Soviet military planners might target the US industrial system in an attack and how many atomic bombs would be needed to have a specified level of impact on U.S. military capability.

What is it used for?
It is widely used for more peaceful purposes today, but with the same underlying rationale: to establish as objectively as possible a consensus on a complex problem, in circumstances where accurate information does not exist or is impossible to obtain economically, or inputs to conventional decision making for example by a committee meeting face-to-face are so subjective that they risk drowning out individuals’ critical judgments.

The typical features of a Delphi procedure are an expert panel; a series of rounds in which information is collected from panelists, analyzed and fed back to them as the basis for subsequent rounds; an opportunity for individuals to revise their judgments on the basis of this feedback; and some degree of anonymity for their individual contributions.

The UK government commonly uses Delphi to make decisions or allocate resources in the health service, a classic context in which demand for resources will always outstrip their availability. Delphi is also mentioned in business texts under decision making techniques, along with other structured approaches such as the Nominal Group Technique. They allow complex decision­making and creative problem­-solving in a way which avoids the drawbacks of conventional meetings with unstructured, free­-flowing interaction and minimal direction such as

• High variability in participant behavior and group social behavior · Discussion falls into a rut or goes off at tangents
• The absence of an opportunity to think through independent ideas results in generalizations
• High status or dominant personalities dominate discussions and decisions
• Unequal participation among those present
• Meetings conclude with a perceived lack of accomplishment

The basic method to implement Delphi as described by Delbecq et al is:

1. develop an initial questionnaire and distribute it to the panel
2. panelists independently form ideas to answer the questionnaire and return it
3. the moderator summarizes the responses to the first questionnaire and
   develops a feedback report along with the second set of questionnaires
   for the panelists
4. panelists independently evaluate earlier responses and vote on the second questionnaire
5. the moderator develops a final summary and feedback report to the group
   and decision makers

Some variations to this very basic method include:

• The number of iterations (the more rounds, the closer the consensus likely to be reached)
• The method of selection and size of the panel
• The scoring system and the rules used to aggregate the judgments of the panelists
• The extent of anonymity afforded to the panelists
  
• How consensus is defined and how disagreements are dealt with

Conclusion
This article, written by Nic Underhill, provides a very concise overview of what the Delphi technique is to the average person. Unfortunately, he does not identify any potential strengths and/or weaknesses of the technique which significantly diminishes the utility of his article. Although he introduces the reader to the Delphi technique, he does not give that person any direction in how to apply it to his/her own unique situation.

Using Delphi Technique in a Consensual Curriculum for Periodontics

Fried and Leao conducted an experiment using the Delphi technique to obtain a group consensus on the curriculum for periodontics. The goal of this study was to use the Delphi technique to identify a consensus about what topics should be included in a periodontics curriculum for undergraduate dental students.

METHODS
Nine periodontics faculties at dental schools in two Brazilian cities participated in this curriculum planning process. Forty undergraduate professors, all dentists, were invited to participate in the study. For the first phase of the experiment, lecturers were asked to list, in writing, items that should be included in a periodontics curriculum for dental students. Suggested items were split into two groups. The first group involved theory-related items associated with foundational concepts and basic principles of periodontics. The second group of items, related to dental practice, was associated with laboratory training and clinical experience with patients.

For the second phase of the experiment, previously obtained responses were scrutinized and collated into items. Then, a comprehensive Likert-scale questionnaire was compiled for submission to the panel. The questionnaire offered the following options for rating the importance of proposed curriculum items: "indispensable"; "important"; "relatively important"; "of little importance"; and "should not be included." The researchers sent the questionnaire to all forty respondents and evaluated the answers as follows:

• item was "kept" if it reached a "50 percent plus one" consensus classification as "essential/important"
• item was eliminated if it reached a "50 percent plus one" consensus classification as "of little importance/should not be included"

• item was included in the next round of the questionnaire if it did not reach a "50 percent plus one" consensus agreement or a "50 percent plus one" consensus rejection

During stage three, the only participants were individuals who provided the most extreme positive ratings or the most extreme negative ratings during the previous stage for items where no clear consensus was reached.

For the fourth and last stage, for each non-consensual item singled out in the third stage, the positive and negative justifications required from third-stage participants were transcribed into a new questionnaire. All the original participants were then asked to for inclusion or against inclusion of each pending item. The format used for the fourth stage questionnaire was that of a sequence of items, each of them carrying its associated positive and negative justifications, followed by corresponding "yes" and "no" answering options. Frequencies for the answers thus obtained were calculated with the SPSS statistical package.

RESULTS
Out of the forty initial participants in the study, six (15 percent) eventually dropped out. Two moved abroad, and the other four were unable to stay involved until the end of the experiment. In accordance with the literature, this 85 percent response rate may be considered as "good."¹⁵ Out of the thirty-four lecturers who participated in each of the four Delphi technique stages, thirteen (38 percent) were females, and twenty-one (62 percent) were males. They had an average of ten years (±8.19) teaching experience.

Throughout the four stages of the consensus-building Delphi technique approach used in this study, participants identified various up-to-date scientific research trends within periodontics. However, in spite of that, items associated with some recent technological advances were not included in the final set of "indispensable" items. Items related to technological advances in odontology, such as human molecular genetics and DNA probing, were initially listed as important to a periodontics curriculum. However, half way through their study, they were excluded from the final list of selected items. The results of their study indicate that some issues such as a public health-oriented vision, were not addressed by the panel. This suggests that the majority of the individuals involved are focused on the treatment model rather than on health promotion. However, such a drawback may also be credited to limitations of the Delphi technique, which does not allow an interpersonal discussion of topics. Overall, use of the Delphi process allowed accomplishment of the primary goal of the study, which was to identify a consensus about what items should be included in the periodontics curriculum for undergraduate dental students based on the perspectives of a panel of faculty who teach periodontics at several different institutions.

Source: http://www.jdentaled.org/cgi/content/full/71/11/1441

An Overview of Four Futures Methodologies (Delphi, Environmental Scanning, Issues Management and Emerging Issue Analysis)

In this paper on futures research by Trudi Lang, the author compares and contrasts the Delphi method with other futures methodologies. The Delphi technique is found to be different from the other three methodologies chosen by the author for this study viz., Environmental Scanning, Issues Management and Emerging Issues Analysis (EIA).

For the Delphi technique, the positives are that Delphi studies have an excellent record of forecasting computer capability advances, nuclear energy expansion, energy demand and population growth and the technique is also said to expose real agreements and disagreements among respondents as well as giving the facilitator simple and direct control over the scope of the study.

The key drawbacks of the technique are that there is a strong response of the group to conform with the statistical feedback of the panel, extreme points of view, which may provide new insights tend to be suppressed. Also, the way the questionnaire and process is structured can lead to a bias and a Delphi study is at the mercy of the view and biases of the coordinating team, who choose respondents, interpret the information and structure the question.

The author's evaluation is that it is difficult to evaluate the accuracy and reliability of the Delphi method, because the technique is based on determining the opinion of panel members and the findings thus become person and situation specific. In addition, much of the work undertaken to evaluate the Delphi technique has been done with university students asking almanac-type questions.

The author however mentions that the Delphi technique is found to be best suited for exploration of issues that involve a mixture of scientific evidence and social values and that the Delphi method to be considered as one of last resort - to deal with extremely complex problems for which there are no other models.

In conclusion, the author states that the Delphi technique is an "inside - out" methodology when compared to the other techniques which are "outside-in" and by their very nature are all inter-related. The author advocates an integrated approach in using the futures methodologies available so that the strengths of one can make up for the weaknesses of the other.

Comments:

The author's research is thorough and a wide variety of resources have been referenced in her article. The author's conclusion is also valid; as for instance, the Delphi technique is far more successful than other techniques in gaining a consensus from a group but is weaker on allowing independent voices to be heard. In this regard, EIA provides a methodology that is more sensitive to these independent voices and could compensate for this weakness of the Delphi methodology.

Pragmatic Research Design: an Illustration of the Use of the Delphi Technique

This study conducted by two scholars at Rhodes University in South Africa used the Delphi Technique in hopes of forecasting the educational needs that will have to be met to prepare students to be entrepreneurs over the next 20-40 years. This paper does not walk readers through the results of the study, but rather the practical challenges in definitions and the organization of applying the Delphi Technique.

Definitions:
Based on the author’s review of the literature, they identified five main characteristics which define the technique:

1. Its focus on researching the future or things about which little is known,
2. Reliance on the use of expert opinion,
3. Utilizing remote group processes,
4. The adoption of an iterative research process, and
5. The creation of a consensus of opinion.

The authors also identify three versions of the Delphi Technique:

• Numeric – aims to specify a single or minimum range of numeric estimates through the use of summary statistics.
• Policy - on the exploration, generation and definition of several alternatives and the arguments for and against each of these alternatives.
• Historic - aims to explain the range of issues that fostered a specific decision, identification of several scenarios that could have led to the resolution of a past problem.

Entrepreneurship was defined as person who provides innovation in an economy, not owners of micro-businesses in saturated markets. The foundation of their study is based on a review of literature that suggests entrepreneurship can be formed through education.

The Experiment:
To comprehensively forecast the answer to their question, they asked three separate questions. The questions were the following:

1. What sector of the South African economy will most likely offer the greatest potential for entrepreneurial opportunities in the next 25 to 40 years?
2. What qualities are needed by graduates to equip them to be innovative entrepreneurs in the future?
3. What should Higher Education in South Africa do to prepare/develop students to constructively participate in the future economy as innovative entrepreneurs?

Three separate panels were created because each question requires answers from a different set of experts.

1. The panel for the first question was referred to members of government departments and research councils.
2. The panel for the second question was referred to endowed Chairs in the area of entrepreneurship.
3. The panel for the third question was referred to alumni of entrepreneurship programs and educationalists and academics in these programs.

The Future of High-Technology Crime: A Parallel Delphi Study

This study conducted by Larry E. Coutorie in 1995 is a follow-up to a 1980s study using the Delphi Technique to forecast the future of high-technology crimes. One of the purposes of this study is to give law enforcement a forecast of where high-tech crimes are headed, since most other techniques only allow reactionary responses.

The Experiment:
The study was conducted using two panels. One was comprised of “traditional” experts, or people already in the high-tech law enforcement field, and “nontraditional” experts, member of hacker and cracker groups recommended by other experts. Two groups on experts were sent three rounds of questionnaires with the following questions, refined each time by the groups’ responses to the previous questionnaire.

1. In your opinion, what area(s) of high technology will be the focus of criminal activity in the next ten years?
2. What form(s) do you believe this activity will take?
3. What steps should be taken now to prepare the police to combat this criminal activity?
4. Do you believe the responsibility for criminal investigation of high-technology crimes will be primarily that of government or private businesses? Why?
5. Do you believe the responsibility for crime prevention activities regarding high-technology crimes will be primarily that of government or private businesses? Why?

Findings:
Each groups’ perspective diverged significantly from the first round of questioning onward. However, at the end of the three questionnaires, a consensus on several issues was identified.

• Likely high-tech future crime areas include computer system attacks via telecommunications, a growing increase in computer-assisted fraud, and computer assisted data manipulation or theft.
• Crime will take the form of software piracy, increased incidents of computer assisted counterfeiting, increased incidents of financial fraud, and increased attacks on computer systems via advanced technologies.
• Preventative steps recommended include recruitment of individual with computer knowledge, increased public/private partnership, more training for law enforcement officers earlier in their career, and legislation that better defines jurisdiction.
• At the time of this study experts forecasted private business would conduct the initial investigation and have an active participatory role in government investigations.
• They also forecasted that private businesses would be responsible for protecting their own assets, with government assistance in identifying potential threats.