Delphi Method (Rating: 4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the  articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2015 regarding Delphi as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Delphi is an analytic methodology that utilizes a panel of subject matter experts to come to a consensus on an issue.  Successive rounds of surveys are given to the experts and are completed anonymously.  Feedback, consisting of other answers and justifications for the answer, is provided to the experts between each round. The feedback is taken into account in next round of responses.  At the end of the rounds, final answers are delivered to the panelists and the customer for whom the method was conducted.

The Delphi method attempts to achieve several objectives: develop a range of possible alternatives, explore underlying assumptions, seek out information that may generate a consensus, and educate the participants on different ideas and aspects of a specific topic.

Strengths:

• Method provides anonymity to participants
• Method uses independence
• It is very useful to come up with a consensus
• Anonymity and independence help overcome groupthink
• Method is versatile
• Controlled feedback gives participants option to hear others’ opinions and then go back and modify their answers
• Statistical analytic ensure equal representation among participants

Weaknesses:

• Method can take extensive periods of time
• Panels must be composed of experts
• It can be manipulated by the researcher due to his or her extreme control over the process
• Technique requires feedback, but some participants may not respond
• Method hinges on ability/training of facilitator/researcher

How-To:

1. Develop a questionnaire/survey/scorecard to evaluate an estimate or recommendation
2. Gather a group of experts related to the subject at hand
3. Provide them the questionnaire/score card and ask them to evaluate the options of the given scenario
4. Gather the data and provide the scores of everyone & the average for each option. Keep everything anonymous for the participants in the study.
5. Conduct at least 2 rounds of scoring and then evaluate the results.
6. Finally, evaluate the best option or at least eliminate the worst option(s).

Personal Application of Technique:

Delphi exercise involved a UN type simulation decision on which policy was more effective in determining what policy to adapt in regards to Nuclear policy. In order to do this, the group first created a “role” so that the person could act as an expert. The roles that were chosen included a person from the Institute for Science and International Security, Pakistani Prime Minister, Chinese Foreign Minister, Indian Prime Minister, Japanese Foreign Minister, Secretary of Defense, Secretary of State, President of Iran.

Some of the lessons learned include: visual representation of results, incorporating technology (i.e. Google Forms) to take advantage of time, and also having different questions for each round. It would be more beneficial to present the results on a histogram.

For Further Information:

Delphi Exercise folder

The Delphi Technique: Making Sense Of Consensus

Summary

The Delphi technique is a common method used to gather data from a variety of different professional fields, and is designed to achieve a convergence of opinion on a specific issue through group communication. This technique is well suited as a method for consensus-building by using questionnaires delivered by multiple iterations to collect data from participants. Participant selection, time frames for conducting and completing the study, the possibility of low response rates, and unintentionally guiding feedback from respondent groups are issues that need to be accounted for when using the Delphi method.

The Delphi technique was originally designed by Dalkey and Helmer at the Rand Corporation in the 1950s in order to solicit data on specific issues for the purpose of goal setting, policy investigation, or predicting the occurrence of future events. According to Delbecq, Van de Ven, and Gustafason (1975), the Delphi technique can be used to achieve the following objectives: 

1.      Determine or develop a range of possible program alternatives.

2.      Explore or expose underlying assumptions or information leading to different judgments.

3.      Seek out information which may generate a consensus on the part of the respondent group.

4.      Correlate informed judgments on a topic spanning a wide range of disciplines.

5.      Educate the respondent group as to the diverse and interrelated aspects of the topic.

Unlike other data gathering techniques, the Delphi technique utilizes multiple iterations (feedback processes) designed to develop a consensus on a specific topic. The feedback process allows and encourages all participants to reassess their initial judgments about the information provided in previous iterations. Other characteristics of the technique is the ability to provide anonymity to respondents, a controlled feedback process, and the suitability of a variety of statistical analysis techniques to interpret the data.

1.      Anonymity is one of the strongest characteristic and advantage of the technique, as it prevents stronger personalities from drowning out timid individuals, and prevents coercion and manipulation.

2.      Controlled feedback is designed to reduce the effect of common communication that occurs during group discussions that derails the original purpose of the discussion.

3.      Statistical analysis techniques ensures each opinion is represented equally after each iteration, and reduces pressure to succumb to group conformity.

The Delphi Process

The Delphi technique can be employed until a consensus is reached, however, several researchers indicate 3 iterations are sufficient to collect information and reach consensus in most scenarios. The following guidelines show 4 rounds in order to present an example of the process when more data is needed. 

Round 1: The technique begins with an open-ended questionnaire that serves as the cornerstone of gathering information about a specific topic from the participants. After the investigators (facilitators) collect each participant’s responses, a well-structured questionnaire is created. The well-structured questionnaire is used as the survey in round 2.

Round 2: Each participant receives a second questionnaire, and reviews the topics summarized by the investigators. The summarizations are based off the information the participants provided in the first round. The participants also rate or order each topic in order to establish priorities. The process of establishing priorities identifies agreements or disagreements among the participants. 

Round 3: Each participant receives a third questionnaire that includes all the topics and ratings summarized from the previous round. The participants are asked to revise their judgements, or specify why they won’t change their answers. 

Round 4: In the final round, each participant receives another questionnaire with the remaining topics, ratings, minority opinions and topics that reached a consensus. 

The most important process of the Delphi technique is selecting participants. According to various researchers, the three groups most qualified to be participants in a Delphi method are:

1: Top management decision makers who will utilize the outcomes of the Delphi study.

2: Professional staff members together with their support team.

3: Respondents to the Delphi questionnaire whose judgements are being sought.      

Critique

Though the Delphi technique is used regularly in various professional settings, it seems to be extremely situational. One of the many elements that are needed to ensure the Delphi technique is employed properly is time. Depending on the number of the participants and the data being evaluated, it may take several days or even weeks to complete the technique. Other methodologies can be used in place of the Delphi technique that would likely result in the same quality of information, but with less time. The technique also has other potential shortcomings, like low response rates, which have to be accounted for. Striking a balance between the requested data, number of participants and timeframe, appears to be key element when deciding to utilize the methodology.

      Source:

Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique: Making Sense of Consensus. Practical Assessment Research & Evaluation, 12(10). Available online: http://pareonline.net/getvn.asp?v=12&n=10

The Delphi Method for Graduate Research

Gregory J. Skulmoski, Francis T. Hartman, and Jennifer Krahn (2007)

Summary:

The authors of this article for the Journal of Information Technology Education are from Zayed University in Dubai and the University of Calgary, Canada.  Their approach is to provide graduate students with the knowledge necessary to employ the Delphi method in their own academic research, be it for a thesis or dissertation.  The primary scope of the piece is based around the Information Systems (IS) field as well as the field of Information Technology (IT), but they believe that as a research technique the Delphi method can be used in a wide variety of fields, not just their own.  This is because they view the Delphi method as a “flexible research technique” and define it as follows:

The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback.

The authors also put forth that that this method is most suitable when “the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts.” 

In order to improve the reader’s understanding of the methodology there is a brief historical overview of what the authors refer to as “Classical Delphi”.  This is the methodology developed by Norman Dalkey for the RAND Corporation in the 1950’s.  In the Delphi method a panel of experts is given a survey with questions to answer.  After they have returned the survey, a second survey is sent out based on the results of the first, this proceeds over a pre-determined number of rounds. Following the completion of the last survey analysis of the final results is conducted.  Classical Delphi is defined by four key features:  

1. Anonymity of Delphi participants: participants are freed from social and professional pressures by the use of anonymous surveys.  Those who are regarded as greats in the field will be judged solely on their answers and not by their reputations.  This also frees participants to think outside traditional lines without fear of reprisal.
2. Iteration: participants are given the opportunity to develop their own understanding and beliefs as the study progresses.
3. Controlled feedback: as rounds of the Delphi progress, participants will be informed of the anonymous perspectives of other participants.  This is one the core advantages of the Delphi method, that the group as a whole will generate a broad array of ideas to begin with and zero in on the best ones as times moves on 
4. Statistical aggregation of group response: this allows for statistical analysis of results.

This “Classical Delphi” model has been adapted by many in the decades following its creation, this has made it more widely applicable and more adaptive to diverse requirements.  Below is a visualization of a more modern take on the model which has been used in some of the authors’ graduate students’ projects:

One of the features of the paper is the design considerations that should be taken into account while utilizing the Delphi method in graduate research.  The authors discuss pros and cons of:  

• Methodological Choices
• The Broadness of the Initial Question
• Criteria for who is to be Considered an Expert
• The Number of Participants
•  The Number of Rounds
• The Mode of Interaction with Participants
• Methodological Rigor
• The Results
•  Further Verification
•  Publication

The authors close with what they consider two important points, “First, the Delphi approach can be aggressively and creatively adapted to a particular situation. Second, when adapting the approach, there is a need to balance validity with innovation. In other words, the greater the departure from classical Delphi, the more likely it is that the researcher will want to validate the results, by triangulation, with another research approach.”

Critique:

This article provides a lot of good information for graduate students who may be unfamiliar with the Delphi method.  It is specifically focused on how they can get out and use the methodology without too much difficulty, and what they need to be thinking about as they do it.  One of the largest limitations of the piece in my opinion is that spends too little time discussing the situations in which the method is not appropriate.  The article goes to great length to explain just how adaptable the methodology is, but even in the Executive Summary it states Delphi, “ is not a method for all types of IS research questions.”  The article raises this concern but rarely returns to it.  This as a result makes it feel like the use of the Delphi method is a foregone conclusion and less that it is just one more valuable tool in your analytic toolbox.  

A Delphi Consensus Approach to Challenging Case Scenarios in Moderate-to-Severe Psoriasis

Summary:

Psoriasis is a condition in which skin cells build up and form scales and itchy, dry patches.  It is a challenging condition to treat considering the lack of literature, the simultaneous presence of 2 chronic diseases in patients, and in its difficulty to diagnose.  A consensus panel of 14 experts in the psoriasis field was formed to use a Delphi method exercise for the purpose of identifying challenging clinical scenarios and to rank treatment approaches, in an effort to provide guidance to the practicing clinician.  The Delphi method is well suited to address healthcare-related issues since the outcome is the representation of the collective judgment of the panel of experts.  The 3 basic characteristics of the Delphi method include:

1.       Repeated individual questioning of the experts

2.       The avoidance of direct confrontation among the experts

3.       Interspersed controlled opinion and feedback\

The Delphi method works to achieve a consensus on complex scenarios where rigorous data is lacking.  The panelists extensively review all available data before presenting and discussing it.  One of the most important aspects is the use of anonymous voting by the panelists as it eliminates the effects of reputation in order to settle controversy.  The anonymity also allows panelists to vote honestly, thus avoiding “groupthink” and as well as any following of charismatic panelists and dogmatism.  Delphi is applied in 3 steps over about 5 months to difficult-to-treat clinical scenarios in patients with moderate-to-severe psoriasis.  The steps are:

1.       Selection of difficult-to-treat psoriasis clinical scenarios;

2.       Selection of potential psoriasis treatment

3.       The matching, through systematic, iterative rounds of voting of clinical scenarios with the most appropriate treatments based on data assessment of peer-reviewed literature.

Once 14 psoriasis experts from the U.S. were identified, each individual panelist was asked to list challenging clinical scenarios and therapeutic options for psoriasis.  The scenarios were then selected and ranked, and the treatment options were listed.  The panelists discussed 24 of the top-ranked scenarios during a live meeting and they voted and ranked the treatment choices for each.  The article presents 5 of the 24 discussed case scenarios.  The Delphi exercise resulted in guidelines for practicing physicians to use when confronted with patients with challenging cases of psoriasis.

Critique:

While the Delphi method is well suited to address healthcare-related issues as the panel of experts select rational treatment choices for each of their discussed scenarios, their solutions are not yet supported by rigorous studies to back up their conclusions as well as the effectiveness of Delphi.  Delphi has potential limitations with conflicting interests among the panelists and their experiences and backgrounds.  Additionally, the experts were only chosen from the U.S. along with treatment options based on what is locally available in the U.S. so conclusions may not be relevant world-wide.  Nonetheless, Delphi’s use of anonymity provides an unbiased view of available clinical data which leads to a more objective consensus in accomplishing the goal.

Source: 
"A Delphi Consensus Approach to Challenging Case Scenarios in Moderate-to-Severe Psoriasis: Part 2" 
By: Bruce E. Strober, Jennifer Clay Cather, David Cohen, Jeffrey J. Crowley, Kenneth B. Gordon, Alice B. Gottlieb, Arthur F. Kavanaugh, Neil J. Korman, Gerald G. Krueger, Craig L. Leonardi, Sergio Schwartzman, Jeffrey M. Sobell, Gary E. Solomon, and Melodie Young
http://link.springer.com/article/10.1007%2Fs13555-012-0002-x

The Delphi method: a powerful tool for strategic management

Summary

The Delphi method structures and facilitates group communication that focus, upon a complex problem so that, over a series of iterations, a group consensus can be achieved about some future direction. It has five major characteristics:

1. The sample consists of a "panel" of carefully selected experts representing a broad spectrum of opinion on the topic or issue being examined. 
2. Participants are usually anonymous. 
3. The "moderator" (i.e. researcher) constructs a series of structured questionnaires and feedback reports for the panel over the course of the Delphi.
4. It is an iterative process often involving three to four iterations or “rounds” of questionnaires and feedback reports.       
5. There is an output typically in form of a research report with the Delphi results, the forecasts, policy and program options with their strengths and weaknesses, recommendations to senior management and, possibly, action plans for developing and implementing the policies and programs.

        It has some advantages over some other group decision-making techniques like the nominal group technique (NGT) and interacting group method (IGM). First, panel members are not swayed by group pressures or vocal members as can easily happen with NGT and IGM. Second, interpersonal conflicts and communication problems are virtually nonexistent because panel members do not interact. Third, travel costs and the problem of coordination to get everyone at the same place at the same time are not factors. The Delphi method consists of four key planning and execution activities: 

1.       Problem definition: It is necessary to define the nature and scope of the problem, expected outcomes of the study and the appropriateness of the Delphi method.
2.       Panel Selection: The Delphi method requires a panel of subject-matter experts (SMEs). The criteria for determining who qualifies as a SME may rest not only on knowledge, but could include criteria such as personal experiences or being stakeholders. Besides, it is important to inform prospective panel members that their commitment to participate would involve several rounds of questionnaires and feedback possibly extending over a period of months. On the hand, Panel selection might not be random because, in some research fields, there might be very few SMEs; thus, one might select all known SMEs.
3.       Determining the panel size: While there is no one sample size advocated for Delphi studies, rules-of-thumb suggest that 15-30 carefully selected SMEs could be used for a heterogeneous population and as few as five to ten for a homogeneous population. The careful selection of SMEs is a key factor in the Delphi method that enables a researcher confidently to use a small panel. That is not to say that large panels are never used. Indeed, some Delphi studies have used large panels numbering over 100 members.
4.       Conducting the Delphi rounds: A Delphi study usually involves three to four rounds or iterations, not just a one-time effort; thus, the moderator is able to set up Round 1 according to some strategy knowing that another two to three rounds could be conducted to achieve consensus or other goals. Although three to four rounds are typically used, the moderator should stop the rounds when the criteria for consensus are achieved, when results become repetitive, or when an impasse is reached. Following the final round, the moderator prepares a comprehensive report and distributes it or a short version to all members.              

The Delphi method deserves serious consideration because the careful design and execution of a Delphi study should lead to useful findings for policy makers and program managers. However, it is recommended that researchers consider using a triangulation of methods rather than the reliance upon a single method. For many situations, researchers may find the combination of a Delphi study and a survey with two independent samples useful and practical.

      

        Critique

        This paper describes the characteristics of the Delphi method, including criticisms of the method, and steps in conducting a Delphi study as well as pitfalls to avoid. It is a good overview in order to understand the basics of the method. However, the author does not mention an important weakness of the Delphi method. From my point of view, this method is heavily contingent upon the researcher’s ability. If the researcher have enough knowledge and ability to conduct this method, it can produce valuable results. Otherwise, it is just a fruitless time-consumption.   

      

        Source:

        Robert Loo, (2002) "The Delphi method: a powerful tool for strategic management", Policing: An International Journal of Police Strategies & Management, Vol. 25 Iss: 4, pp.762 – 769. Retrieved from http://search.proquest.com.ezproxy.mercyhurst.edu/docview/211297480/A90AEE4047D44C01PQ/1?accountid=27687

The Selection of Delphi Panels for Strategic Planning Purposes

John F. Preble (1984)

Summary:

Preble develops and answers three main questions in this article:

1. Do results obtained using an intracompany Delphi panel tend to differ significantly from those obtained using an intercompany Delphi panel?
2. Are the forecasts generated consistent?
3. If the results are consistent, which panel type is recommended?

Previous literature on the Delphi panels, particularly Martino (1972) and Johnson (1976), recommend using a panel of experts from outside the organization. These studies operated on the implicit assumption that external panelists were likely to be better qualified or more expert than any panelists within the organization, despite a lack of evidence supporting this assumption.

The author composed two 15-member Delphi panels which consisted of top-level employees from large, successful life insurance firms headquartered in the north-eastern United States. The positions represented included Legal, Public Relations, Human Research, and other such diverse roles. The intracompany panel was composed of 15 members from the same company, while the intercompany panel consisted of 15 members from 15 different companies; three participants dropped out midway through the study, leaving 14 and 12 panelists, respectively. The panelists were asked to provide estimates as to the likelihood and timing of 27 different events, dated 1985, 1990, 1995, and 2000, and provide their degree of familiarity with each event. The author provided three rounds of questionnaires to the panelists, including statistical feedback after the first round and qualitative reasoning behind outliers after the second round.

After collecting the data, the author conducted t-tests to determine statistically significant differences. Seventy-six percent of the t-tests were not significant, meaning that in most cases the forecast from the intracompany was “quite close” to the corresponding intercompany estimate. Considering estimates categorized as unlikely (0.0-2.49), slightly likely (2.50-4.99), likely (5.00-7.49), or very likely (7.50-10.0), 95 of 96 comparisons were either in the same category or the next closest category (see Figure 1).  These results show that intracompany panel estimates are about the same as intercompany panel estimates. Because these estimates are consistent, Preble recommends that intracompany panels be used by strategic planners in order to increase administrative control, decrease the number of dropouts and overall costs, and satisfy the need for confidentiality of proprietary information.

Figure 1 - Mean scores and classification comparisons

Critique:

Methodologically, the only significant change is that a few female members should have been included in the intercompany Delphi panel; otherwise, the panel demographics are very similar (see Figure 2). 

Figure 2 - Panel demographics

One of the downsides to this method is that it took four months to complete the three rounds of the Delphi, therefore, this method could be overlooked under tight deadlines. This method could also drift dangerously close to group-think. While anonymity prevents panelists from being dominated by stronger personalities, the opportunity for panelists to change their responses in the second round based on the statistical data of the first round could eliminate any significant dissent. While the goal of Delphi is to converge on a central estimate, the outliers may have unique insights or experiences that are not reflected in statistical syntheses, but only come through in the qualitative comments.

Source:

Preble, J.F. (1984). The Selection of Delphi Panels for Strategic Planning Purposes. Strategic Management Journal, 5(2), 157-170.

EBSCOhost Mercyhurst access link