Green Team Summary of Findings: Delphi Analysis (3 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 16 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 22 April 2010 regarding Delphi Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

DESCRIPTION

The Delphi method is based on a structured process for collecting and distilling knowledge from a group of experts by means of a series of questionnaires combined with controlled opinion feed back. It was developed in the early 1950s, and mainly applied to technology forecasting, but also to many types of policy analysis.

Delphi focuses on researching the future or things about which little is known. It relies on the use of expert opinion, utilizes remote group processes, adopts an iterative research process, and creates a consensus of opinion.

There are three versions of the Delphi Technique:

• Numeric – aims to specify a single or minimum range of numeric estimates through the use of summary statistics.

• Policy - on the exploration, generation and definition of several alternatives and the arguments for and against each of these alternatives.

• Historic - aims to explain the range of issues that fostered a specific decision, identification of several scenarios that could have led to the resolution of a past problem.

Delphi is particularly appropriate when decision-making is required in a political or emotional environment. The tool works formally or informally, in large or small contexts.

STRENGTHS AND WEAKNESSES

Strengths:

• Its ability to ascertain expert opinions and potential policy options

• Allows a group response without the attendant disadvantages sometimes experienced with group problem solving or decision-making.

• Expert participants are more likely to generate reasoned, independent, and well-considered opinions in the absence of exposure to the "persuasively stated opinions of others". Because the experts do not ever participate in a face-to-face discussion, there is no danger of one or more individuals’ opinions being swayed by a more dominant or more experienced individual.

• Efficiency and flexibility, especially in light of modern communication technologies such as e-mail and the Internet. Experts may be drawn from a wide geographic area, and the participants’ commitment in terms of time and money invested is minimal.

• Delphi method has been shown to be an effective way to conduct research when the responses being sought are value judgments rather than factual information. Although it is more difficult to assess the "correctness" of value judgments, it is generally agreed upon that value judgments are not all equal but can in fact be more "right" or more "wrong."

Weaknesses:

• A low level reliability of judgments among experts

• Dependency of forecasts on the particular judges selected

• Should not be used when any of the following three critical conditions are not present: adequate time, participant skill in written communication, and high participant motivation. It is estimated that a minimum of 45 days is required to carry out a Delphi study

• A high degree of motivation is needed to offset the tendency for participant dropout as the study progresses. Because there is no direct contact between participants, those who are not highly motivated and interested in the subject at hand may feel isolated or detached from the process (It is easy for participants to drop out of the process)

• There is a strong response of the group to conform with the statistical feedback of the panel, extreme points of view, which may provide new insights tend to be suppressed

• The way the questionnaire and process is structured can lead to a bias and a Delphi study is at the mercy of the view and biases of the coordinating team, who choose respondents, interpret the information and structure the question

• Participants must be knowledgeable and able to clearly communicate their ideas

• Sensitivity of the results to ambiguity in the questionnaire that is used for

• Difficulty in assessing the degree of expertise incorporated in the forecast

• Good responses can be altered by those without a strong belief in the participants' answers

HOW TO

There are many ways in which to conduct a Delphi exercise depending on the desired outcome. However, from the research several consistent trends emerge. They are as follows:

• A questionnaire is sent to experts.

• Each expert gives his answers to the questions in an independent and anonymous way.

• The responses of each expert are analyzed by the monitoring team. The moderator summarizes the responses to the first questionnaire and develops a feedback report along with the second set of questionnaires for the panelists

• The set of responses is then sent back to experts and they are asked if they wish to revise the initial predictions.

• The process is reiterated until a degree of consensus is reached by experts. Generally the number of rounds is determined in advance.

• The moderator develops a final summary and feedback report to the group and decision makers

PERSONAL APPLICATION

We started with the a question: "Of the 21 first year graduate students, how many will graduate on May 21, 2011 having successfully completed their thesis or comprehensive exam?" Each person wrote out their answer without discussion and handed it in. While one person found the average, the facilitator introduced data regarding the thesis completion rate from 2005 to 2009. The average was written on the board and then a one minute discussion regarding the trend of incompletes. Then the group wrote out their next answer and another set of calculations were done. Another one minute discussion was introduced and then a third round of responses. A final discussion noted how/if the process changed answers due to the introduction of facts or consensus from the "experts."