Summary of Findings: Delphi Technique (4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 5 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2014  regarding Delphi Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

The Delphi technique is a method that relies on expert and group knowledge to make more accurate forecasts using incomplete information.  The individual forecasts are compiled after a series of rounds.  Then, the individuals’ responses are anonymized and dispersed to the remainder of the group for consideration, and new individuals forecasts are given.  

The RAND corporation created the Delphi technique in order to support accurate decision making in the face of incomplete information.  There is a substantial amount of research on the validity of the Delphi technique dating back to its creation in the 1950s, but the methodologies scholars have used to test Delphi’s effectiveness have varied in almost every study.  

Strengths:

1. Conducted in writing or electronically and does not require face-to-face meetings

2. Helps generate consensus or identify divergence of opinions among group members

3. Participants are relatively free of social pressure, influence, and dominance from other group members

4. Anonymous responses allow respondents to keep opinion until they are comfortable changing an estimate

5. Is inexpensive

Weaknesses:

1.  Time for answers may not be given to the problem and consensus may not be obtained

2. Participants may ignore feedback

3. Experts may not be defined among the group

4. Requires adequate time and participant commitment

5. More time consuming than other group methods

6.Broad guidelines-- there are at least 27 different ways to conduct the method

Step by Step:  

1. Use a group of 5-20 heterogeneous experts or people with appropriate knowledge of the subject.
2. The entire process must use a systematic process, particularly with anonymous feedback and a controlled method of dispersing responses and feedback.
3. A minimum of three iterations should be conducted with polling continuing until there is a stability in responses.

Exercise:

We used Delphi Decision Aid online software to conduct three 5 minute rounds of Delphi to forecast how many second year Applied Intelligence graduate students will have at least one full time job offer in an intelligence-related field by graduation and how many pages second year Applied Intelligence will have completed on average by October 29 toward a thesis. The first round also contained a ranking question to rank panel expertise on various topics to inform further Delphi questions for subsequent rounds. Subsequent rounds asked the two original questions in addition to predicting the outcome of the National Football League AFC division this session, how many selfies Kim Kardashian will have in her book scheduled for publishing in April 2015, and what the S&P 500 index will be in early November. After each round, the panel had a few minutes to review the feedback of the round through statistical aggregation of responses and written comments explaining why a panelist made the estimate that they did. 

What did we learn from the Delphi Exercise

1. Delphi works well with broad questions where the expertise of one person is not sufficient to encompass the entire scope of the question.
2. Literature suggests that panelists tend to perform poorly on questions asking them to rank various items from best to worst and that self-reported expertise is not a best practice for panel selection.
3. Delphi is designed to collect expert estimates  in cases where a variety of relevant factors (economic, technical, etc.) ensure that individual panelists have limited knowledge and could reasonably benefit from communicating with other experts possessing different information.   
4. Estimates from panelists do not have to be quantitative such as in prediction markets.

Additional Resources Of Interest:

Delphi Decision Aid