The Delphi method: An experimental study of group opinion

Summary:

The Delphi method aims to elicit and refine forecasts within a group.  Its logical foundation is the assumption that group knowledge is better than an individual’s when a judgment is required with incomplete information.  In the spring and summer of 1968, RAND conducted a series of experiments to see how the Delphi method affects decision-making.  During the time of the study, the Delphi method was comprised of three components: anonymous responses, iteration and controlled feedback, and statistical group response.  Anonymous responses allow complete honesty without the fear of retaliation.  Iteration and controlled feedback help to establish a systematic process so that other groups may repeat it.  Finally, statistical group responses refer to the aggregate judgment or opinion of the group, which is intended to minimize the error caused by biased or motivated judgments.

Using upperclassmen and graduate student from UCLA, 14 groups ranging from 11 to 30 members answered 350 questions combined in 10 experiments, resulting in about 13,000 answers.  RAND examined whether face-to-face discussion or the traditional systematic, controlled feedback generates judgments that are more accurate.  The results indicated that the traditional systematic, controlled feedback generated estimates that are more accurate.

In their briefing, RAND goes on to discussion some of the implications of their findings.  First, the traditional Delphi method, which happens to be more accurate, also provides quantitative data that may be examined for insight into the group interactions and improves the techniques validity.  Second, the addition of rating confidence and understanding of the question can help to provide accuracy measurements of the Delphi-generated assessment.

Critique:

The Delphi method is most beneficial when there is anonymous, systematic feedback.  However, the creation of controlled responses and procedures is far more time-consuming than face-to-face feedback.  Time is a valuable and rare resource in the intelligence community.  If time were more available, creating Delphi procedures for estimates would not only enhance accuracy, but it would also identify where in the analysis errors are made through quantitative data.  Consequently, themes in the analytical errors may emerge (i.e. most analytical errors occur when making economic forecasts of Baltic states).  Due to the time required for a Delphi session, it is reserved for intelligence products that do not have an immediate implication.  Perhaps strategic products would best, rather than tactical or operational.

Source:

Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinion (No. RM-5888-PR) (pp. 1–79). Santa Monica, CA: RAND. Retrieved from http://www.rand.org/content/dam/rand/pubs/research_memoranda/RM5888/RM5888.pdf