Thursday, October 23, 2014

The Delphi method: An experimental study of group opinion



Summary:
The Delphi method aims to elicit and refine forecasts within a group.  Its logical foundation is the assumption that group knowledge is better than an individual’s when a judgment is required with incomplete information.  In the spring and summer of 1968, RAND conducted a series of experiments to see how the Delphi method affects decision-making.  During the time of the study, the Delphi method was comprised of three components: anonymous responses, iteration and controlled feedback, and statistical group response.  Anonymous responses allow complete honesty without the fear of retaliation.  Iteration and controlled feedback help to establish a systematic process so that other groups may repeat it.  Finally, statistical group responses refer to the aggregate judgment or opinion of the group, which is intended to minimize the error caused by biased or motivated judgments.

Using upperclassmen and graduate student from UCLA, 14 groups ranging from 11 to 30 members answered 350 questions combined in 10 experiments, resulting in about 13,000 answers.  RAND examined whether face-to-face discussion or the traditional systematic, controlled feedback generates judgments that are more accurate.  The results indicated that the traditional systematic, controlled feedback generated estimates that are more accurate.

In their briefing, RAND goes on to discussion some of the implications of their findings.  First, the traditional Delphi method, which happens to be more accurate, also provides quantitative data that may be examined for insight into the group interactions and improves the techniques validity.  Second, the addition of rating confidence and understanding of the question can help to provide accuracy measurements of the Delphi-generated assessment.

Critique:
The Delphi method is most beneficial when there is anonymous, systematic feedback.  However, the creation of controlled responses and procedures is far more time-consuming than face-to-face feedback.  Time is a valuable and rare resource in the intelligence community.  If time were more available, creating Delphi procedures for estimates would not only enhance accuracy, but it would also identify where in the analysis errors are made through quantitative data.  Consequently, themes in the analytical errors may emerge (i.e. most analytical errors occur when making economic forecasts of Baltic states).  Due to the time required for a Delphi session, it is reserved for intelligence products that do not have an immediate implication.  Perhaps strategic products would best, rather than tactical or operational.

Source:
Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinion (No. RM-5888-PR) (pp. 1–79). Santa Monica, CA: RAND. Retrieved from http://www.rand.org/content/dam/rand/pubs/research_memoranda/RM5888/RM5888.pdf

4 comments:

  1. Kyle,

    What are the key elements of systemic feedback?

    ReplyDelete
    Replies
    1. I found only two key elements. 1) the feedback must be controlled, meaning the respondent can only choose from a pre-created set of answers; or the respondent can only give an answer within a set of guidelines. 2) There must be iterations of feedback.

      Delete
  2. Kyle,

    While anonymity may seem like a advantage as it can help overcome biases towards individuals ideas, there are some disadvantages as well. For example, we have worked together enough where I have an understanding of some of the things you say or the meaning you have behind them based off our work experience. Do you feel that the benefits of Delphi outweigh the negatives of having that work experience with colleagues?

    ReplyDelete
    Replies
    1. Good point to bring up. If the Delphi feedback is not anonymous, the session is not systematic. Rather, it's more of the face-to-face interaction type that is not as effective as the systematic alternative. So while your point is valid, it contains negatives as well. For instance, you may give more weight to what I'm saying about a particular subject based on our previous experience or what you know about my education, but I could still be wrong. On the flip-side, I could be really nervous giving my critique and may not seem as reliable, when in fact I may be highly accurate.

      Delete