Wednesday, October 22, 2014

The Delphi Technique

By: National Security Agency

In December 2011, the NSA released a document explaining how the Delphi technique could minimize the subjective elements in the decision making process. The document went on to explain the technique and the weaknesses it presents within the intelligence community.  The Delphi technique was developed to make the “most effective use of informed intuitive judgment by creating conditions under which a group of experts can perform most ably and their answers can be combined into a single group opinion.”

This technique utilizes anonymous responses, controlled feedback, and a statistical derivation of responses.  Participants are questioned individually to avoid problems that face-to-face conversations can have such as: dominating personalities, group members fearing abandoning the overall public opinion, groups making lots of noise that do not contribute to the issue at hand, the bandwagon effect, and the idea that experts are hard to get together in one place at one time. According to RAND, anonymous responses do produce better results than face-to-face group discussions.

After responding to the first set of questions, participants are asked to respond again using the information provided by the other responses. This cycle can continue for several iterations. Participants are sometimes allowed to ask further questions or give reasons for additional data. By utilizing this technique, participants can produce new ideas which are then disseminated among the group and respondents can get a feel for the majority’s opinion. This technique aggregates every opinion to make sure it is contributing to the group’s final opinion. In addition to the normal Delphi technique, the document suggests to rank experts and weigh their opinions.

The article also goes on to conclude that the Delphi technique needs to have the right type of experts to the answer questions. The intelligence community has two issues regarding this, first: it is not normal for experts using the Delphi technique to carry technical data to back up their judgments; and second, the structure of the intelligence community suggests that “persons with the requisite technical knowledge are usually not on a level to make judgments, while those at the appropriate levels often must depend on others for data.” Instead, the intelligence community is made up of specialists that may not be capable of judging all facets of a complex problem.

In the field of intelligence, validating analytical techniques such as the Delphi technique is significant. It can help find the consensus of the community which can help determine policy decisions. In addition to the weaknesses stated above, the Delphi technique can also ignore facts and cause bias. Although the concept of “two heads are better than one” makes logical sense, the idea that the coordinator can alter the feedback to cause bias makes this technique very subjective.


National Security Agency. (2011). The Delphi Technique (NSA DOCID: 3928741).


  1. Joy,

    A modifier the article suggests is ranking experts and weighing their judgments. What are your thoughts on this modifier, does it make the technique more robust or does it take away from the intended use of Delphi?

  2. Ricardo,

    I think it would depend on the situation. Ranking experts and weighing judgments could cause the technique to produce an entirely different answer, which leads to biases.