Expert Opinions in Forecasting: The Role of the Delphi Technique

Rowe and Wright in this paper discuss principles that must be applied in forming Delphi groups. Rowe and Wright argue that the application of these principles create a substantially more accurate estimate than one that would come from an individual. These are the following principles:

1.      Use experts with appropriate knowledge
Experts will resist changing their estimates unless valuable evidence is presented to them. The risk of using non-experts, or subjects with a limited knowledge base, is that the final estimate’s accuracy might be no greater than the first round due because they may be more willing to change their responses.

2.      Use heterogeneous experts
The combining of judgments increases the reliability of an aggregate estimate. Interaction between experts is likely to fill knowledge gaps which will be beneficial in developing a final estimate.

3.      Use between 5 and 20 experts
The size of the group will depend on the number of experts available. In a study done by Brockhoff (1975), Delphi groups of five, seven, nine, and 11 showed no difference accuracy.

4.     For Delphi feedback, one must provide experts the mean or median of the estimates plus the rationales for the estimate
The restriction of the flow of information denies collaboration and the support for estimates made. With a reason, an expert is likely to stick with their estimate than change it.

5.      Continue Delphi polling until responses show stability (Three rounds)
Erffmeyer, Erffmeyer and Lane (1986) found that the quality of Delphi estimates increased up to the fourth round but not after.  Additonally, Brockhoff (1975) found that the accuracy of estimates increased up to round three and decreased with each additional round.

6.      Weight all experts’ estimates equally in developing the final forecast

7.      In phrasing questions, use clear and succinct definition and avoid emotive terms

8.      Frame questions in a balanced manner
Consistently state both outcomes of a response so that each expert is aware of the meaning of their responses.

9.      Avoid including irrelevant information in questions

Critique
How is determined if someone is an expert at something? Stating that one must find heterogeneous experts with a similar knowledge based sounds contradictory. For Delphi to be successful, participants must possess appropriate knowledge. How does one have a differing expertise with the same knowledge base?

Additionally, there is a great degree of variation between a group of five and a group of 20. Although studies found know statistical difference between groups of five and 11, there were no studies cited that examined the accuracy of estimates between groups of five and 20.

Source:
Rowe, G. & Wright, G. Expert opinions in forecasting: The role of the Delphi technique. Principles of forecasting, 125-144.

The Delphi method: An experimental study of group opinion

Summary:

The Delphi method aims to elicit and refine forecasts within a group.  Its logical foundation is the assumption that group knowledge is better than an individual’s when a judgment is required with incomplete information.  In the spring and summer of 1968, RAND conducted a series of experiments to see how the Delphi method affects decision-making.  During the time of the study, the Delphi method was comprised of three components: anonymous responses, iteration and controlled feedback, and statistical group response.  Anonymous responses allow complete honesty without the fear of retaliation.  Iteration and controlled feedback help to establish a systematic process so that other groups may repeat it.  Finally, statistical group responses refer to the aggregate judgment or opinion of the group, which is intended to minimize the error caused by biased or motivated judgments.

Using upperclassmen and graduate student from UCLA, 14 groups ranging from 11 to 30 members answered 350 questions combined in 10 experiments, resulting in about 13,000 answers.  RAND examined whether face-to-face discussion or the traditional systematic, controlled feedback generates judgments that are more accurate.  The results indicated that the traditional systematic, controlled feedback generated estimates that are more accurate.

In their briefing, RAND goes on to discussion some of the implications of their findings.  First, the traditional Delphi method, which happens to be more accurate, also provides quantitative data that may be examined for insight into the group interactions and improves the techniques validity.  Second, the addition of rating confidence and understanding of the question can help to provide accuracy measurements of the Delphi-generated assessment.

Critique:

The Delphi method is most beneficial when there is anonymous, systematic feedback.  However, the creation of controlled responses and procedures is far more time-consuming than face-to-face feedback.  Time is a valuable and rare resource in the intelligence community.  If time were more available, creating Delphi procedures for estimates would not only enhance accuracy, but it would also identify where in the analysis errors are made through quantitative data.  Consequently, themes in the analytical errors may emerge (i.e. most analytical errors occur when making economic forecasts of Baltic states).  Due to the time required for a Delphi session, it is reserved for intelligence products that do not have an immediate implication.  Perhaps strategic products would best, rather than tactical or operational.

Source:

Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinion (No. RM-5888-PR) (pp. 1–79). Santa Monica, CA: RAND. Retrieved from http://www.rand.org/content/dam/rand/pubs/research_memoranda/RM5888/RM5888.pdf

The Delphi Technique

By: National Security Agency

Summary:

In December 2011, the NSA released a document explaining how the Delphi technique could minimize the subjective elements in the decision making process. The document went on to explain the technique and the weaknesses it presents within the intelligence community.  The Delphi technique was developed to make the “most effective use of informed intuitive judgment by creating conditions under which a group of experts can perform most ably and their answers can be combined into a single group opinion.”

This technique utilizes anonymous responses, controlled feedback, and a statistical derivation of responses.  Participants are questioned individually to avoid problems that face-to-face conversations can have such as: dominating personalities, group members fearing abandoning the overall public opinion, groups making lots of noise that do not contribute to the issue at hand, the bandwagon effect, and the idea that experts are hard to get together in one place at one time. According to RAND, anonymous responses do produce better results than face-to-face group discussions.

After responding to the first set of questions, participants are asked to respond again using the information provided by the other responses. This cycle can continue for several iterations. Participants are sometimes allowed to ask further questions or give reasons for additional data. By utilizing this technique, participants can produce new ideas which are then disseminated among the group and respondents can get a feel for the majority’s opinion. This technique aggregates every opinion to make sure it is contributing to the group’s final opinion. In addition to the normal Delphi technique, the document suggests to rank experts and weigh their opinions.

The article also goes on to conclude that the Delphi technique needs to have the right type of experts to the answer questions. The intelligence community has two issues regarding this, first: it is not normal for experts using the Delphi technique to carry technical data to back up their judgments; and second, the structure of the intelligence community suggests that “persons with the requisite technical knowledge are usually not on a level to make judgments, while those at the appropriate levels often must depend on others for data.” Instead, the intelligence community is made up of specialists that may not be capable of judging all facets of a complex problem.

Critique:

In the field of intelligence, validating analytical techniques such as the Delphi technique is significant. It can help find the consensus of the community which can help determine policy decisions. In addition to the weaknesses stated above, the Delphi technique can also ignore facts and cause bias. Although the concept of “two heads are better than one” makes logical sense, the idea that the coordinator can alter the feedback to cause bias makes this technique very subjective.

Source:

National Security Agency. (2011). The Delphi Technique (NSA DOCID: 3928741). https://www.nsa.gov/public_info/_files/cryptologic_spectrum/The_Delphi_Technique.pdf

The effectiveness of Nominal, Delphi, and Interacting group decision making processes

By: Andrew H. Van de Ven and Andrew L. Delbecq

Summary:
In their 1974 research, Van de Ven and Delbecq compared the effectiveness of nominal group technique (NGT), delphi technique, and traditional interacting group technique to identify which performed best in the decision making process.

NGT follows a process of first allowing members to silently, and on their own, generate a list of ideas and thoughts on how to answer the proposed problem.  Then, all of the members list their proposed solutions to the rest of the team.  All the responses are recorded and written out for the entire team to read.  Once all of the proposed solutions have been recorded, the team discusses each solution for its validity and to clarify what is meant by each solution.  Finally, the group independently votes on the solutions and the group's decision is derived from the outcome of the vote.

Interacting group technique is the widely used method of having team members have an unstructured discussion of the problem.  Participants propose solutions the group actively debates what the best course of action is.

Delphi allows for decision making when team members are not physically in the same location. The process used by Van de Ven and Delbecq began by issuing questionnaires by mail (today could be electronically) to participants who did not know the identify of any of the other participants.  Once the questionnaires were returned, the ideas generated where recorded and sent out in a second questionnaire that dove more in depth into the ideas.  Respondents voted on different solutions and returned the questionnaire.

Their study was conducted at a Midwestern university were students were asked to  define the job description of part-time student dormitory counselors.  The effectiveness of each technique was judged based on the quantity of unique ideas generated by the technique and the overall satisfaction of the outcome by the group.  They hypothesized that NGT would be more effective than delphi and delphi would be more effective than interacting.

The size of the groups was set to seven participants, which Van de Ven and Delbecq identified as about the size of many real world teams.  420 individuals were assigned to 60 heterogeneous groups, creating 20 groups for each of the three methods.

The results of the research found six main conclusions,

1. No significant difference in the quantity of ideas generated by the NGT and Delphi groups
2. Significantly more unique ideas were generated by delphi groups than interacting
3. The highest difference in unique ideas was between NGT and interacting groups
4. NGT groups had significantly greater satisfaction with the process than delphi
5. NGT had a highly significant satisfaction with the process and interacting
6. No significant difference in satisfaction between delphi and interacting

Critique:

I am very pleased with the experimental design and result of Van de Ven and Delbecq's research.   I was suprised at fist as to why participants were more satisfied with NGT than with the delphi process.  The delphi process is a very impersonal process, where you don't know who the other participants are and lack the ability of the in-depth, back and fourth discussion that NGT provides.  I believe the lack of the social element does take away from the overall process.  The exit interview of the delphi participants stated the same answers.  The lack of social interaction can make participants feel less engaged in the process and lacked the ability to have an in depth discussion to clarify or develop new ideas.

Source:
Van De Ven, A. H and Delbecq, A. L. (1974).  The effectiveness of nominal, delphi, and interacting group decision making processes.  Academy of Management Journal. Pages 605 - 621.

Summary of Findings: Wellness Activities (4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 5 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2014  regarding wellness activities specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Wellness, which includes exercise, diet, and sleep, is an analytic modifier which has been shown in many studies to  improve performance on a variety of cognitive tasks, but does not appear to have been studied in the context of forecasting abilities.  The strength of the current evidence coupled with low cost and relative simplicity of many of the most effective activities, however, suggests that additional  research should be conducted to conclude the effect of wellness activities on typical intelligence analyst activities.

Strengths:

1. Body of research suggests increased productivity across a variety of performance measures

2. Mitigates performance losses associated with involuntary sleep deprivation

3. Potential to increase group cohesion when conducted in a group setting

4. Enables auxiliary quality of life improvements such as enhanced sleep and life expectancy

5. Wellness activities are low-cost in terms of time and money

Weaknesses:

1. There is no research establishing diet and exercise’s effect on forecasting accuracy

2. There is moderate conflict among existing research describing diet and exercises effect on cognition

3. Stress is interpreted differently among individuals, so mandating a workout or diet regime is difficult

4. Further research on wellness activities and forecasting accuracy must overcome the presence of confounding variables

Step by Step:  

While this modifier is not amenable to a traditional step-by step approach, based on our limited research, here are five recommendations supported by the literature we examined:

1. Establish a sleep schedule that you can maintain
2. Regular exercise reduces stress levels
3. Caffeine and tyrosine improved task performance over a period of four hours
4. A standard caffeine dose (200mg) reduces cognitive fatigue
5. A high-fruit content diet shows enhanced neurogenesis, which is likely to increase mental cognition

Exercise:

Participants played a memory game with six sets of shapes.  The participants were allowed 15 seconds to view the pieces for 15 seconds to try and memorize the location of the sets.  They then turned around and waited for 15 seconds while the shapes were flipped face down.  After the 15 second wait period, the analysts turned around and then completed the matching game as quickly as possible.  The second time through the exercise, participants were once again given 15 seconds to look at a new layout of the pieces.  This time, the participants did an exercise (jump squat, push up, sit-ups, wall sit, forward leaning rest) during the 15 second wait period.  Upon finishing 15 seconds of exercise, the participants attempted to match all the sets of pieces.

What did we learn from the Wellness Exercise

Four of the five participants had a slower completion time of the matching game with an elevated heart rate as opposed to their resting heart rate.  One participant had a significant reduction in time (> 30 seconds), one participant had only a slight increase (5 seconds), and the other three had a significant increase in time (> 30 seconds) while having an elevated heart rate.

Due to time constraints and research design, there was no conclusive answer to if having an elevated heart rate helped improve the participant's memory.  The participants each did a different exercise, in which there was no control for each exercise and its individual effects on the participant’s memory.  In the end, the participants agreed that a change in the research design and the length of the experiment may have produced more conclusive results.

Additional Resources Of Interest:

Snake Oil 2

Effects of Sensory-Enhanced Yoga on Symptoms of Combat Stress in Deployed Military Personnel

Effects of Sensory-Enhanced Yoga on Symptoms of Combat Stress in Deployed Military Personnel

By: Carolyn C. Stoller, Jon H. Greuel, Lucy S. Cimini, Mary S. Fowler, Jane A. Koomar

Summary:

The authors of this study examined the effects of “sensory-enhanced hatha yoga on symptoms of combat stress in deployed military personnel, compared their anxiety and sensory processing with that of stateside civilians, and identified any correlations between the State–Trait Anxiety Inventory scales and the Adolescent/Adult Sensory Profile quadrants.” Studies have shown that traditional healthcare treatments for post-traumatic stress disorder(PTSD) such as talk therapies have had limited success. Non-conventional healthcare treatments such as yoga incorporates breath work and movement which increases heart rate and reduces symptoms of PTSD. According to a Walter Reed Medical Center study, yoga nidra reduced the severity of insomnia, depression, anxiety, and fear, all symptoms on the PTSD list.

The authors of this study used a randomized control trial to research the effects of sensory-enhanced hatha yoga on combat stress. Participants had to be deployed to Forward Operating Base Warrior, Kirkuk, or Iraq. Military personnel were contacted by email and flyers to participate in the study. The study consisted of 35 treatment and 35 control participants. According to the authors, “Of the 70 participants, 20 were in the U.S Army and 50 were in the U.S Air Force; 22 were women and 48 were men.” The treatment participants took part in hatha yoga classes for three weeks, seven times a week, a minimum of nine times during the three week period.

Overall, sensory-enhanced hatha yoga was effective in reducing anxiety. Military personnel “showed significantly greater improvements than control participants on 16 of 18 mental health and quality-of-life factors.” Of the 70 participates, 54 percent showed sleep improvements, 37 percent felt more relaxed, 26 percent felt an increase in physical benefits, and 11 percent reported better frustration and anger management. The results of this study supports the use of sensory-enhanced hatha yoga to control and manage combat stress.

Critique:

The authors of this study provided strong evidence that sensory-enhanced hatha yoga reduced PTSD and combat stress. However, I would have liked to gain a better understanding of the background of the participants such as, how long they were deployed and what military occupation specialty they held while in deployment.

Source:

Stoller, C. C., Greuel, J. H., Cimini, L. S., Fowler, M. S., & Koomar, J. A. (2012). Effects of Sensory-Enhanced Yoga on Symptoms of Combat Stress in Deployed Military Personnel. American Journal of Occupational Therapy, 66(1), 59–68. doi:10.5014/ajot.2012.001230

Exercise and the brain: something to chew on

Summary:

Concerning dieting and cognition, Van Praag (2009) found that limited intake reduced the risk of attention deficit.  The intake of a variety of dietary supplements enhanced learning for both animals and humans.  Specifically, fish oil, teas, fruits, folate, spices, and vitamin improved cognitive functions.  While results were only found in rats, particular foods that increased memory are plant-derives foods like grapes, blueberries, strawberries, tea, and cocoa.  Van Praag hypotheses that flavanol in plant-derived foods are the primary ingredient responsible for this increase in cognition.  When dieting is combined with exercise, epicatechin is very effective in enhancing memory and synaptic plasticity.

Exercise is also an effective mechanism to increase cognition.  Past studies have shown that exercise enhances neurogenesis.  Neurogenesis refers to the generation of neurons and connections, or synapses, between those neurons.  An increase in the number of neurons and synapses equates to an increase in memory generation, memory recall, and learning.  Most interestingly, most of the neurogenesis occurs in the hippocampus, the area of the brain most responsible for learning and memory.  As shown by the diagram, diet and exercise work simultaneously to enhance cognitive functions.  

Diagram of diet & exercise effects on cognition. Source: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680508/

Critique:

Assuming that brain cognition does benefit intelligence analysis in some fashion, diet and exercise does improve cognition (thus intelligence analysis).  It appears that analysts would benefit most from a high-fruit diet combined with regular exercise.  The duration, frequency, and intensity of exercise are unknown, especially since many of Van Praag’s conclusions are based on rats.  However, the genetic similarities between humans and rats, as uncomfortable an idea that is to some, may provide enough grounds for analysts to improve their wellness.

Source:

Van Praag, H. (2009, May 12). Exercise and the brain: something to chew on. Retrieved October 14, 2014, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680508/