Expert Opinions in Forecasting: The Role of the Delphi Technique

Rowe and Wright in this paper discuss principles that must be applied in forming Delphi groups. Rowe and Wright argue that the application of these principles create a substantially more accurate estimate than one that would come from an individual. These are the following principles:

1.      Use experts with appropriate knowledge
Experts will resist changing their estimates unless valuable evidence is presented to them. The risk of using non-experts, or subjects with a limited knowledge base, is that the final estimate’s accuracy might be no greater than the first round due because they may be more willing to change their responses.

2.      Use heterogeneous experts
The combining of judgments increases the reliability of an aggregate estimate. Interaction between experts is likely to fill knowledge gaps which will be beneficial in developing a final estimate.

3.      Use between 5 and 20 experts
The size of the group will depend on the number of experts available. In a study done by Brockhoff (1975), Delphi groups of five, seven, nine, and 11 showed no difference accuracy.

4.     For Delphi feedback, one must provide experts the mean or median of the estimates plus the rationales for the estimate
The restriction of the flow of information denies collaboration and the support for estimates made. With a reason, an expert is likely to stick with their estimate than change it.

5.      Continue Delphi polling until responses show stability (Three rounds)
Erffmeyer, Erffmeyer and Lane (1986) found that the quality of Delphi estimates increased up to the fourth round but not after.  Additonally, Brockhoff (1975) found that the accuracy of estimates increased up to round three and decreased with each additional round.

6.      Weight all experts’ estimates equally in developing the final forecast

7.      In phrasing questions, use clear and succinct definition and avoid emotive terms

8.      Frame questions in a balanced manner
Consistently state both outcomes of a response so that each expert is aware of the meaning of their responses.

9.      Avoid including irrelevant information in questions

Critique
How is determined if someone is an expert at something? Stating that one must find heterogeneous experts with a similar knowledge based sounds contradictory. For Delphi to be successful, participants must possess appropriate knowledge. How does one have a differing expertise with the same knowledge base?

Additionally, there is a great degree of variation between a group of five and a group of 20. Although studies found know statistical difference between groups of five and 11, there were no studies cited that examined the accuracy of estimates between groups of five and 20.

Source:
Rowe, G. & Wright, G. Expert opinions in forecasting: The role of the Delphi technique. Principles of forecasting, 125-144.