The Selection of Delphi Panels for Strategic Planning Purposes

John F. Preble (1984)

Summary:

Preble develops and answers three main questions in this article:

1. Do results obtained using an intracompany Delphi panel tend to differ significantly from those obtained using an intercompany Delphi panel?
2. Are the forecasts generated consistent?
3. If the results are consistent, which panel type is recommended?

Previous literature on the Delphi panels, particularly Martino (1972) and Johnson (1976), recommend using a panel of experts from outside the organization. These studies operated on the implicit assumption that external panelists were likely to be better qualified or more expert than any panelists within the organization, despite a lack of evidence supporting this assumption.

The author composed two 15-member Delphi panels which consisted of top-level employees from large, successful life insurance firms headquartered in the north-eastern United States. The positions represented included Legal, Public Relations, Human Research, and other such diverse roles. The intracompany panel was composed of 15 members from the same company, while the intercompany panel consisted of 15 members from 15 different companies; three participants dropped out midway through the study, leaving 14 and 12 panelists, respectively. The panelists were asked to provide estimates as to the likelihood and timing of 27 different events, dated 1985, 1990, 1995, and 2000, and provide their degree of familiarity with each event. The author provided three rounds of questionnaires to the panelists, including statistical feedback after the first round and qualitative reasoning behind outliers after the second round.

After collecting the data, the author conducted t-tests to determine statistically significant differences. Seventy-six percent of the t-tests were not significant, meaning that in most cases the forecast from the intracompany was “quite close” to the corresponding intercompany estimate. Considering estimates categorized as unlikely (0.0-2.49), slightly likely (2.50-4.99), likely (5.00-7.49), or very likely (7.50-10.0), 95 of 96 comparisons were either in the same category or the next closest category (see Figure 1).  These results show that intracompany panel estimates are about the same as intercompany panel estimates. Because these estimates are consistent, Preble recommends that intracompany panels be used by strategic planners in order to increase administrative control, decrease the number of dropouts and overall costs, and satisfy the need for confidentiality of proprietary information.

Figure 1 - Mean scores and classification comparisons

Critique:

Methodologically, the only significant change is that a few female members should have been included in the intercompany Delphi panel; otherwise, the panel demographics are very similar (see Figure 2). 

Figure 2 - Panel demographics

One of the downsides to this method is that it took four months to complete the three rounds of the Delphi, therefore, this method could be overlooked under tight deadlines. This method could also drift dangerously close to group-think. While anonymity prevents panelists from being dominated by stronger personalities, the opportunity for panelists to change their responses in the second round based on the statistical data of the first round could eliminate any significant dissent. While the goal of Delphi is to converge on a central estimate, the outliers may have unique insights or experiences that are not reflected in statistical syntheses, but only come through in the qualitative comments.

Source:

Preble, J.F. (1984). The Selection of Delphi Panels for Strategic Planning Purposes. Strategic Management Journal, 5(2), 157-170.

EBSCOhost Mercyhurst access link