Wednesday, September 9, 2015

The Selection of Delphi Panels for Strategic Planning Purposes

John F. Preble (1984)

Summary:
Preble develops and answers three main questions in this article:
  1. Do results obtained using an intracompany Delphi panel tend to differ significantly from those obtained using an intercompany Delphi panel?
  2. Are the forecasts generated consistent?
  3. If the results are consistent, which panel type is recommended?
Previous literature on the Delphi panels, particularly Martino (1972) and Johnson (1976), recommend using a panel of experts from outside the organization. These studies operated on the implicit assumption that external panelists were likely to be better qualified or more expert than any panelists within the organization, despite a lack of evidence supporting this assumption.

The author composed two 15-member Delphi panels which consisted of top-level employees from large, successful life insurance firms headquartered in the north-eastern United States. The positions represented included Legal, Public Relations, Human Research, and other such diverse roles. The intracompany panel was composed of 15 members from the same company, while the intercompany panel consisted of 15 members from 15 different companies; three participants dropped out midway through the study, leaving 14 and 12 panelists, respectively. The panelists were asked to provide estimates as to the likelihood and timing of 27 different events, dated 1985, 1990, 1995, and 2000, and provide their degree of familiarity with each event. The author provided three rounds of questionnaires to the panelists, including statistical feedback after the first round and qualitative reasoning behind outliers after the second round.

After collecting the data, the author conducted t-tests to determine statistically significant differences. Seventy-six percent of the t-tests were not significant, meaning that in most cases the forecast from the intracompany was “quite close” to the corresponding intercompany estimate. Considering estimates categorized as unlikely (0.0-2.49), slightly likely (2.50-4.99), likely (5.00-7.49), or very likely (7.50-10.0), 95 of 96 comparisons were either in the same category or the next closest category (see Figure 1).  These results show that intracompany panel estimates are about the same as intercompany panel estimates. Because these estimates are consistent, Preble recommends that intracompany panels be used by strategic planners in order to increase administrative control, decrease the number of dropouts and overall costs, and satisfy the need for confidentiality of proprietary information.

Figure 1 - Mean scores and classification comparisons
Critique:
Methodologically, the only significant change is that a few female members should have been included in the intercompany Delphi panel; otherwise, the panel demographics are very similar (see Figure 2). 
Figure 2 - Panel demographics
One of the downsides to this method is that it took four months to complete the three rounds of the Delphi, therefore, this method could be overlooked under tight deadlines. This method could also drift dangerously close to group-think. While anonymity prevents panelists from being dominated by stronger personalities, the opportunity for panelists to change their responses in the second round based on the statistical data of the first round could eliminate any significant dissent. While the goal of Delphi is to converge on a central estimate, the outliers may have unique insights or experiences that are not reflected in statistical syntheses, but only come through in the qualitative comments.

Source:
Preble, J.F. (1984). The Selection of Delphi Panels for Strategic Planning Purposes. Strategic Management Journal, 5(2), 157-170.

7 comments:

  1. Do you believe this research itself enough to prove the idea that there is not a significant difference between intercompany and intracompany Delphi panel groups? And have you found any other researches that supports this idea?

    ReplyDelete
    Replies
    1. Osman, the only major benefit I see to using an intercompany group is to look at a problem from a different point of view; however, that may not outweigh the cost savings and control abilities of using an intracompany group. Martino (1972) hypothesized that using a group of outsiders would yield better results to the method (and recommended that companies do so), but the results of his study concluded that the differences were insignificant.

      Delete
  2. Since the results showed intracompany panel estimates were about the same as intercompany panel estimates, did Preble give a reason why they recommended intracompany panels be used by strategic planners over intercompany panels?

    ReplyDelete
    Replies
    1. Katie, Preble recommends that intracompany panels be used by strategic planners in order to increase administrative control, decrease the number of dropouts and overall costs, and satisfy the need for confidentiality of proprietary information (if this is applicable to the problem).

      Delete
  3. There are obvious advantages to the intracompany panels like Andrew mentions at the end of his summary and these need to be weighed accordingly. Where I feel this particular study has a weakness is that it only took place in one field, life insurance. I am far from a knowledgeable source on the field, but as it is a well established and highly quantitatively oriented field I could easily see the industry as a whole having some level of group think. I would be interested in seeing this study repeated in some other fields where companies are more diversified in their approaches and see if there are similar results.

    ReplyDelete
    Replies
    1. I really like the point you made there. I didn't think about testing this in another field. I personally am surprised that there was no statistically significant difference here. I also think it would be really interesting to see this method tested out in a field like intelligence that has very different viewpoints scattered among the field.

      Delete
  4. I agree in the time point, one of the advantages of this technique I think is to come to a consensus in a reasonable amount of time when a consensus opinion is needed. The large amount of time used in this scenario greatly inhibits the major advantage of this technique,

    ReplyDelete