The Delphi Method for Graduate Research

Gregory J. Skulmoski, Francis T. Hartman, and Jennifer Krahn (2007)

Summary:

The authors of this article for the Journal of Information Technology Education are from Zayed University in Dubai and the University of Calgary, Canada.  Their approach is to provide graduate students with the knowledge necessary to employ the Delphi method in their own academic research, be it for a thesis or dissertation.  The primary scope of the piece is based around the Information Systems (IS) field as well as the field of Information Technology (IT), but they believe that as a research technique the Delphi method can be used in a wide variety of fields, not just their own.  This is because they view the Delphi method as a “flexible research technique” and define it as follows:

The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback.

The authors also put forth that that this method is most suitable when “the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts.” 

In order to improve the reader’s understanding of the methodology there is a brief historical overview of what the authors refer to as “Classical Delphi”.  This is the methodology developed by Norman Dalkey for the RAND Corporation in the 1950’s.  In the Delphi method a panel of experts is given a survey with questions to answer.  After they have returned the survey, a second survey is sent out based on the results of the first, this proceeds over a pre-determined number of rounds. Following the completion of the last survey analysis of the final results is conducted.  Classical Delphi is defined by four key features:  

1. Anonymity of Delphi participants: participants are freed from social and professional pressures by the use of anonymous surveys.  Those who are regarded as greats in the field will be judged solely on their answers and not by their reputations.  This also frees participants to think outside traditional lines without fear of reprisal.
2. Iteration: participants are given the opportunity to develop their own understanding and beliefs as the study progresses.
3. Controlled feedback: as rounds of the Delphi progress, participants will be informed of the anonymous perspectives of other participants.  This is one the core advantages of the Delphi method, that the group as a whole will generate a broad array of ideas to begin with and zero in on the best ones as times moves on 
4. Statistical aggregation of group response: this allows for statistical analysis of results.

This “Classical Delphi” model has been adapted by many in the decades following its creation, this has made it more widely applicable and more adaptive to diverse requirements.  Below is a visualization of a more modern take on the model which has been used in some of the authors’ graduate students’ projects:

One of the features of the paper is the design considerations that should be taken into account while utilizing the Delphi method in graduate research.  The authors discuss pros and cons of:  

• Methodological Choices
• The Broadness of the Initial Question
• Criteria for who is to be Considered an Expert
• The Number of Participants
•  The Number of Rounds
• The Mode of Interaction with Participants
• Methodological Rigor
• The Results
•  Further Verification
•  Publication

The authors close with what they consider two important points, “First, the Delphi approach can be aggressively and creatively adapted to a particular situation. Second, when adapting the approach, there is a need to balance validity with innovation. In other words, the greater the departure from classical Delphi, the more likely it is that the researcher will want to validate the results, by triangulation, with another research approach.”

Critique:

This article provides a lot of good information for graduate students who may be unfamiliar with the Delphi method.  It is specifically focused on how they can get out and use the methodology without too much difficulty, and what they need to be thinking about as they do it.  One of the largest limitations of the piece in my opinion is that spends too little time discussing the situations in which the method is not appropriate.  The article goes to great length to explain just how adaptable the methodology is, but even in the Executive Summary it states Delphi, “ is not a method for all types of IS research questions.”  The article raises this concern but rarely returns to it.  This as a result makes it feel like the use of the Delphi method is a foregone conclusion and less that it is just one more valuable tool in your analytic toolbox.