Friday, September 11, 2015

The Delphi Method for Graduate Research

Gregory J. Skulmoski, Francis T. Hartman, and Jennifer Krahn (2007)

The authors of this article for the Journal of Information Technology Education are from Zayed University in Dubai and the University of Calgary, Canada.  Their approach is to provide graduate students with the knowledge necessary to employ the Delphi method in their own academic research, be it for a thesis or dissertation.  The primary scope of the piece is based around the Information Systems (IS) field as well as the field of Information Technology (IT), but they believe that as a research technique the Delphi method can be used in a wide variety of fields, not just their own.  This is because they view the Delphi method as a “flexible research technique” and define it as follows:

The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback.

The authors also put forth that that this method is most suitable when “the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts.” 
In order to improve the reader’s understanding of the methodology there is a brief historical overview of what the authors refer to as “Classical Delphi”.  This is the methodology developed by Norman Dalkey for the RAND Corporation in the 1950’s.  In the Delphi method a panel of experts is given a survey with questions to answer.  After they have returned the survey, a second survey is sent out based on the results of the first, this proceeds over a pre-determined number of rounds. Following the completion of the last survey analysis of the final results is conducted.  Classical Delphi is defined by four key features:  
  1. Anonymity of Delphi participants: participants are freed from social and professional pressures by the use of anonymous surveys.  Those who are regarded as greats in the field will be judged solely on their answers and not by their reputations.  This also frees participants to think outside traditional lines without fear of reprisal.
  2. Iteration: participants are given the opportunity to develop their own understanding and beliefs as the study progresses.
  3. Controlled feedback: as rounds of the Delphi progress, participants will be informed of the anonymous perspectives of other participants.  This is one the core advantages of the Delphi method, that the group as a whole will generate a broad array of ideas to begin with and zero in on the best ones as times moves on 
  4. Statistical aggregation of group response: this allows for statistical analysis of results.

This “Classical Delphi” model has been adapted by many in the decades following its creation, this has made it more widely applicable and more adaptive to diverse requirements.  Below is a visualization of a more modern take on the model which has been used in some of the authors’ graduate students’ projects:

One of the features of the paper is the design considerations that should be taken into account while utilizing the Delphi method in graduate research.  The authors discuss pros and cons of:  
  • Methodological Choices
  • The Broadness of the Initial Question
  • Criteria for who is to be Considered an Expert
  • The Number of Participants
  •  The Number of Rounds
  • The Mode of Interaction with Participants
  • Methodological Rigor
  • The Results
  •  Further Verification
  •  Publication

The authors close with what they consider two important points, “First, the Delphi approach can be aggressively and creatively adapted to a particular situation. Second, when adapting the approach, there is a need to balance validity with innovation. In other words, the greater the departure from classical Delphi, the more likely it is that the researcher will want to validate the results, by triangulation, with another research approach.”


This article provides a lot of good information for graduate students who may be unfamiliar with the Delphi method.  It is specifically focused on how they can get out and use the methodology without too much difficulty, and what they need to be thinking about as they do it.  One of the largest limitations of the piece in my opinion is that spends too little time discussing the situations in which the method is not appropriate.  The article goes to great length to explain just how adaptable the methodology is, but even in the Executive Summary it states Delphi, “ is not a method for all types of IS research questions.”  The article raises this concern but rarely returns to it.  This as a result makes it feel like the use of the Delphi method is a foregone conclusion and less that it is just one more valuable tool in your analytic toolbox.  


  1. From your point of view, what is the most important difference between the classical Delphi method and the one which has been used in some of the authors’ graduate students’ projects?

    1. It is my understanding that most retain the 4 key features I listed above, but not all. One of the major things to change is the application of the methodology. Originally the RAND Corporation developed it for a U.S. sponsored military project, but now it is used in all manner of research.

    2. This comment has been removed by the author.

  2. Do the authors detail the survey phases? I mean should the experimenters handout a different questionnaire in each session?

    1. Yes they did. They specified that "depending upon the research goals, the
      researcher may direct the focus of the research, or be directed by the opinions of the participants." Either way the researcher is taking the first rounds results and molding the questions for round two based on that feedback. The authors also specify that participants should be given the opportunity to change or expand their answers to the first round after they have seen the results, but before beginning the second round of questions.

  3. The authors state the Delphi technique can be “creatively adapted to a particular situation”, do they mention what kind of changes can be made when the technique is being employed? It seems that drifting to far from the classical method will change the fundamental structure of the technique, which may yield different results.

    1. One of that adaptations that they mention specifically regarding academic research is that while the methodology has primarily been used quantitatively, it can also be applied to qualitative or mixed research methods.

  4. I think one of the reasons that the authors did not go too in-depth into the situations in which Delphi can be used is due to the sheer number of situations. As we've seen from the articles posted, Delphi is applicable in health insurance, medical diagnoses, information technology, and likely as well in numerous other situations. While it is not applicable in every situation, it seems to be a very useful, flexible, and applicable methodology.

  5. Do you think Delphi would be useful for collaborative thinking in a disaster response situation? For instance, the initial response to another Hurricane Katrina. It seems as though the technique would be implemented and the data would sit in a filing cabinet.