Pragmatic Research Design: an Illustration of the Use of the Delphi Technique

This study conducted by two scholars at Rhodes University in South Africa used the Delphi Technique in hopes of forecasting the educational needs that will have to be met to prepare students to be entrepreneurs over the next 20-40 years. This paper does not walk readers through the results of the study, but rather the practical challenges in definitions and the organization of applying the Delphi Technique.

Definitions:
Based on the author’s review of the literature, they identified five main characteristics which define the technique:

1. Its focus on researching the future or things about which little is known,
2. Reliance on the use of expert opinion,
3. Utilizing remote group processes,
4. The adoption of an iterative research process, and
5. The creation of a consensus of opinion.

The authors also identify three versions of the Delphi Technique:

• Numeric – aims to specify a single or minimum range of numeric estimates through the use of summary statistics.
• Policy - on the exploration, generation and definition of several alternatives and the arguments for and against each of these alternatives.
• Historic - aims to explain the range of issues that fostered a specific decision, identification of several scenarios that could have led to the resolution of a past problem.

Entrepreneurship was defined as person who provides innovation in an economy, not owners of micro-businesses in saturated markets. The foundation of their study is based on a review of literature that suggests entrepreneurship can be formed through education.

The Experiment:
To comprehensively forecast the answer to their question, they asked three separate questions. The questions were the following:

1. What sector of the South African economy will most likely offer the greatest potential for entrepreneurial opportunities in the next 25 to 40 years?
2. What qualities are needed by graduates to equip them to be innovative entrepreneurs in the future?
3. What should Higher Education in South Africa do to prepare/develop students to constructively participate in the future economy as innovative entrepreneurs?

Three separate panels were created because each question requires answers from a different set of experts.

1. The panel for the first question was referred to members of government departments and research councils.
2. The panel for the second question was referred to endowed Chairs in the area of entrepreneurship.
3. The panel for the third question was referred to alumni of entrepreneurship programs and educationalists and academics in these programs.

The Future of High-Technology Crime: A Parallel Delphi Study

This study conducted by Larry E. Coutorie in 1995 is a follow-up to a 1980s study using the Delphi Technique to forecast the future of high-technology crimes. One of the purposes of this study is to give law enforcement a forecast of where high-tech crimes are headed, since most other techniques only allow reactionary responses.

The Experiment:
The study was conducted using two panels. One was comprised of “traditional” experts, or people already in the high-tech law enforcement field, and “nontraditional” experts, member of hacker and cracker groups recommended by other experts. Two groups on experts were sent three rounds of questionnaires with the following questions, refined each time by the groups’ responses to the previous questionnaire.

1. In your opinion, what area(s) of high technology will be the focus of criminal activity in the next ten years?
2. What form(s) do you believe this activity will take?
3. What steps should be taken now to prepare the police to combat this criminal activity?
4. Do you believe the responsibility for criminal investigation of high-technology crimes will be primarily that of government or private businesses? Why?
5. Do you believe the responsibility for crime prevention activities regarding high-technology crimes will be primarily that of government or private businesses? Why?

Findings:
Each groups’ perspective diverged significantly from the first round of questioning onward. However, at the end of the three questionnaires, a consensus on several issues was identified.

• Likely high-tech future crime areas include computer system attacks via telecommunications, a growing increase in computer-assisted fraud, and computer assisted data manipulation or theft.
• Crime will take the form of software piracy, increased incidents of computer assisted counterfeiting, increased incidents of financial fraud, and increased attacks on computer systems via advanced technologies.
• Preventative steps recommended include recruitment of individual with computer knowledge, increased public/private partnership, more training for law enforcement officers earlier in their career, and legislation that better defines jurisdiction.
• At the time of this study experts forecasted private business would conduct the initial investigation and have an active participatory role in government investigations.
• They also forecasted that private businesses would be responsible for protecting their own assets, with government assistance in identifying potential threats.

Essential Components Of Curricular Learning Communities In Higher Education

The purpose of this study was to identify the essential components of curricular learning communities in higher education. A panel of experts participated in a four-round Delphi process designed to identify these essential features. The writer used a modified Delphi process to first elicit and then rate the importance of characteristics of curricular learning communities in higher education.

Delphi Advantages:
-The Delphi technique offers the advantage of group response without the attendant disadvantages sometimes experienced with group problem solving or decision-making.
-Expert participants are more likely to generate reasoned, independent, and well-considered opinions in the absence of exposure to the "persuasively stated opinions of others". Because the experts do not ever participate in a face-to-face discussion, there is no danger of one or more individuals’ opinions being swayed by a more dominant or more experienced individual.
-Efficiency and flexibility, especially in light of modern communication technologies such as e-mail and the Internet. Experts may be drawn from a wide geographic area, and the participants’ commitment in terms of time and money invested is minimal.
-Delphi method has been shown to be an effective way to conduct research when the responses being sought are value judgments rather than factual information. Although it is more difficult to assess the "correctness" of value judgments, it is generally agreed upon that value judgments are not all equal but can in fact be more "right" or more "wrong."

Delphi Limitations:-Delphi should not be used when any of the following three critical conditions are not present: adequate time, participant skill in written communication, and high participant motivation. It is estimated that a minimum of 45 days is required to carry out a Delphi study.
-Participants must be knowledgeable and able to clearly communicate their ideas. A high degree of motivation is needed to offset the tendency for participant dropout as the study progresses. Because there is no direct contact between participants, those who are not highly motivated and interested in the subject at hand may feel isolated or detached from the process.
-Another is the problem of bias in Delphi studies that can occur from poorly worded or leading questions or selective interpretation of the results.

Instrument Design and Implementation
Round One: Initial Survey:
The first round in the current study consisted of a brief survey, designed to collect some demographic data on the participants, and one open-ended question.

Round Two: Questionnaire One:
A list of 79 features was compiled from the information obtained in the initial survey. Obvious repetitions were eliminated, though items that were similar but not exactly the same were maintained. Items were sorted into four categories: Curricular Features, Pedagogical Features, Structural Features, and Environmental Features. Participants were asked to rate each feature on a Likert-type scale, identifying each feature as an "essential" (5), "very important" (4), "moderately important" (3), "slightly important" (2), or "not important"(1) characteristic of a curricular learning community.

Round Three: Questionnaire Two:
Questionnaire Two listed only the features that had received a mean rating of 4.0 or higher in the previous round. Once again the items were placed into the four categories of Curricular, Pedagogical, Structural, and Environmental.

Round Four: Questionnaire Three:
The third and final questionnaire listed the forty features that received the highest rating (determined by mean and mode) on the previous questionnaire. Panelists were given the following information: ranking of the items from first and second questionnaire, mean score of the items from both rounds, and the number of times each item was selected as one of the three to five most important items.

In this round, panelists were asked to assign a total of 100 value points to the forty items. At the end of this questionnaire, participants were asked to answer the another open-ended question.

Summary of Data Collection and Analysis Procedures:

The following table outlines the four-round Delphi procedure that was followed in this study: (see link for table)

Link: http://www.winona.edu/advising/lcchapter3.htm

Using the Delphi Technique to Achieve Consensus

How it is leading us away from representative government to an illusion of citizen participation.

This article explained the Delphi Technique and how it is used as a methodology in the education system. It was critical of the technique and explains the consequences and motivations of those who are using it.

The Delphi Technique and consensus building are both founded in the same principle - the Hegelian dialectic of thesis, antithesis, and synthesis, with synthesis becoming the new thesis. The goal is a continual evolution to "oneness of mind" (consensus means solidarity of belief) -the collective mind, the wholistic society, the wholistic earth, etc. In thesis and antithesis, opinions or views are presented on a subject to establish views and opposing views. In synthesis, opposites are brought together to form the new thesis. All participants in the process are then to accept ownership of the new thesis and support it, changing their views to align with the new thesis. Through a continual process of evolution, "oneness of mind" will supposedly occur.
In group settings, the Delphi Technique is an unethical method of achieving consensus on controversial topics. It requires well-trained professionals, known as "facilitators" or "change agents," who deliberately escalate tension among group members, pitting one faction against another to make a preordained viewpoint appear "sensible," while making opposing views appear ridiculous.

The facilitators or change agents encourage each person in a group to express concerns about the programs, projects, or policies in question. They listen attentively, elicit input from group members, form "task forces," urge participants to make lists, and in going through these motions, learn about each member of a group. They are trained to identify the "leaders," the "loud mouths," the "weak or non-committal members," and those who are apt to change sides frequently during an argument.

Suddenly, the amiable facilitators become professional agitators and "devil's advocates." Using the "divide and conquer" principle, they manipulate one opinion against another, making those who are out of step appear "ridiculous, unknowledgeable, inarticulate, or dogmatic." They attempt to anger certain participants, thereby accelerating tensions. The facilitators are well trained in psychological manipulation. They are able to predict the reactions of each member in a group. Individuals in opposition to the desired policy or program will be shut out.

The Delphi Technique works. It is very effective with parents, teachers, school children, and community groups. The "targets" rarely, if ever, realize that they are being manipulated. If they do suspect what is happening, they do not know how to end the process. The facilitator seeks to polarize the group in order to become an accepted member of the group and of the process. The desired idea is then placed on the table and individual opinions are sought during discussion. Soon, associates from the divided group begin to adopt the idea as if it were their own, and they pressure the entire group to accept their proposition.

How the Delphi Technique Works
First, a facilitator is hired. While his job is supposedly neutral and non-judgmental, the opposite is actually true. The facilitator is there to direct the meeting to a preset conclusion. The facilitator begins by working the crowd to establish a good-guy-bad-guy scenario. Anyone disagreeing with the facilitator must be made to appear as the bad guy, with the facilitator appearing as the good guy.

Next, the attendees are broken up into smaller groups of seven or eight people. Each group has its own facilitator. The group facilitators steer participants to discuss preset issues, employing the same tactics as the lead facilitator. Why hold such meetings at all if the outcomes are already established? The answer is because it is imperative for the acceptance of the School-to-Work agenda, or the environmental agenda, or whatever the agenda, that ordinary people assume ownership of the preset outcomes. If people believe an idea is theirs, they'll support it. If they believe an idea is being forced on them, they'll resist.


How to Diffuse the Delphi Technique

Three steps can diffuse the Delphi Technique as facilitators attempt to steer a meeting in a specific direction.

Always be charming, courteous, and pleasant. Smile. Moderate your voice so as not to come across as belligerent or aggressive.

Stay focused. If possible, jot down your thoughts or questions. When facilitators are asked questions they don't want to answer, they often digress from the issue that was raised and try instead to put the questioner on the defensive. Do not fall for this tactic. Courteously bring the facilitator back to your original question. If he rephrases it so that it becomes an accusatory statement (a popular tactic), simply say, "That is not what I asked. What I asked was . . ." and repeat your question.

Be persistent. If putting you on the defensive doesn't work, facilitators often resort to long monologues that drag on for several minutes. During that time, the group usually forgets the question that was asked, which is the intent. Let the facilitator finish. Then with polite persistence state: "But you didn't answer my question. My question was . . ." and repeat your question.

Never become angry under any circumstances. Anger directed at the facilitator will immediately make the facilitator the victim. This defeats the purpose. The goal of facilitators is to make the majority of the group members like them, and to alienate anyone who might pose a threat to the realization of their agenda. People with firm, fixed beliefs, who are not afraid to stand up for what they believe in, are obvious threats.

At a meeting, have two or three people who know the Delphi Technique dispersed through the crowd so that, when the facilitator digresses from a question, they can stand up and politely say: "But you didn't answer that lady/gentleman's question."

Establish a plan of action before a meeting. Everyone on your team should know his part. Later, analyze what went right, what went wrong and why, and what needs to happen the next time. Never strategize during a meeting.

link: http://www.eagleforum.org/educate/1998/nov98/focus.html

This process sounds very political in this type of setting and that personal agendas, or vendettas may come into play if a facilitator is not in control of the situation. The illusion of a representative participation is obvious in this type of situation.

Apprehending the Future: Emerging Technologies, from Science Fiction to Campus Reality

Bryan Alexander, Director of Research at the National Institute for Technology and Liberal Education (NITLE) talks about the various techniques used to decipher emerging technology trends from the perspective of higher education in his article. The author talks about how the Delphi technique has been particularly useful in forecasting technology in higher education.

One of the cases where the Delphi technique is used is in the Horizon Project. Launched in 2002, this project draws on a large body of experts across academia. Over several months, the group identifies trends, ranks their impact, compares estimates, and progressively builds up a profile of emerging technologies. This is then published as the annual Horizon Report. The January 2009 report identified the following technologies:

• Mobiles (time-to-adoption: one year or less)
• Cloud computing (time-to-adoption: one year or less)
• Geo-Everything (time-to-adoption: two to three years)
• The Personal Web (time-to-adoption: two to three years)
• Semantic-Aware Applications (time-to-adoption: four to five years)
• Smart Objects (time-to-adoption: four to five years)

Another application of the Delphi Technique in higher education was the "The Future of Internet III" project by Pew Internet & American Life Project and Elon University. This study was much broader in scope and had a much longer timeline. The outcome from this exercise is as below:

• The mobile device will be the primary connection tool to the Internet for most people in the world in 2020.
• The transparency of people and organizations will increase, but that will not necessarily yield more personal integrity, social tolerance, or forgiveness.
• Talk and touch user-interfaces with the Internet will be more prevalent and accepted by 2020.
• Those working to enforce intellectual property law and copyright protection will remain in a continuing "arms race," with the crackers who will find ways to copy and share content without payment.
• The divisions between "personal" time and work time and between physical and virtual reality will be further erased for everyone who's connected, and the results will be mixed in terms of social relations.
• Next-generation engineering of the network to improve the current Internet architecture is more likely than an effort to rebuild the architecture from scratch.

In conclusion, the author states that no technique can effectively predict the future and using a combination of techniques can only help us have some idea of the future. The future is increasingly complex and "black swans" continue to occur and have enormous effects on the future.

Comments:

As can be seen from the above conclusions, they do not seem to be too radical or innovative in any way but instead seem to tug the line of what is believed to be the general consensus (for instance mobile devices being the primary connecting tool in 2020 or touch and talk user interfaces being prevalent does not necessarily need to be deciphered by experts). This is in fact the major drawback of this technique - that Delphi outcomes can be driven by a desire for consensus, rather than actual agreement, meaning that divergent ideas can and often do get quashed.

Cruise Missile Proliferation: An Application of Bayesian Analysis To Intelligence Forecasting

Michael William Gannon

Summary:
The author applies Bayesian analysis to the problem of cruise missile proliferation. The author defines Bayesian analysis as, "a quantitative procedure in which alternative hypothetical outcomes are postulated and their prior probabilities estimated. As additional relevant events occur, the probabilities of their association with each hypothesis are used to calculate a revised probability for each alternative outcome." He notes that Bayesian analysis has been used by the CIA to provide Indicators & Warnings (I&W) as to the probability of outbreak of armed conflict. Any observed event has a probability associated with its actual occurrence, depending on initial causes. By observing and evaluating events that do occur, "posterior probabilities" can be assigned to each cause, creating a likelihood of an event that may occur in the future based on similar initial causes.

Strengths of the method, according to the author, are "The principal advantage of the method is the establishment of a formal analytical framework which accommodates weighted inputs of all observed events, makes differing interpretations of a given event more explicit, and provides a readily available chronological record of the analytical process." A major weakness to the method, however, is that it "is limited to situations which can be expressed as a number of
mutually exclusive outcomes. An ample flow of data which is logically related to the hypotheses to be tested must be available, and analysts must be qualified to assign realistic probabilities associating the observed events to their hypothetical causes."

In assessing the history of Bayesian analysis, the author notes that the CIA found that Bayesian analysis and the Delphi method to be highly complimentary. Furthermore, the CIA found that Bayesian analysis had distinct advantages over other methods. These advantages were:

(1) More information can be extracted from the available data.. .and probabilities are not at the mercy of the most recent or most visible item.
(2) The formal procedure has been shown to be less conservative than the analysts' informal opinions, and to drive the probabilities away from fifty-fifty faster and farther than the analysts' overall subjective judgments do....
(3) The procedure provides a reproducible sequence for arriving at the final figures ....
(4) The formulation of the questions forces the analyst to consider alternative explanations of the evidence he sees.. . [and] to look at how well the evidence explains hypotheses other than the one he has already decided is the most likely.
(5) The use of quantified judgments allows the results of the analysis to be displayed on a numerical scale, rather through through the use of [subjective terms].

Limitations discovered by the CIA included:
(1) The question must lend itself to formulation in mutually exclusive categories .
(2) The question must be expressed as a specific set of hypothetical outcomes.
(3) There should be a fairly rich flow of data which is at least peripherally related to the question.
(4) The question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event.

Ultimately, Bayesian analysis is intended as a forecasting tool, but has the added benefit of utilizing raw data (which can be "graded" for source reliability and assessed for explanatory hypotheses). This provides the analyst with a "quick reference" source to conduct "snap shot evaluations" on current program evaluations, such as the state of a nations missile program.

Authors Comment:
This paper is a master's thesis that used Bayesian Analysis to assess future cruise missile proliferation. I did not include any findings of the thesis in this summary; instead I summarized the sections that dealt with the method of Bayesian analysis itself.