Cruise Missile Proliferation: An Application of Bayesian Analysis To Intelligence Forecasting

Michael William Gannon

Summary:
The author applies Bayesian analysis to the problem of cruise missile proliferation. The author defines Bayesian analysis as, "a quantitative procedure in which alternative hypothetical outcomes are postulated and their prior probabilities estimated. As additional relevant events occur, the probabilities of their association with each hypothesis are used to calculate a revised probability for each alternative outcome." He notes that Bayesian analysis has been used by the CIA to provide Indicators & Warnings (I&W) as to the probability of outbreak of armed conflict. Any observed event has a probability associated with its actual occurrence, depending on initial causes. By observing and evaluating events that do occur, "posterior probabilities" can be assigned to each cause, creating a likelihood of an event that may occur in the future based on similar initial causes.

Strengths of the method, according to the author, are "The principal advantage of the method is the establishment of a formal analytical framework which accommodates weighted inputs of all observed events, makes differing interpretations of a given event more explicit, and provides a readily available chronological record of the analytical process." A major weakness to the method, however, is that it "is limited to situations which can be expressed as a number of
mutually exclusive outcomes. An ample flow of data which is logically related to the hypotheses to be tested must be available, and analysts must be qualified to assign realistic probabilities associating the observed events to their hypothetical causes."

In assessing the history of Bayesian analysis, the author notes that the CIA found that Bayesian analysis and the Delphi method to be highly complimentary. Furthermore, the CIA found that Bayesian analysis had distinct advantages over other methods. These advantages were:

(1) More information can be extracted from the available data.. .and probabilities are not at the mercy of the most recent or most visible item.
(2) The formal procedure has been shown to be less conservative than the analysts' informal opinions, and to drive the probabilities away from fifty-fifty faster and farther than the analysts' overall subjective judgments do....
(3) The procedure provides a reproducible sequence for arriving at the final figures ....
(4) The formulation of the questions forces the analyst to consider alternative explanations of the evidence he sees.. . [and] to look at how well the evidence explains hypotheses other than the one he has already decided is the most likely.
(5) The use of quantified judgments allows the results of the analysis to be displayed on a numerical scale, rather through through the use of [subjective terms].

Limitations discovered by the CIA included:
(1) The question must lend itself to formulation in mutually exclusive categories .
(2) The question must be expressed as a specific set of hypothetical outcomes.
(3) There should be a fairly rich flow of data which is at least peripherally related to the question.
(4) The question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event.

Ultimately, Bayesian analysis is intended as a forecasting tool, but has the added benefit of utilizing raw data (which can be "graded" for source reliability and assessed for explanatory hypotheses). This provides the analyst with a "quick reference" source to conduct "snap shot evaluations" on current program evaluations, such as the state of a nations missile program.

Authors Comment:
This paper is a master's thesis that used Bayesian Analysis to assess future cruise missile proliferation. I did not include any findings of the thesis in this summary; instead I summarized the sections that dealt with the method of Bayesian analysis itself.

Bayesian Analysis For Intelligence: Some Focus on the Middle East

Bayesian Analysis For Intelligence: Some Focus on the Middle East
By Nicholas Schweitzer
Approved For Release 1994
CIA Historical Review Program
02 July 96

Summary:
Nicholas Schweitzer suggests that advanced analytical methods, such as Bayesian analysis, should be used to aid analysts in an age where information flows continue to rise. In an effort to test Bayesian Analysis as a tool for intelligence analysts, he used the technique among a group of intelligence analysts to assess complex political-military problems. The Middle East was chosen as a discussion point because of the level regional complexities.

Schweitzer defines Bayesian Analysis as “a tool of statistical inference used to deduce the probabilities of various hypothetical causes from the observation of a real event. It also provides a convenient method for recalculating those probabilities in the light of a continuing flow of new events…the ‘rule of Bayes’ states that the probability of an underlying cause (hypothesis) equals its previous probability multiplied by the probability that the observed event was caused by that hypothesis.”

How to:

Because of limitations, the Bayesian technique can only be applied where certain criteria are met. First, the question to be answered must lend itself to formulation in mutually exclusive categories (i.e. war vs. no war); the insertion of overlapping possibilities reduces accuracy of the Bayesian technique. Second, the question must be expressed as a specific set of hypothetical outcomes. Third, there should be a fairly rich flow of data that is at least peripherally related to the question. Lastly, the question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event. If this criteria is met then:

1. Assign numeric probabilities to hypotheses. The sum of the values must equal .1 or 100%. Because the examination of political/military affairs and events do not automatically yield quantified results, the possible outcomes (hypotheses) have to be quantified. Schweitzer asserts that implementing a Delphi method is the best solution to quantify possible outcomes. He suggests the following procedure to do this:

• Use analysts who are experts on the subject matter (preferably ones who are working on the situation with you)
• Establish a periodic routine for reporting
• On the first day of the period, each of a number of participating analysts submits the items of evidence they have seen since the last round.
• Submissions should be in the form of 1-2 sentences summarizing the item, along with the date, source, & classification.
• The inclusion of relevant items and exclusion of irrelevant items is up to the discretion of the analyst.
• A coordinator consolidates the items, resolving differences of wording, emphasis, and meaning, and returns the complete list of items to the participants.
• On the following day, the analysts (working individually) evaluate the items and return the numerical assessments
• *the use of a group of analysts, as opposed to a single expert, is highly recommended*
  

2. Assess and quantify the evidence that supports/negates the hypotheses.
3. Calculate the new probabilities according to the rule of Bayes:

E is an event, an “item” of intelligence
H is a hypothesis, a hypothetical cause of events
Hi is one of a set of n mutually exclusive hypotheses
P(Hi) is the starting, or “prior” probability of a hypothesis
P(E/Hi) is the probability of an event given Hi, of an event occurring, given a particular underlying cause
P(Hi/E) is the probability of a hypothesis given E, the “revised” probability of a hypothesis, given that a particular event has occurred.

Strengths (please see article for further explanation):

• Allows for the weighting of evidence
• Provides transparency in intelligence assessments
• Forces the consideration of alternative possibilities
• Quantifies analysis instead of using words of estimative probability
• Displays the trend toward an outcome quicker than the analyst can typically realize it on their own
• Incorporating the Delphi method adds credibility to the assessment when presented to managers and decision makers.
  

Weaknesses (please see article for further explanation):

• Limited applicability
• Data problems – can exist in deciding which information is relevant and should be included, as well as what weight values should be given to evidence.
• Source reliability – what is the best practice to account for this
• “Negative evidence”- the absence of any positive evidence may in itself be highly indicative of something
• Problems over time – problems in using this method in a project continuing over many months
• Problem with numbers – cannot use the probability of ‘zero’ (doesn’t work mathematically or analytically) therefore extremely low probabilities must be indicated by a very small number. Also, some people have difficulty thinking in, and assigning, probabilities.
• Subject to bias and manipulation – this is one of the reasons for which the author suggests using a group of experts/analysts to assign probabilities.
  

Homeland Security Employs Imagination

Washington Post, 18 June 2004

Summary:
The Department of Homeland Security (DHS) has developed a special program designed to think creatively as to how terrorists may attack the US. Known as the Analytic Red Cell office, DHS sought out various professionals to form a "Red Team" that would serve to "think outside the box" as to new ways we may be targeted by terrorists. Various team members were "futurists, philosophers, software programmers, a pop musician and a thriller writer"

According to the article, "Typically the Red Cell team assembles 20 or so participants for a day-long session at leased offices in the Washington area. Each session divides into smaller groups and takes up a different question, such as: If you were a terrorist, how would you target the G-8 economic summit, held last week in Georgia? Another recent topic was: Why haven't terrorists hit the United States since Sept. 11, 2001?" After the Red Cell comes to its conclusions, the final reports are forwarded to intelligence analysts throughout the Intelligence Community (IC), who vet the results and compare them to actual threats and information. "Most Red Cell reports note they are 'alternative assessments intended to provoke thought and stimulate discussion'."

This technique is not new to the IC; the CIA and Pentagon have used the method since the Cold War to expand their thinking on on how the Soviets and other foreign militaries may attack the US. The reasoning behind the method is to get non-intelligence professionals to tackle intelligence related issues. According to Brad Meltzer, a thriller writer, when he was appraoched, "They said, 'We want people who think differently from the ones we have on staff.' "