Bayesian Analysis For Intelligence: Some Focus on the Middle East

Bayesian Analysis For Intelligence: Some Focus on the Middle East
By Nicholas Schweitzer
Approved For Release 1994
CIA Historical Review Program
02 July 96

Summary:
Nicholas Schweitzer suggests that advanced analytical methods, such as Bayesian analysis, should be used to aid analysts in an age where information flows continue to rise. In an effort to test Bayesian Analysis as a tool for intelligence analysts, he used the technique among a group of intelligence analysts to assess complex political-military problems. The Middle East was chosen as a discussion point because of the level regional complexities.

Schweitzer defines Bayesian Analysis as “a tool of statistical inference used to deduce the probabilities of various hypothetical causes from the observation of a real event. It also provides a convenient method for recalculating those probabilities in the light of a continuing flow of new events…the ‘rule of Bayes’ states that the probability of an underlying cause (hypothesis) equals its previous probability multiplied by the probability that the observed event was caused by that hypothesis.”

How to:

Because of limitations, the Bayesian technique can only be applied where certain criteria are met. First, the question to be answered must lend itself to formulation in mutually exclusive categories (i.e. war vs. no war); the insertion of overlapping possibilities reduces accuracy of the Bayesian technique. Second, the question must be expressed as a specific set of hypothetical outcomes. Third, there should be a fairly rich flow of data that is at least peripherally related to the question. Lastly, the question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event. If this criteria is met then:

1. Assign numeric probabilities to hypotheses. The sum of the values must equal .1 or 100%. Because the examination of political/military affairs and events do not automatically yield quantified results, the possible outcomes (hypotheses) have to be quantified. Schweitzer asserts that implementing a Delphi method is the best solution to quantify possible outcomes. He suggests the following procedure to do this:

• Use analysts who are experts on the subject matter (preferably ones who are working on the situation with you)
• Establish a periodic routine for reporting
• On the first day of the period, each of a number of participating analysts submits the items of evidence they have seen since the last round.
• Submissions should be in the form of 1-2 sentences summarizing the item, along with the date, source, & classification.
• The inclusion of relevant items and exclusion of irrelevant items is up to the discretion of the analyst.
• A coordinator consolidates the items, resolving differences of wording, emphasis, and meaning, and returns the complete list of items to the participants.
• On the following day, the analysts (working individually) evaluate the items and return the numerical assessments
• *the use of a group of analysts, as opposed to a single expert, is highly recommended*
  

2. Assess and quantify the evidence that supports/negates the hypotheses.
3. Calculate the new probabilities according to the rule of Bayes:

E is an event, an “item” of intelligence
H is a hypothesis, a hypothetical cause of events
Hi is one of a set of n mutually exclusive hypotheses
P(Hi) is the starting, or “prior” probability of a hypothesis
P(E/Hi) is the probability of an event given Hi, of an event occurring, given a particular underlying cause
P(Hi/E) is the probability of a hypothesis given E, the “revised” probability of a hypothesis, given that a particular event has occurred.

Strengths (please see article for further explanation):

• Allows for the weighting of evidence
• Provides transparency in intelligence assessments
• Forces the consideration of alternative possibilities
• Quantifies analysis instead of using words of estimative probability
• Displays the trend toward an outcome quicker than the analyst can typically realize it on their own
• Incorporating the Delphi method adds credibility to the assessment when presented to managers and decision makers.
  

Weaknesses (please see article for further explanation):

• Limited applicability
• Data problems – can exist in deciding which information is relevant and should be included, as well as what weight values should be given to evidence.
• Source reliability – what is the best practice to account for this
• “Negative evidence”- the absence of any positive evidence may in itself be highly indicative of something
• Problems over time – problems in using this method in a project continuing over many months
• Problem with numbers – cannot use the probability of ‘zero’ (doesn’t work mathematically or analytically) therefore extremely low probabilities must be indicated by a very small number. Also, some people have difficulty thinking in, and assigning, probabilities.
• Subject to bias and manipulation – this is one of the reasons for which the author suggests using a group of experts/analysts to assign probabilities.