Summary Of Findings: Bayesian Analysis (4 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 12 articles read in advance of (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 6 MAY 2009 regarding Bayesian Analysis specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:
Bayesian analysis is a method that uses Bayesian statistics to assess the likelihood of an event happening in light of new evidence. It generates an estimate and the use of Bayesian statistics in Intelligence analysis allows for the uncertainty of the traditional intelligence data set to be understood in a scientifically valid manner.

Strengths:

*can limit analyst biases by reducing the weight of evidence simply because it is new or vivid

*forces the analyst to resassess evidence and consider alternative possibilites

*adheres to rigid mathematical formulas

*provides a numerical likelihood

*provides audit trail and ability to reproduce results

Weaknesses:

*Probabilities are based largely on subjectivity

*Susceptible to biases

*Highly complex problems require heavy computations

*Can be mathematically complex

*Not always useful as a stand alone method (works well in tandem with methods like Delphi); may require SMEs for determining probability distributions

*Some reliance on ambiguous validities

*"Negative evidence"--absence of positive evidence

How-To:

This method loosely follows the guidance suggested by his line of research into the use of natural frequencies in teaching and explaining Bayes to beginners.

1.) Create a 2x2 matrix. Label the quadrants with the respective information that creates true positive, false negative, false positive, and true negative quadrants.

2.) Take the given information, the base line (for example, 100 out of 1,000) with the new information (for example, a new document that is 90% credible saying that war is immiment) which means that your true positive and your false negative must equal 100 and the false positive and true negative must equal 900.

3.) To calculate the true positive quandrant, take 90% of the 100 from the base line (which equals 90).

4.) To calculate the false negative quadrant, take the numerator of the base line (100) and subtract the true positive quadrant (90), creating an answer of 10.

5.) To calculate the true negative quadrant, take 90% of your non-war cases (900), equalling 810.

6.) To calculate the false positives, subtract the sum of the three quadrants known from the total number of cases (1,000), which equals 90.

7.) To calculate the new probablitiy, divide what the numerator of the base line (100) from the new total of positive caes (90+90=180), which equals 55.5%

The 55.5% means that there is a 55% probability that countries X and Y are likely to go to war.

Experience:

To understand the basic mathematical principles behind Bayes, the class worked through some sample problems. One of the problems was based on a medical test with an 80% accuracy rate for a cancer with 2% affliction rate in the general population. The class applied this to a sample population of 1000 cases. We established a matrix and assessed the true positive, false positive, false negative, and true negative quantities (16, 116, 4, and 784 respectively). We plugged these numbers into the appropriate matrix fields. We then divided the number of actual cases of cancer (20 or 2% of 1000) into the number of positive tests (132--the 16 true positives and 116 false postives). The result was 15% rate of those who have the cancer from the positive tests, a rather stark difference from the 2% base rate! This problem actually reflects the number of breast cancer rates from a medical treatment from around a two decades ago!

Note: see the matrix for a synopsis of another of the problems we worked through (a peice of evidence emerging suggesting a cause for war).

The class also used a Bayesian application to assess the likelihood we would contract swine flu. We started with the initial hypothesis that we would contract swine flu or we would not contract swine flu, and assigned an initial probability to each hypothesis (the latter >5%) We then added weighted evidence which influenced the base rate of the hypothesis. After all the evidence was entered, the class assessed the likelihood of contracting swine flu.

Using Search Engine Optimization For Intelligence Analysis

SEO Analysis Now - A Site For Using SEO in Intelligence Analysis
Rated: 4 Stars out of 5

As a final requirement of this course, I had to research and report findings on an analytical technique of my choosing. In addition, I had to test-out and apply the technique in order to reveal its true strengths and shortcomings; as well as come up with a how-to guide to use the technique. For my project, I chose to conduct a Search Engine Optimization (SEO) Analysis on the Websites of two popular coffee shops, Starbucks (which is well established) and Caribou Coffee (which is slowly gaining popularity) and reflect on how this process can be applied to Competitive Intelligence (CI), to Law Enforcement Intelligence (LEI), or to the National Security sector. I chose SEO not only because it is an emerging analytical process and can be useful to intelligence analysts, but also because I found it quite intriguing and fun.

SEO provides a way to gain insight into a Website’s audience.

If we know who an audience is (age, gender, ethnicity, education, affluence, etc), how they behave (what other Websites, or types of Websites, they visit), from where they log on, when they visit, and what other sites direct their traffic (as well as what their general interests are), then we can make assessments and predictions about how to either promote their behavior (steering them toward a particular site – useful for CI & marketing purposes) or how to counter that behavior (keeping them away from sites – useful for LEI & N’tl Sec purposes).

For more detailed information about how SEO can be used for Intel analysis, please check out my project:

SEO Analysis Now - A Site For Using SEO in Intelligence Analysis

(The image below is a geographical search index comparison of online users searching for Caribou Coffee [left] and Starbucks [right]. Images provided by Google Insights for Search)

Bayesian statistics: principles and benefits

http://library.wur.nl/frontis/bayes/03_o_hagan.pdf

This article is meant to summarize the basics of Bayesian statistics for beginners.

In Bayesian statistics:

• a statistical model is created to link data to parameters
• prior information about parameters is formulated
• inferences about parameters are derived from the resulting posterior distribution

Graphically, the narrower the curve, the tighter the parameters. The difference between frequentist methods and Bayesian analysis is the use of past information, which is principally subjective. It is important for the prior information to be defensible and reasonable. The author believes subjectivity is a strength of the system because it allows for the examination of posterior distributions from different informed observers.

Until the 1990s, computational tools for conducting Bayesian analysis were nascent or non-existent. While there are tools for the specialist available today, the general practitioner of Bayesian analysis will find there are few user friendly tools available.

The author enumerates a number of benefits for using Bayesian analysis. They are:

• It provides meaningful and intuitive inferences.
• It can answer complex questions cleanly and exactly.
• It makes use of all available information.
• It is well suited for decision-making.
  

Introduction to Bayesian Analysis

http://www.mimuw.edu.pl/~pokar/StatystykaSemMgr/WalshBayesIntro.pdf

"Bayesian statistics is concerned with generating the posterior distribution of the unknown parameters given both the data and some prior density for these parameters. Bayesian statistics creates a much more complete picture of the uncertainty in the estimation of the unknown parameters."

The article's main usefulness lies in suggestions on how to improve the Bayesian process. The author's first suggestion is to remove the effect of non-critical (or at least non interesting) parameters from the overall findings. They label these non-critical parameters as "nuisance" parameters.

Second, the authors recognize the lack of hypothesis testing in the scholarly works of their peers. They underhandedly advocate for such studies to be done.

The Logic of Intelligence Failure

http://www.cdi.org/blair/logic.cfm
By Bruce G. Blair, Ph.D.

The author of this article asserts that critics of the intelligence community concerning both of the major recent US intelligence failures--Iraq WMD and 9/11--ought to come to the realization that threats of varying levels of uncertainty are typically inaccurately assessed. Only successive, repeated assessments of updated data can narrow the gap between perceptions and reality. Since data users and intelligence analysts tend to initially process information subjectively (against their own rational beliefs, judgments, opinions), and modify their rationality as new information or intelligence comes in.

Blair argues that decision making was the result of intelligence analysis that basically followed the laws of reason. He contends that, "applying a rule of logic known as Bayes' law to these cases (9/11 WMD) shows that the intelligence process produced conclusions that were not only plausible but reasonable."

Blair utilized the following formula in his study:
____________________________________________________________________
While all of the probabilities come from the minds of people and are inherently subjective, the analysis itself depends on the product of successive analyses of real (objective) data, the probabilities of which are likely to converge with reality, so long as the individuals involved are thinking logically/rationally. Lower rates of error are likely to accelerate this convergent process.

Here are two scenarios, applied to Baye's formula:

WMD

Terrorist Attack


The author then provides possible scenarios , based on Bayesian calculations and iterations, that determine the point at which the relationship of warnings converge with the reality of the event.

___________________

Thus, concerning situations that involve preemption or preventive war, the author suggests that neither inductive reasoning nor even Bayesian analysis can truly clarify the validity of warnings, intelligence interpretations, or new information in certain situations--"two observers with different preexisting beliefs will often believe that the same bit of behavior confirms their beliefs - hawks seeing aggresive behavior and doves seeing evidence of conciliatory behavior".

What is Bayesian Analysis?

http://www.bayesian.org/bayesexp/bayesexp.html

This article is a summary of the International Society for Bayesian Analysis's (ISBA) definition of the basics of the Bayesian theorem and analysis. The according to the ISBA website, the organization "was founded in 1992 to promote the development and application of Bayesian analysis useful in the solution of theoretical and applied problems in science, industry and government. By sponsoring and organizing meetings, publishing the electronic journal of Bayesian statistics Bayesian Analysis, and other activities ISBA provides a focal point for those interested in Bayesian analysis and its applications".

Summary
Bayesian analysis, a statistical tool for handling probability distributions, got its start in the mid 18th century. It was not, however, until the 1980s when modern computers were able to handle the complex computations that made Bayesian implementation difficult, that Bayesian analysis gained more widespread acceptance. Since then, its use has increased in popularity, being used in many different applications--from healthcare, to weather, to criminal justice. Despite the many nuanced manifestations of Bayesian analysis, it serves a common application: to analyze the probability of unknown and uncertain occurences.



How To:

The left side of the equation expresses the known quantities--"parameters"--as a probability of the current data--"prior distribution". 'y' represents the new data that enters into the calculation. Thus, the "'likelihood,' [is] proportional to the distribution of the observed data given the model parameters.

On the right, the equation's new probablity distribution (posterior distribution) is read: "posterior is proportional to the prior times the likelihood".

Strengths:

• Many diverse applications
• "Philosophical consistency"
• Lacks problems that are associated with other 'frequentist' methods
• Produce clear answers, products
• Reformulates for each variable

Weaknesses:

• Subjective nature of prior probabilities-"your prior information is different from mine".
• More complex problems require more powerful computational tools

Cruise Missile Proliferation: An Application of Bayesian Analysis To Intelligence Forecasting

Michael William Gannon

Summary:
The author applies Bayesian analysis to the problem of cruise missile proliferation. The author defines Bayesian analysis as, "a quantitative procedure in which alternative hypothetical outcomes are postulated and their prior probabilities estimated. As additional relevant events occur, the probabilities of their association with each hypothesis are used to calculate a revised probability for each alternative outcome." He notes that Bayesian analysis has been used by the CIA to provide Indicators & Warnings (I&W) as to the probability of outbreak of armed conflict. Any observed event has a probability associated with its actual occurrence, depending on initial causes. By observing and evaluating events that do occur, "posterior probabilities" can be assigned to each cause, creating a likelihood of an event that may occur in the future based on similar initial causes.

Strengths of the method, according to the author, are "The principal advantage of the method is the establishment of a formal analytical framework which accommodates weighted inputs of all observed events, makes differing interpretations of a given event more explicit, and provides a readily available chronological record of the analytical process." A major weakness to the method, however, is that it "is limited to situations which can be expressed as a number of
mutually exclusive outcomes. An ample flow of data which is logically related to the hypotheses to be tested must be available, and analysts must be qualified to assign realistic probabilities associating the observed events to their hypothetical causes."

In assessing the history of Bayesian analysis, the author notes that the CIA found that Bayesian analysis and the Delphi method to be highly complimentary. Furthermore, the CIA found that Bayesian analysis had distinct advantages over other methods. These advantages were:

(1) More information can be extracted from the available data.. .and probabilities are not at the mercy of the most recent or most visible item.
(2) The formal procedure has been shown to be less conservative than the analysts' informal opinions, and to drive the probabilities away from fifty-fifty faster and farther than the analysts' overall subjective judgments do....
(3) The procedure provides a reproducible sequence for arriving at the final figures ....
(4) The formulation of the questions forces the analyst to consider alternative explanations of the evidence he sees.. . [and] to look at how well the evidence explains hypotheses other than the one he has already decided is the most likely.
(5) The use of quantified judgments allows the results of the analysis to be displayed on a numerical scale, rather through through the use of [subjective terms].

Limitations discovered by the CIA included:
(1) The question must lend itself to formulation in mutually exclusive categories .
(2) The question must be expressed as a specific set of hypothetical outcomes.
(3) There should be a fairly rich flow of data which is at least peripherally related to the question.
(4) The question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event.

Ultimately, Bayesian analysis is intended as a forecasting tool, but has the added benefit of utilizing raw data (which can be "graded" for source reliability and assessed for explanatory hypotheses). This provides the analyst with a "quick reference" source to conduct "snap shot evaluations" on current program evaluations, such as the state of a nations missile program.

Authors Comment:
This paper is a master's thesis that used Bayesian Analysis to assess future cruise missile proliferation. I did not include any findings of the thesis in this summary; instead I summarized the sections that dealt with the method of Bayesian analysis itself.

Bayes' Theorem and Intelligence

Net Wars

Summary:
According to the author, "Beliefs are based on probabilistic information. Bayes Theorem says that our initial beliefs are updated to to posterior beliefs after observing new conditions." As an analytic method, Bayesian analysis provides a formula which allows the analyst to upgrade original assertions as new evidence is discovered, and assign likelihoods to events; the more we observe the better we can predict the likelihood of a certain event. The formula used to update the analysts initial beliefs to posterior beliefs is: p(C|O) = p(O|C)p(C)/p(O|C)p(C) + p(O|¬C)p(¬C). According to Bayes, initial beliefs have a high margin of error; this is alleviated by incorporating new evidence through this formula. This formula "produces interesting results because it accounts for uncertainties created by False Positives and False Negatives."

The author provides the following example of using Bayesian analysis to update your beliefs:

"There is a case of this occurring. Europeans believed that swans were always White and there could be no Black Swans. They updated their probability of a Swan being White to 99% based on their limited experiences. As they explored the world, they found Black Swans in Australia. This reduced the probability of a swan being white and increased the probability of a swan being black. This process of inductive reasoning can be explained via Bayesian probability."

The author provides the following example of using Bayesian analysis as an intelligence methodology:

"There are 10,000 civilians. 1% of whom are insurgents pretending to be civilians. Police can investigate individuals and determine if they are an insurgent or civilian with 95% certainty.

Prior Probability is this: 0.01 (10,000) and 0.99(10,000). So
Group 1: 100 insurgents
Group 2: 9,900 Civilians

The Police investigate the entire population. This produces four groups:
Group 1: Insurgents - Positive test (0.95)
Group 2: Insurgents - False Negative test (0.05)
Group 3: Civilians - False Postive test (0.05)
Group 4: Civilians - Negative test (0.95)

How certain are the police that the men they captured are actually insurgents? The answer is 16%.
(0.95 x 0.01)/ (0.95 x 0.01) + (0.05 x 0.99) =
0.0095/0.0590 = 0.161"

The 16% certainty rate stems from the uncertainty that always exists as some insurgents escape detection while some innocents test positive as insurgents. The author notes that this is an extremely oversimplified example; actual Bayesian analysis in this situation would require some serious computing power that takes into account many other factors, as well as multiple testing to insure that the most accurate results were reached. Nonetheless, the example highlights the use of Bayesian analysis as a method of predictive analysis. However, the method predicts the probability of a particular event happening, and not whether that event will actually occur.

The author returns to the "black swan" example, stating that these highly unlikely, yet possible, intelligence "black swans" are events that can occur, but are highly unlikely too. Just because they haven't happened, doesn't mean they won't. Bayesian analysis provides a method for determining their likelihood. The author concludes by reiterating that intelligence analysis is not about predicting future events, but rather about predicting the likelihood of future events. "The inability to stop a Black Swan event, or a false prediction of a Black Swan event, does not always mean that the intelligence community 'failed'." Rather, the notion that Intel failed comes from the distorted view of Intel analysts as fortune tellers, rather than the reducers of uncertainty that they truly are.

Bayesian Statistics In The Real World Of Intelligence Analysis: Lessons Learned

Bayesian Statistics In The Real World Of Intelligence Analysis: Lessons Learned
By Kristan Wheaton, Jennifer Lee, & Hemangini Deshmukh
Journal of Strategic Studies, vol. 2 n.1
February 2009

Summary:
In this article, Kris Wheaton, in collaboration with Jennifer Lee and Hemangini Deshmukh, agree that alternative methods, ones that are more structured, should be applied to the intelligence process in order to improve intelligence analysis. Recognizing the potential that Bayesian Statistics can bring to the field of intelligence, the author questions the ease in which entry-level intelligence analysts can apply and use this advanced statistical method.

Following the model of an experiment conducted by Gerd Gigenrenzer to test the accuracy of a diagnosis by two groups of doctors (with one group using traditional statistic formulations [5% or .05] and the other using natural frequencies [5 times out of 100]), the author tested 67 Senior Intelligence Studies students at Mercyhurst College. The findings of Wheaton’s experiment were extremely similar to that of Gigenrenzer’s, showing that groups who receive natural frequencies have a much higher rate of being accurate when using Bayesian statistics (79% versus 18% accuracy). This accuracy is attributed to the power of ‘framing’ questions. The author concludes, “natural frequencies are an effective method for encouraging Bayesian reasoning.”

In addition to the experiment, the article covers a brief how-to and overview of Bayesian Analysis. In short, Bayesian statistics is particularly useful due to its ability to take-in-account probabilities of one event affecting another, allowing the analyst to rationally update a prior assessment in light of new evidence. This process also helps in the reduction of two very common cognitive biases, the vividness and recency biases. See article for relevant examples of how the Bayes Theorem can be applied to intelligence-related issues.

The Bayes Theorem is illustrated below:

Bayesian Analysis For Intelligence: Some Focus on the Middle East

Bayesian Analysis For Intelligence: Some Focus on the Middle East
By Nicholas Schweitzer
Approved For Release 1994
CIA Historical Review Program
02 July 96

Summary:
Nicholas Schweitzer suggests that advanced analytical methods, such as Bayesian analysis, should be used to aid analysts in an age where information flows continue to rise. In an effort to test Bayesian Analysis as a tool for intelligence analysts, he used the technique among a group of intelligence analysts to assess complex political-military problems. The Middle East was chosen as a discussion point because of the level regional complexities.

Schweitzer defines Bayesian Analysis as “a tool of statistical inference used to deduce the probabilities of various hypothetical causes from the observation of a real event. It also provides a convenient method for recalculating those probabilities in the light of a continuing flow of new events…the ‘rule of Bayes’ states that the probability of an underlying cause (hypothesis) equals its previous probability multiplied by the probability that the observed event was caused by that hypothesis.”

How to:

Because of limitations, the Bayesian technique can only be applied where certain criteria are met. First, the question to be answered must lend itself to formulation in mutually exclusive categories (i.e. war vs. no war); the insertion of overlapping possibilities reduces accuracy of the Bayesian technique. Second, the question must be expressed as a specific set of hypothetical outcomes. Third, there should be a fairly rich flow of data that is at least peripherally related to the question. Lastly, the question must revolve around the type of activity that produces preliminary signs and is not largely a chance or random event. If this criteria is met then:

1. Assign numeric probabilities to hypotheses. The sum of the values must equal .1 or 100%. Because the examination of political/military affairs and events do not automatically yield quantified results, the possible outcomes (hypotheses) have to be quantified. Schweitzer asserts that implementing a Delphi method is the best solution to quantify possible outcomes. He suggests the following procedure to do this:

• Use analysts who are experts on the subject matter (preferably ones who are working on the situation with you)
• Establish a periodic routine for reporting
• On the first day of the period, each of a number of participating analysts submits the items of evidence they have seen since the last round.
• Submissions should be in the form of 1-2 sentences summarizing the item, along with the date, source, & classification.
• The inclusion of relevant items and exclusion of irrelevant items is up to the discretion of the analyst.
• A coordinator consolidates the items, resolving differences of wording, emphasis, and meaning, and returns the complete list of items to the participants.
• On the following day, the analysts (working individually) evaluate the items and return the numerical assessments
• *the use of a group of analysts, as opposed to a single expert, is highly recommended*
  

2. Assess and quantify the evidence that supports/negates the hypotheses.
3. Calculate the new probabilities according to the rule of Bayes:

E is an event, an “item” of intelligence
H is a hypothesis, a hypothetical cause of events
Hi is one of a set of n mutually exclusive hypotheses
P(Hi) is the starting, or “prior” probability of a hypothesis
P(E/Hi) is the probability of an event given Hi, of an event occurring, given a particular underlying cause
P(Hi/E) is the probability of a hypothesis given E, the “revised” probability of a hypothesis, given that a particular event has occurred.

Strengths (please see article for further explanation):

• Allows for the weighting of evidence
• Provides transparency in intelligence assessments
• Forces the consideration of alternative possibilities
• Quantifies analysis instead of using words of estimative probability
• Displays the trend toward an outcome quicker than the analyst can typically realize it on their own
• Incorporating the Delphi method adds credibility to the assessment when presented to managers and decision makers.
  

Weaknesses (please see article for further explanation):

• Limited applicability
• Data problems – can exist in deciding which information is relevant and should be included, as well as what weight values should be given to evidence.
• Source reliability – what is the best practice to account for this
• “Negative evidence”- the absence of any positive evidence may in itself be highly indicative of something
• Problems over time – problems in using this method in a project continuing over many months
• Problem with numbers – cannot use the probability of ‘zero’ (doesn’t work mathematically or analytically) therefore extremely low probabilities must be indicated by a very small number. Also, some people have difficulty thinking in, and assigning, probabilities.
• Subject to bias and manipulation – this is one of the reasons for which the author suggests using a group of experts/analysts to assign probabilities.