Summary of Findings: Bayesian Forecasting (4 out of 5 stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 5 articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2014  regarding Bayesian Forecasting specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Bayesian analysis is a statistical procedure used to combine prior information with new evidence to produce an estimate.  Bayes theorem, also known as conditional probability, is a means for revising predictions as new relevant evidence is collected.

Strengths:

1. Bayes presents a mathematical approach to how analysts do intelligence  

2. Outputs a single percentage as opposed to a range, which can be updated as new evidence is found

3. Bayes is the accepted gold standard of probabilistic mathematics

4. Clearly outlines the process, allowing for modifiers, such as nominal group technique, to be incorporated

5. Simplistic Bayes can be learned quickly by analysts lacking strong mathematical skills

Weaknesses:

1. Quality of estimate contingent on appropriate selection and weighing of evidence

2. Confirmation of appropriate selection and weighing of evidence not possible until after the fact due to present unknowns and future uncertainties

3. Evidence factored into Bayesian model can fluctuate significantly in importance

4. Output of Bayesian model not always in format that satisfies intelligence requirement

5. The absence of standard deviation and other “traditional” statistical measures creates barriers to communicating estimate to decision makers

Step by Step:  

Note: There are many different to complete a Bayesian model. This step by step process was identified as a common one across different this Bayesian exercises.

1. Identify a conceptual problem
2. Create a baseline
3. Take pieces of evidence and apply them to potential outcomes of the problem
4. Follow Nate Silver’s example to solve for the probability:   

5. This methodology can be repeated as new information and/or evidence is introduced to problem 
    

Silver, N. (2012). The signal and the noise: Why so many predictions fail--but some don't. New York: Penguin Press.

Exercise:

Participants were provided a Bayesian calculation Excel sheet and a form containing 8 pieces of evidence pertaining to Iran nuclear development (available here).  Participants were required to assess the prior probability, the probability of the evidence being directly related to Iran nuclear development, and the probability of the evidence occurring/appearing regardless of Iran’s nuclear weapon intentions.  The prior probability for the next evidence valuation was the revised probability result from the previous evaluation.  For example, the revised probability after evaluation Evidence A served as the prior probability before evaluating for Evidence B.

Once the evaluations were completed, all of the participants’ results were automatically recorded and graphed.  A aggregated average was also included (available here).

What did we learn from the Bayesian Forecasting Exercise

Although Bayesian is considered the ‘gold standard’ for probability revisions, the complex nature of intelligence work makes it troublesome to use simple Bayesian.  First, it is difficult to accurately weigh evidence without hindsight.  Second, selecting which evidence is worth considering without hindsight is equally cumbersome.