Jennifer Lee’s 2007 Mercyhurst thesis contains three experiments that suggest an efficient way for teaching a Bayesian reasoning technique to intelligence analysts and indicate that using natural frequencies when modeling a problem increases forecasting accuracy compared to modeling a problem using conditional probabilities when solving intelligence-related Bayesian problems. However, Lee found no significant difference in forecasting accuracy between a group using an experimental paper and pencil natural frequency tree method and a group using their own form of calculation to solve an intelligence-related Bayesian problem worded using natural frequencies as applied by a previous experiment conducted by Ulrich Hoffrage and Gerd Gigerenzer. Both groups using natural frequencies to solve Bayesian problems outperformed the group using traditional statistical language.
Lee tested two hypotheses:
- Intelligence-related Bayesian problems worded using natural frequencies will elicit higher Bayesian reasoning amongst intelligence analysts compared to intelligence-related Bayesian problems that are worded using traditional statistical language.
- Findings proved this hypothesis true.
- A paper and pencil frequency tree method that utilizes natural frequencies can easily be taught to intelligence analysts within 90 minutes and elicit a higher level of Bayesian reasoning than Bayesian problems that are only worded using natural frequencies.
- Findings proved the first part (ease of teaching) true, but the second part (superiority of frequency tree method) false.
- Bayes forces analysts to assess all pieces of evidence in a systematic way, thus eliminating any biases resulting from recentness or visibility.
- Bayesian procedures are transparent and therefore can be reproduced by other analysts who disagree with the final estimate.
- The way in which analysts model problems using Bayes forces the analysts to consider alternative explanations of the facts perceived
- Bayes forces numerical assignments when weighing pieces of evidence
- Bayes is less conservative than informal opinions and tend to drive the probabilities away from 50/50 faster and farther than subjective intuitive judgments do.
The article indicates that it is possible to teach analysts a Bayesian method in a small amount of time. Existing literature indicated that Bayesian reasoning compliments other analytic approaches when applied to strategic warning intelligence because Bayes helps ensure that analysts don't assign more meaning to the evidence available than is warranted. Strategic warning analysis focuses on the odds favoring an imminent attack over no imminent attack. Bayes enables analysts to quantify judgments, enables evaluation of evidence against hypotheses, and enables judgments about individual pieces of evidence instead of the sum of the evidence overall.
One of the hypotheses Lee describes to account for the failure of the natural frequency tree method to statistically outperform the natural frequency methods developed by Sedlmeier and Gigerenzer is the composition of the natural frequency tree group, which was composed primarily of freshmen and sophomores. Seniors made up the sample population of the natural frequency group without the tree method. The qualitative differences in maturity between the two groups could have been enough to affect the results of the natural frequency tree method forecasts but this could not be proven. An implication for future research is selection of participants that are more similar than different in class level dimensions.
Bayesian Reasoning Method For Intelligence Using Natural Frequencies