Jennifer Lee’s 2007 Mercyhurst thesis contains three experiments that suggest an efficient way for teaching a Bayesian reasoning technique to intelligence analysts and indicate that using natural frequencies when modeling a problem increases forecasting accuracy compared to modeling a problem using conditional probabilities when solving intelligence-related Bayesian problems. However, Lee found no significant difference in forecasting accuracy between a group using an experimental paper and pencil natural frequency tree method and a group using their own form of calculation to solve an intelligence-related Bayesian problem worded using natural frequencies as applied by a previous experiment conducted by Ulrich Hoffrage and Gerd Gigerenzer. Both groups using natural frequencies to solve Bayesian problems outperformed the group using traditional statistical language.
 |
| Performance of traditional statistics group forecasts compared to Hoffrage/Gigerenzer group forecasts. In the traditional group, 6 out of 34 analysts were within ten points of the correct answer. In the Hoffrage/Gigerenzer group, 26 of the 33 analysts were within ten points of the correct answer. |
- Intelligence-related Bayesian problems worded using natural frequencies will elicit higher Bayesian reasoning amongst intelligence analysts compared to intelligence-related Bayesian problems that are worded using traditional statistical language.
- Findings proved this hypothesis true.
- A paper and pencil frequency tree method that utilizes natural frequencies can easily be taught to intelligence analysts within 90 minutes and elicit a higher level of Bayesian reasoning than Bayesian problems that are only worded using natural frequencies.
- Findings proved the first part (ease of teaching) true, but the second part (superiority of frequency tree method) false.
- Bayes forces analysts to assess all pieces of evidence in a systematic way, thus eliminating any biases resulting from recentness or visibility.
- Bayesian procedures are transparent and therefore can be reproduced by other analysts who disagree with the final estimate.
- The way in which analysts model problems using Bayes forces the analysts to consider alternative explanations of the facts perceived
- Bayes forces numerical assignments when weighing pieces of evidence
- Bayes is less conservative than informal opinions and tend to drive the probabilities away from 50/50 faster and farther than subjective intuitive judgments do.
Critique
The article indicates that it is possible to teach analysts a Bayesian method in a small amount of time. Existing literature indicated that Bayesian reasoning compliments other analytic approaches when applied to strategic warning intelligence because Bayes helps ensure that analysts don't assign more meaning to the evidence available than is warranted. Strategic warning analysis focuses on the odds favoring an imminent attack over no imminent attack. Bayes enables analysts to quantify judgments, enables evaluation of evidence against hypotheses, and enables judgments about individual pieces of evidence instead of the sum of the evidence overall.
One of the hypotheses Lee describes to account for the failure of the natural frequency tree method to statistically outperform the natural frequency methods developed by Sedlmeier and Gigerenzer is the composition of the natural frequency tree group, which was composed primarily of freshmen and sophomores. Seniors made up the sample population of the natural frequency group without the tree method. The qualitative differences in maturity between the two groups could have been enough to affect the results of the natural frequency tree method forecasts but this could not be proven. An implication for future research is selection of participants that are more similar than different in class level dimensions.
Source:
Bayesian Reasoning Method For Intelligence Using Natural Frequencies
Posts
Even though Lee found that Bayesian is easy to teach, could it be possible that the students used in the experiment truly didn't understand Bayesian? That was quite a discrepancy between the two studies.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteKyle,
ReplyDeleteLee performed three different experiments. The first one tested performance on Bayesian problems that were worded using natural frequency compared to Bayesian problems worded using traditional statistical language.
The second experiment tested an experimental Bayesian reasoning method using the natural frequency tree format.
The third experiment tested the same experimental Bayesian reasoning method using the natural frequency tree format again to address concerns about the second experiment's methodology and results.
In the first experiment Bayesian reasoning as described by Ulrich Hoffrage and Gerd Gigerenzer outperformed Bayesian reasoning using traditional statistical language.
In experiments 2 and 3 the experimental natural frequency tree format failed to outperform the Bayesian reasoning as prescribed by Hoffrage and Gigerenzer. However the natural frequency tree format did not decrease forecasting accuracy. The natural frequency tree format just failed to yield better results than letting analysts use their own form of calculation to solve Bayesian problems like in the Hoffrage/Gigerenzer group from the first experiment.
Appendix G of the article contains a step by step approach to the frequency tree method with visuals.
Lee considered the possibility that the participants did not have the background to understand Bayesian, but discounted this hypothesis because out of the 66 combined freshman and sophomore participants in experiment 2, only 29 had not yet taken a statistics class.
Neither experiments 2 or 3 found statistically significant improvements of forecasting accuracy utilizing the natural frequency tree method compared to simply wording Bayesian problems in natural frequencies.
Participants did report a statistically significant improvement of their understanding of Bayesian after learning the natural frequency tree method so there was an auxiliary benefit.
Ricardo,
ReplyDeleteThe experiment found that students using Bayes did not have a significantly larger forecasting accuracy than those using the frequency tree method. Do you think this is a valid method that improves forecasting accuracy?
Harrison,
ReplyDeleteAll three experiments used Bayesian techniques. In the first experiment consisting of two groups, the independent variable was the use of natural frequencies in the wording of the problem instead of traditional statistical language. The dependent variable was the number of participants who were able to answer the question correctly, both in the natural frequencies
group and the traditional statistical language group.
The natural frequencies group outperformed the traditional frequencies group as shown in the picture.
In experiments 2 and 3, the performance of participants using an experimental structured natural frequency tree method was compared to the performance of the natural frequency group in experiment 1.
Participants in experiments 2 and 3 using the experimental structured natural frequency tree method did not perform better than the natural frequency group in experiment 1 at a statistically significant level. Performance was comparable. However, participants in experiments 2 and 3 reported an improved understanding of Bayesian reasoning at a statistically significant level.