Using Subjective Logic to Evaluate Competing Hypotheses

Summary and Critique by Claude Bingham
Original research by Simon Pope, Audun Jøsang of the CRC for Enterprise Distributed Systems Technology.


Summary
Intelligence analysis is difficult to define and even harder to evaluate in terms of quality. To analyze and improve upon the Analysis of Competing Hypothesis (ACH) process, one of the CIA's primary analysis methodologies, the researchers at the CRC for Enterprise Distributed Systems Technology turned to Subjective Logic. This addition to the ACH is intended to serve as a starting point for reasoning through hypotheses and their outputs. One of the major benefits, of the ACH-SL variant, is claimed to be removing the step that requires evaluating evidence biases. 

The CRC researchers give an extensive overview of the basic steps of the ACH process, highlighting important steps that can affect the validity of the analysis' outcome. The Steps to ACH are listed below:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities. 
2. Make a list of significant evidence and arguments for and against each hypothesis. 
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “diagnosticity” of the evidence and arguments–that is, identify which items are most helpful in judging the relative likelihood of the hypotheses. 
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value. 
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them. 
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation. 
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one. 
8. Identify possible milestones for future observation that may indicate events are taking a different course than expected.

Step 5 is seen as particularly important. In this step, an analyst must make a judgment about individual evidence; does it support or refute the hypotheses, does it have counterfactuals that negate its importance? These questions are important because the ACH methodology does not expressly require them to be asked for the methodology to function. Not doing so can result in too narrow a framing of the impact of a piece of evidence.

The researchers also examined ACH-Counter Deception (ACH-CD), a variant that looks at ways to counteract possible deception in evidence. The key point of mentioning this variant is that it lessens the chance that positively confirming evidence can lead to reasoning errors. False positives are likely when taking positively confirming evidence at face value. The researchers of this study contend that the ACH-CD has a minor flaw in that it requires the analyst to decide between evidence OR its counterfactual rather than making a subjective judgment call and evaluating both.  

Finally, the researchers lay out how the ACH-Subjective Logic (ACH-SL) methodology is different and what the steps entail. 

1. (ACH Step 1) 
2. (ACH Step 2) 
3. Prepare a model consisting of: 
1. A set of exhaustive and exclusive hypotheses – where one and only one must be true. 
2. A set of items of evidence that are relevant to one or more hypotheses; are influences that have a causal influence on one or more hypotheses; or, would disconfirm one or more hypotheses. 
4. Consider the evidence with respect to the hypotheses: 
1. For each hypothesis and item of evidence, assess its base rate. 
2. Should the evidence be treated as causal or derivative? Decide and record for each item of evidence or evidence/hypothesis pair. 
3. Make judgments for causal evidence as to the likelihood of each hypothesis if the evidence were true and if the evidence were false. 
4. Make judgments for derivative evidence as to the likelihood that the evidence will be true if the hypothesis were true, and if the hypothesis were false. 
5. From the judgments provided, compute the diagnosticity for each item of evidence. 
5. Measure the evidence itself and decide the likelihood that the evidence is true. Supply the measured evidence as input into the constructed model, and use the Subjective Logic calculus to compute the overall likelihood of each hypothesis. 
6. Analyze how sensitive the conclusion is to a few critical items of evidence. Changes in the value of evidence with high diagnosticity will alter the calculated likelihoods more than evidence with low diagnosticity. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation. 
7. (ACH Step 7) 
8. (ACH Step 8)

Because Subjective Logic is a form of calculus, ACH-SL requires analyst-supplied values for subjective judgments on each hypothesis and compares those values to empirically established base rates. Intelligence hypotheses are often one-off events so the base rate really does not exist and the curve is smoothed with a formula based on the number of competing hypotheses. The evidence goes through a similar process of comparing the base rates to the subjectively valued rates. 

Traditional ACH uses deductive reasoning through causal links between evidence and the hypotheses while ACH-CD uses abductive reasoning about derivative evidence, evidence that is not directly related to the hypothesis. There is a breakdown in these two methods because deductive reasoning about derivative evidence is difficult because the evidence is not causally related and the reverse is true of doing abductive reasoning about causal evidence. There, the evidence is more closely related to why a given hypothesis is possible and reasoning breaks down over what is and is not possible. 

Finally, the researchers introduce the calculus behind Subjective Logic and explain how the Bayesian data points of systems and subjective value judgments of analysts can be combined to form probabilistic conclusions. 

The researchers of this study conclude by stating that not only does ACH-SL work but that the Distributed Systems Technology Centre has created a technology framework called s ShEBA based upon it. 

Critique

This study is very well thought out and takes a broad approach to explaining the path ACH methodology has taken to get to its current iteration. While the math is dense and difficult to understand critically at a novice level, it shows a more open-minded approach. It give the chance for the semantic rationalizations and subjective valuations analysts already give to evidence to be quantifiable and less arbitrary. It gives more shape and order to the resulting Words of Estimative Probability.