Summary of Findings: Analysis of Competing Hypotheses (3 Of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2017 regarding ACH as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Analysis of Competing Hypotheses is a matrix-based analysis methodology created to analyze a complex problem using weighted evidence that allows  for the elimination of confirmation bias.  It accomplishes this via use of scientific methodology and placing numerical value on weight of evidence, relevancy, and the consistency or inconsistency that is determined by the analyst.  

Strengths:

• The thorough elimination of confirmation bias within the analytic process.
• ACH forces an analyst to confront a competing hypothesis via numerous bodies of evidence.
• ACH is a great analytical method to use when working with structured data.
• ACH is a good starting point in the analytic process.
• Helps show relationship, if any, between hypothesis and evidence.
• ACH is a good thinking tool, but it is one of many tools.
• ACH utilizes the scientific method and applies it to the practice of intelligence analysis.

Weaknesses:

• Still only a piece software.
• Time consuming.
• Still subject to cognitive biases in how evidence is weighed

• Only an algorithm at the end of the calculation, requires human judgment.
• ACH is dependent on discrete judgement of the analyst

How-To:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities.
2. Make a list of significant evidence and arguments for and against each hypothesis.
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “diagnosticity” of the evidence and arguments–that is, identify which items are most helpful in judging the relative likelihood of the hypotheses.
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them.
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.
8. Identify possible milestones for future observation that may indicate events are taking a different course than expected.

Application of Technique:

To demonstrate the application of the ACH matrix the class took a look at the question of will Hurricane Harvey cost more in federal spending than Hurricane Katrina? The class tested two hypotheses. The first was that Hurricane Harvey will cost more than Hurricane Katrina in federal spending  and the second hypothesis was that Hurricane Harvey will cost less than Hurricane Katrina in federal spending. The class did independent research for approximately ten minutes, and then combined their information into a single ACH matrix. The class then debated the credibility, relevancy, and the consistency of each piece of evidence to determine a weighted score in the ACH matrix. The class was able to objectively measure the strengths as well as the weaknesses of the methodology through the exercise.

For Further Information:

Chapter 8-Analysis of Competing Hypotheses

Palo Alto Research Center PARC ACH Download

Using Subjective Logic to Evaluate Competing Hypotheses

Summary and Critique by Claude Bingham
Original research by Simon Pope, Audun Jøsang of the CRC for Enterprise Distributed Systems Technology.


Summary
Intelligence analysis is difficult to define and even harder to evaluate in terms of quality. To analyze and improve upon the Analysis of Competing Hypothesis (ACH) process, one of the CIA's primary analysis methodologies, the researchers at the CRC for Enterprise Distributed Systems Technology turned to Subjective Logic. This addition to the ACH is intended to serve as a starting point for reasoning through hypotheses and their outputs. One of the major benefits, of the ACH-SL variant, is claimed to be removing the step that requires evaluating evidence biases. 

The CRC researchers give an extensive overview of the basic steps of the ACH process, highlighting important steps that can affect the validity of the analysis' outcome. The Steps to ACH are listed below:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities. 
2. Make a list of significant evidence and arguments for and against each hypothesis. 
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “diagnosticity” of the evidence and arguments–that is, identify which items are most helpful in judging the relative likelihood of the hypotheses. 
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value. 
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them. 
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation. 
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one. 
8. Identify possible milestones for future observation that may indicate events are taking a different course than expected.

Step 5 is seen as particularly important. In this step, an analyst must make a judgment about individual evidence; does it support or refute the hypotheses, does it have counterfactuals that negate its importance? These questions are important because the ACH methodology does not expressly require them to be asked for the methodology to function. Not doing so can result in too narrow a framing of the impact of a piece of evidence.

The researchers also examined ACH-Counter Deception (ACH-CD), a variant that looks at ways to counteract possible deception in evidence. The key point of mentioning this variant is that it lessens the chance that positively confirming evidence can lead to reasoning errors. False positives are likely when taking positively confirming evidence at face value. The researchers of this study contend that the ACH-CD has a minor flaw in that it requires the analyst to decide between evidence OR its counterfactual rather than making a subjective judgment call and evaluating both.  

Finally, the researchers lay out how the ACH-Subjective Logic (ACH-SL) methodology is different and what the steps entail. 

1. (ACH Step 1) 
2. (ACH Step 2) 
3. Prepare a model consisting of: 
1. A set of exhaustive and exclusive hypotheses – where one and only one must be true. 
2. A set of items of evidence that are relevant to one or more hypotheses; are influences that have a causal influence on one or more hypotheses; or, would disconfirm one or more hypotheses. 
4. Consider the evidence with respect to the hypotheses: 
1. For each hypothesis and item of evidence, assess its base rate. 
2. Should the evidence be treated as causal or derivative? Decide and record for each item of evidence or evidence/hypothesis pair. 
3. Make judgments for causal evidence as to the likelihood of each hypothesis if the evidence were true and if the evidence were false. 
4. Make judgments for derivative evidence as to the likelihood that the evidence will be true if the hypothesis were true, and if the hypothesis were false. 
5. From the judgments provided, compute the diagnosticity for each item of evidence. 
5. Measure the evidence itself and decide the likelihood that the evidence is true. Supply the measured evidence as input into the constructed model, and use the Subjective Logic calculus to compute the overall likelihood of each hypothesis. 
6. Analyze how sensitive the conclusion is to a few critical items of evidence. Changes in the value of evidence with high diagnosticity will alter the calculated likelihoods more than evidence with low diagnosticity. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation. 
7. (ACH Step 7) 
8. (ACH Step 8)

Because Subjective Logic is a form of calculus, ACH-SL requires analyst-supplied values for subjective judgments on each hypothesis and compares those values to empirically established base rates. Intelligence hypotheses are often one-off events so the base rate really does not exist and the curve is smoothed with a formula based on the number of competing hypotheses. The evidence goes through a similar process of comparing the base rates to the subjectively valued rates. 

Traditional ACH uses deductive reasoning through causal links between evidence and the hypotheses while ACH-CD uses abductive reasoning about derivative evidence, evidence that is not directly related to the hypothesis. There is a breakdown in these two methods because deductive reasoning about derivative evidence is difficult because the evidence is not causally related and the reverse is true of doing abductive reasoning about causal evidence. There, the evidence is more closely related to why a given hypothesis is possible and reasoning breaks down over what is and is not possible. 

Finally, the researchers introduce the calculus behind Subjective Logic and explain how the Bayesian data points of systems and subjective value judgments of analysts can be combined to form probabilistic conclusions. 

The researchers of this study conclude by stating that not only does ACH-SL work but that the Distributed Systems Technology Centre has created a technology framework called s ShEBA based upon it. 

Critique

This study is very well thought out and takes a broad approach to explaining the path ACH methodology has taken to get to its current iteration. While the math is dense and difficult to understand critically at a novice level, it shows a more open-minded approach. It give the chance for the semantic rationalizations and subjective valuations analysts already give to evidence to be quantifiable and less arbitrary. It gives more shape and order to the resulting Words of Estimative Probability. 

 

Extending Heuer's Analysis of Competing Hypotheses Method to Support Complex Decision Analysis

Extending Heuer’s Analysis of Competing Hypotheses Method to Support Complex Decision Analysis

Summary:

Marco Valtorta, Michael Huhns, Jiangbo Dang, Hrishikesh Goradia, and Jingshan Huang of the University of South Carolina used Bayesian Networks to extend the Analysis of Competing Hypotheses. They first explained what ACH is, followed by an explanation of Bayesian Networks. After the initial explanation of each matrix, they used a fictional situation to formulate an ACH matrix into a Bayesian matrix.

Before examining ACH and Bayesian Networks more in-depth, they give a brief explanation of another methodologies: Porter’s 5 Forces, Win-Loss Analysis, and Scenario Planning as possible rivals to the ACH matrix. Porter’s 5 Forces is an intelligence model designed for businesses as a method of examining threats in the industry: threat of entry, threat of substitution, buyer bargaining power, supplier bargaining power, rivalry between current competitors. Win-Loss analysis examines sales from a historical point of view to give feedback and possibilities for future sales. Scenario Planning is more strategic, as it uses multiple possible scenarios as a method of achieving a specific goal.

The researchers explained how ACH aids the intelligence process first by examining what the intelligence process is. Their explanation of the process divided it into three phases: the collection phase, analysis phase, and reporting phase. They went on to explain ACH as a methodology which weighs varying hypotheses using evidence for and against each hypothesis. This method gives reasoning for the validation or invalidation of the conclusions as well as explaining how the analysts arrived at their conclusions.

Bayesian Networks are acyclic, graphical models that shows probabilistic relationships between variables as well as combining data with prior knowledge. The complex nature of the model allows for a more in-depth analysis of ACH. In the Bayesian Network, the hypotheses of ACH are shown as modules connecting to the pieces of evidence.

They examined the limitations of using a Bayesian Network to model ACH hypotheses with the evidence. Basic Bayesian models cannot fully show all the variables without the context of the evidence without using a more complex model. The researchers got around this by adding more nodes to their model that provide the context for the evidence and giving lines between the hypothesis node and the evidence nodes if they were weighted for or against the hypothesis.

Critique:

While the use of Bayesian Networks to extend ACH into a visual model would be a good way to show hypotheses and their evidence for more visual decision makers, it has not been tested in the field. Furthermore, unless the audience knows how Bayesian networks work, the visual could be more confusing than a normal ACH matrix. A simpler explanation of Bayesian Networks would be useful when presenting it as a method of extending the ACH matrix. On a different note, the examination of rival methodologies i.e. Porter’s 5 Forces etc., was useful for the comparison with ACH, but not necessarily relevant to the paper.

Citation: Valtorta, Marco, Huhns, Michael, Dang Jiangbo, Goradia, Hrishikesh, and Huang, Jingshan. “Extending Heuer’s Analysis of Competing Hypotheses Method To Support Complex Decision Making.” 10 February 2005. University of South Carolina. Accessed 29 August 2017. https://cse.sc.edu/~MGV/reports/TR2005-01.pdf. 

 

How Does Analysis of Competing Hypotheses (ACH) Improve Intelligence Analysis?

Summary and critique by Jared Leets

Summary:

Richards Heuer Jr discusses the three different approaches to Analysis of Competing Hypotheses (ACH) and four significant steps in the intelligence analysis process which include researching relevant information, organizing information to help with analysis, reviewing the information to make a credible evaluation, and finally writing the product.

Heuer Jr states that when searching for information, the purpose is to get the analyst to challenge his or her initial mindset. Researching alternative hypotheses helps the analyst broaden his or her search for information that they typically would not look for. When analysts initially work with ACH they consider it beneficial due to the fact that it forces them to think about evidence and outcomes that had never occurred to them. In the first step ACH is an easy strategy for questioning a complex problem. Every analyst should seriously consider questioning their traditional way of answering a question to a problem.

The second step is organizing the information. Two principles, decomposition and externalization, exist to help cope with human mental perception and memory. Decomposition refers to breaking down the problem while externalization refers to removing the problem out of someone’s head and writing it down in order to simplify the problem and show the variables. The purpose is to show all of the evidence and what type relationship they share if any. The three types of ACH manual, automated, and Bayesian all have their differences. The manual ACH views only the relationship between each individual piece of evidence and hypotheses. The matrix will assist the analyst in evaluating evidence that is most indicative in opposing hypotheses

The automated ACH allows analysts to place evidence in categories. The categories include date, type of source, credibility of source, and the relevance of the source. By sorting the evidence it makes it much easier for the analyst to put all his or her efforts towards the most convincing evidence. The Bayesian ACH places more importance on the relationship between the evidence and hypotheses. It typically has multiple sets of hypotheses, which tend to increase judgments that must be made in the analytical process.

Heuer Jr states that in the third step, analyzing the information, the automated ACH will sort the evidence by categories while the Bayesian ACH will give a mathematical algorithm that shows the analyst a possible answer. Sorting evidence by relevance in categories significantly helps with intelligence analysis. Heuer Jr explains that in the automated version an analyst can compare evidence from a clandestine source or open source can help reveal deception from a source. While the inconsistency and weighted inconsistency scores can produce an accurate estimate, the analyst must use their own judgment in the end. According to Bayesian ACH proponents, they claim that using an Inconsistency Score or Weighted Inconsistency Score is problematic since it relies on incomplete information. A Bayesian inference is ideal for making probabilities bases on judgments from intelligence analysts. However, it is very complex, time consuming, and requires a Bayesian methodologist to help the analyst when it is being conducted. Proponents state that this displays how complex an intelligence problem can be and that one single analyst cannot make a judgment call without help from other sources. In the final step, writing the product, Heuer Jr says each ACH has its benefits and can help the analyst in reinforcing what their research has revealed or shows the analyst an alternative path for additional research.

Critique:

ACH has it benefits and its obstacles. For example, it is very good at helping an analyst see alternative hypotheses and pieces of evidence and weighing that evidence to help make a correct decision. In any intelligence project an analyst must think from different points of views. ACH can help an analyst go step by step and see if their current hypothesis will answer the intelligence requirement. However, ACH can also have its issues. As Heuer Jr explains, when weighing evidence it is easy to be biased, especially when deciding how credible or relevant a piece of evidence is. In addition, software can only help an analyst come to a decision. If an analyst says that ACH told them the answer and base everything off of that, they will likely lose credibility. In the end, ACH can help with intelligence analysis by helping the analyst think of different ideas to address the problem and support their research, but it can also be a problematic if analysts rely too much on it as software cannot answer problems in the intelligence profession.

Source: Heuer Jr, Richards. (October 16, 2005). How Does Analysis of Competing Hypotheses Improve Intelligence Analysis? Pherson Associates. http://www.pherson.org/wp-content/uploads/2013/06/06.-How-Does-ACH-Improve-Analysis_FINAL.pdf

Why are we not evaluating multiple competing hypotheses in ecology and evolution?

By : Praveen Kumar Neelappa

This article suggests that there is the gap between theory and practice in the use of analysis of competing hypotheses (ACH). It identifies several intellectual and practical barriers that discourage the use of multiple hypotheses in the field of ecology and evolution. This article points out that scientist have a bias or a motivation to consider one hypothesis over other (Intellectual barriers) and there are practical limitations inherent to factorial design, the standard experimental design that allows researchers to evaluate several explanatory variables and their interactions in the same study, one variable at a time (Practical barrier).

Cognitive bias makes us think that we are making a logical, rational and effective decision while considering the alternative hypothesis, but our unconscious bias influences the experiment and its outcome. There is a tendency of scientist to put more weight on evidence that supports favored ideas more than other evidence that is available (Confirmation bias), seek for the pattern in their experiment (Pattern seeking bias) and be judged only by their internal consistency (Belief bias). There are several ways one might minimize the effect of cognitive bias in science so that one does not rely exclusively on one’s perceptions. It can be achieved by masking (kept) information about the experiment from the participant, to reduce or eliminate bias, until after a trial outcome is known (Blind bias), working with other scientists with different perspectives (Work with the enemy) and a null model which generates a pattern in the absence of any biological process, forcing the researcher to think about many different hypotheses, which could potentially minimize the negative impacts of cognitive biases in science.

Any study that has a simple, easy to understand explanation will be preferred over a study that employs complex and perhaps less-elegant ideas (Simplicity bias) to avoid practical barrier. Editors and reviewers tend to rely on prior knowledge when evaluating a manuscript, creating additional difficulties for researchers when publishing studies that confront well-established ideas. This tension between new and old ideas could reflect a conflict between new and old generations (Publication bias).

The article concludes by suggesting that ecological and evolutionary research is aimed at understanding patterns arising from nonlinear and stochastic interactions among a multitude of processes and agents at multiple spatial and temporal scales. If we wish to truly advance scientific progress despite this complexity, we must better commit to strong inference in our scientific inquiries by simultaneously evaluating multiple competing hypotheses.

Critique:

The use of ACH is widely promoted to enhance the effectiveness of the scientific investigation. This article points out some valid draw backs of using ACH in the different field of studies and discusses these drawbacks and solutions to them in detail. The article clearly illustrates various types of biases and the various stages where they can be encountered while carrying out experiments. It is imperative that the individual carrying out the experiment is objective in data collection and maintains an objective view at every stage of the experiment which will be the best way to counter any possibility of the final results being biased. Additionally, measures need to be put in place to reassess for bias along the way to ensure the results are void of any form of bias.

Citation:

Betini GS, Avgar T, Fryxell JM. 2017 Why are we not evaluating multiple competing hypotheses in ecology and evolution?.R. Soc. open sci. 4: 160756. http://dx.doi.org/10.1098/rsos.160756

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

By: Matt Haines

Source: https://pdfs.semanticscholar.org/50a1/730ce13238d17c6c2f48c3621d4489c86d6a.pdf

Summary:

Gregorio Convertino, Dorrit Billman, Peter Pirolli, JP Masur, and Jeff Shrager created a virtual environment to conduct standard analysis of competing hypotheses(ACH) and then they analyzed the effects of that environment. The authors begin by defining CACHE as a collaborative analysis of competing hypotheses environment. Then, they explain the difficulties an analyst faces in everyday intelligence tasks. An analyst is faced with tasks that span a vast multitude of areas of expertise on a daily basis and biases influence all of those analytical products. The authors then go into detail of what the CACHE framework actually does in order to combat this challenge. CACHE allows a user to search through all available evidence, input that evidence into a personal ACH matrix, view other team members ACH matrix, and communicate with other team members through an instant messaging system.

Before completing the actual test of the CACHE framework, the authors hypothesized that:

Heterogeneous groups would show less confirmation bias than Homogeneous groups. Because CACHE supports sharing information among participants, the differing views in the heterogeneous groups should mitigate cognitive biases by 1) exposure to more, and less-biased, evidence and 2) access to alternative analyses provided by partners.

and that,  “Heterogeneous groups would show no net process loss relative to the Solo/Nominal Group. CACHE will mitigate the process costs, producing equivalent or better performance in heterogeneous groups.” The results of the experiments were concurrent with the authors’ hypotheses.

Critique:

The CACHE framework is a great idea and prototype for groups where not every person can be in the same location at the same time. However, the authors of this paper did not do much to actually prove anything. This paper laid out a product. It did not add to the ACH process nor did it attempt to contest normal assumptions of ACH. However, CACHE has achieved something just by allowing analysts to be in two different places at once and collaborate with each other. This feature can help eliminate some group think biases because it takes some power away from those members of the team who are better presenters. For example, one of the major complaints many international students have, is that they feel like their ideas are not heard, because they cannot vocally command a room. By allowing analysts to work remotely, international students can have the same voice as a native english speaker.