Summary of Findings: Analysis of Competing Hypotheses ( 3 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University, in September 2018 regarding Analysis of Competing Hypotheses as an Analytic Method, specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

“Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.” - Richard Heuer, Central Intelligence Agency

Analysis of Competing Hypotheses (ACH) is a structured analytic technique designed to give analysts a framework for organizing information to formulate an estimate. ACH also is useful in resolving goal conflicts in a simple and repeatable manner. ACH gives analysts a skeleton in which to enter in definitive, contrasting hypotheses and assess the evidence on how it confirms or disconfirms the given hypotheses. The hypothesis with more confirming evidence becomes the more likely hypothesis. While ACH is a well-known methodology, it has not been thoroughly tested to assess its validity in academic literature.

Strengths:

• Consideration of alternative viewpoints beyond the ultimately proposed analytical theory
• Designed to simplify and analyze intelligence problems
• Mitigates cognitive biases
• Analyst’s reasoning is more transparent
• Easy to use
• Allows for categorizing and sorting of evidence in useful ways, e.g., time, relevance, source reliability

Weaknesses:

• Unclear how much evidence is appropriate/most effective
• Too much or too little evidence can negatively influence the analyst’s estimate
• Relationships between pieces of evidence unclear/unidentified
• Some evidence is only important when considered in conjunction with other evidence
• Doesn’t identify pre-existing assumptions  
• ACH can eliminate necessary context

How-To:

1.      Identify possible hypotheses

2.     Compile a list of evidence

3.     Enter evidence into matrix

a.     Hypotheses across the top

b.     Evidence down the side

4.     Determine whether evidence is consistent or inconsistent with each hypothesis

a.     Consider (and weigh when necessary) the credibility of source for evidence

b.     Consider (and weigh when necessary) the relevance of evidence to the hypothesis

5.     Draw conclusion regarding which hypothesis is most consistent/inconsistent with the evidence included in the matrix

a.     Use conclusion to support estimate

b.     If conclusion does not support either hypothesis, collect more evidence and reevaluate ACH matrix

Application of Technique:

We wrote an investigative narrative regarding a crime and the analysts were asked to determine the likelihood that a suspect did or did not murder the victim. The narrative was presented to the analysts in paragraph form and evidence was included throughout the narrative.

After reading the narrative, analysts were asked to conduct the ACH as individuals, identifying their own key pieces of evidence and determining for themselves whether they were consistent or inconsistent with each hypothesis. After an allotted time to conduct their own ACH’s, analysts were then asked to share their conclusions with the class. The ensuing discussion brought forth issues regarding which evidence was important to the investigation, which evidence was most relevant to the hypotheses, and which sources for the evidence were most credible.

For Further Information:

1.      Heuer on ACH: Chapter 8 of “Psychology of Intelligence Analysis” from the Central Intelligence Agency

2.     Wheaton on ACH: Analysis of Competing Hypotheses and Structured Analysis of Competing Hypotheses: What Are The Links With Foresight?

3.     Brasfield on ACH: Forecasting Accuracy and Cognitive Bias in ACH

4.     Van Gelder on ACH: https://timvangelder.com/2007/12/31/hypothesis-testing-whats-wrong-with-ach/

How Does Analysis of Competing Hypotheses (ACH) Improve Intelligence Analysis?

Summary and Critique by Michael Z
Richards J. Heuer, Jr., 16 October 2005

The meat of Heuer’s article really begins with him first explaining how Analysis of Competing Hypotheses (ACH) aids in the search for information at the very beginning stages of analysis. He notes that it is in this first step that “ACH does the most to help the analyst avoid being caught by surprise.” This is achieved by forcing the analyst to consider a much wider array of information that would otherwise be pursued by the analyst and is further achieved by casting a different perspective on what the analysts already knows as a result of their focus on rejecting hypotheses. Heuer takes time to contrast ACH with what he calls a “satisficing” approach in which an analyst knowledgeable on the topic of analysis develops a favored hypothesis and seeks evidence to confirm it. Ultimately Heuer states that while such an approach is efficient and and oftentimes safe, that it does not protect the analyst from surprise and cannot distinguish that certain pieces of evidence may be consistent with more than the favored hypothesis.

Heuer then explains how ACH is beneficial in the assembly and organization of information used in analysis. He explains that all three forms of ACH abide by the basic principles of decomposition and externalization, which he notes as two principles which are present in all simple tools that analysts use to overcome limitation in human cognition. All three forms of ACH achieve through the use of of an analytical matrix that allows the user to break down the issue into its component parts. Heuer then goes into detail as to how each three forms of ACH, (Manual ACH, Automated ACH, and Bayesian ACH), utilize their own matrix in order to assist in making judgements.

The next section of the article focuses on how ACH assists the analyst in the actual analysis of the information. In this section he notes a great distinction between Bayesian ACH and the other two forms of ACH, and this is that Bayesian ACH provides the analyst with an algorithm that supplies the analyst with an answer, while the other two forms of ACH are instead utilized as a process that aids in the analysts ultimate judgement on the issue. After which he then goes into further detail about each ACH method.

In the next section of the article Heuer takes note of the advantages of ACH during the writing of a report. One of the greatest benefits provided by ACH is the consideration of alternative viewpoints beyond the ultimately proposed analytical theory. Another advantage listed is that ACH allows for one to present the consideration and rationale that each piece of evidence was given and provided for the ultimate analysis. In provided such information in the report readers can more easily understand how exactly the final analytical judgement was achieved.

Critique:

Overall I am very satisfied with Heuere’s explanation and presentation on how analysis can benefit from ACH at each step of the denoted analytical process. This method of presenting and explaining ACH flowed logically and allowed the ready to immediately grasp how ACH would improve their analysis.

However I felt like the article would have benefited from further discussion of how the analyst navigates each step of the process along each type of ACH. For example an explanation of how an analyst decides which categories to utilize in Automated ACH and how they make judgements about the weight and reliability of each piece of evidence. Ultimately though this is a very minor criticism of an otherwise great explanatory article by the developer of ACH himself.

Resolving Goal Conflicts Via Argumentation-Based Analysis of Competing Hypotheses

Resolving Goal Conflicts Via Argumentation-Based Analysis of Competing Hypotheses

Pradeep K. Murukannaiah, Anup K. Kalia, Pankaj R. Telang, and Munindar P. Singh, Department of Computer Science, North Carolina State University, Cisco Systems Inc.

Pradeep Murukannaiah (et. al) asks the fundamental question of: what makes stakeholders’ goals conflict? The purpose of the writings and the study done by the authors is to understand how stakeholders process goals to identify the sources of their conflicts and to resolve goal conflicts by studying stakeholders’ beliefs about various goals. They do this with the key understanding that goals conflict when stakeholders have contradictory beliefs about supporting or opposing those goals. Murukannaiah (et. al) resolve these goals by using Analysis of Competing Hypotheses (ACH) technique and infusing an argument based ACH technique into their problem-solving process (Arg-ACH).  

The first section gives a background of what the study is analyzing and what some of the details behind the study are. Section two looks at ACH and applies it for resolving goal conflicts for stakeholders in a sample scenario involving protections of a city against terrorist attacks. The scenario includes the city’s train systems and the city’s hotels and their security. Section three describes Arg-ACH and shows how it assists in systematically eliciting stakeholders’ beliefs and building arguments from beliefs to support or opposition of goals. Sections four and five detail the design of the authors empirical study and the findings that they received from it. Section 6 showed related work and section 7 provided the conclusion that included the finding that Arg-ACH yields high quality analysis reports.

Critique:

Overall, I believe the authors of this article and study did a good job of explaining the ins and outs of ACH. After reading it for a second time, the Arg-ACH technique began to make sense. Their examples and scenarios for tying the two concepts together were very helpful and helped to put it in a real-world perspective. The related works section and parts of the information on the study itself were at times hard to keep up with and keep a clear line of understanding while reading. As the authors talked about in the conclusion, with the ever-improving realm of technology, it is likely that information retrieval abilities will continue to grow and ultimately make Arg-ACH a more popular technique for solving goal conflicts. 

Murukannaiah, P. K., Kalia, A. K., Telangy, P. R., & Singh, M. P. (2015). Resolving goal conflicts via argumentation-based analysis of competing hypotheses. 2015 IEEE 23rd International Requirements Engineering Conference (RE). doi:10.1109/re.2015.7320418. Retrieved from https://www.csc2.ncsu.edu/faculty/mpsingh/papers/mas/RE-15-Arg-ACH.pdf

Critical Epistemology for Analysis of Competing Hypotheses by Prof. Nicholaos Jones, Ph.D

Summary:

In his 2018 article in the Journal of Intelligence and National Security, Mr. Nicholaos Jones, a Professor of Philosophy at the University of Alabama at Huntsville, gives an epistemological critique of Analysis of Competing Hypotheses (ACH).

Mr. Jones begins by laying out his process for evaluating hypothesis testing methodologies. He first asserts that a methodology is “good” or “better” based on how it meets the reasoning goals of the analytical process. He identifies two initial reasoning goals: reliability and discrimination. Jones defines reliability in terms of how some body of evidence regards one hypothesis is more likely than a second hypothesis; a hypothesis that is more truthful than its alternative(s) will rank higher. In this case, discrimination means being able to partition hypotheses into ranks. Jones states that the reasoning criteria/goal of discrimination is easier to meet than reliability. He also states that discrimination is necessary, but not necessary to be approximate, to meet the criteria of reliability.

Before identifying alternative/competing methodologies to ACH, in this case Falsificationism (strict falsification), Bayesianism, Explanationism (inference of best explanation), Jones identifies additional reasoning goals of tractability and objectivity. Jones states that a methodology has tractability if it is efficient and elegant (i.e. simple) to use. Objectivity means the ability to minimize the effects of cognitive bias on reasoning by reducing the reliance on “subject evidence.” In assessing alternative methodologies, Jones states, in his assessment, that determining reliability of a method is difficult and avoids it. Each of the three identified alternatives have two strong and one weak reasoning goals.

Jones believes that ACH does about as well as alternative methodologies in terms of three reasoning goals that he stated were easier to assess. Jones believe the core of ACH is the third through fifth steps of ACH as laid out by Heuer. Jones interprets these steps through either strict or colloquial interpretations of Heuer’s third step on evidence consistency with regards to the hypothesis. Strict interpretation requires the absence of any logical contradictions in evidence whereas the colloquial interpretation requires evidence be “at least weakly or moderately plausible.” Under both interpretations Jones, regards ACH as stronger than alternative methodologies.

Jones then decides to complicate his ongoing analysis with a final reasoning goal: stability. In order to define stability, Jones breaks down hypothesis testing into three steps: 1) Inputs; 2) Operation; and 3) Rankings. Stability in this case means that every execution of a hypothesis testing (methodology) operation, using the same evidence or inputs, will yield the same result. Jones suggests that once stability is added, ACH becomes unstable. He lists three scenarios to highlight his critique: “first, when multiple competing hypotheses are consistent with all available evidence; second, when exactly one hypothesis is consistent with all available evidence; third, when none of the competing hypotheses are consistent with all available evidence.” The first case represents a problem of abundant fit, which Jones believes can be remedied with more evidence which changes the nature of the problem into the second or third case. Meanwhile the second case represents a problem of redundancy because if all evidence fits a single hypothesis, then revising competing theses, deleting or simplifying evidence, and further refining the process will yield an identical result thus adding complexity to the methodology, making it less tractable.

The third case highlights Jones case that ACH is unstable under both strict and colloquial interpretations of evidence consistency, albeit for different reasons. In a strict interpretation of the third case, ACH gives no way of determining which of the inconsistent hypotheses is more likely. In a colloquial interpretation, Jones argues that in cases where a generalized piece of evidence is separated into its constituent pieces of evidence, arbitrary factors influence the outcome. So, in a case where the likely hypothesis doesn’t chance, the individualization of evidence can affect which of the other two hypotheses is more or less likely, making ACH unstable.

Jones the suggests that by favoring generality of evidence or increasing nuance or subtlety of evidence diagnosticity, the stability critique can be avoided. Jones argues that the generality rule fails because 1) it is too strong; 2) the rule is ad hoc and presents bias; and 3) the rule is unnecessary. Jones then argues that considering the diagnosticity of the evidence, as suggested by Heuer, may provide the solution to the generality problem. Jones concludes that weighting diagnosticity of evidence further complicates ACH because of the way to select for individualized evidence is arbitrary and therefore does not lead to stability.

Jones believes that ACH is superior to other discriminatory methodologies by using a procedure to “aggregate consistency judgements.” While it is superior in this regard, ACH’s shortcoming is in the necessity to summarize evidence over specific individual items within the evidence. Jones sees this a structural flaw in ACH. In his view, there is no adequate solution to what he terms as the Generality Problem. Jones ultimately suggests that the methodology of ACH does not matter as much as the way the analyst uses the method and the analysts “luck or intuition.” While an analyst can yield effective results from ACH, Jones’ key criticism is in the lack of universal tactics for ranking/individuating between analysts’ evidence. He suggests that further research on ACH focus on techniques for individuating evidence, which will enhance transparency of the ACH operation and eliminating arbitrary or subjective bias into their analytic products.

Criticsim:

My primary disagreement with Jones’ approach to ACH is that he only tackles the methodology from the approach of a pure academic, as opposed to that of a practitioner. As an analyst, we are taught that there are logical shortcomings from the get go. The application of ACH to an analytic problem is intended to be a guide when examining your hypotheses and evidence. Jones’ critique that the process is redundant is correct but, in my opinion, I believe he misses the primary purpose of ACH. The methodology provides a structured approach to evaluating evidence that may be self-evident but also allows the analyst to ask essential questions of the process: are the hypotheses too broad/narrow? Is there more evidence to be found? Is the evidence redundant? Etc… In Chapter 8 of The Psychology of Intelligence Analysis, Heuer starts by stating,

“Analysis of competing hypotheses…is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve… Because of its thoroughness, it is particularly appropriate for controversial issues when analysts want to leave an audit trail to show what they considered and how they arrived at their judgment.”

In the excerpt shown above, Heuer clearly writes that ACH is a tool for the analyst to question his process and show their steps in their work. Again, by purely approaching ACH from an academic or logical standpoint, I believe Jones overplays ACH’s pitfalls which are clearly highlighted upfront by Heuer. 

Going back to the redundancy issues that Jones highlights, I believe that, again, Jones’ misses the point. The grand simplicity of ACH is that the process is easily repeated to show all possible alternatives. While that adds, in Jones’ view, complexity, it is as Heuer states, the auditing process for the analyst to show how he came to his final analytic conclusion.

Overall, I agree with Jones’ conclusion that a procedure or technique for objectively individualizing evidence would enhance ACH’s reliability and improve analytic products. 

Link: https://www.tandfonline.com/doi/abs/10.1080/02684527.2017.1395948

Hypothesis testing—What’s Wrong with ACH?

Tim van Gelder, 31 December 2007

Summary and Critique by Jillian J

Van Gelder begins by describing the ACH method—whereby an analyst can “determine which of a range of hypotheses is most likely to be true, given the available evidence”  and identifies the hypothesis-testing structure and external representation aid as some of the method’s strengths. He then lodges five complaints about the method: too many judgements, no e is an island, flat structure of hypotheses, subordinate deliberation, and decontextualisation and discombobulations.

Van Gelder writes that analysts must make a judgement of consistency for each piece of evidence they enter into the matrix, further stating that the number of judgements can quickly become cumbersome. On top of that workload, the relationship between a given piece of evidence (e) and the given hypothesis (h) may be irrelevant or inconclusive.

Next he cites the matrix structure as a problem that makes ACH treat “an item of evidence as consistent of inconsistent on its own with each of the hypothesis”. He writes that while the analyst may deem e consistent with h, this judgment is only valid in the context of other relevant information (or auxiliary hypotheses). If that other information said the opposite, then that same e would be inconsistent with h—in essence, a multi-premise structure. He acknowledges the short term utility of organizing h’s and e’s, but writes that further along the process, if the analyst finds information that challenges an initial assumption such that e1 is inconsistent with h only until combined with additional information (a), she/he has a problem. ACH doesn’t allow for that type of nuance.

The flat structure of hypotheses presents an issue because hypotheses can be, and often are complex. Van Gelder posits that ACH doesn’t efficiently address the multiple facets of complex hypotheses.

His fourth qualm is that ACH doesn’t have a way of weighting the salience of a given e. ACH allows the analyst to judge the magnitude of consistency (very consistent, consistent, neutral, not applicable, inconsistent, or very inconsistent), but doesn’t let the analyst delineate how seriously she/he takes the e itself.

The final issue is that while ACH tries to strip away excess details, the result is an e without context. This leaves the analyst uncertain of the relationship between e and h which leads to a muddied analysis.

Critique: 

I would add that ACH can also perpetuate cognitive biases. When I search for evidence, sometimes the result is a(n) “(in)consistency heavy” matrix. Then I might actively seek out disconfirming evidence to make the inconsistency to consistency ratio a little closer. While it’s useful to search for disconfirming evidence, I have to stop somewhere. I run the risk of choosing a stopping point that fits my bias.

I also think there’s a quick fix for van Gelder’s fourth issue about weighting salience. ACH allows the analyst to control the order in which the evidence appears on the matrix. It would be easy for the analyst to rank the evidence according to importance.

I found the flat structure critique only moderately valid. ACH allows the analyst to create virtually unlimited matrices. It may be tedious, but it is possible to break down a complex hypothesis into multiple matrices and apply evidence to each facet.

Overall, I agree with van Gelder. ACH has problems and its utility is limited. The issues he identified resonated with the frustrations I've had while using the method. But I maintain that challenging our analyses is as important as coming up with our analyses in the first place. Therefore, even with its flaws, analysts should apply the principles of ACH to their analyses, if not the ACH method itself. 

Extending Heuer’s Analysis of Competing Hypotheses

Extending Heuer’s Analysis of Competing Hypotheses

Method to Support Complex Decision Analysis

Marco Valtorta (mgv@cse.sc.edu), Michael Huhns, Jiangbo Dang, Hrishikesh Goradia, and Jingshan Huang Center for Information Technology TR-CIT05-03 Department of Computer Science and Engineering TR-2005-001 University of South Carolina

Marco Valtorta (et al.) looks into how the analysis of competing hypotheses can be combined with Bayesian networks.  The authors break down the article into eight sections.  The first section following the introduction compares ACH to other related work but indicate that the related work mostly applies to industrial problems.  The related work detailed includes: Porter’s 5 Forces, Win-Loss Analysis, and Scenario Planning.

The body of the research begins with the third section. The third section contains a detailed breakdown on how an individual can utilize analysis of competing hypothesis in a simulated example involving a terrorist attack on the Iran oil industry. The example Hypotheses are H1: Terrorists will bomb the oil refineries in Abadan, H2: Terrorists will bomb the oil pipelines in Abadan, H3:  Terrorists will bomb the oil wells in Abadan,  H4: Terrorists will bomb the oil facilities in Shiraz,  in Shiraz. H5: Terrorists will not launch an attack.  The hypotheses are displayed with corresponding evidence in an ACH table.  

 The fourth section, continuing with the example used with the ACH method, uses Bayesian networks analyze the problem. ACH tables are represented by Valtorta (et al.) via bipartite graphs creating sets and respective nodes for the hypotheses and evidence.

The authors compare ACH and Bayesian Networks in the fifth section and specifically note that evidence can be refined more through BN using probability tables instead of the “diagnosticity” in ACH.  The article cites Heuer stating that simple linear additive scoring mechanism to derive a probability for an indicated hypothesis. 

Sections 6 and 7 detail how Bayesian networks enhance ACH and how the two models can be integrated.  Integrating the two requires addressing three limitations with BN, showing dependency between hypotheses, showing dependency between evidence nodes, and modeling context for hypotheses.  The authors enhance dependency and context by making their model more complex through the introduction of intermediate nodes.  Utilizing the terrorist attack example, they introduce “Terrorist action” and “threat level” as intermediate nodes linking they hypotheses and evidence.

Critique:

Overall, I believe the authors did a decent job explaining ACH and the example was relevant to the intelligence community.  The explanation of Bayesian networks was somewhat confusing however the use of visuals assisted, particularly in displaying how intermediate variables are utilized when ACH and BN are integrated.  The related work section, although interesting, was irrelevant considering they did not attempt to tie the work into the example problem regarding the terrorist attack, although in some instances it may not be possible with the given example.  The authors note that their research has not been tested operationally, therefore I don’t know if it adds much to the community.  In order to test their research new tools need to be created to generate a BN fragment from an ACH and address inadequacies in the bipartite BN.  I think their research could have benefited from a more detailed explanation of the tools required. 

Valtorta, M., Huhn, M., Dang, J., Goradia, H., Huang, J. (2005).  Extending Heuer’s Analysis of Competing Hypotheses. Center for Information Technology TR-CIT05-03 Department of Computer Science and Engineering TR-2005-001 University of South Carolina. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.3422&rep=rep1&type=pdf

Summary of Findings:  Intuition ( 4 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures, and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University, in November 2017 regarding Intuition as an Analytic Method, specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Intuition is an analytic method that relies on a) quick and ready insight, b) immediate apprehension or cognition, c) knowledge or conviction gained by intuition, d) the power or faculty of attaining to direct knowledge or cognition without evident rational thought and inference.

Strengths:

• Processes information extremely quickly
• Free (Built-in function of human brain)  
• Focuses on well-defined patterns, relationships, and possibilities
• Intuition involves factors such as expertise, processing styles, task structure, feedback, and time pressure in making decisions

Weaknesses:

• Heavily influenced by individual biases
• Subjective is a matter of opinion
• Current research suggests that intuition works best only in certain situations
• Hard to articulate decision-making process
• Flawed information's can lead to wrong intuition
• Short-term emotional bias can lead to bad intuition
• Unstructured questions damage the effectiveness of intuition
• Insufficient consideration of alternative
• Lack of openness; every person has a different experience base

How-To: Holistic Hunch

1. Be presented a problem that you have never seen before
2. Internalization of associated mental frames and value judgements, into habitualized and internalized patterns of thinking and understanding
3. Make a decision based on system two thinking based on past experiences combined with novel information in new ways

How-To: Automated Expertise

1. Be presented a problem that is well within your subject matter expertise
2. Internalization of associated mental frames and value judgements, into habitualized and internalized patterns of thinking and understanding
3. Make a decision based on past situation-specific experiences and a replay of those experiences.

For Further Information:

1. Intuition Definitions
2. Intuition Vs. Structured Analytic Methods
3. Pros and Cons of Business Intuition