Summary of Findings: Analysis of Competing Hypotheses (3 Of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2017 regarding ACH as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Analysis of Competing Hypotheses is a matrix-based analysis methodology created to analyze a complex problem using weighted evidence that allows  for the elimination of confirmation bias.  It accomplishes this via use of scientific methodology and placing numerical value on weight of evidence, relevancy, and the consistency or inconsistency that is determined by the analyst.  

Strengths:

• The thorough elimination of confirmation bias within the analytic process.
• ACH forces an analyst to confront a competing hypothesis via numerous bodies of evidence.
• ACH is a great analytical method to use when working with structured data.
• ACH is a good starting point in the analytic process.
• Helps show relationship, if any, between hypothesis and evidence.
• ACH is a good thinking tool, but it is one of many tools.
• ACH utilizes the scientific method and applies it to the practice of intelligence analysis.

Weaknesses:

• Still only a piece software.
• Time consuming.
• Still subject to cognitive biases in how evidence is weighed

• Only an algorithm at the end of the calculation, requires human judgment.
• ACH is dependent on discrete judgement of the analyst

How-To:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities.
2. Make a list of significant evidence and arguments for and against each hypothesis.
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “diagnosticity” of the evidence and arguments–that is, identify which items are most helpful in judging the relative likelihood of the hypotheses.
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them.
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.
8. Identify possible milestones for future observation that may indicate events are taking a different course than expected.

Application of Technique:

To demonstrate the application of the ACH matrix the class took a look at the question of will Hurricane Harvey cost more in federal spending than Hurricane Katrina? The class tested two hypotheses. The first was that Hurricane Harvey will cost more than Hurricane Katrina in federal spending  and the second hypothesis was that Hurricane Harvey will cost less than Hurricane Katrina in federal spending. The class did independent research for approximately ten minutes, and then combined their information into a single ACH matrix. The class then debated the credibility, relevancy, and the consistency of each piece of evidence to determine a weighted score in the ACH matrix. The class was able to objectively measure the strengths as well as the weaknesses of the methodology through the exercise.

For Further Information:

Chapter 8-Analysis of Competing Hypotheses

Palo Alto Research Center PARC ACH Download