Tuesday, September 4, 2018

Summary of Findings: Analysis of Competing Hypotheses ( 3 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University, in September 2018 regarding Analysis of Competing Hypotheses as an Analytic Method, specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.
“Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.” - Richard Heuer, Central Intelligence Agency

Analysis of Competing Hypotheses (ACH) is a structured analytic technique designed to give analysts a framework for organizing information to formulate an estimate. ACH also is useful in resolving goal conflicts in a simple and repeatable manner. ACH gives analysts a skeleton in which to enter in definitive, contrasting hypotheses and assess the evidence on how it confirms or disconfirms the given hypotheses. The hypothesis with more confirming evidence becomes the more likely hypothesis. While ACH is a well-known methodology, it has not been thoroughly tested to assess its validity in academic literature.
  • Consideration of alternative viewpoints beyond the ultimately proposed analytical theory
  • Designed to simplify and analyze intelligence problems
  • Mitigates cognitive biases
  • Analyst’s reasoning is more transparent
  • Easy to use
  • Allows for categorizing and sorting of evidence in useful ways, e.g., time, relevance, source reliability
  • Unclear how much evidence is appropriate/most effective
    • Too much or too little evidence can negatively influence the analyst’s estimate
  • Relationships between pieces of evidence unclear/unidentified
    • Some evidence is only important when considered in conjunction with other evidence
  • Doesn’t identify pre-existing assumptions  
  • ACH can eliminate necessary context
1.      Identify possible hypotheses
2.     Compile a list of evidence
3.     Enter evidence into matrix
a.     Hypotheses across the top
b.     Evidence down the side
4.     Determine whether evidence is consistent or inconsistent with each hypothesis
a.     Consider (and weigh when necessary) the credibility of source for evidence
b.     Consider (and weigh when necessary) the relevance of evidence to the hypothesis
5.     Draw conclusion regarding which hypothesis is most consistent/inconsistent with the evidence included in the matrix
a.     Use conclusion to support estimate
b.     If conclusion does not support either hypothesis, collect more evidence and reevaluate ACH matrix
Application of Technique:
We wrote an investigative narrative regarding a crime and the analysts were asked to determine the likelihood that a suspect did or did not murder the victim. The narrative was presented to the analysts in paragraph form and evidence was included throughout the narrative.
After reading the narrative, analysts were asked to conduct the ACH as individuals, identifying their own key pieces of evidence and determining for themselves whether they were consistent or inconsistent with each hypothesis. After an allotted time to conduct their own ACH’s, analysts were then asked to share their conclusions with the class. The ensuing discussion brought forth issues regarding which evidence was important to the investigation, which evidence was most relevant to the hypotheses, and which sources for the evidence were most credible.
For Further Information:

1 comment:

  1. I incorporated ACH in many of my high impact deliverables in the intelligence in business category. From helping a Fortune 500 estimate whether a projected shift from open pit mining to underground mining represents significant opportunities for them so that they could make their next multi-million product development decision to arguing that the US river barge coal industry would actually increase in the short-term during an international M&A deal despite the "lore" indicating otherwise; I at least find it to be an efficient tool for walking through findings with executive decision-makers for specific types of intelligence questions.

    If nothing else, ACH externalizes how the team weighed and evaluated each key piece of evidence to come to their conclusion in a explicit way enabling review and critique.

    I can however think of a few ways that I've seen ACH misapplied and become actively harmful, and they relate to "the failure to generate appropriate hypothesis" as touched upon in Chapter 4 of "The Psychology of Intelligence Analysis". Any mental model of the world, no matter how well thought out; fails to capture the complexity of the world in actuality. Thus I require key assumptions check and exhaustive hypothesis generation before anyone on my team can even suggest ACH.

    Here is a recent example of an invalid application of ACH in my opinion: the B2B middle-market manufacturing and global distribution conglomerate whose market intelligence program I manage constantly reevaluates market-specific pricing for the portfolio of products we sell into each country. Someone proposed ACH while conflating intelligence and operational variables by postulating hypotheses such as "Product line X priced at $19.99 is likely to sell more volume" and "Product line X priced at $24.99 is likely to sell more volume". One fundamental problem with this is that it implicitly assumes price-driven competition and that the determining factor of a product's success is price. This is invalid. In one EU market where our imported brands are #1, demand for our product increased despite an increase in price due to completely unrelated factors; and in another EU market we are clinging on to we've failed to gain traction despite price decreases due to external competitive pressures and internal marketing decisions related to failure to tailor our packaging to the wants of the consumers in the market.

    Thanks to my experience in management consulting on matters of intelligence & research prior to working for a particular company, I've been successful in setting up the market intelligence function in a way where the C-Suite; marketing department, sales department, etc. are our internal clients and not groups we need to defer to on matters of products, process, and how we frame our findings around the business decisions we support. As Godfather would suggest, "consigliere".

    ~ Have a $wagilicious day!