Note:
This post represents the synthesis of the thoughts, procedures and experiences
of others as represented in the articles read in advance (see previous posts)
and the discussion among the students and instructor during the Advanced
Analytic Techniques class at Mercyhurst University, in September 2018 regarding
Analysis of Competing Hypotheses as an Analytic Method, specifically. This
technique was evaluated based on its overall validity, simplicity, flexibility
and its ability to effectively use unstructured data.
Description:
“Analysis of competing hypotheses,
sometimes abbreviated ACH, is a tool to aid judgment on important issues
requiring careful weighing of alternative explanations or conclusions. It helps
an analyst overcome, or at least minimize, some of the cognitive limitations
that make prescient intelligence analysis so difficult to achieve.” - Richard Heuer, Central
Intelligence Agency
Analysis
of Competing Hypotheses (ACH) is a structured analytic technique designed to
give analysts a framework for organizing information to formulate an estimate.
ACH also is useful in resolving goal conflicts in a simple and repeatable
manner. ACH gives analysts a skeleton in which to enter in definitive,
contrasting hypotheses and assess the evidence on how it confirms or
disconfirms the given hypotheses. The hypothesis with more confirming evidence
becomes the more likely hypothesis. While ACH is a well-known methodology, it
has not been thoroughly tested to assess its validity in academic literature.
Strengths:
- Consideration of alternative viewpoints beyond the
ultimately proposed analytical theory
- Designed to simplify and analyze intelligence problems
- Mitigates cognitive biases
- Analyst’s reasoning is more transparent
- Easy to use
- Allows for categorizing and sorting of evidence in
useful ways, e.g., time, relevance, source reliability
Weaknesses:
- Unclear how much
evidence is appropriate/most effective
- Too much or too
little evidence can negatively influence the analyst’s estimate
- Relationships
between pieces of evidence unclear/unidentified
- Some evidence is
only important when considered in conjunction with other evidence
- Doesn’t identify
pre-existing assumptions
- ACH can eliminate
necessary context
How-To:
1.
Identify possible hypotheses
2.
Compile a list of evidence
3.
Enter evidence into matrix
a.
Hypotheses across the top
b.
Evidence down the side
4.
Determine whether evidence is consistent or inconsistent with each
hypothesis
a.
Consider (and weigh when necessary) the credibility of source for
evidence
b.
Consider (and weigh when necessary) the relevance of evidence to
the hypothesis
5.
Draw conclusion regarding which hypothesis is most
consistent/inconsistent with the evidence included in the matrix
a.
Use conclusion to support estimate
b.
If conclusion does not support either hypothesis, collect more
evidence and reevaluate ACH matrix
Application of Technique:
We wrote an investigative narrative regarding a crime and the
analysts were asked to determine the likelihood that a suspect did or did not
murder the victim. The narrative was presented to the analysts in paragraph
form and evidence was included throughout the narrative.
After reading the narrative, analysts were asked to conduct
the ACH as individuals, identifying their own key pieces of evidence and
determining for themselves whether they were consistent or inconsistent with
each hypothesis. After an allotted time to conduct their own ACH’s, analysts
were then asked to share their conclusions with the class. The ensuing
discussion brought forth issues regarding which evidence was important to the
investigation, which evidence was most relevant to the hypotheses, and which
sources for the evidence were most credible.
For
Further Information:
2.
Wheaton on ACH: Analysis of Competing Hypotheses and Structured Analysis
of Competing Hypotheses: What Are The Links With Foresight?
3.
Brasfield on ACH: Forecasting Accuracy and Cognitive Bias in ACH
4.
Van Gelder on ACH: https://timvangelder.com/2007/12/31/hypothesis-testing-whats-wrong-with-ach/
Posts
I incorporated ACH in many of my high impact deliverables in the intelligence in business category. From helping a Fortune 500 estimate whether a projected shift from open pit mining to underground mining represents significant opportunities for them so that they could make their next multi-million product development decision to arguing that the US river barge coal industry would actually increase in the short-term during an international M&A deal despite the "lore" indicating otherwise; I at least find it to be an efficient tool for walking through findings with executive decision-makers for specific types of intelligence questions.
ReplyDeleteIf nothing else, ACH externalizes how the team weighed and evaluated each key piece of evidence to come to their conclusion in a explicit way enabling review and critique.
I can however think of a few ways that I've seen ACH misapplied and become actively harmful, and they relate to "the failure to generate appropriate hypothesis" as touched upon in Chapter 4 of "The Psychology of Intelligence Analysis". Any mental model of the world, no matter how well thought out; fails to capture the complexity of the world in actuality. Thus I require key assumptions check and exhaustive hypothesis generation before anyone on my team can even suggest ACH.
Here is a recent example of an invalid application of ACH in my opinion: the B2B middle-market manufacturing and global distribution conglomerate whose market intelligence program I manage constantly reevaluates market-specific pricing for the portfolio of products we sell into each country. Someone proposed ACH while conflating intelligence and operational variables by postulating hypotheses such as "Product line X priced at $19.99 is likely to sell more volume" and "Product line X priced at $24.99 is likely to sell more volume". One fundamental problem with this is that it implicitly assumes price-driven competition and that the determining factor of a product's success is price. This is invalid. In one EU market where our imported brands are #1, demand for our product increased despite an increase in price due to completely unrelated factors; and in another EU market we are clinging on to we've failed to gain traction despite price decreases due to external competitive pressures and internal marketing decisions related to failure to tailor our packaging to the wants of the consumers in the market.
Thanks to my experience in management consulting on matters of intelligence & research prior to working for a particular company, I've been successful in setting up the market intelligence function in a way where the C-Suite; marketing department, sales department, etc. are our internal clients and not groups we need to defer to on matters of products, process, and how we frame our findings around the business decisions we support. As Godfather would suggest, "consigliere".
~ Have a $wagilicious day!