Extending Heuer's Analysis of Competing Hypotheses Method to Support Complex Decision Analysis

Extending Heuer’s Analysis of Competing Hypotheses Method to Support Complex Decision Analysis

Summary:

Marco Valtorta, Michael Huhns, Jiangbo Dang, Hrishikesh Goradia, and Jingshan Huang of the University of South Carolina used Bayesian Networks to extend the Analysis of Competing Hypotheses. They first explained what ACH is, followed by an explanation of Bayesian Networks. After the initial explanation of each matrix, they used a fictional situation to formulate an ACH matrix into a Bayesian matrix.

Before examining ACH and Bayesian Networks more in-depth, they give a brief explanation of another methodologies: Porter’s 5 Forces, Win-Loss Analysis, and Scenario Planning as possible rivals to the ACH matrix. Porter’s 5 Forces is an intelligence model designed for businesses as a method of examining threats in the industry: threat of entry, threat of substitution, buyer bargaining power, supplier bargaining power, rivalry between current competitors. Win-Loss analysis examines sales from a historical point of view to give feedback and possibilities for future sales. Scenario Planning is more strategic, as it uses multiple possible scenarios as a method of achieving a specific goal.

The researchers explained how ACH aids the intelligence process first by examining what the intelligence process is. Their explanation of the process divided it into three phases: the collection phase, analysis phase, and reporting phase. They went on to explain ACH as a methodology which weighs varying hypotheses using evidence for and against each hypothesis. This method gives reasoning for the validation or invalidation of the conclusions as well as explaining how the analysts arrived at their conclusions.

Bayesian Networks are acyclic, graphical models that shows probabilistic relationships between variables as well as combining data with prior knowledge. The complex nature of the model allows for a more in-depth analysis of ACH. In the Bayesian Network, the hypotheses of ACH are shown as modules connecting to the pieces of evidence.

They examined the limitations of using a Bayesian Network to model ACH hypotheses with the evidence. Basic Bayesian models cannot fully show all the variables without the context of the evidence without using a more complex model. The researchers got around this by adding more nodes to their model that provide the context for the evidence and giving lines between the hypothesis node and the evidence nodes if they were weighted for or against the hypothesis.

Critique:

While the use of Bayesian Networks to extend ACH into a visual model would be a good way to show hypotheses and their evidence for more visual decision makers, it has not been tested in the field. Furthermore, unless the audience knows how Bayesian networks work, the visual could be more confusing than a normal ACH matrix. A simpler explanation of Bayesian Networks would be useful when presenting it as a method of extending the ACH matrix. On a different note, the examination of rival methodologies i.e. Porter’s 5 Forces etc., was useful for the comparison with ACH, but not necessarily relevant to the paper.

Citation: Valtorta, Marco, Huhns, Michael, Dang Jiangbo, Goradia, Hrishikesh, and Huang, Jingshan. “Extending Heuer’s Analysis of Competing Hypotheses Method To Support Complex Decision Making.” 10 February 2005. University of South Carolina. Accessed 29 August 2017. https://cse.sc.edu/~MGV/reports/TR2005-01.pdf. 

 

How Does Analysis of Competing Hypotheses (ACH) Improve Intelligence Analysis?

Summary and critique by Jared Leets

Summary:

Richards Heuer Jr discusses the three different approaches to Analysis of Competing Hypotheses (ACH) and four significant steps in the intelligence analysis process which include researching relevant information, organizing information to help with analysis, reviewing the information to make a credible evaluation, and finally writing the product.

Heuer Jr states that when searching for information, the purpose is to get the analyst to challenge his or her initial mindset. Researching alternative hypotheses helps the analyst broaden his or her search for information that they typically would not look for. When analysts initially work with ACH they consider it beneficial due to the fact that it forces them to think about evidence and outcomes that had never occurred to them. In the first step ACH is an easy strategy for questioning a complex problem. Every analyst should seriously consider questioning their traditional way of answering a question to a problem.

The second step is organizing the information. Two principles, decomposition and externalization, exist to help cope with human mental perception and memory. Decomposition refers to breaking down the problem while externalization refers to removing the problem out of someone’s head and writing it down in order to simplify the problem and show the variables. The purpose is to show all of the evidence and what type relationship they share if any. The three types of ACH manual, automated, and Bayesian all have their differences. The manual ACH views only the relationship between each individual piece of evidence and hypotheses. The matrix will assist the analyst in evaluating evidence that is most indicative in opposing hypotheses

The automated ACH allows analysts to place evidence in categories. The categories include date, type of source, credibility of source, and the relevance of the source. By sorting the evidence it makes it much easier for the analyst to put all his or her efforts towards the most convincing evidence. The Bayesian ACH places more importance on the relationship between the evidence and hypotheses. It typically has multiple sets of hypotheses, which tend to increase judgments that must be made in the analytical process.

Heuer Jr states that in the third step, analyzing the information, the automated ACH will sort the evidence by categories while the Bayesian ACH will give a mathematical algorithm that shows the analyst a possible answer. Sorting evidence by relevance in categories significantly helps with intelligence analysis. Heuer Jr explains that in the automated version an analyst can compare evidence from a clandestine source or open source can help reveal deception from a source. While the inconsistency and weighted inconsistency scores can produce an accurate estimate, the analyst must use their own judgment in the end. According to Bayesian ACH proponents, they claim that using an Inconsistency Score or Weighted Inconsistency Score is problematic since it relies on incomplete information. A Bayesian inference is ideal for making probabilities bases on judgments from intelligence analysts. However, it is very complex, time consuming, and requires a Bayesian methodologist to help the analyst when it is being conducted. Proponents state that this displays how complex an intelligence problem can be and that one single analyst cannot make a judgment call without help from other sources. In the final step, writing the product, Heuer Jr says each ACH has its benefits and can help the analyst in reinforcing what their research has revealed or shows the analyst an alternative path for additional research.

Critique:

ACH has it benefits and its obstacles. For example, it is very good at helping an analyst see alternative hypotheses and pieces of evidence and weighing that evidence to help make a correct decision. In any intelligence project an analyst must think from different points of views. ACH can help an analyst go step by step and see if their current hypothesis will answer the intelligence requirement. However, ACH can also have its issues. As Heuer Jr explains, when weighing evidence it is easy to be biased, especially when deciding how credible or relevant a piece of evidence is. In addition, software can only help an analyst come to a decision. If an analyst says that ACH told them the answer and base everything off of that, they will likely lose credibility. In the end, ACH can help with intelligence analysis by helping the analyst think of different ideas to address the problem and support their research, but it can also be a problematic if analysts rely too much on it as software cannot answer problems in the intelligence profession.

Source: Heuer Jr, Richards. (October 16, 2005). How Does Analysis of Competing Hypotheses Improve Intelligence Analysis? Pherson Associates. http://www.pherson.org/wp-content/uploads/2013/06/06.-How-Does-ACH-Improve-Analysis_FINAL.pdf

Why are we not evaluating multiple competing hypotheses in ecology and evolution?

By : Praveen Kumar Neelappa

This article suggests that there is the gap between theory and practice in the use of analysis of competing hypotheses (ACH). It identifies several intellectual and practical barriers that discourage the use of multiple hypotheses in the field of ecology and evolution. This article points out that scientist have a bias or a motivation to consider one hypothesis over other (Intellectual barriers) and there are practical limitations inherent to factorial design, the standard experimental design that allows researchers to evaluate several explanatory variables and their interactions in the same study, one variable at a time (Practical barrier).

Cognitive bias makes us think that we are making a logical, rational and effective decision while considering the alternative hypothesis, but our unconscious bias influences the experiment and its outcome. There is a tendency of scientist to put more weight on evidence that supports favored ideas more than other evidence that is available (Confirmation bias), seek for the pattern in their experiment (Pattern seeking bias) and be judged only by their internal consistency (Belief bias). There are several ways one might minimize the effect of cognitive bias in science so that one does not rely exclusively on one’s perceptions. It can be achieved by masking (kept) information about the experiment from the participant, to reduce or eliminate bias, until after a trial outcome is known (Blind bias), working with other scientists with different perspectives (Work with the enemy) and a null model which generates a pattern in the absence of any biological process, forcing the researcher to think about many different hypotheses, which could potentially minimize the negative impacts of cognitive biases in science.

Any study that has a simple, easy to understand explanation will be preferred over a study that employs complex and perhaps less-elegant ideas (Simplicity bias) to avoid practical barrier. Editors and reviewers tend to rely on prior knowledge when evaluating a manuscript, creating additional difficulties for researchers when publishing studies that confront well-established ideas. This tension between new and old ideas could reflect a conflict between new and old generations (Publication bias).

The article concludes by suggesting that ecological and evolutionary research is aimed at understanding patterns arising from nonlinear and stochastic interactions among a multitude of processes and agents at multiple spatial and temporal scales. If we wish to truly advance scientific progress despite this complexity, we must better commit to strong inference in our scientific inquiries by simultaneously evaluating multiple competing hypotheses.

Critique:

The use of ACH is widely promoted to enhance the effectiveness of the scientific investigation. This article points out some valid draw backs of using ACH in the different field of studies and discusses these drawbacks and solutions to them in detail. The article clearly illustrates various types of biases and the various stages where they can be encountered while carrying out experiments. It is imperative that the individual carrying out the experiment is objective in data collection and maintains an objective view at every stage of the experiment which will be the best way to counter any possibility of the final results being biased. Additionally, measures need to be put in place to reassess for bias along the way to ensure the results are void of any form of bias.

Citation:

Betini GS, Avgar T, Fryxell JM. 2017 Why are we not evaluating multiple competing hypotheses in ecology and evolution?.R. Soc. open sci. 4: 160756. http://dx.doi.org/10.1098/rsos.160756

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

By: Matt Haines

Source: https://pdfs.semanticscholar.org/50a1/730ce13238d17c6c2f48c3621d4489c86d6a.pdf

Summary:

Gregorio Convertino, Dorrit Billman, Peter Pirolli, JP Masur, and Jeff Shrager created a virtual environment to conduct standard analysis of competing hypotheses(ACH) and then they analyzed the effects of that environment. The authors begin by defining CACHE as a collaborative analysis of competing hypotheses environment. Then, they explain the difficulties an analyst faces in everyday intelligence tasks. An analyst is faced with tasks that span a vast multitude of areas of expertise on a daily basis and biases influence all of those analytical products. The authors then go into detail of what the CACHE framework actually does in order to combat this challenge. CACHE allows a user to search through all available evidence, input that evidence into a personal ACH matrix, view other team members ACH matrix, and communicate with other team members through an instant messaging system.

Before completing the actual test of the CACHE framework, the authors hypothesized that:

Heterogeneous groups would show less confirmation bias than Homogeneous groups. Because CACHE supports sharing information among participants, the differing views in the heterogeneous groups should mitigate cognitive biases by 1) exposure to more, and less-biased, evidence and 2) access to alternative analyses provided by partners.

and that,  “Heterogeneous groups would show no net process loss relative to the Solo/Nominal Group. CACHE will mitigate the process costs, producing equivalent or better performance in heterogeneous groups.” The results of the experiments were concurrent with the authors’ hypotheses.

Critique:

The CACHE framework is a great idea and prototype for groups where not every person can be in the same location at the same time. However, the authors of this paper did not do much to actually prove anything. This paper laid out a product. It did not add to the ACH process nor did it attempt to contest normal assumptions of ACH. However, CACHE has achieved something just by allowing analysts to be in two different places at once and collaborate with each other. This feature can help eliminate some group think biases because it takes some power away from those members of the team who are better presenters. For example, one of the major complaints many international students have, is that they feel like their ideas are not heard, because they cannot vocally command a room. By allowing analysts to work remotely, international students can have the same voice as a native english speaker.

Analysis of Competing Hypothesis for Investigating Lone Wolf Terrorism

Summary and Critique by Michael Pouch

Summary:

Lisa Kaati and Pontus Svenson uses Analysis of Competing Hypothesis (ACH) to help introduce a model to target similarities between different lone wolf cases. The purpose of the study is to outline this analytic tool to investigate lone wolf terrorists by showing how this method could be applied.

Before the authors began to examine their study, they introduce and define what lone wolf terrorism is. Next, they looked at the characteristics regarding the lone wolves background, and behavior that lone wolves share. They also point out the difficulty that law enforcement and intelligence community have to prevent lone wolf terrorist attacks. They specify that it is unfeasible for analysts to gather information and evaluate all data concerning radicalization processes of possible lone wolf terrorists, without any analytic method or process. However, analytic tools that assist the analysts could help facilitate the process. This will help gather and scope the information to gather more data and investigate more possibilities for lone wolf radicalization. One these tools that the authors mention was ACH.

When identifying a hypothesis for lone wolf terrorism, while using ACH, the analyst needs to pose hypotheses regarding them and their behavior. Foremost, they need to brainstorm possible hypothesis by making a list of significant evidence for and against each hypothesis. Next, the evidence needs to be evaluated by the likelihood of alternative hypotheses or helpfulness during the investigation. This will help by drawing tentative conclusions about the hypothesis and have an objective during the investigation. Lastly, the analyst proceeds to collect information about an individual with the ACH format, as shown in figure 1.

Figure 1: Two simplified examples to illustrate template hypothesis. The hypotheses are at the top and the evidence down the side. The left matrix is a hypothesis regarding the characteristics in background and behavior of lone wolf terrorists. The right matrix illustrates hypotheses regarding a terrorist attack.

The researchers describe an outline that can be used to categorize and analyze about possible lone wolf terrorists in the effort to prevent an attack while using ACH (Shown in figure 2).  First, the analyst identifies a hypothesis to begin a framework for the examination of a possible lone wolf. Second, the hypothesis is constantly developing and cultivating to help scope and specify a likely lone wolf. Third, is the process of collecting information that confirms or refute the hypotheses that

Figure 2: Mode of operation for the framework.

were started. The information can originate from a diverse number of sources such as Twitter, Facebook, web blogs, police reports, intelligence reports, tips and web forums. Fourth, is the progression of collecting relevant information and linking it to the hypotheses that it supports. After analyzing and connecting relevant information about each individual, the hypothesis is fragmented into additional explicit statements until the statements become observable actions called indicators. Lastly, when there is enough evidence, ACH warns the analyst so that appropriate action can be taken.

Critique:

The use of ACH to help prevent an attack from a lone wolf is likely an effective analytic method to use to help organize and have a framework for identifying a lone wolf. However, there is room for bias in selecting the template hypothesis, relevant evidence, and weighing the individual to likely become a possible lone wolf. Due to this, there is no guarantee that ACH will automatically select a possible lone wolf attacker. Despite bias in the selecting and evaluating process, ACH does help create a systematic process that increases the odds of preventing a lone wolf attack by giving valuing and indicating  a possible lone wolf. Additionally, ACH helps the analyst leave a trail of evidence that can be interpreted. Overall, ACH is a useful analytic tool to establish a framework of potential lone wolves that can be measured.

Citation: Kaati, L., & Svenson, P. (2011). Analysis of Competing Hypothesis for Investigating Lone Wolf Terrorist. 2011 European Intelligence and Security Informatics Conference. doi:10.1109/eisic.2011.60. https://www.foi.se/download/18.3bca00611589ae79878192/1480076487550/FOI-S--3800--SE.pdf

Summary of Findings: Torture (Not Valid)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2016 regarding Torture as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use structured data.

Description:

Torture or “enhanced interrogation techniques” (EIT) is a modifier for intelligence collection that utilizes the infliction physical or mental stressors/pain in order to elicit information from a given human subject.

Strengths:

• Will likely produce information, however the reliability of that information is highly suspect.

Weaknesses:

• Not a simple process.
• May lead to the collection of misinformation, if any is even given.
• Where do you find an expert (10,000 hours=10 Years Experience) in torturing?
• Data on torture use is severely limited, and thus makes it difficult to assess.
• Difficult to replicate and validity is questionable

How-To:

• Find someone who you believe has the information you want.
• Capture them and place them in an environment where you have complete control
• Inflict pain or threaten its use via a desired method in order to elicit information
• Application or threat of application of mind-altering substances may be used to enhance the fear within the subject that they will divulge useful information
• Threats of imminent death of the subject or family, friends, associates, etc. can also be an effective means of encouraging subjects to give up withheld information

Application of Technique:

A group of students in class participated in a socratic discussion after being presented information on the history, uses, current techniques, and events where torture was used in order to determine the logical effectiveness of the modifier.  Macro and micro issues were presented as evidence for and against the use of the modifier in order to weigh costs and potential benefits.

For Further Information:

Perspectives on Enhanced Interrogation Techniques:

https://www.fas.org/sgp/crs/intel/R43906.pdf

CIA Interrogation Manual:

https://cryptome.org/2013/09/cia-interrogation-guide.pdf

These Are The 13 ‘Enhanced Interrogation Techniques’ The CIA Used On Detainees

http://www.businessinsider.com/the-13-enhanced-interrogation-techniques-the-cia-used-on-detainees-2014-12

Senate Select Committee on Intelligence: Committee Study of the Central Intelligence Agency’s Detention and Interrogation Program:

https://fas.org/irp/congress/2014_rpt/ssci-rdi.pdf

FM2-22.3 (FM 34-52) Human Intelligence Collector Operations:

http://fas.org/irp/doddir/army/fm2-22-3.pdf

Regarding the Torture of Others:

http://blogs.stlawu.edu/digitalmedia/files/2009/08/SONTAG_REGARDING_TORTURE2-4.pdf

Torture at Abu Ghraib:

http://www.veronaschools.org/cms/lib02/NJ01001379/Centricity/Domain/588/Torture%20at%20Abu%20Ghraib.pdf

Liberalism, Torture, and the Ticking Bomb:

http://scholarship.law.georgetown.edu/cgi/viewcontent.cgi?article=1163&context=facpub

Torture, Henry Shue:

http://www.jstor.org/stable/2264988?seq=1#page_scan_tab_contents

Interrogation of Abu Zubaydah:

https://en.wikipedia.org/wiki/Interrogation_of_Abu_Zubaydah

Hard Measures, part 1:

https://www.youtube.com/watch?v=MC9PP94f4OQ

Hard Measures, part 2:

https://www.youtube.com/watch?v=roS4LeUvny8