Why are we not evaluating multiple competing hypotheses in ecology and evolution?

By : Praveen Kumar Neelappa

This article suggests that there is the gap between theory and practice in the use of analysis of competing hypotheses (ACH). It identifies several intellectual and practical barriers that discourage the use of multiple hypotheses in the field of ecology and evolution. This article points out that scientist have a bias or a motivation to consider one hypothesis over other (Intellectual barriers) and there are practical limitations inherent to factorial design, the standard experimental design that allows researchers to evaluate several explanatory variables and their interactions in the same study, one variable at a time (Practical barrier).

Cognitive bias makes us think that we are making a logical, rational and effective decision while considering the alternative hypothesis, but our unconscious bias influences the experiment and its outcome. There is a tendency of scientist to put more weight on evidence that supports favored ideas more than other evidence that is available (Confirmation bias), seek for the pattern in their experiment (Pattern seeking bias) and be judged only by their internal consistency (Belief bias). There are several ways one might minimize the effect of cognitive bias in science so that one does not rely exclusively on one’s perceptions. It can be achieved by masking (kept) information about the experiment from the participant, to reduce or eliminate bias, until after a trial outcome is known (Blind bias), working with other scientists with different perspectives (Work with the enemy) and a null model which generates a pattern in the absence of any biological process, forcing the researcher to think about many different hypotheses, which could potentially minimize the negative impacts of cognitive biases in science.

Any study that has a simple, easy to understand explanation will be preferred over a study that employs complex and perhaps less-elegant ideas (Simplicity bias) to avoid practical barrier. Editors and reviewers tend to rely on prior knowledge when evaluating a manuscript, creating additional difficulties for researchers when publishing studies that confront well-established ideas. This tension between new and old ideas could reflect a conflict between new and old generations (Publication bias).

The article concludes by suggesting that ecological and evolutionary research is aimed at understanding patterns arising from nonlinear and stochastic interactions among a multitude of processes and agents at multiple spatial and temporal scales. If we wish to truly advance scientific progress despite this complexity, we must better commit to strong inference in our scientific inquiries by simultaneously evaluating multiple competing hypotheses.

Critique:

The use of ACH is widely promoted to enhance the effectiveness of the scientific investigation. This article points out some valid draw backs of using ACH in the different field of studies and discusses these drawbacks and solutions to them in detail. The article clearly illustrates various types of biases and the various stages where they can be encountered while carrying out experiments. It is imperative that the individual carrying out the experiment is objective in data collection and maintains an objective view at every stage of the experiment which will be the best way to counter any possibility of the final results being biased. Additionally, measures need to be put in place to reassess for bias along the way to ensure the results are void of any form of bias.

Citation:

Betini GS, Avgar T, Fryxell JM. 2017 Why are we not evaluating multiple competing hypotheses in ecology and evolution?.R. Soc. open sci. 4: 160756. http://dx.doi.org/10.1098/rsos.160756

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

Collaborative Intelligence Analysis with CACHE: Bias Reduction and Information Coverage

By: Matt Haines

Source: https://pdfs.semanticscholar.org/50a1/730ce13238d17c6c2f48c3621d4489c86d6a.pdf

Summary:

Gregorio Convertino, Dorrit Billman, Peter Pirolli, JP Masur, and Jeff Shrager created a virtual environment to conduct standard analysis of competing hypotheses(ACH) and then they analyzed the effects of that environment. The authors begin by defining CACHE as a collaborative analysis of competing hypotheses environment. Then, they explain the difficulties an analyst faces in everyday intelligence tasks. An analyst is faced with tasks that span a vast multitude of areas of expertise on a daily basis and biases influence all of those analytical products. The authors then go into detail of what the CACHE framework actually does in order to combat this challenge. CACHE allows a user to search through all available evidence, input that evidence into a personal ACH matrix, view other team members ACH matrix, and communicate with other team members through an instant messaging system.

Before completing the actual test of the CACHE framework, the authors hypothesized that:

Heterogeneous groups would show less confirmation bias than Homogeneous groups. Because CACHE supports sharing information among participants, the differing views in the heterogeneous groups should mitigate cognitive biases by 1) exposure to more, and less-biased, evidence and 2) access to alternative analyses provided by partners.

and that,  “Heterogeneous groups would show no net process loss relative to the Solo/Nominal Group. CACHE will mitigate the process costs, producing equivalent or better performance in heterogeneous groups.” The results of the experiments were concurrent with the authors’ hypotheses.

Critique:

The CACHE framework is a great idea and prototype for groups where not every person can be in the same location at the same time. However, the authors of this paper did not do much to actually prove anything. This paper laid out a product. It did not add to the ACH process nor did it attempt to contest normal assumptions of ACH. However, CACHE has achieved something just by allowing analysts to be in two different places at once and collaborate with each other. This feature can help eliminate some group think biases because it takes some power away from those members of the team who are better presenters. For example, one of the major complaints many international students have, is that they feel like their ideas are not heard, because they cannot vocally command a room. By allowing analysts to work remotely, international students can have the same voice as a native english speaker.

Analysis of Competing Hypothesis for Investigating Lone Wolf Terrorism

Summary and Critique by Michael Pouch

Summary:

Lisa Kaati and Pontus Svenson uses Analysis of Competing Hypothesis (ACH) to help introduce a model to target similarities between different lone wolf cases. The purpose of the study is to outline this analytic tool to investigate lone wolf terrorists by showing how this method could be applied.

Before the authors began to examine their study, they introduce and define what lone wolf terrorism is. Next, they looked at the characteristics regarding the lone wolves background, and behavior that lone wolves share. They also point out the difficulty that law enforcement and intelligence community have to prevent lone wolf terrorist attacks. They specify that it is unfeasible for analysts to gather information and evaluate all data concerning radicalization processes of possible lone wolf terrorists, without any analytic method or process. However, analytic tools that assist the analysts could help facilitate the process. This will help gather and scope the information to gather more data and investigate more possibilities for lone wolf radicalization. One these tools that the authors mention was ACH.

When identifying a hypothesis for lone wolf terrorism, while using ACH, the analyst needs to pose hypotheses regarding them and their behavior. Foremost, they need to brainstorm possible hypothesis by making a list of significant evidence for and against each hypothesis. Next, the evidence needs to be evaluated by the likelihood of alternative hypotheses or helpfulness during the investigation. This will help by drawing tentative conclusions about the hypothesis and have an objective during the investigation. Lastly, the analyst proceeds to collect information about an individual with the ACH format, as shown in figure 1.

Figure 1: Two simplified examples to illustrate template hypothesis. The hypotheses are at the top and the evidence down the side. The left matrix is a hypothesis regarding the characteristics in background and behavior of lone wolf terrorists. The right matrix illustrates hypotheses regarding a terrorist attack.

The researchers describe an outline that can be used to categorize and analyze about possible lone wolf terrorists in the effort to prevent an attack while using ACH (Shown in figure 2).  First, the analyst identifies a hypothesis to begin a framework for the examination of a possible lone wolf. Second, the hypothesis is constantly developing and cultivating to help scope and specify a likely lone wolf. Third, is the process of collecting information that confirms or refute the hypotheses that

Figure 2: Mode of operation for the framework.

were started. The information can originate from a diverse number of sources such as Twitter, Facebook, web blogs, police reports, intelligence reports, tips and web forums. Fourth, is the progression of collecting relevant information and linking it to the hypotheses that it supports. After analyzing and connecting relevant information about each individual, the hypothesis is fragmented into additional explicit statements until the statements become observable actions called indicators. Lastly, when there is enough evidence, ACH warns the analyst so that appropriate action can be taken.

Critique:

The use of ACH to help prevent an attack from a lone wolf is likely an effective analytic method to use to help organize and have a framework for identifying a lone wolf. However, there is room for bias in selecting the template hypothesis, relevant evidence, and weighing the individual to likely become a possible lone wolf. Due to this, there is no guarantee that ACH will automatically select a possible lone wolf attacker. Despite bias in the selecting and evaluating process, ACH does help create a systematic process that increases the odds of preventing a lone wolf attack by giving valuing and indicating  a possible lone wolf. Additionally, ACH helps the analyst leave a trail of evidence that can be interpreted. Overall, ACH is a useful analytic tool to establish a framework of potential lone wolves that can be measured.

Citation: Kaati, L., & Svenson, P. (2011). Analysis of Competing Hypothesis for Investigating Lone Wolf Terrorist. 2011 European Intelligence and Security Informatics Conference. doi:10.1109/eisic.2011.60. https://www.foi.se/download/18.3bca00611589ae79878192/1480076487550/FOI-S--3800--SE.pdf

Summary of Findings: Torture (Not Valid)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2016 regarding Torture as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use structured data.

Description:

Torture or “enhanced interrogation techniques” (EIT) is a modifier for intelligence collection that utilizes the infliction physical or mental stressors/pain in order to elicit information from a given human subject.

Strengths:

• Will likely produce information, however the reliability of that information is highly suspect.

Weaknesses:

• Not a simple process.
• May lead to the collection of misinformation, if any is even given.
• Where do you find an expert (10,000 hours=10 Years Experience) in torturing?
• Data on torture use is severely limited, and thus makes it difficult to assess.
• Difficult to replicate and validity is questionable

How-To:

• Find someone who you believe has the information you want.
• Capture them and place them in an environment where you have complete control
• Inflict pain or threaten its use via a desired method in order to elicit information
• Application or threat of application of mind-altering substances may be used to enhance the fear within the subject that they will divulge useful information
• Threats of imminent death of the subject or family, friends, associates, etc. can also be an effective means of encouraging subjects to give up withheld information

Application of Technique:

A group of students in class participated in a socratic discussion after being presented information on the history, uses, current techniques, and events where torture was used in order to determine the logical effectiveness of the modifier.  Macro and micro issues were presented as evidence for and against the use of the modifier in order to weigh costs and potential benefits.

For Further Information:

Perspectives on Enhanced Interrogation Techniques:

https://www.fas.org/sgp/crs/intel/R43906.pdf

CIA Interrogation Manual:

https://cryptome.org/2013/09/cia-interrogation-guide.pdf

These Are The 13 ‘Enhanced Interrogation Techniques’ The CIA Used On Detainees

http://www.businessinsider.com/the-13-enhanced-interrogation-techniques-the-cia-used-on-detainees-2014-12

Senate Select Committee on Intelligence: Committee Study of the Central Intelligence Agency’s Detention and Interrogation Program:

https://fas.org/irp/congress/2014_rpt/ssci-rdi.pdf

FM2-22.3 (FM 34-52) Human Intelligence Collector Operations:

http://fas.org/irp/doddir/army/fm2-22-3.pdf

Regarding the Torture of Others:

http://blogs.stlawu.edu/digitalmedia/files/2009/08/SONTAG_REGARDING_TORTURE2-4.pdf

Torture at Abu Ghraib:

http://www.veronaschools.org/cms/lib02/NJ01001379/Centricity/Domain/588/Torture%20at%20Abu%20Ghraib.pdf

Liberalism, Torture, and the Ticking Bomb:

http://scholarship.law.georgetown.edu/cgi/viewcontent.cgi?article=1163&context=facpub

Torture, Henry Shue:

http://www.jstor.org/stable/2264988?seq=1#page_scan_tab_contents

Interrogation of Abu Zubaydah:

https://en.wikipedia.org/wiki/Interrogation_of_Abu_Zubaydah

Hard Measures, part 1:

https://www.youtube.com/watch?v=MC9PP94f4OQ

Hard Measures, part 2:

https://www.youtube.com/watch?v=roS4LeUvny8

The Effects and Effectiveness of Using Torture as an Interrogation Device: Using Research to Inform the Policy Debate

Summary

Hot on the heels of the Abu Ghraib incident, Mark A. Costanzo and Ellen Gerrity took a qualitative look the circumstances and results of torture. Early on, the point was raised on how do you collect data on torture. In fact, that point was made several times throughout the article in what appeared to be a bid to highlight the issue of conducting scientific research on what they call “one of the most extreme forms of human violence.” To codify this, the report used the United Nations definition of torture that states that any act that causes severe suffering or pain (mental or physical) with the intent to extract information and is executed under the authority of a public official is torture. 

That aside, they did pull from many different studies concerning problems from coerced information. One instance they looked at was in law enforcement and the confessions pulled from suspects via long interrogations and even less legal means. What they found was 24% of wrongful convictions came from false confessions that were gained via means far less detrimental than torture. Costanzo and Gerrity use this as a basis to show that information gained from torture would have a shadow cast upon its validity. Suspects in criminal cases were often kept awake and interrogated for hours on end until they had lost major cognitive control. They eventually just confessed to something that they had not done to make it stop. 

Greater parallels are draw versus prolonged law enforcement interrogations and torture. The report asked that any information that was gained in marathon interrogation was often false. Add in mental and physical pain and the probability of false information increases. Ulpian in 200 CE raised questions on the validity of information extracted under torture was suspect. Further evidence was provided from CIA operatives that people being tortured would be willing to confess anything, true or not, just to make the torture stop. 

Critique

This is a very well sourced article that draws from many different fields for the purpose of looking at something that is hard to study. After going through it, some of my ideas on torture are now not as justifiable. While I am very aware of intelligence professionals who state that the Arab culture is not conducive to normal methods of interrogation and that torture is necessary, the report stated that people who made use of torture attempt to justify it as a means of dismissing or allaying what they have done. I am by no means saying that supporters of torture are wrong or are falsely attempting to justify torture as a valid means of getting information out of someone. I have not been in a firsthand situation to see positive results from torture (and I rather hope I do not have to). It is when there has been nearly 2000 years of concerns as to the validity of torture for attaining actionable information that I start to have misgivings.

The Interrogation Game: Using Coercion and Rewards to Elicit Information from Groups

Summary

This study created a model called the "Interrogation Game" to test the differences between how knowledgeable and ignorant sources react when faced with both coercive and rewarding interrogators. The primary purpose of the study was to determine when detainees provide accurate information to interrogators. The study also highlighted the importance of understanding when detainees truthfully tell interrogators that they do not know the desired information. The authors, David Blake Johnson and John Barry Ryan, acknowledge both the lack and necessity of research on the efficacy of interrogation within the intelligence community. Johnson and Ryan argue that both knowledgeable and ignorant sources have the ability to provide inaccurate information, and that interrogators are unable to decipher the difference between the two during an interrogation. If a detainee is truly ignorant, then further interrogation may only lead to false information.

 The "Interrogation Game" model identifies a coercive interrogator as one who reduces a detainee's utility for not providing accurate information, and a rewarding interrogator as someone who provides a detainee with utility gains in exchange for accurate information. To test the differences between the two, this study performed an experiment using 160 Florida State University students in ten sessions each. During each session the participants were divided into groups of four. Each group was assigned a playing card, which could only be one of the four aces. Two participants within each group were given the identity of the card, while the other two were given no information. The "interrogator" in the experiment consisted of a computer that prompted a random participant to identify the group's card. The participants received different earnings depending on the type of interrogator. Most importantly, ignorant participants (participants who were asked to reveal the card but did not know the card's identity) who truthfully stated "I don't know" to the rewarding interrogator received the same amount of earnings as the other participants. However, ignorant participants who truthfully stated "I don't know" to the coercive interrogator received less earnings than the other participants. In other words, the coercive interrogator punished participants for not knowing the identity of the card. 

The study found that ignorant participants truthfully responded "I don't know" to the coercive interrogator 12.5% of the time, whereas ignorant participants truthfully responded "I don't know" to the rewarding interrogator 32.5% of the time. Johnson and Ryan explain that this is because revealing the truth to a rewarding interrogator leads to a fair outcome, whereas telling the truth to a coercive interrogator leads to an unfair outcome. In turn, the study found that participants revealed the incorrect card to the coercive interrogator 43.8% of the time, which was 7.1% more often than the amount of times the incorrect card was revealed to the rewarding interrogator. This is because participants were more willing to admit to not knowing the identity of the card to a rewarding interrogator than they were to a coercive interrogator. Johnson and Ryan concluded that not only did the participants tell the truth less frequently to coercive interrogators, but the ignorant participants were also less likely to reveal their ignorance to coercive interrogators. 

Critique

This study was confusing to understand at first due to the several assumptions and details included in its model. Johnson and Ryan incorporate multiple formulas and theories that add to this confusion, though they also seem to increase the model's reliability. A major concern, though, is the lack of reality mixed in with the experiment. Johnson and Ryan mention how torture is a form of interrogation, but it is hard to truly view this experiment as a form of "coercive interrogation". This is not to say that it cannot be applied to real-life torture situations, but it is difficult to see a complete connection. 

Source

Johnson, D. B., & Ryan, J.B. (2015). The interrogation game: Using coercion and rewards to elicit information from groups. Journal of Peace Research, 52(6), 822-837. Retrieved from: http://jpr.sagepub.com/content/52/6/822.full.pdf
 

Interrogational Torture: Or How Good Guys Get Bad Information with Ugly Methods

Summary

This research aimed to provide insight on an important problem: the effectiveness of interrogational torture in revealing clear and accurate information to the state.  There is little consensus among policy makers, academics, or even interrogators on whether interrogational torture works. This article examines whether torture is effective in revealing reliable information and at what cost in terms of frequency and intensity of the torture employed, even under limited conditions and with certain restrictions in place. Due to the absence of any reliable and systematic data from either the field or laboratory on the effectiveness of torture, Schiemann investigates the question theoretically by employing the tools of game theory.

Game theory allows Schiemann to build an analytical model of restricted, limited interrogational torture incorporating realistic elements and corresponding to what one might call the pragmatically normative view of interrogational torture: that “limited” or “restrained” torture is unfortunately necessary in limited cases. This allows Schiemann to compare the outcomes of the analytical model with the criteria defining the normative model of interrogational torture’s proponents. The model’s outcomes substitute for empirical data that we will likely never have to assess torture’s effectiveness.

In the first section, Schiemann outlines his normative model of pragmatic interrogational torture.  The point of this model is to first identify constraints on the use of torture in interrogations, and second, the anticipated consequences of employing interrogational torture, including the expected reliability of the information gathered and the degree to which torture is employed.  Together, they define how torture should work on the pragmatic view.  The second section develops two variants of an analytical model of interrogational torture that is consistent with both the constraints of the normative model and other real-world features of interrogational torture to see how torture would work.  Obviously, if data was actually available one could assess how torture does work. The third section presents the results of the analytical model and compares them to the predictions of the normative model, with the goal being to see if the outcomes of the analytical model support or undermine the normative model. Finally, Schiemann concludes in the last section by discussing some implications for state-sanctioned interrogational torture.

The Game

In Schiemann’s model, there are three players in the game, depicted in Figure 1: nature, a detainee, and an interrogator for the state.  The nature player is game theoretic convention for modeling uncertainty about other players in the game, in this case about the characteristics of the detainee and the characteristics of the interrogator.  The interrogator is uncertain whether the detainee is knowledgeable and weak, knowledgeable and strong, or possesses no information (innocent).  The knowledgeable and weak detainee is willing to give up information in exchange for not being tortured, while the knowledgeable and strong detainee prefers to suffer torture rather than give up information.  The innocent detainee has no valuable information to divulge.  The uncertainty about exactly which detainee is actually facing the interrogator is captured by the first move of the game, by nature, dividing it into three branches.

Findings

Schiemann concludes that the outcomes of the analytical model provide little support for the pragmatic defense of interrogational torture.  The outcome predicted by the pragmatic defenders of interrogational torture – valuable information extracted by (the threat of) torture from a knowledgeable detainee but no torture to innocent detainees – does not occur in equilibrium.  For torture to ever generate valuable information, the state must be willing to torture innocent detainees.  Furthermore, this “ever generate” is not the same as “reliably generate,” despite coming at the cost of torturing innocents.  An outcome resulting in valuable information is possible, but the conditions supporting it are empirically unlikely.  They require a knowledgeable detainee to believe the promise he will not be tortured if he divulges information and an interrogator to be very confident the detainee has told all he knows.  Moreover, even when these conditions are met, interrogational torture fails to satisfy the necessary reliability condition.

Critique

The question of whether interrogational torture is effective in providing valuable information to the state is ultimately an empirical question that cannot be conclusively decided by a game theoretic (or any other) model. However, formal models can be helpful tools for probing reality. Examining if – in reality – interrogational torture actually provides the state with vital information they could otherwise not get – and at what human cost – is a pressing moral question of our time. The debate over this question implies that this reality needs probing, and the probing Schiemann offers here suggests that torture games have no winners.

Source

Schiemann, J. W. (2012). Interrogational torture or how good guys get bad information with ugly methods. Political Research Quarterly, 65 (1), 3-19.