Summary of Findings: Devil’s Advocacy (3.5 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in September 2016 regarding Devil’s Advocacy as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Devil’s Advocacy is an analytic modifier that tests assumptions and critiques existing analysis. This is done through the use of a specific individual or team that actively looks for potential inaccuracies or problems within an analysis. This can also be done by the advocate(s) supporting a minority point of view and championing it versus the prevailing analysis.

Strengths:

• Works with unstructured data

• Provides alternative viewpoints if the Devil’s Advocate is an expert and at the same level as the rest of the team.
• It can be simple to set up and create, and is flexible enough to use in any environment.
• Can help to uncover biases and assumptions in reasoning

Weaknesses:

• May not be successful if the Devil’s Advocate role is not formalized and  respected within the team
• Has the potential to create tensions in analytical teams
• If done incorrectly, it has the potential to unnecessarily reduce confidence (by intelligence consumers) in an adequately supported estimate
• May not work if team members are unable to accept or perceive the ideas of the Devil’s Advocate
• If pressed for time, it is difficult to adequately implement

How-To:

1. Designate a person/team to look at a previously established assumption or analysis.
2. That person/team then proceeds to use available data and experiences to challenge the assumption or analysis by providing alternative explanations, outcomes, or reasonings that are just as likely to occur given the information available.  
3. Present these finding to an individual or group willing to actively listen to the Devil’s Advocate’s findings.

Application of Technique:

A group of Graduate Students at Mercyhurst University developed a Devil’s Advocate exercise about North Korean nuclear development strategies.  Various types of role-played national security  intelligence assessments were developed: OSINT, HUMINT, SIGINT, GEOINT, IMINT.  See “National Security oriented Devil’s Advocate Exercise used in class [North Korea]” below.

In class the participatory team challenged “All-Source Team Black’s” overall estimate on North Korean nuclear weapons and developed reasons to accept or reject the overall analysis for it’s legitimacy of sources and built in key assumptions.  Instructions on how to use the process were issued from the “CIA’s Analytic Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis” Also, students were able to provide personal input on their personal understandings of the situation.  The team had 15-minutes to work through this problem.

For Further Information:

National Security oriented Devil’s Advocate Exercise used in class [North Korea]: https://drive.google.com/file/d/0BxOGezwoHVUCTEl5VTllRmRNbmc/view?usp=sharing

CIA Analytic Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis, pg. 17: https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/Tradecraft%20Primer-apr09.pdf

Devil’s Advocate Wikipedia:

https://en.wikipedia.org/wiki/Devil%27s_advocate

Red Team: How to Succeed By Thinking Like the Enemy, pg. 216-219:

https://www.amazon.com/Red-Team-Succeed-Thinking-Enemy/dp/0465048943