Summary Of Findings: Gap Analysis (3 Out Of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 12 articles read in advance of (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 29 APR 2009 regarding Gap Analysis specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:
Traditionally, "gap analysis" is a method used to conduct an internal operational analysis, whereas the gap analysis identifies the "gap" between a current state and a desired endstate within a company or agency. From an intelligence analysis perspective, "gap analysis" can be used as a tool to identify the likely pathway or pathways a target may take to arrive at a given endstate from a known position. Thus, "gap analysis" does not necessarily provide an estimate, but rather provides the analyst with a list of possible actions a target may likely take. Gap analysis as an analytic technique bears a striking resemblance to several other methods, such as Indicators & Warnings and Decision Trees.

Strengths:

1) Identify the target

2) Characterize the current status of the target as well as the target's goals

3) Identify what you want to know about the target

4) List the pieces of information that you have

5) Use a systematic approach to infer what the target is likely to do in order for the target to reach its goals

Weaknesses:

* No structured method to conduct the analysis

* May not leave you with a clear estimate

* Open to bias and other cognitive downfalls (satisficing, mirror imaging, etc.)

* Overlaps with the process of other methods and modifiers (i.e. decision treees, I&W, Brainstorming, SWOT, etc.)

* Susceptible to deception

* Danger of pitfalls

How-To:

1) Identify the target

2) Characterize the current status of the target as well as the target's goals

3) Identify what you want to know about the target

4) List the pieces of information that you have

5) Use a systematic approach to infer what the target is likely to do in order for the target to reach its goals

Experience:

For the first application, the group tried to determine what thesis topic Mary, a fictional first year graduate, would write about. Professor Wheaton acted in place of Mary, and we role played in questions and answer format. The group determined the gaps needing filled would be what intelligence track she was most interested in (national security, law enforcement, or competitive), what area or topic in her previous classes interested her the most, the choice of her primary reader, and the reader's academic interests. Upon filling these gaps, the group ascertained a plausible topic for Mary's thesis.

For the second application, the group discussed Russia's long held ambitions for a warm water port, preferably on the Mediterranean. The first part of the discussion centered on hostilities between Georgia and Russia, and actions Russia could take against Georgia to maintain their sphere of influence in the Black Sea. Additionally, the group discussed what actions may be necessary in their diplomatic relations with other nations bordering the sea. After discussing how Russia could potentially gain a Black Sea port by bringing Georgia into their sphere of influence, the group discussed how Russia could proceed toward gaining access to Mediterranean ports. The group determined that Turkey would be central in future Russian objectives for ports in the Mediterranean. The group put together lists of things the Russians could do that they are not doing now that would indicate their goals of extending their influence in Georgia and Turkey.

These applications illustrated the following:

* Helped in thinking through the steps that lead to a decision

* Allowed for open discussion and debate, helping the critical thinking process

* At some points it felt like stabbing in the dark, but the estimates later became more clear as the group discussed the options

* Eliminated peripheral influences not directly related to the topic, such as administrative processes.

Gap Analysis: As Is or To Be, What is the Question?

Gap Analysis: As Is or To Be, What is the Question?
By Dorothy Ball,
Four Thought Group

Summary:
This article describes how to use gap analysis in Business Process Improvement (BPI) strategies, relating the method to Health Care organizations. In this summary, I have attempted to relate the gap analysis process, as the author uses it, to the intelligence field - identifying and overcoming intelligence gaps.

Dorothy Ball, a senior policy and business consultant at Four Thought Group, Inc., states that gap analysis is an effective solution for businesses and organizations that have a service orientation (which includes for-profits, non-profits, and government agencies). To paraphrase, gap analysis is a method for an organization to improve performance or make gains by identifying the potential needs for that organization to move from where they are (or what intelligence you have) to where they want to be (or what intelligence is still needed). Ball describes gap analysis as a method to develop a roadmap that gets you from “where you are now (As Is) to where you want to be (To Be).”

How to:

1. Identify what your organization looks like now (or what intelligence/information you currently have available) and what you want your organization to look like (or what intelligence/information you desire). Needs for improvement (more intelligence) is often indicated by changes such as policy, resources, environment (or events that spark inquiry). Changes may be event-driven, or ongoing.
2. Understand the Business Process (or collection process). Ball describes the business process as “a collection of related, structured activities, or chain of business functions, activities and tasks, that produce each specific service or product… Each business process consists of inputs, method, and outputs. The inputs are required before the method can be put into practice to achieve the outcome. When the method is applied to the inputs then certain outputs will be created.”
3. Use comparative analysis techniques to identify what is needed to get you from where you are (the Intel you have) to where you want to be (the Intel desired). Examination of the current process, and the modeling of the new process, may be necessary in order to discover the interrelationships/interconnectedness of how the current process can get you the desired results.
4. Create a plan for implementation.
   

Summary Of Findings: Game Theory (4 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 12 articles read in advance of (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 22 APR 2009 regarding Game Theory specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Game theory is a method based on applied mathematics and economic theory. It can be useful when attempting to analyze (and ultimately predict) the strategic interactions between two or more actors and the way in which their actions influence future decisions. Game theory assumes that all actors are rational, and can be influenced by various individuals and factors. Games typically involve five common elements: players, strategies, rules, outcomes, and payoffs.

Strengths:

-assumes rational actors

-assumes actors will adjust their actions based on the actions of other actors

-not clearly differentiated from role-playing, simulations, and/or decision trees

-very mathematically based (can be intimidating)

-difficult to quantify options, strategies, and motivations

-may not be a valid method to produce an accurate estimation (see Game Theory, Simulated Interaction, and Unaided Judgment For Forecasting Decisions in Conflict: Further Evidence)

--In real world applications, identifying all of the key players and outcomes can be difficult

Weaknesses:

-Visual step-by-step trail to a conclusion/estimate

-Ability to quantify variables in play

-Emphasis on mathematics and scientifc method

-Applicable to multiple fields (economics, conflict, etc)

-90% rate of success according to BDM

How-To:

Game Theory varies in complexity and in application, however, each application has the following in common:

*Establish the players and the complexity of the game being played, so as to understand the rules which govern the players and the game.

*Identify the possible outcomes for the choices the players can make (although this is particularly difficult as not all decisions can be predicted)

*Establish measurable values for predicted outcomes.

*Eliminate dominated strategies and employ dominate strategies. Repeat this step until a clear, singular strategy emerges or equilibrium is reached between the players.

*Employ selected strategy.

Experience:

As a class, we visited www.gametheory.net and played the repeatable version of Prisoner's Delemma under the "Interactive Materials" tab. Each student played the game at their personal computers. Our objective as we played against the five "personalities" was to identify the particular strategies employed by the computer (in addition to scoring the most utility points). Some of the strategies employed by the computers personalities included "tit-for-tat" and "tit-for-two tats."

Game theory, simulated interaction, and unaided judgement for forecasting decisions in conflicts: Further evidence

Kesten C. Green

Summary
Green's article, published in the International Journal of Forcasting (2005), is a validity study that compares the accuracy of game theory, simulated interactions (role playing), and unaided judgment as methods in forecasting decisions in conflicts. It provides further evidence to support his earlier research, which indicated that expert game theorists' forecasts had less accuracy than those of novice (student) role players.

It is important to note here that these methods were tested in experiments involving specific conflict situations only. Not all game theorists recommend game theory for generating forecasts in specific situations.

Green's 2002 study reflected the accuracy of the three methods for forecasting conflict decisions. Participants in the study were asked to select the most likely decision (in 5 conflict situations) from a list. The results of the study did not bode well for the game theory experts (participants, % accuracy):

• University Students (using unaided judgment), 27%
• Game Theory Experts, 28%
• University Students (using role playing/simulated interactions), 61%

Green's 2002 study drew some attention from his peers and game theorists, who commented that perhaps some conflict situations are more appropriate for use with game theory than others. The 2005 study utilized the same basic process, and was administered to participants representing the aforementioned three methods. It was directed at three (actual) conflicts, unrelated to the five used in the 2002 study:

1. Personal Grievance. An employee of a New Zealand student association felt there was a disparity between her work and the level of pay she was receiving. Upon the administration's evaluation of her situation, it was discovered that she was being compensated above the top-level of pay in her salary bracket. While her manager did not consider cutting her salary, she did arrange for a mediator to set up a meeting between the parties, as it was obvious she would not be elligible for a pay increase in the near future. Participants were to choose from several decisions.
2. Telco Takeover. This situation presented the participants with the conflict stemming from a corporate takeover battle between Alltel and CenturyTel. In 2001, executives from the much smaller CenturyTel company presented an offer that involved the sale of their mobile phone division to Alltel, which Alltel declined. Soon after, Alltel returned to CenturyTel with an offer to buy all of the company at 40% more than the share price. The board of CenturyTel fought to prevent an Alltel takeover. Participants were to forecast the execution of the deal and predict the basis of terms.
3. Water Dispute. This conflict raged over Syria's and Iraq's claims to access of the Euphrates River in 1975. The flow of water into Iraq had been slowed following Syria's construction of a dam across the Euphrates to create a reservoir in Syria. Both sides mobilized for war, and Saudi Arabia called both parties together for mediation. Participants in the study were to forecast whether Syria would increase water flow from the dam or whether Iraq would declare war and bomb Syria's dam.

Since the findings of this research were to be compared with those of the 2002 study, Green took steps to evenly match the conflict situations with game theorists' and other decision makers' interests, i.e. types of organizations involved, nature of the disputes, and familiarity of situations.

Results
The findings of this follow-up study are entirely consistent with the 2002 study. Game theory experts and unaided novices had a much lower forecast accuracy than novices who implemented role playing (simulated interactions) in predicting the eight conflict decisions.

Summary Findings: Argument Mapping (3 Stars Out Of 5)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 12 articles read in advance of (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 1 APR 2009 regarding Argument Mapping
specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Definition:

Argument mapping (AM) is an analytic modifier that can be used to examine the logic behind the development of a particular conclusion and/or hypothesis. The product of AM is a visual representation (typically a box-and-line diagram) of the reasons that support and oppose the claim. Constructing a visual depiction of a complex argument reduces the level of abstraction in evaluating a decision.

Strengths:

Provides Audit Trail

Breaks down prose arguments and visualizes arguments for decision makers.

Shows strengths and weaknesses of argument

Improves critical thinking

Reduces cognitive bias and blindspots

Weaknesses:

Not estimative in nature.

The method does not explicitly seek out contradictory evidence.

The language specific to argument mapping is not uniform from scholar to scholar.

Dependent on the quality of presentation of original argument

The technique does not consider the impact of deliberately deceptive information

How-To:

* Must locate a central claim for a position (this is the conclusion of an argument)

--conclusion indicators include: therefore, thus, so, hence

* extrapolate true and logistical reasons and objections which support or refute the central claim.

--rewrite statements as individual sentences

--Reasons must answer the question: "How do we know that [insert claim] is true/warranted?"

* include all premises for reasons - leave nothing to be implied

--premise indicators include: since, because, for, given that

* With objections, list any possible refutations underneath the objection

* Form a 'warrant' - a statement that justifies the step from a reason to a claim

--warrants reveals an arguer's core implicit beliefs

--used to scrutinize the soundness of an argument

--used to accept or reject controversial arguments

*Use arrows to indicate which premises or co-premises support claims; or premises refute oppositions.

*indicate premises that need to be combined in order to support a conclusion, and premises that are each seperate reasons to believe a conclusion

*Use a color-coding scheme (if possible) to help visualize the arguments

--Supporting claims - green

--opposing claims - red

--nuetral claims - gray or no color

Experience:
We attempted to argument map a one page NY times editorial. The comments below were generated as a result of that exercise.

Argument mapping is difficult to synthesize from prose that is not written in a BLUF (Bottom Line Up Front) format.

The ability to effectively argument map is too dependent on quality of the articles given.

People will disagree on structure of argument and levels of relevance of data.

Argument mapping is not appropriate to use to develop an analytic estimate, rather it can be best implemented as a modifier to validate an already determined estimate.

Argument mapping would be an excellent modifier to test the validity of a conclusion/forecast. Would be an excellent supplement to ACH - suggest conducting an ACH to find a claim that is most likely, then test that claim for validity using Argument Mapping. This will help to reduce cognitive blindspots and biases, as well as test the reasoning behind the conclusion.

Extrapolating the main claim of the argument was easier once all of the premises supporting it were laid out.

The original format of the 'argument" really matters when it come to the ease or difficulty of applying argument mapping to the text.

Argument Mapping (Efebia)

Summary
Argument mapping is a logical sequencing method that employs box-and-line diagrams to "map" a course of possible decisions for a given "argument" and the ramifications of each option. The purpose of the map is to provide a visual depiction of the relationships between the overall contention and the logical (evidential) claims supporting or opposing it. Upon examination of the supporting and opposing claims, an argument map should serve as a useful tool for use in the decision making process, by reducing the complexity of a dilemma, conflict, or argument. Since argument mapping distinguishes itself from other box-and-line methods by utilizing different shapes, colors, and positioning, it appeals to the strong visual comprehension abilities of humans, allowing for improved processing and understanding, ultimately simplifying the complexity of an argument.

Basic Construction of an Argument Map
The components of an argument map are listed below with a brief explanation. Those pieces of the map are identified in the example below (click on the map to link to the source and to access a higher resolution map).

• The main claim: referred to in this article as the "position" or "contention" -is the hypothesis to be logically examined. The argument map's purpose is to help the decision maker accept or reject this hypothesis.
  
• Reason: a positive claim, or one that directly supports the main claim. Reasons not only support the main position, they may also support another reason.
• Objection: a negative claim, or one that opposes the overall main claim.
• Rebuttal: opposes an objection directly above it; or, "objection against an objection".

Strengths of Argument Mapping
According to the article, by using colors, shapes, etc. in argument mapping, the mind can better process the complexity and abstract nature of a difficult argument. Additionally, the logical structure of argument mapping is easier to present than traditional prose, which requires the reader to deconstruct the argument for himself, demanding a large investment in time and cognition.

The article also cites "extensive research" conducted by the University of Melbourne to gauge the effectiveness of argument mapping in the context of critical thinking. The study compared the achieved critical thinking abilities of students who utilized the Reason!able (Rationale) argument mapping tool versus those who used traditional prose. The study determined that, in a 12-week course, students who used argument mapping gained 12 IQ points.

Author's Comment:
Since this article may have commercial implications (as it names a particular brand of software), it is obviously a strong advocate for argument mapping, and does not approach the method with much objectivity. While there may be some obvious disadvantages to using argument mapping in certain situations, none are provided here. Moreover, the only other method this article discussed was traditional prose, which is not an analytic method by itself, but a medium of production or dissemination. Despite a mere mention of conceptual modeling and flow charts, there was no substantive, formal comparative discussion on them.

Argument Mapping and Analysis of Competing Hypotheses (AM vs ACH)

Summary

This article presents a comparative assessment of argument mapping (AM) and Analysis of Competing Hypotheses (ACH) as analytic diagramming techniques. While the paper focuses primarily on the differing techniques in diagramming a decision making pathway, the author considers both as "complementary analytical frameworks".

Comparing the Techniques

Both techniques involve analyzing a given proposition (hypotheses in ACH; conclusions in AM), evaluating the evidence (evidence in ACH; reasons and objections in AM), and diagramming a visual representation of the relationships (matrices in ACH; box-and-line diagrams in AM). In ACH, the letters C, I, and N represent positive (consistent), negative (inconsistent), or nuetral relationships, respectively. In AM, arrows indicate a relationship between the reasons supporting or opposing the conclusion; color indicates whether the evidence is a reason (green) or an objection (red) for the argument. When a piece of evidence is neither a reason nor an objection (neutral or irrelevant), the creator of the AM may choose either to eliminate it from the map or to make it a different color (typically gray).

Multiple Propositions
Due to the manner in which each of these methods handles multiple propositions, ACH gets the advantage over AM. ACH incorporates multiple propositions simply by adding another column in the matrix. However, adding any additional propositions to an AM may result in a cluttered diagram with crossed lines and complicated pathways. To resolve this issue, it may be more appropriate to duplicate pieces of evidence, or even create an additional map, making it less efficient.

Multi-tiered Evidence
The paper uses an example from Psychology of Intelligence Analysis to demonstrate a situation in which AM may be a more efficient method. The chosen proposition is "Iraq will not retaliate forUS bombing of its intelligence headquarters," and the piece of evidence in question is "Saddam [has made a] public statement of intent not to retaliate". Often, as cited in this example, an additional piece of evidence may be necessary to establish a piece of evidence. When using an ACH, the analyst may need to construct another matrix to validate evidence. The AM technique allows for the addition another level of evidence within the same map. The paper refers to the multi-tiered evidence as "granularity". Granularity allows evidence of differing levels of abstraction to be present on the same map; "the higher the level on map, the more general or abstract the reason or objection".

Assumptions
AM also allows the user to add assumptions or warrants alongside the pieces of evidence. Incorporating an assumption into an ACH would force the user either to add a justification with the conclusion in its cell in the matrix, or combine multiple pieces of evidence together.

Summary Findings: Dialectic And The Socratic Method (3 Stars Out Of 5)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the 12 articles read in advance of (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst College on 25 MAR 2009 regarding Dialectics generally and the Socratic Method specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Definition:

The Dialectic Method is an analytic technique designed to force the participants to re-examine their internal beliefs, biases, and conclusions through an open and directed dialogue.

The Dialectic Method uses questioning techniques with the intention of creating a better understanding of a problem or concept. In the realm of intelligence analysis, it should be used as an analytic modifier; i.e. a technique to reassess the validity of the analytic process, not as a forecasting method.

Strengths:

--The primary strength is the ability to identify and challenge initial assumptions about a target, and in effect, it reduces prejudice and bias.

--The dialectic method is also useful throughout the intelligence cycle from requirements, estimative conclusions, and feedback.

--Using dialectic demands the analyst think critically about the certainty of the analysis generated.

Weaknesses:

-- It does not provide an analytical forecast by itself.

-- The questioner needs to be highly skilled in managing the process.

-- As a cautionary note, thinkers caught in their own illogical concepts may become irritated or even angered by such an approach.

-- The approach can be time consuming, and should not be used under time constraints.

How-To:

-- The first step is to provide an initial, well-formulated question with group-wide understanding of the hypothesis at-hand.

-- After the initial hypothesis is presented, the group undetakes an opposing line of questioning to disect the hypothesis and its sub-components.

-- Use the discussion to synthesize arguements for and against the initial hypothesis to determine its truth and validity.

Experience:

We applied the Socratic Method specifically and the principles of dialectics generally to a variety of realistic intelligence situations. We explored how a formal questioning approach that assumes an antithesis, for example, could perhaps have impacted the estimate regarding the presence of WMD's in pre-war Iraq. Even if a Socratic approach to questioning the conclusions of that estimate would not have changed the overall finding, the group generally agreed that it would probably have altered the final level of confidence.

We also applied the method to the requirement phase where the decisionmaker who wants "everything" could be seen as establishing a thesis while the intelligence professional who knows that the decisionmaker doesn't need everything essentially establishes an antithesis. The ultimate intelligence requirement could then be seen as the synthesis of the two positions.

Welcome!

Welcome to the blog for the Spring 2009 Advanced Analytic Techniques class at Mercyhurst College. Each week we will be taking a hard look at a particular technique used by analysts in the law enforcement, business or national security intelligence communities.

Students in this class are required to identify and summarize two "resources" on each technique before Tuesday of each week. Resources include primary and secondary sources of any kind on the particular technique we are studying. This includes, books, articles, case studies, websites, company profiles, etc.

The summaries, however, will focus on four main questions. Based on what is contained in the selected resource:

• What does this technique do (or purport to do)?
• What are its strengths?
• What are its weaknesses?
• What step-by-step actions should you take to use this method?

Obviously, not all of these questions will be answered in each of the resources the students find and summarize. However, the collected set of resources that all of the students find should, ideally, create a consistent series of answers to the four questions above.

Ideally. That is the operative word here. Anyone who has worked with a variety of intelligence analysis techniques knows that consistency is not necessarily their hallmark. My expectation is that we will learn as much (if not more) from those techniques that are poorly defined as we will from those whose definitions are clear, concise and consistent.

Let the learning begin!