Author: Lauren C Culver

August 2017

Critique: Bryant C Kimball

**Summary**

Standard Monte Carlo
simulations consist of a collection of hundreds of thousands of random functions
that cycle through a given distribution.
The results of these thousands of interactions is considered the
generation of potential outcomes. While historical
risk assessment forces the analyst to consider possible outcomes based off of
all the possibilities that have already happened, the Monte Carlo method combines
stochastics and simulation; cycling a random sampling of inputs into a virtual representation
of a problem over and over and over again to obtain a distribution of results.

In 2017, analysts
supporting decision making at the intersection of energy and U.S. foreign
policy teamed Monte Carlo analysis with decision analysis, predictive scenario
analysis, and exploratory modeling to understand the threat of sixteen
countries’ energy demands on US foreign policy.
However, the experiment itself pits the four models of uncertainty
analysis against one other with the intentions of issuing a recommendation
about the most appropriate approach to uncertainty analysis for foreign
policy.

Figure 1 |

Figure 2 Figure 3 |

The simulation produced a series of outcomes for each country as well as policies to match (Figure 4). The researcher found that
the results of the Monte Carlo analysis correctly convey the policy and that one
policy will not always be beneficial. It requires additional, time-consuming
analysis by the analyst to more specifically identify the input set that drives
net benefits of a particular policy.

**Critique**

Essentially, Monte Carlo
simulations can present an analyst with a distribution of outcomes for a given
situation. This research serves as evidence that Monte Carlo simulations can help reduce uncertainty as well as issue recommendations based on specific outcomes across a distribution. This validated tool clearly
reduces uncertainty by allowing the analyst to easily deconstruct the potential
outcomes. Even more so in intelligence analysis that forces the analyst to
discriminate between numeric ranges of likelihood, Monte Carlo as a tool, can
help add to an analyst’s judgment by either validating or helping guide the
analyst to the most accurate WEP.

What this particular
model does an even better job of depicting is that no one methodology is
holistic. Each uncertainty model relies
on the others to complete the picture. This holds true with all the methodologies
we’ve studied thus far, and highlights the importance of building a toolkit
with a wide range of methods and modifiers to use when deemed relevant.

https://ngi.stanford.edu/sites/default/files/Culver_dis%5B1%5D.pdf

https://ngi.stanford.edu/sites/default/files/Culver_dis%5B1%5D.pdf

Although I agree with you when you say that "no one methodology is holistic," this summary is informative and presents an understanding of the monte carlo method as well as its value. Evidently, this method can provide various outcomes, if done correctly, and it seems it is designed to possibly introduce the analyst to a new way of approaching the issue.

ReplyDeleteChelsie,

ReplyDeleteThe articles summarized by both Tom and myself highlighted Monte Carlo as a method that strengthens analytic confidence in intelligence analysis. Monte Carlo's design itself provides the somewhat unique ability of quantifying what we typically view as issues too complex to quantify with any sort of precision.

After reading these summaries, I think of monte carlo as 10,000 analysts running a simulation with the same data and producing an estimate, then we look at how all those estimates compare and we are inclined to take the most prevalent estimate as the estimate most likely to be correct (except it's a computer doing the work). I don't know if this is actually true, but polling the audience on "Who Wants to be a Millionaire?" always seemed to give the right answer. However, the phrase "no one saw that coming" leads me to believe just because everyone is seeing the picture one way doesn't necessarily mean they're right. Do you think software running these simulations 10,000 times is more or less reliable than 10,000 analysts running simulations?

ReplyDeleteJillian - personally I believe the answer to this comes down to how messy the data is. I think there are some layers to intelligence analysis that requires an actual analyst and not a simulation

DeleteYour blog post definitely displays the value of MC sims. You mention how it is not holistic and all of the uncertainty models rely on each other. Where would you say MC falls short?

ReplyDeleteTotal after thought but I cant help but think about cruising around in a 1986 Monte Carlo whenever I hear about this methodology. What a Car!

I think its significant to be able to quantify a portion of your estimate. In terms of limitations, I just think, as I said above to Jillian, there are some layers to intelligence analysis that requires an actual analyst and not a simulation.

DeleteYes its a nice car.

I find it interesting how in this circumstance the authors recognize the need for more analysis after running multiple forms of analysis, including the Monte Carlo Simulation. It seems in the context of energy policy and national security would be a complex topic where intersecting objectives create a situation where the net benefits of the policy must exceed the net negatives of the policy. Based on your understanding of the analysis above, do you think that MC simulations would provide a "better" result that would provide more definitive analysis on a simpler or less complex issue than the one ascribed above?

ReplyDelete