Wednesday, November 7, 2018

Modeling for Decision Making Under Uncertainty In Energy and US Foreign Policy

Pages 136-148
Author: Lauren C Culver
August 2017
Critique: Bryant C Kimball


Summary
Standard Monte Carlo simulations consist of a collection of hundreds of thousands of random functions that cycle through a given distribution.  The results of these thousands of interactions is considered the generation of potential outcomes.  While historical risk assessment forces the analyst to consider possible outcomes based off of all the possibilities that have already happened, the Monte Carlo method combines stochastics and simulation; cycling a random sampling of inputs into a virtual representation of a problem over and over and over again to obtain a distribution of results. 
In 2017, analysts supporting decision making at the intersection of energy and U.S. foreign policy teamed Monte Carlo analysis with decision analysis, predictive scenario analysis, and exploratory modeling to understand the threat of sixteen countries’ energy demands on US foreign policy.  However, the experiment itself pits the four models of uncertainty analysis against one other with the intentions of issuing a recommendation about the most appropriate approach to uncertainty analysis for foreign policy. 
Figure 1

The model run time for a single country took about 20 minutes to produce 1,000 unique input combinations for each policy (Figure 1)), so 6 hours for all sixteen countries, not including time for analysis.  Although run time is lengthy, when the results are calculated there is both a clear sense of the best policy, the one that maximizes expected net benefits, and the risk associated with it and there are hundreds of scenarios to compare. In addition to providing the analyst with rich results, the output of Monte Carlo analysis is intuitive by nature and the visuals make it easy to detect the trends that lead to analysis (Figures 2 and 3). 
Figure 2

Figure 3

The simulation produced a series of outcomes for each country as well as policies to match (Figure 4). The researcher found that the results of the Monte Carlo analysis correctly convey the policy and that one policy will not always be beneficial. It requires additional, time-consuming analysis by the analyst to more specifically identify the input set that drives net benefits of a particular policy.


 
Figure 4


Critique

Essentially, Monte Carlo simulations can present an analyst with a distribution of outcomes for a given situation.  This research serves as evidence that Monte Carlo simulations can help reduce uncertainty as well as issue recommendations based on specific outcomes across a distribution.  This validated tool clearly reduces uncertainty by allowing the analyst to easily deconstruct the potential outcomes. Even more so in intelligence analysis that forces the analyst to discriminate between numeric ranges of likelihood, Monte Carlo as a tool, can help add to an analyst’s judgment by either validating or helping guide the analyst to the most accurate WEP. 

What this particular model does an even better job of depicting is that no one methodology is holistic.  Each uncertainty model relies on the others to complete the picture. This holds true with all the methodologies we’ve studied thus far, and highlights the importance of building a toolkit with a wide range of methods and modifiers to use when deemed relevant. 

https://ngi.stanford.edu/sites/default/files/Culver_dis%5B1%5D.pdf

8 comments:

  1. Although I agree with you when you say that "no one methodology is holistic," this summary is informative and presents an understanding of the monte carlo method as well as its value. Evidently, this method can provide various outcomes, if done correctly, and it seems it is designed to possibly introduce the analyst to a new way of approaching the issue.

    ReplyDelete
  2. Chelsie,

    The articles summarized by both Tom and myself highlighted Monte Carlo as a method that strengthens analytic confidence in intelligence analysis. Monte Carlo's design itself provides the somewhat unique ability of quantifying what we typically view as issues too complex to quantify with any sort of precision.

    ReplyDelete
  3. After reading these summaries, I think of monte carlo as 10,000 analysts running a simulation with the same data and producing an estimate, then we look at how all those estimates compare and we are inclined to take the most prevalent estimate as the estimate most likely to be correct (except it's a computer doing the work). I don't know if this is actually true, but polling the audience on "Who Wants to be a Millionaire?" always seemed to give the right answer. However, the phrase "no one saw that coming" leads me to believe just because everyone is seeing the picture one way doesn't necessarily mean they're right. Do you think software running these simulations 10,000 times is more or less reliable than 10,000 analysts running simulations?

    ReplyDelete
    Replies
    1. Jillian - personally I believe the answer to this comes down to how messy the data is. I think there are some layers to intelligence analysis that requires an actual analyst and not a simulation

      Delete
  4. Your blog post definitely displays the value of MC sims. You mention how it is not holistic and all of the uncertainty models rely on each other. Where would you say MC falls short?

    Total after thought but I cant help but think about cruising around in a 1986 Monte Carlo whenever I hear about this methodology. What a Car!

    ReplyDelete
    Replies
    1. I think its significant to be able to quantify a portion of your estimate. In terms of limitations, I just think, as I said above to Jillian, there are some layers to intelligence analysis that requires an actual analyst and not a simulation.

      Yes its a nice car.

      Delete
  5. I find it interesting how in this circumstance the authors recognize the need for more analysis after running multiple forms of analysis, including the Monte Carlo Simulation. It seems in the context of energy policy and national security would be a complex topic where intersecting objectives create a situation where the net benefits of the policy must exceed the net negatives of the policy. Based on your understanding of the analysis above, do you think that MC simulations would provide a "better" result that would provide more definitive analysis on a simpler or less complex issue than the one ascribed above?

    ReplyDelete