Summary of Findings: Prediction Markets (3.5 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2017 regarding Prediction Markets as an Analytic Technique Method specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Prediction markets aggregate collective knowledge of the crowd through the use of a trading based system, allowing participants to invest a stake in a desired outcome they believe will come true. Prediction Markets, when conducted correctly, can provide a range of reasonably accurate forecasts and provide strong reliability to the answered question.

Strengths:

• Can be more efficient than some bureaucratic processes
• Able to aggregate disparate pieces of information to accurately predict resolvable questions
• The incentive to gain profits largely eliminates the occurrence of groupthink
• Works well even when people have limited knowledge about their surrounding environment and the people with whom they transact
• Can incorporate insight from experts across many different fields
• Experts not required

Weaknesses:

• No system for assessing the difficulty of the question
• Doesn’t work well with open ended questions
• Prone to question interpretation gap failures - difficult to determine if question answered actually satisfies intelligence requirement
• Can be manipulated if a speculative trade influences the beliefs of other traders, whether by playing by the rules or creating ideas that cheat the market
• Some moderate level of expertise is required to be a forecaster
• Long term estimates are at risk of forecaster apathy
• Requires a large volume of analysts to create the number of estimates needed

How-To:

1. Choose a resolvable question
2. Gather participants that have a general knowledge of the issue
3. Stipulate incentives
4. Have participants wager on the likely answer, candidate, or outcome
5. Calculate market valuations
6. Evaluate results
7. Inform participants

Application of Technique:

The class received handouts with a list of NFL Football Teams and their standings in 2002 along with $100. They were asked to bet money on which teams they thought would win the SuperBowl in 2003 based on their final standings. After looking at the statistics, students would place bets on the top 3 teams they thought would win. After the bets were placed, we looked at the teams with the most bets and calculated the results. Without a binary choice, the class was capable of making the accurate prediction that the New England Patriots would win the 2003 Super Bowl.

For Further Information:

1. Predictit
2. How Prediction Markets Works
3. Robin Hanson - Idea Futures
4. Philip Tetlock

Using Prediction Markets to Forecast Research Evaluations

By Sam Farnan

Summary

Researchers utilized prediction markets to determine its feasibility with predicting the accuracy of research evaluations. Specifically, they sought to explore if a prediction market model would produce similar results to the Research Excellence Framework of 2014 (REF2014) The REF2014 was six-year process of evaluating research quality in educations in the United Kingdom. REF2014 came under criticism due to the lengthy, costly, and complex way it was conducted.

Researchers in the UK hypothesized whether a prediction market would offer the same results with much less bureaucracy than the REF2014 brought upon academic institutions. For a sample size, they examined 33 chemistry departments within the UK's higher education system. A total of 16 participants aided in the study, and ultimately concluded that in this case, the prediction market actually had less errors overall and showed similar results to the REF2014 as it related to the selected chemistry departments. There were still a number of errors, specifically with regards to institutions sacrificing research quality and ranking to gain research income. 

Critique

I feel the number of participants that the researchers utilized was far too small to implicate that prediction market models could replicate the imperfect, yet expansive, REF2014. Additionally, prediction markets may not account for the more detailed aspects of large evaluations similar to this one as shown in the study above. Although this study shows potential in utilizing prediction markets in this case and had a solid overall design, I believe much more research is needed in order to make a claim that prediction markets are able to reliably replicate results from a large evaluation such as REF2014. 

Source: http://rsos.royalsocietypublishing.org/content/royopensci/2/10/150287.full.pdf

Manipulation in Prediction Markets - Chasing the Fraudsters

Summary by Keith Robinson Jr.

Summary: 

In this article, the researchers address the issues of manipulation and fraud in prediction markets and examine fraud detection approaches. While the authors acknowledge the versatility and forecasting accuracy of prediction markets in comparison to polls or even statistical models, they are not without issues. First, the researchers concluded that to some extent, prediction markets can be manipulated (manipulation defined as a speculative attack that achieves its objective of changing prices). Researchers have come to mixed conclusions regarding manipulation in prediction markets. While anecdotal evidence has shown that the manipulation affects the information aggregation aspect, not reducing predictive accuracy of forecast, other evidence has revealed that "manipulators highly incited for inaccurate predictions, can diminish the predictive power of the markets down to a level that is no better than random guessing" (p. 2981).

Second, the literature looks at fraud in prediction markets. Frequently, extant literature on prediction markets consider traders playing by the rules; however, traders may play by the rules and create other ideas how to cheat the market. They focus on the visualization of the Fraud Cube (see Figure 1.), a framework to understand and uncover where a prediction market may be manipulated or cheated. In order fraud to occur in prediction markets three dimensions must occur: 1) desire/objective (whether to disrupt the market, self-enrichment, or both, 2) temporal horizon (short-term or in the long run - realize quick profits or destroy market prediction or vested interest in the outcome or decisions derived), and 3) source of incentive (the incentive is caused by an inner incentive scheme inside the market or externally).

Figure 1. Framework to understand and uncover where a prediction market may be manipulated or cheated. 

Next, existing fraud detection and trading patterns recognized by Blume et al. (2010) are highlighted. Prominent detection strategies revolve around "ping-pong indicators," focusing on transfer or money, and prominent-edge indicators," taking a look at stocks. Realizing Blume's indicator shortcomings, the researchers developed a simple algorithm, easily applicable for practitioners. This algorithm utilizes a scoring system, scoring traders with "suspicious points" whereas the top ranks have the highest probability to commit prediction market fraud. The researchers evaluate the algorithm during a 12 week data collection encompassing 2,111 participants conducting 112,386 transactions. The algorithm was able to find 484 suspects, 6 traders having more than 200 suspicious points with 551 points being the highest yield.

Critique:

The authors utilized existing literature to simplify a method for application by practitioners. While it is not an end all be all solution, the authors acknowledge their algorithm's own limitations. All in all, the article highlights an under-addressed issue that can damper prediction market accuracy and implications for such.



Source: Kloker, Simon & T. Kranz, Tobias. (2017). Manipulation in Prediction Markets - Chasing the Fraudsters. Retrieved from https://www.researchgate.net/publication/318563054_Manipulation_in_Prediction_Markets_-_Chasing_the_Fraudsters.

Using Prediction Markets to Enhance US Intelligence Capabilities - Puong Fei Yeh

Summary and Critique by Evan Garfield

Summary

 Prediction markets allow participants to stake bets on the likelihood of various events taking place. This method involves the trading of contracts tied to future outcomes.The market price is essentially an estimate of the probability of a future outcome occurring. These market prices reflect the  participants collective confidence that an outcome will occur. In this article the author discusses the value in using prediction markets for the intelligence community. Shes emphasizes that prediction markets are reliable aggregate measures of disparate and dispersed information and can result in forecasts that are more accurate than those of experts. 

According to the author, use of prediction markets in the intelligence community can be traced back to 2001 within the Defense Advanced Research Project Agency (DARPA). DARPA’s Future Markets Applied to Prediction (FutureMAP) program tested the effectiveness of prediction markets in forecasting future events. Under the FutureMap program, the Policy Analysis Market (PAM) offered trading on a variety of different contracts (political, economic, military indicators, etc). The program, however, was very brief and terminated in 2003.

The author continues to discuss the Hayek hypothesis that market prices are the means in which disparate pieces of information are aggregated. According to Hayek, 

"The mere fact that there is one price for any commodity…brings about the solution which…might have been arrived at by one single mind possessing all the information which is, in fact, dispersed among all the people involved in the process"

Going off this point, the author stresses that prediction markets works even when people have limited knowledge about their surrounding environment and the people with whom they transact. She then applies this utility to address needs of the intelligence community. She cites the 9/11 Commission, and its conclusion that lack of unity of effort in information sharing as the biggest impediment to all-source analysis. 

The author then discusses some concerns with prediction markets including market design issues and market manipulation and bias. There are questions with regards to number of traders in a market. Furthermore, Analysts might engage in trading behavior to fit a certain policy outcome. Behavioral bias may also occur when traders trade according to the outcomes they desire rather than a dispassionate assessment of what is likely. She states, "An analogy is that in the run-up to the Iraq war, intelligence analysts were so convinced that Iraq had reconstituted their WMD programs that any evidence, regardless of its veracity, only served to harden their earlier convictions".

Critique

The author does a good job discussing the utility is using prediction markets for strategic intelligence as well as potential concerns when using prediction markets. However, more research is needed with regards to  as whether real-money markets produce better accuracy than play-money markets. Furthermore, more study is needed on whether prediction markets can be applied for tactical analysis. The author also does a good job stressing the utility of prediction markets in promoting collaborative intelligence and enhanced forecasting. The nature of this methodology also allows for non-subject matter experts to contribute without significantly hurting forecasting accuracy. This approach allows for broad contribution across the entire intelligence community and helps reduce the risk of group think. Given the compartmentalization of the intelligence community, prediction markets allow for invaluable aggregation of analyst input for complex issues.


Source: https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol50no4/using-prediction-markets-to-enhance-us-intelligence-capabilities.html

Use of Prediction Markets to Forecast Infectious Disease Activity

Summary and critique by: Ian Abplanalp

Summary:

In this article researchers took the methodology of prediction markets also known as information markets, or future markets, and applied it to the medical field. The purposed study purposed that a group of medical experts from various field who were privy to different bodies of information could aggregate and accurately forecast what FLU strains would be prevalent in the upcoming year. 

Prediction markets work by having "traders" invest money in outcomes that they believe will come true. As time progresses and more traders invest in differing options, the buy in prices for probable outcomes shift to become more expensive and less expensive to invest in low probability outcomes. These market prices reflect the traders collective confidence that an outcome will happen. Due to the potential to earn money for a correct selection the participant is encouraged to invest in something that will yield a payoff. 

As the more people invest into the market that as time progresses that market will come closer and closer to mirroring what will actually occur in the future. Prediction markets have a very high forecasting accuracy. For example if eighty-five percent of investors suggest this will happen, then that has an eighty-five percent chance of happen. This mimics the principle that the more one flips a coin over time the closer it will be to fifty percent heads and fifty percent tails. 

Three things are highlighted that are key to making a prediction market valid as a methodology. The first is that a prediction market needs a diverse group of traders, to pull information from a large amount of sources. Sufficient amount of traders is also required as there must be enough to encourage market value changes but not so many that there is a large amount of error. The correct number of traders is still up for debate as research has yielded a wide array of results with different amount of traders and there has been no consensus of the issue. The third criteria a prediction market must have is that there must be incentives to trade within the market or it would essentially be a betting pool and not a market. 

When in application 62 medical professionals from various subfields in medicine participated in the market. The market introduced new contracts, which are the equivalent to shares, into the market every two weeks. As the market went on the predictions where more accurate throughout,  both with correctly forecasting the strain color, but also within one color variation (See Figure 1).  The researchers concluded that even though there was some wiggle room within the results that given the volatility of the FLU market it was an overall success. 

Critique:

Prediction markets have inherent strengths of having a malleable method throughout the course of an experiment to accurately represent what is likely to happen in the future. Prediction markets are versatile, as they can be applied to many different fields. They also trump surveys as they allow for informed decisions throughout the entire process by being allowing trade through the process, as opposed to a survey which is filled out once. This allows for confidence in an event, to rise and fall appropriately leading to an overall better forecasting agent. The downfall to prediction markets are they require a great deal of time and effort to set up as well as run. They also require people who care about an outcome to invest into a market with money. This investment raises an ethical question in some fields were it may seem inappropriate to be investing physical money such as medicine, or college sport outcomes. The last pitfall prediction markets have is that they can be swayed by manipulators who are able to purchase large portions of a market if said big buyer wanted to suppress an unpopular opinion to them.

Sources: Use of Prediction Markets to Forecast Infectious Disease Activity

  

Have Corporate Prediction Markets Had Their Heyday?

Summary and critique by Kevin Muvunyi

Summary

Thomas Wolfram in his article “Have Corporate Prediction Markets Had Their Heyday”, examines the reasons behind the relatively low adoption of prediction markets by large business entities as a forecasting tool, and then proceeds to provide possible avenues to revive this important prediction technique. To achieve the purpose of his research, the author reviewed existing literature on the subject and conducted interviews with 32 key business executives.

 In his attempt to understand the contradiction between the strong academic support for prediction markets and its slow uptake by businesses, Wolframm first examines the rationale behind the popularity of this methodology in the world of academia. According to the author, multiple research studies have proven that prediction markets are more efficient in decision making and forecasting, because they eliminate the problematic caused by bias and group pressure in traditional decision making settings by allowing the participants to make their decision anonymously. Furthermore, the technique incentivizes individuals through the promise of possible remuneration to bring forth new information, which can then be aggregated to predict the outcome of a future event, thus, making them more efficient in comparison to long-standing forecasting instruments like questionnaires, surveys, and polls according to the author. 

After examining the merits of the methodology, Wolframm then moves to provide possible explanations to the current withering status of prediction markets in the business world, primarily basing his assertions on interviews conducted with key corporate decision makers. The author summarizes the explanations into three key points as follows:

• Finding appropriate and knowledgeable experts (traders) is complicated; it does not help if participants are diverse but ignorant of the issue as it undermines their predictions

• Lack of trust from the top echelons of management: management trusts consulting groups more than own employees, and also believes that prediction markets disturb the concept of hierarchy in an organization due to the fact that their ideas have a chance of being rejected based on this model, thus, undermining their authority.

•  Businesses “going digital” in a different direction from prediction markets; as businesses turn digital prediction markets become obsolete, because everything becomes data driven, therefore, big data analysis and other similar techniques are more insightful and appropriate.

In the light of apparent insurmountable obstacles to the revival of prediction markets in corporate circles, Wolframm advocates that they are possible ways to remediate this issue. For example, the author suggests that to ensure knowledgeable traders, a convenient graphical user interface and communication exercises accompanying prediction market implementations would be more appropriate. Furthermore, Wolframm brings forth the notion of idea markets as an innovative approach to the forecasting tool. In contrast to traditional prediction market whereby participants are allowed to trade on the outcome of uncertain future events, idea markets could provide a platform for the generation and assessment of ideas through the trading of virtual stocks representing products and concepts.

Critique:

The author does a great job at analyzing the root cause of disaffection of prediction markets as a forecasting technique in the corporate world. He also equally provides sound alternatives to revive the technique. Nonetheless, the only problem with his article is that it fails to demonstrate in a tangible manner, the superiority of improved prediction markets in comparison to big data analytical tools in the new data driven world that we live in today.

source: http://eds.a.ebscohost.com.ezproxy.mercyhurst.edu/ehost/pdfviewer/pdfviewer?vid=10&sid=4bed5a56-fc91-4969-aac9-62fa90b8808f%40sessionmgr4008   

The Power of Prediction Markets

Prediction markets can be uncannily accurate — sometimes. Scientists have begun to understand why they work, and how they can fail.

Summary and Critique by Oddinigwe Onyemenem

Summary

In 2012, a group of international psychologists embarked on a project dubbed the “Reproducibility Project” in an effort to repeat dozens of psychology experiments to see which held up. One of the participants, Ann Dreber, who leads a team of behavioral economists at the Stockholm School of Economics, viewed it as an avenue to mix science with gambling and thought it would be fantastic to bet on the outcome. The team was specifically interested to see whether scientists could make good use of prediction markets: mini Wall Streets in which participants buy and sell ‘shares’ in a future event at a price that reflects their collective wisdom about the chance of the event happening. As a control, Dreber and her colleagues first asked a group of psychologists to estimate the odds of replication for each study on the project’s list. Then the researchers set up a prediction market for each study, and gave the same psychologists USD 100 apiece to invest. In 2015, the project had replicated fewer than half of the studies examined and Dreber found that her experts hadn’t done much better than chance with their individual predictions. But working collectively through the markets, they had correctly guessed the outcome 71% of the time. According to Mann, experiments of this nature depict the power of prediction markets to turn individuals’ guesses into forecasts of sometimes startling accuracy.

Mann points out that prediction markets are increasingly being used to make various kinds of forecasts such as the outcomes of sporting events and business decisions. Prediction markets advocates claim that it allows people to aggregate information without the biases that affect traditional forecasting methods such as polls or expert analysis. The application in science was shown by Dreber and her team by giving researchers a fast and low-cost way to identify potential problems with replicated studies. On the other hand, skeptics point out that prediction markets are far from being perfect. This is due to an incorrect notion that a great prediction is almost, always guaranteed when a market is set up. According to Eric Zitzewitz, an economist at Dartmouth College, it is an area of active research to determine the best designs for prediction markets and the limitations. Nevertheless, advocates of prediction markets argue that even imperfect forecasts can be beneficial. For instance, hearing there’s an 80 or 90% chance of rain can make an individual take an umbrella.

The prediction-market idea was revived by the spread of the Internet, which dramatically lowered the entry barriers for creating and participating in prediction markets. In 1988, the University of Iowa’s Tippie College of Business launched the not-for-profit Iowa Electronics Market (IEM) as a network- based teaching and research tool. Over the years, IEM has set up several markets to predict election outcomes which a 2008 study found that the its predictions across five presidential elections were more accurate than the polls 74% of the time. The success of the IEM helped to inspire the creation of dozens of other prediction markets.

The article also addresses the fact that prediction markets have also missed the mark by a long shot in several cases. In the Brexit case, the prediction market gave the odds of a stay vote as 85% on the day of the referendum, whereas the outcome was narrowly in favor to leave. Also, prediction markets were off the mark in predicting the outcome of the 2016 US presidential election, which elected Donald Trump instead of the highly-favored to win, Hillary Clinton. These sorts of instances have caused academics to probe prediction markets about why they work so well, their limits and reasons for failures? Mann points out that if prediction markets offer a way to update guesses considering new information, they will do as well or better than other forecasting methods. Prediction markets in general still need to deal with challenges such as how to limit manipulation and overcome biases.

Critique

As rightly stated in the article, prediction markets are used in a wide array of markets such as sports, politics, movie, business, technology, etc. While some fully embrace it, others remain highly skeptical due to the possibility of manipulation or infusion of biases.  Research into improving it still needs to be done to ensure, as best as possible, the integrity of the process. In the next decade, prediction markets can be a major driving force behind significantly improving decision-making as more tools are implemented.

Source:

https://www.nature.com/polopoly_fs/1.20820!/menu/main/topColumns/topLeftColumn/pdf/538308a.pdf

Summary of Findings: Wargaming (3.75 out of 5 Stars)

Note: This post represents the synthesis of the thoughts, procedures and experiences of others as represented in the articles read in advance (see previous posts) and the discussion among the students and instructor during the Advanced Analytic Techniques class at Mercyhurst University in October 2017 regarding Wargaming as an Analytic Technique specifically. This technique was evaluated based on its overall validity, simplicity, flexibility and its ability to effectively use unstructured data.

Description:

Wargaming, when done correctly, can provide a range of reasonably accurate forecasts and , in these conditions, is a best considered an analytic method. When the simulation is not validated or has been simplified for the purposes of playability, wargaming is best thought of as a analytic modifier designed to teach the basic elements of a conflict without actually being used to predict outcomes.  It can and has been used in military, law enforcement, and business. It is used by the military to recreate historic battles or simulate future conflicts in order to prepare for all possible contingencies before an actual war begins. Similarly, teams within businesses play as their company and the competition too. Teams are made to reflect all the other competitors that affect a given industry to simulate a number of possible outcomes. This can take into account numerous aspect of business such as strategy, marketing, and finance.

Strengths:

• When executed correctly, Wargaming can simulate realistic conditions
• Its structure allows for replicability of the simulation
• Can create insight in possible avenues of attack/approach not previously considered
• Able to identify unforeseen obstacles in current strategy, procedures, or tactics
• Gives alternative perspectives for evaluating scenarios
• Provides accurate, holistic depiction of active, dynamic competition

Weaknesses:

• Requires extensive lead-time to prepare (game design, playtesting)
• Difficult to balance realism vs playability
• Realism is often difficult to achieve
• Susceptible to over-confidence or systematic bias
• Often complex, costly, and time-consuming
• Validity can be an issue; bad information can give false hope
• Can’t plan for unexpected variables/events

How-To:

1. Choose a situation or environment for which to simulate in a Wargame
2. Design the simulation to encapsulate all possible factors within the environment
3. Run the simulation until complete
4. Compare and analyze the results toward reality
5. Re-run the simulation to increase confidence level in results
6. Reevaluate factors and re-run as necessary

Application of Technique:

The class received handouts of grid paper that had obstacles courses outlined on them. The class was then divided in half so that each person had a partner to participate in the activity against. The class then was instructed on the rules of how they were to move through the obstacle course, with the objective to make it through the course as quickly as possible and ahead of their opponent. The rules were that each person was assigned an identifier (Shape on the grid as a game piece) that they were able to move either horizontally or vertically in a given turn. Each player was able to accelerate their piece by one block per turn so they could gain speed, the same principle was applied to deceleration. If a player hit an object or a barrier in the course the player's momentum would drop to zero. The same principle applied to if players pieces were to come in contact with each both players momentum would drop to zero.

For Further Information:

1. Wargaming - Wiki
2. Business War Games - Wiki
3. Pathologies of Wargaming

Why Wargaming Works

Summary:
by Matthew Haines

       Dr. Perla and Dr. McGrady outline what wargaming is and how it can be both successful and unsuccessful in its attempt to inform and instruct its players. They both agree that wargaming has a significant impact on its players’ decisions outside of the simulation and they propose a combination of reasons for this. They begin by comparing a games narrative to that of literature and the effect it has on a reader. That while reading prose the reader builds an imaginary space, that encompasses the work of fiction, but is perceived as real in the moment. This literary term is called the l’entre deux, or the “between place”. This phenomenon is what ties the emotional response a reader has towards problems that a fictional narrative is proposing. The author’s use President Clinton’s fear of the repercussions of biological terrorism.
Dr. Perla and Dr. McGrady then outline the neuroscience behind this idea. That when reading a fictional work that applies suspense and emotion to otherwise historical facts, a person must pause and remember what is factual and what is not. They site a study done on participants who were given a factual “cut and dry” recounting of President George Washington’s campaign, and a narrative that painted a scenario that the race was down to the wire. After the study participants were asked if George Washington became the first president of the United States, and the participants that read the “cut and dry” piece answered significantly quicker than those that read the other piece. The authors state that this place between is heightened even further in wargaming.
They state that wargaming does not only create a narrative designed to build emotion and suspense, but it is also influenced by the players’ actions. Therefore, wargaming is the closest place that a person can get to the “between space”, thus intensifying the effects it has on the players. The author’s state that these effects can be both good or bad depending on the design of the game. A well designed game can be used to
       
        help players learn how better to balance the equation between the cost of preparing for the
        uncertain future and the risk of not doing so; can help enlighten players about the fact that
        unexpected and unpredictable events, including embarrassing ones, do happen and that there are
        real consequences when they do.


However, a poorly designed game can under or over estimate the effects variables have on an outcome, and create a false sense of reality for the player. Wargaming is also hindered by its inability to account for unknown unknowns. This can make it extremely ineffective when dealing with problems that are outside the scope of a game designer’s cognitive biases.

Critique:

        What this article shows is that if wargaming can even be described as an analytical/forecasting tool it is a very poor and dangerous one. The authors were able to show that wargaming and simulation are one of the best ways to make the problems simulated a top priority for participants. This is just another way of creating a bias. To use these simulations, as described above, for intelligence purposes is a bad idea. More biases do not allow for best possible intelligence products. That said the article does highlight some good points for using wargaming as a teaching tool. If wargaming is used in conjunction with brainstorming processes, decision trees, and unique factors then it could be used in a more effective way.

Source:

http://eds.b.ebscohost.com/eds/pdfviewer/pdfviewer?vid=2&sid=0cb67813-c777-4daa-a98f-8b2af3c1a712%40sessionmgr101 

Wargaming: Training, Educational Tool for the Future

Colonel Thomas M. Lafleur examines the use of wargaming as a strategic  tool. One of the first questions he posed was the ability to transfer results of a war game conducted in earlier years to a later time period. Later in the study, he found elements of maneuvers in war games can be transferred to any strategy in later periods, but if the gaps between time periods is large enough, the maneuvers themselves cannot be due to changes in circumstances.

To ensure a complete transference of a war game to strategy, Colonel Lafleur presents three things to be done: a nuanced scenario must be in place. The nuances mentioned are a realistic scenario with leaders who know the scenario extremely well. Second, the participants in the war game must have specific and detailed knowledge of how to proceed in the war game as well as skill sets relating to the war game. Third, examine the qualities developed out of the scenario as possible solutions for future problems.

Critique:
Colonel Lafleur does a good job examining the ability to transfer strategies developed in war games to actual military strategies. He goes into detail on what is needed for an effective war game.

Citation: Lafleur, Thomas M. http://www.arcic.army.mil/App_Documents/UQ/Wargaming.pdf.