Saturday, October 29, 2016

The Effect of Simulation Order on Level Accuracy and Power of Monte Carlo Tests




In this article authors Hall and Titterington test the effectiveness of Monte Carlo Tests against the asymptotic tests. The authors begin by defining their chief question as to whether or not the Monte Carlo testing method increasing statistical accuracy. The authors stated that they believed from the beginning that because of the nature of the Monte Carlo testing, the method would logically increase the accuracy of such tests.

The authors describe the nature of Monte Carlo testing and how it differs from asymptotic testing.  They also discuss the history of the testing method and its base theories.  Their descriptions provide a well-defined basis of understanding for the readers to work from.  Hall and Titterington show the basic mathematical formula that Monte Carlo tests are built from and explain the equations step by step.

Deeper issues are then explained with Monte Carlo tests such as the issues of 'pivotalness'.  Meaning that the accuracy of the experiment can actually be effected by the number of experiments that are run.  If this is not the case with a specific experiment being run then the results of the testing would mathematically prove to be no more accurate than asymptotic testing.  However, it is also explained that the methodology maintains its accuracy even with a smaller number of samples because of the way in which tests are run.

In order to test the effectiveness of the models, the authors ran test two different experiments using both models and compared the predictions to the actual results and to each other.  The authors found that Monte Carlo tests proved to maintain their accuracy even with limited sample sizes.

Critique:

While the authors when into great detail explaining the arithmetic and the logic behind Monte Carlo testing, there is a lot more that could have been done to explain their experiments to test the theory.  The authors were vague on how the models were being applied in order to test their accuracy and so it diminishes the generalizability and verifiability of the experiment run.

Hall, P., & Titterington, D. M. (1989). The effect of simulation order on level accuracy and power of Monte Carlo tests. Journal of the Royal Statistical Society. Series B (Methodological), 51(3), 459–467.

6 comments:

  1. Did the authors explain the main differences between Monte Carlo testing and asymptotic testing?

    ReplyDelete
    Replies
    1. Sort of, they broke down some of the fundamentals of both but only truly defined and explained Monte Carlo tests.

      Delete
  2. Sam, in the authors' explanation of the formula for Monte Carlo methods, did they mention the different types of distributions that are used to model data? Using the correct type of distribution is incredibly important to receiving valid probabilities.

    ReplyDelete
    Replies
    1. the authors said that the data should be normally distributed.

      Delete
  3. Would you say that a MC simulation is an effective tool for forecasting based off this article?

    ReplyDelete
    Replies
    1. I would say that it is great for many scenarios but not necessarily all. The beginning of the article sets the optimal conditions for such testing.

      Delete