In this article
authors Hall and Titterington test the effectiveness of Monte Carlo Tests
against the asymptotic tests. The authors begin by defining their chief
question as to whether or not the Monte Carlo testing method increasing
statistical accuracy. The authors stated that they believed from the beginning
that because of the nature of the Monte Carlo testing, the method would
logically increase the accuracy of such tests.

The authors describe
the nature of Monte Carlo testing and how it differs from asymptotic
testing. They also discuss the history
of the testing method and its base theories.
Their descriptions provide a well-defined basis of understanding for the
readers to work from. Hall and
Titterington show the basic mathematical formula that Monte Carlo tests are
built from and explain the equations step by step.

Deeper issues are
then explained with Monte Carlo tests such as the issues of 'pivotalness'. Meaning that the accuracy of the experiment
can actually be effected by the number of experiments that are run. If this is not the case with a specific
experiment being run then the results of the testing would mathematically prove
to be no more accurate than asymptotic testing.
However, it is also explained that the methodology maintains its
accuracy even with a smaller number of samples because of the way in which
tests are run.

In order to test the
effectiveness of the models, the authors ran test two different experiments
using both models and compared the predictions to the actual results and to
each other. The authors found that Monte
Carlo tests proved to maintain their accuracy even with limited sample sizes.

Critique:

While the authors
when into great detail explaining the arithmetic and the logic behind Monte
Carlo testing, there is a lot more that could have been done to explain their
experiments to test the theory. The
authors were vague on how the models were being applied in order to test their
accuracy and so it diminishes the generalizability and verifiability of the
experiment run.

Hall, P., &
Titterington, D. M. (1989). The effect of simulation order on level accuracy
and power of Monte Carlo tests. Journal of the
Royal Statistical Society. Series B (Methodological), 51(3), 459–467.

Did the authors explain the main differences between Monte Carlo testing and asymptotic testing?

ReplyDeleteSort of, they broke down some of the fundamentals of both but only truly defined and explained Monte Carlo tests.

DeleteSam, in the authors' explanation of the formula for Monte Carlo methods, did they mention the different types of distributions that are used to model data? Using the correct type of distribution is incredibly important to receiving valid probabilities.

ReplyDeletethe authors said that the data should be normally distributed.

DeleteWould you say that a MC simulation is an effective tool for forecasting based off this article?

ReplyDeleteI would say that it is great for many scenarios but not necessarily all. The beginning of the article sets the optimal conditions for such testing.

Delete