The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
On the Assessment of Monte Carlo Error in Simulation-Based Statistical Analyses
Open accessAbstract
Statistical experiments, more commonly referred to as Monte Carlo or simulation studies, are used to study the behavior of statistical methods and measures under controlled situations. Whereas recent computing and methodological advances have permitted increased efficiency in the simulation process, known as variance reduction, such experiments remain limited by their finite nature and hence are subject to uncertainty; when a simulation is run more than once, different results are obtained. However, virtually no emphasis has been placed on reporting the uncertainty, referred to here as Monte Carlo error, associated with simulation results in the published literature, or on justifying the number of replications used. These deserve broader consideration. Here we present a series of simple and practical methods for estimating Monte Carlo error as well as determining the number of replications required to achieve a desired level of accuracy. The issues and methods are demonstrated with two simple examples, one evaluating operating characteristics of the maximum likelihood estimator for the parameters in logistic regression and the other in the context of using the bootstrap to obtain 95% confidence intervals. The results suggest that in many settings, Monte Carlo error may be more substantial than traditionally thought.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Markov Chain Monte Carlo in Practice
(1995) · Chapman & Hall/CRC eBooks · doi:10.1201/b14835 · Cited by 7,757
Metropolis, N., & Ulam, S. (1949) · Journal of the American Statistical Association · doi:10.1080/01621459.1949.10483310 · Cited by 6,474
Monte Carlo Statistical Methods
Kim, H. (2000) · Technometrics · doi:10.1080/00401706.2000.10485722 · Cited by 5,508
Ripley, B. D. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316726 · Cited by 2,241
Practical markov chain monte carlo
Geyer, C. J. (1992) · Statistical Science · doi:10.1214/ss/1177011137 · Cited by 2,045
Logistic disease incidence models and case-control studies
PRENTICE, R. L., & PYKE, R. (1979) · Biometrika · doi:10.1093/biomet/66.3.403 · Cited by 1,085
Metropolis, N., & Ulam, S. M. (1949) · Journal of the American Statistical Association · doi:10.2307/2280232 · Cited by 768
Gilks, W. R. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a14021 · Cited by 711
Hierarchical Spatio-Temporal Mapping of Disease Rates
Waller, L. A., Carlin, B. P., Xia, H., & Gelfand, A. E. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474012 · Cited by 509
van Belle, G. (2002) · Wiley-Interscience eBooks · Cited by 461
Markov Chain Monte Carlo: Can We Trust the Third Significant Figure?
Flegal, J. M., Haran, M., & Jones, G. L. (2008) · Statistical Science · doi:10.1214/08-sts257 · Cited by 280
On the Number of Bootstrap Simulations Required to Construct a Confidence Interval
Hall, P. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350169 · Cited by 242
Jackknife-After-Bootstrap Standard Errors and Influence Functions
Efron, B. (1992) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1992.tb01866.x · Cited by 222
Markov chain Monte Carlo: Can we trust the third significant figure
Flegal, J. M., Haran, M., & Jones, G. L. (2007) · Cited by 211
Elements of Computational Statistics
Gentle, J. E. (2002) · Springer eBooks · doi:10.1007/b97337 · Cited by 196
Monte Carlo Approximation of Bootstrap Variances
Booth, J. G., & Sarkar, S. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480596 · Cited by 102
A Survey regarding the Reporting of Simulation Studies
Hauck, W. W., & Anderson, S. (1984) · The American Statistician · doi:10.1080/00031305.1984.10483206 · Cited by 40
Efficiency of Monte Carlo EM and Simulated Maximum Likelihood in Two-Stage Hierarchical Models
Jank, W., & Booth, J. G. (2003) · Journal of Computational and Graphical Statistics · doi:10.1198/1061860031338 · Cited by 32
Monte Carlo error estimation for multivariate Markov chains
Kosorok, M. R. (2000) · Statistics & Probability Letters · doi:10.1016/s0167-7152(99)00090-5 · Cited by 24
Cited by
Using simulation studies to evaluate statistical methods
Morris, T. P., White, I. R., & Crowther, M. J. (2019) · Statistics in Medicine · doi:10.1002/sim.8086 · Cited by 1,305
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Brain-wide mapping of neural activity controlling zebrafish exploratory locomotion
Dunn, T. W., Mu, Y., Narayan, S., Randlett, O., Naumann, E. A., Yang, C.-T., Schier, A. F., Freeman, J., Engert, F., & Ahrens, M. B. (2016) · eLife · doi:10.7554/elife.12741 · Cited by 312
“Specifically, we defined N to be the number of simulated trajectories needed for this quantity (gathered resources) to stabilize (Koehler et al., 2009), and we considered the quantity to be stable when the local Fano factor (variance/mean over a moving window of 400 simulations) first dropped below…”
BackgroundLovibond, K., Jowett, S., Barton, P., Caulfield, M., Heneghan, C., Hobbs, F. R., Hodgkinson, J., Mant, J., Martin, U., Williams, B., et al. (2011) · The Lancet · doi:10.1016/s0140-6736(11)61184-7 · Cited by 311
Trifactor Models for Multiple-Ratings Data
Shin, H. J., Rabe-Hesketh, S., & Wilson, M. (2019) · Multivariate Behavioral Research · doi:10.1080/00273171.2018.1530091 · Cited by 52
Metadata
- Authors
- Koehler, E., Brown, E., & Haneuse, S. J.-P. A.
- Venue
- The American Statistician
- Publisher
- Informa UK Limited
- Published
- 2009-05
- Volume
- 63
- Issue
- 2
- Pages
- 155-162
- ISSN
- 0003-1305 (ISSN-L)
- 1537-2731
- OpenAlex
- W2033582774
- Cited by
- 303
- References
- 20
- FWCI
- 10.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.