The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Using simulation studies to evaluate statistical methods
Open accessAbstract
Simulation studies are computer experiments that involve creating data by pseudo‐random sampling. A key strength of simulation studies is the ability to understand the behavior of statistical methods because some “truth” (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analyzed, and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. In particular, this tutorial provides a structured approach for planning and reporting simulation studies, which involves defining aims, data‐generating mechanisms, estimands, methods, and performance measures (“ADEMP”); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine , which included at least one simulation study and identify areas for improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · cited 1×
“The sample estimate V̂ar(̂) should be a consistent estimate of the sampling variance of ̂ (see for example (18)).”
BackgroundMultiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072
Ripley, B. D. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316726 · Cited by 2,241
Information and the Accuracy Attainable in the Estimation of Statistical Parameters
Rao, C. R. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_16 · Cited by 1,808
Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician
Rubin, D. B. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346785 · Cited by 1,343
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086
Imputing missing covariate values for the Cox model
White, I. R., & Royston, P. (2009) · Statistics in Medicine · doi:10.1002/sim.3618 · Cited by 985
Self-citationComputer Age Statistical Inference: Algorithms, Evidence, and Data Science
Efron, B., & Hastie, T. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316576533 · Cited by 902
Multiple-Imputation Inferences with Uncongenial Sources of Input
Meng, X.-L. (1994) · Statistical Science · doi:10.1214/ss/1177010269 · Cited by 756
The design of simulation studies in medical statistics
Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2673 · Cited by 752
White, I. R., & Carlin, J. B. (2010) · Statistics in Medicine · doi:10.1002/sim.3944 · Cited by 744
Self-citationNo rationale for 1 variable per 10 events criterion for binary logistic regression analysis
van Smeden, M., de Groot, J. A. H., Moons, K. G. M., Collins, G. S., Altman, D. G., Eijkemans, M. J. C., & Reitsma, J. B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0267-3 · Cited by 595
Tuning multiple imputation by predictive mean matching and local residual draws
Morris, T. P., White, I. R., & Royston, P. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-75 · Cited by 568
Self-citationAdjusting for partially missing baseline measurements in randomized trials
White, I. R., & Thompson, S. G. (2005) · Statistics in Medicine · doi:10.1002/sim.1981 · Cited by 378
Self-citationOn the Assessment of Monte Carlo Error in Simulation-Based Statistical Analyses
Koehler, E., Brown, E., & Haneuse, S. J.-P. A. (2009) · The American Statistician · doi:10.1198/tast.2009.0030 · Cited by 303
Inference for imputation estimators
Robins, J. (2000) · Biometrika · doi:10.1093/biomet/87.1.113 · Cited by 276
Should We Adjust for Covariates in Nonlinear Regression Analyses of Randomized Trials?
Hauck, W. W., Anderson, S., & Marcus, S. M. (1998) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00147-5 · Cited by 218
Kuss, O. (2015) · Statistics in Medicine · doi:10.1002/sim.6383 · Cited by 206
Li, P., & Redden, D. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6344 · Cited by 187
Seaman, S. R., Bartlett, J. W., & White, I. R. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-46 · Cited by 177
Self-citationVariance reduction in randomised trials by inverse probability weighting using the propensity score
Williamson, E. J., Forbes, A., & White, I. R. (2014) · Statistics in Medicine · doi:10.1002/sim.5991 · Cited by 172
Self-citationMultiple Imputation for Multilevel Data with Continuous and Binary Variables
Audigier, V., White, I. R., Jolani, S., Debray, T. P. A., Quartagno, M., Carpenter, J., van Buuren, S., & Resche-Rigon, M. (2018) · Statistical Science · doi:10.1214/18-sts646 · Cited by 166
Self-citationSimulating biologically plausible complex survival data
Crowther, M. J., & Lambert, P. C. (2013) · Statistics in Medicine · doi:10.1002/sim.5823 · Cited by 164
Self-citationLangan, D., Higgins, J. P. T., & Simmonds, M. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1198 · Cited by 149
Optimal full matching for survival outcomes: a method that merits more widespread use
Austin, P. C., & Stuart, E. A. (2015) · Statistics in Medicine · doi:10.1002/sim.6602 · Cited by 116
Crowther, M. J., Look, M. P., & Riley, R. D. (2014) · Statistics in Medicine · doi:10.1002/sim.6191 · Cited by 95
Self-citationThe Reporting of Computation-Based Results in Statistics
Hoaglin, D. C., & Andrews, D. F. (1975) · The American Statistician · doi:10.1080/00031305.1975.10477393 · Cited by 90
Comparison of different approaches to estimating age standardized net survival
Lambert, P. C., Dickman, P. W., & Rutherford, M. J. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0057-3 · Cited by 85
Boulesteix, A., Wilson, R., & Hapfelmeier, A. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0417-2 · Cited by 85
Simsum: Analyses of Simulation Studies Including Monte Carlo Error
White, I. R. (2010) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1001000305 · Cited by 82
Self-citationDesign and Analysis of Monte Carlo Experiments: Attacking the Conventional Wisdom
Skrondal, A. (2000) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3502_1 · Cited by 82
Feiveson, A. H. (2002) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0200200201 · Cited by 74
Presenting simulation results in a nested loop plot
Rücker, G., & Schwarzer, G. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-129 · Cited by 74
Royston, P. J., Barthel, F. M.-S., Parmar, M., Choodari‐Oskooei, B., & Isham, V. S. (2011) · Trials · doi:10.1186/1745-6215-12-81 · Cited by 66
Bias and inference from misspecified mixed‐effect models in stepped wedge trial analysis
Thompson, J. A., Fielding, K., Davey, C., Aiken, A. M., Hargreaves, J., & Hayes, R. (2017) · Statistics in Medicine · doi:10.1002/sim.7348 · Cited by 66
Bias in randomised factorial trials
Kahan, B. C. (2013) · Statistics in Medicine · doi:10.1002/sim.5869 · Cited by 56
Senn, S. J., Chambless, L. E., & Roeback, J. R. (1994) · Statistics in Medicine · doi:10.1002/sim.4780132110 · Cited by 47
Multiple imputation in Cox regression when there are time‐varying effects of covariates
Keogh, R. H., & Morris, T. P. (2018) · Statistics in Medicine · doi:10.1002/sim.7842 · Cited by 44
Self-citationGutman, R., & Rubin, D. B. (2012) · Statistics in Medicine · doi:10.1002/sim.5627 · Cited by 43
Meta‐analysis of Gaussian individual patient data: Two‐stage or not two‐stage?
Morris, T. P., Fisher, D. J., Kenward, M. G., & Carpenter, J. R. (2018) · Statistics in Medicine · doi:10.1002/sim.7589 · Cited by 41
Self-citationA Survey regarding the Reporting of Simulation Studies
Hauck, W. W., & Anderson, S. (1984) · The American Statistician · doi:10.1080/00031305.1984.10483206 · Cited by 40
Comparison of imputation variance estimators
Hughes, R. A., Sterne, J. A. C., & Tilling, K. M. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214526216 · Cited by 38
Idle thoughts of a ‘well‐calibrated’ Bayesian in clinical drug development
Grieve, A. P. (2016) · Pharmaceutical Statistics · doi:10.1002/pst.1736 · Cited by 37
An alternative pseudolikelihood method for multivariate random-effects meta-analysis
Chen, Y., Hong, C., & Riley, R. D. (2014) · Statistics in Medicine · doi:10.1002/sim.6350 · Cited by 35
A penalized robust semiparametric approach for gene–environment interactions
Wu, C., Shi, X., Cui, Y., & Ma, S. (2015) · Statistics in Medicine · doi:10.1002/sim.6609 · Cited by 35
Estimating a Marginal Causal Odds Ratio Subject to Confounding
Zhang, Z. (2008) · Communication in Statistics- Theory and Methods · doi:10.1080/03610920802200076 · Cited by 30
Pham, T. M., Carpenter, J. R., Morris, T. P., Wood, A. M., & Petersen, I. (2018) · Statistics in Medicine · doi:10.1002/sim.8004 · Cited by 30
Self-citationEstimation after subpopulation selection in adaptive seamless trials
Kimani, P., Todd, S., & Stallard, N. (2015) · Statistics in Medicine · doi:10.1002/sim.6506 · Cited by 29
The consequences of proportional hazards based model selection
Campbell, H., & Dean, C. B. (2013) · Statistics in Medicine · doi:10.1002/sim.6021 · Cited by 22
Sayers, A. E., Crowther, M. J., Judge, A., Whitehouse, M. R., & Blom, A. W. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015397 · Cited by 22
Self-citationProposed best practice for projects that involve modelling and simulation
O’Kelly, M., Anisimov, V. V., Campbell, C., & Hamilton, S. (2016) · Pharmaceutical Statistics · doi:10.1002/pst.1789 · Cited by 17
Taguri, M., & Chiba, Y. (2014) · Statistics in Medicine · doi:10.1002/sim.6329 · Cited by 15
Hsu, C., Taylor, J. M. G., & Hu, C. (2015) · Statistics in Medicine · doi:10.1002/sim.6534 · Cited by 13
He, X., Whitmore, G. A., Loo, G. Y., Hochberg, M. C., & Lee, M. T. (2014) · Statistics in Medicine · doi:10.1002/sim.6356 · Cited by 12
Chaurasia, A. K., & Harel, O. (2014) · Statistics in Medicine · doi:10.1002/sim.6334 · Cited by 12
A new modeling approach for quantifying expert opinion in the drug discovery process
Alonso, A., Milanzi, E., Molenberghs, G., Buyck, C., & Bijnens, L. (2015) · Statistics in Medicine · doi:10.1002/sim.6459 · Cited by 7
Smaller clinical trials for decision making; using p-values could be costly
Graves, N., Barnett, A. G., Burn, E., & Cook, D. (2018) · F1000Research · doi:10.12688/f1000research.15522.1 · Cited by 5
Cited by
Sanderson, E., Glymour, M. M., Holmes, M. V., Kang, H., Morrison, J., Munafò, M. R., Palmer, T., Schooling, C. M., Wallace, C., Zhao, Q., et al. (2022) · Nature Reviews Methods Primers · doi:10.1038/s43586-021-00092-5 · Cited by 2,229
Daniel, R., Zhang, J., & Farewell, D. (2021) · Biometrical Journal · doi:10.1002/bimj.201900297 · Cited by 167
Multiple imputation with missing data indicators.
Beesley, L. J., Bondarenko, I., Elliot, M. R., Kurian, A. W., Katz, S. J., & Taylor, J. M. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211047346 · Cited by 114
Sisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231165001 · Cited by 99
Ju, K., Lin, L., Chu, H., Cheng, L.-L., & Xu, C. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01035-6 · Cited by 61
Metadata
- Authors
- Morris, T. P., White, I. R., & Crowther, M. J.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2019-01-16
- Volume
- 38
- Issue
- 11
- Pages
- 2074-2102
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2775363893
- Cited by
- 1,305
- References
- 58
- FWCI
- 88.51
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
