The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments
Open accessAbstract
Many simulation practitioners can get more from their analyses by using the statistical theory on design of experiments (DOE) developed specifically for exploring computer models. We discuss a toolkit of designs for simulators with limited DOE expertise who want to select a design and an appropriate analysis for their experiments. Furthermore, we provide a research agenda listing problems in the design of simulation experiments—as opposed to real-world experiments—that require more investigation. We consider three types of practical problems: (1) developing a basic understanding of a particular simulation model or system, (2) finding robust decisions or policies as opposed to so-called optimal solutions, and (3) comparing the merits of various decisions or policies. Our discussion emphasizes aspects that are typical for simulation, such as having many more factors than in real-world experiments, and the sequential nature of the data collection. Because the same problem type may be addressed through different design types, we discuss quality attributes of designs, such as the ease of design construction, the flexibility for analysis, and efficiency considerations. Moreover, the selection of the design type depends on the metamodel (response surface) that the analysts tentatively assume; for example, complicated metamodels require more simulation runs. We present several procedures to validate the metamodel estimated from a specific design, and we summarize a case study illustrating several of our major themes. We conclude with a discussion of areas that merit more work to achieve the potential benefits—either via new research or incorporation into standard simulation or statistical packages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Applied Linear Statistical Models
Ziegel, E. R., Neter, J., Wasserman, W., & Kutner, M. (1992) · Technometrics · doi:10.2307/1269588 · Cited by 9,907
Response Surface Methodology: Process and Product Optimization Using Designed Experiments
Gunst, R. F., Myers, R. H., & Montgomery, D. C. (1996) · Technometrics · doi:10.2307/1270613 · Cited by 9,530
Cressie, N. (1992) · Terra Nova · doi:10.1111/j.1365-3121.1992.tb00605.x · Cited by 8,823
Simulation Modeling and Analysis.
Schervish, M. J., Law, A. M., & Kelton, W. D. (1983) · Journal of the American Statistical Association · doi:10.2307/2288169 · Cited by 8,697
McKay, M. D., Beckman, R. J., & Conover, W. J. (2000) · Technometrics · doi:10.2307/1271432 · Cited by 7,415
Applied Linear Statistical Models
(1997) · Technometrics · doi:10.1080/00401706.1997.10485141 · Cited by 7,185
Design and Analysis of Computer Experiments
Sacks, J., Welch, W. J., Mitchell, T. J., & Wynn, H. P. (1989) · Statistical Science · doi:10.1214/ss/1177012413 · Cited by 6,919
Cressie, N. A. C. (1993) · Wiley Series in Probability and Statistics · doi:10.1002/9781119115151 · Cited by 6,507
Empirical Model Building and Response Surfaces
Freeny, A. (1988) · Technometrics · doi:10.1080/00401706.1988.10488371 · Cited by 5,235
Statistics for experimenters. An introduction to design, data analysis, and model building.
Box, G. E. P., Hunter, W. G., & Hunter, J. S. (1978) · Cited by 4,021
Simulation modeling and analysis
(1999) · Journal of Manufacturing Systems · doi:10.1016/s0278-6125(99)90119-x · Cited by 3,833
Myers, R. H., & Montgomery, D. C. (1996) · IIE Transactions · doi:10.1080/15458830.1996.11770760 · Cited by 3,819
Response Surface Methodology: Process and Product Optimization Using Designed Experiments
Gunst, R. F. (1996) · Technometrics · doi:10.1080/00401706.1996.10484509 · Cited by 3,255
Theory of Modeling and Simulation
(2019) · Elsevier eBooks · doi:10.1016/c2016-0-03987-6 · Cited by 2,914
Kemp, C. D., & Tufte, E. R. (1992) · Biometrics · doi:10.2307/2532323 · Cited by 2,713
Design and Analysis of Experiments
Grice, J. V., & Montgomery, D. C. (2000) · Technometrics · doi:10.2307/1271458 · Cited by 2,654
A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output
Saltelli, A., Tarantola, S., & Chan, K. P.-S. (1999) · Technometrics · doi:10.1080/00401706.1999.10485594 · Cited by 2,275
The Design and Analysis of Computer Experiments
Santner, T. J., Williams, B. J., & Notz, W. I. (2003) · Springer Series in Statistics · doi:10.1007/978-1-4757-3799-8 · Cited by 2,170
Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Cited by 2,092
Theory of Optimal Experiments.
Herzberg, A. M., Wynn, H. P., Fedorov, V. V., Studden, W. J., & Klimko, E. M. (1972) · Biometrika · doi:10.2307/2334826 · Cited by 1,897
Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control
Spall, J. C. (2003) · Wiley eBooks · doi:10.1002/0471722138 · Cited by 1,890
Statistics for Spatial Data, Revised Edition.
Cressie, N. A. C. (1994) · Biometrics · doi:10.2307/2533238 · Cited by 1,856
Comparative studies of metamodelling techniques under multiple modelling criteria
Jin, R., Chen, W., & Simpson, T. W. (2001) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-001-0160-4 · Cited by 1,709
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0986 · Cited by 1,540
Design and Analysis of Experiments, 5th Ed.
Copeland, K. A. F. (2001) · Journal of Quality Technology · doi:10.1080/00224065.2001.11980088 · Cited by 1,475
Applied Linear Statistical Models
John, R. C. S. (1986) · Journal of Quality Technology · doi:10.1080/00224065.1986.11978989 · Cited by 1,307
Exploratory designs for computational experiments
Morris, M. D., & Mitchell, T. J. (1995) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(94)00035-t · Cited by 1,197
Multiple Comparisons: Theory and Methods
Christensen, R. (1997) · Technometrics · doi:10.1080/00401706.1997.10485097 · Cited by 1,098
INTRODUCTION TO STOCHASTIC SEARCH AND OPTIMIZATION: ESTIMATION, SIMULATION, AND CONTROL
Nowak, W. (2012) · Medical Entomology and Zoology · Cited by 1,033
Kiefer, J. (1959) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1959.tb00338.x · Cited by 790
Feature Article: Optimization for simulation: Theory vs. Practice
Fu, M. C. (2002) · INFORMS Journal on Computing · doi:10.1287/ijoc.14.3.192.113 · Cited by 742
Hyperdimensional Data Analysis Using Parallel Coordinates
Wegman, E. J. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474926 · Cited by 688
Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building
Parameswaran, R., Box, G. E. P., Hunter, W. G., & Hunter, J. S. (1979) · Journal of Marketing Research · doi:10.2307/3150696 · Cited by 684
Computational Statistics Handbook with MATLAB
Martinez, W. L., & Martinez, A. R. (2001) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420035636 · Cited by 655
Hsu, J. C. (1996) · Springer eBooks · doi:10.1007/978-1-4899-7180-7 · Cited by 593
Statistical tools for simulation practitioners
(1990) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(90)90147-b · Cited by 528
Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses
Clarke, S. M., Griebsch, J. H., & Simpson, T. W. (2005) · Journal of Mechanical Design · doi:10.1115/1.1897403 · Cited by 523
Kleijnen, J. P. C., & Sargent, R. G. (2000) · European Journal of Operational Research · doi:10.1016/s0377-2217(98)00392-0 · Cited by 491
Self-citationOrthogonal Column Latin Hypercubes and Their Application in Computer Experiments
Ye, K. Q. (1998) · Journal of the American Statistical Association · doi:10.1080/01621459.1998.10473803 · Cited by 475
Exploration of Metamodeling Sampling Criteria for Constrained Global Optimization
Sasena, M. J., Papalambros, P. Y., & Goovaerts, P. (2002) · Engineering Optimization · doi:10.1080/03052150211751 · Cited by 428
Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses
Clarke, S. M., Griebsch, J. H., & Simpson, T. W. (2003) · Volume 2: 29th Design Automation Conference, Parts A and B · doi:10.1115/detc2003/dac-48759 · Cited by 414
Sampling Strategies for Computer Experiments
Simpson, T. W., Dennis, L. A., & Chen, W. (2001) · Cited by 396
A Proof of the Conjecture that the Tukey-Kramer Multiple Comparisons Procedure is Conservative
Hayter, A. J. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346392 · Cited by 390
Performance metrics in supply chain management
Kleijnen, J. P. C., & Smits, M. T. (2003) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601539 · Cited by 380
Self-citationRao, C. R. (1967) · Project Euclid (Cornell University) · Cited by 378
Batch Size Effects in the Analysis of Simulation Output
Schmeiser, B. (1982) · Operations Research · doi:10.1287/opre.30.3.556 · Cited by 365
On Sequential Sampling for Global Metamodeling in Engineering Design
Jin, R., Chen, W., & Sudjianto, A. (2002) · Volume 2: 28th Design Automation Conference · doi:10.1115/detc2002/dac-34092 · Cited by 348
Response Surface Alternatives to the Taguchi Robust Parameter Design Approach
Myers, R. H., Khuri, A. I., & Vining, G. (1992) · The American Statistician · doi:10.1080/00031305.1992.10475869 · Cited by 334
Koehler, J. R., & Owen, A. B. (1996) · Handbook of Statistics · doi:10.1016/s0169-7161(96)13011-x · Cited by 322
An overview of the design and analysis of simulation experiments for sensitivity analysis
Kleijnen, J. P. C. (2005) · European Journal of Operational Research · doi:10.1016/j.ejor.2004.02.005 · Cited by 320
Self-citationDave, U. (1994) · Journal of the Operational Research Society · doi:10.1057/jors.1994.153 · Cited by 302
Kriging for interpolation in random simulation
van Beers, W. C. M., & Kleijnen, J. P. C. (2003) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601492 · Cited by 286
Self-citationApplication-driven sequential designs for simulation experiments: Kriging metamodelling
Kleijnen, J. P. C., & Beers, W. C. M. V. (2004) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601747 · Cited by 281
Self-citationSearching for important factors in simulation models with many factors: Sequential bifurcation
Bettonvil, B., & Kleijnen, J. P. C. (1997) · European Journal of Operational Research · doi:10.1016/s0377-2217(96)00156-7 · Cited by 239
Self-citationKriging Interpolation in Simulation: A Survey
van Beers, W. C. M., & Kleijnen, J. P. C. (2005) · Winter Simulation Conference · doi:10.1109/wsc.2004.1371308 · Cited by 232
Self-citationStatistical Tools for Simulation Practitioners
Thode, H. C. (1988) · Technometrics · doi:10.1080/00401706.1988.10488459 · Cited by 227
Simulation: A Statistical Perspective
Robinson, S., Kleijnen, J. P. C., & Groenendaal, W. J. H. V. (1993) · Journal of the Operational Research Society · doi:10.1057/jors.1993.150 · Cited by 178
Self-citationExperimental Design for Sensitivity Analysis, Optimization, and Validation of Simulation Models
Kleijnen, J. P. C. (1998) · Handbook of Simulation · doi:10.1002/9780470172445.ch6 · Cited by 173
Self-citationA User's Guide to the Brave New World of Designing Simulation Experiments
Kleijnen, J. P. C., Sanchez, S. M., Lucas, T., & Cioppa, T. M. (2003) · Data Archiving and Networked Services (DANS) · Cited by 157
Self-citationA comparison of experimental designs in the development of a neural network simulation metamodel
Alam, M. F., McNaught, K. R., & Ringrose, T. J. (2004) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2003.10.006 · Cited by 138
New Procedures to Select the Best Simulated System Using Common Random Numbers
Chick, S. E., & Inoue, K. (2001) · Management Science · doi:10.1287/mnsc.47.8.1133.10229 · Cited by 133
[Design and Analysis of Computer Experiments]: Rejoinder
Sacks, J., Welch, W. J., Mitchell, T. J., & Wynn, H. P. (1989) · Statistical Science · doi:10.1214/ss/1177012420 · Cited by 132
Six (Or So) Things You Can Do with a Bad Model
Hodges, J. S. (1991) · Operations Research · doi:10.1287/opre.39.3.355 · Cited by 125
Kleijnen, J. P. C., & van Beers, W. C. M. (2004) · European Journal of Operational Research · doi:10.1016/j.ejor.2003.09.037 · Cited by 109
Self-citationControlled Sequential Bifurcation: A New Factor-Screening Method for Discrete-Event Simulation
Wan, H., Ankenman, B. E., & Nelson, B. L. (2006) · Operations Research · doi:10.1287/opre.1060.0311 · Cited by 104
Is It You or Your Model Talking?: A Framework for Model Validation
Hodges, J. S., & Dewar, J. (1992) · Cited by 101
ASAP3: a batch means procedure for steady-state simulation analysis
Steiger, N. M., Lada, E. K., Wilson, J. R., Joines, J. A., Alexopoulos, C., & Goldsman, D. M. (2005) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/1044322.1044325 · Cited by 82
12 Multiresponse surface methodology
Khuri, A. I. (1996) · Handbook of statistics · doi:10.1016/s0169-7161(96)13014-5 · Cited by 70
The 1996 performance assessment for the Waste Isolation Pilot Plant
Helton, J. C., & Marietta, M. G. (2000) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(00)00022-3 · Cited by 63
Wavelet regression for random or irregular design
Antoniadis, A., & Pham, D.-T. (1998) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(98)90145-1 · Cited by 62
Analysis of Supersaturated Designs
Holcomb, D. R., Montgomery, D. C., & Carlyle, W. M. (2003) · Journal of Quality Technology · doi:10.1080/00224065.2003.11980188 · Cited by 50
Dealing with the least squares regression assumptions in simulation metamodeling
Tunalı, S., & Batmaz, İ. (2000) · Computers & Industrial Engineering · doi:10.1016/s0360-8352(00)00046-2 · Cited by 27
Credible uses of the distributed interactive simulation (DIS) system
Dewar, J., Hodges, J. S., Saunders‐Newton, D., Bankes, S. C., Lucas, T. W., & Vye, P. (1996) · Cited by 20
Self-citation
Cited by
Flood inundation modelling: A review of methods, recent advances and uncertainty analysis
Teng, J., Jakeman, A. J., Vaze, J., Croke, B. F. W., Dutta, D., & Kim, S. (2017) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2017.01.006 · Cited by 1,456
Sensitivity analysis: A review of recent advances
Borgonovo, E., & Plischke, E. (2016) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.06.032 · Cited by 1,167
Dilated cardiomyopathy: the complexity of a diverse genetic architecture
Hershberger, R. E., Hedges, D. J., & Morales, A. (2013) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2013.105 · Cited by 1,025
O’Sullivan, P. (2005) · Manual Therapy · doi:10.1016/j.math.2005.07.001 · Cited by 985
Socio‐economic differentials in peripheral biology: Cumulative allostatic load
Seeman, T., Epel, E., Gruenewald, T., Karlamangla, A., & McEwen, B. S. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2009.05341.x · Cited by 579
Metadata
- Authors
- Kleijnen, J. P. C., Sanchez, S. M., Lucas, T. W., & Cioppa, T. M.
- Venue
- INFORMS Journal on Computing
- Published
- 2005-08
- Volume
- 17
- Issue
- 3
- Pages
- 263-289
- ISSN
- 0899-1499 (ISSN-L)
- 1091-9856
- 1526-5528
- OpenAlex
- W2148080122
- Cited by
- 619
- References
- 73
- FWCI
- 60.68
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.