The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Verification, validation, and predictive capability in computational engineering and physics
Abstract
Developers of computer codes, analysts who use the codes, and decision makers who rely on the results of the analyses face a critical question: How should confidence in modeling and simulation be critically assessed? Verification and validation (V&V) of computational simulations are the primary methods for building and quantifying this confidence. Briefly, verification is the assessment of the accuracy of the solution to a computational model. Validation is the assessment of the accuracy of a computational simulation by comparison with experimental data. In verification, the relationship of the simulation to the real world is not an issue. In validation, the relationship between computation and the real world, ie, experimental data, is the issue. This paper presents our viewpoint of the state of the art in V&V in computational physics. (In this paper we refer to all fields of computational engineering and physics, eg, computational fluid dynamics, computational solid mechanics, structural dynamics, shock wave physics, computational chemistry, etc, as computational physics.) We describe our view of the framework in which predictive capability relies on V&V, as well as other factors that affect predictive capability. Our opinions about the research needs and management issues in V&V are very practical: What methods and techniques need to be developed and what changes in the views of management need to occur to increase the usefulness, reliability, and impact of computational physics for decision making about engineering systems? We review the state of the art in V&V over a wide range of topics, for example, prioritization of V&V activities using the Phenomena Identification and Ranking Table (PIRT), code verification, software quality assurance (SQA), numerical error estimation, hierarchical experiments for validation, characteristics of validation experiments, the need to perform nondeterministic computational simulations in comparisons with experimental data, and validation metrics. We then provide an extensive discussion of V&V research and implementation issues that we believe must be addressed for V&V to be more effective in improving confidence in computational predictive capability. Some of the research topics addressed are development of improved procedures for the use of the PIRT for prioritizing V&V activities, the method of manufactured solutions for code verification, development and use of hierarchical validation diagrams, and the construction and use of validation metrics incorporating statistical measures. Some of the implementation topics addressed are the needed management initiatives to better align and team computationalists and experimentalists in conducting validation activities, the perspective of commercial software companies, the key role of analysts and decision makers as code customers, obstacles to the improved effectiveness of V&V, effects of cost and schedule constraints on practical applications in industrial settings, and the role of engineering standards committees in documenting best practices for V&V. There are 207 references cited in this review article.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Simulation Modeling and Analysis.
Schervish, M. J., Law, A. M., & Kelton, W. D. (1983) · Journal of the American Statistical Association · doi:10.2307/2288169 · Cited by 8,697
Clarke, E. M., Grümberg, O., & Long, D. (1996) · Cited by 6,813
Computational Methods for Fluid Dynamics
Ferziger, J. H., Peric, M., & Leonard, A. (1997) · Physics Today · doi:10.1063/1.881751 · Cited by 6,005
Clarke, E. M., & Schlingloff, B. (1999) · Cited by 4,226
Structural Reliability Analysis and Prediction
(2017) · Wiley eBooks · doi:10.1002/9781119266105 · Cited by 3,778
Numerical Computation of Internal and External Flows
(2007) · Elsevier eBooks · doi:10.1016/b978-0-7506-6594-0.x5037-1 · Cited by 3,474
Perspective: A Method for Uniform Reporting of Grid Refinement Studies
Roache, P. J. (1994) · Journal of Fluids Engineering · doi:10.1115/1.2910291 · Cited by 3,135
Geostatistics: Modeling Spatial Uncertainty
Stein, M. L., Chiles, J.-P., & Delfiner, P. (2000) · Journal of the American Statistical Association · doi:10.2307/2669569 · Cited by 2,880
Sivia, D. S. (1996) · Cited by 2,678
QUANTIFICATION OF UNCERTAINTY IN COMPUTATIONAL FLUID DYNAMICS
Roache, P. J. (1997) · Annual Review of Fluid Mechanics · doi:10.1146/annurev.fluid.29.1.123 · Cited by 2,319
Structural Reliability Methods
Ditlevsen, O., & Madsen, H. O. (1996) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · Cited by 2,193
A posteriori error estimation in finite element analysis
Ainsworth, M., & Oden, J. T. (1997) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/s0045-7825(96)01107-3 · Cited by 2,166
Verification and Validation in Computational Science and Engineering
Roache, P. J. (1998) · Cited by 2,157
Axelsson, O. (1994) · Cambridge University Press eBooks · doi:10.1017/cbo9780511624100 · Cited by 2,085
(1998) · PDCA/Test · doi:10.1201/9781420048131.axh · Cited by 1,900
Geostatistics: Modeling Spatial Uncertainty
Chilès, J., & Delfiner, P. (1999) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316993 · Cited by 1,834
Experimentation and Uncertainty Analysis for Engineers
Coleman, H. W., Steck, W. G., & Hurlbut, F. C. (1991) · Journal of Engineering for Industry · doi:10.1115/1.2899692 · Cited by 1,805
Assessment and Propagation of Model Uncertainty
Draper, D. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02015.x · Cited by 1,599
The capability maturity model: Guidelines for improving the software process
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90461-7 · Cited by 1,452
Verification and validation in computational fluid dynamics
Oberkampf, W. L., & Trucano, T. G. (2002) · Progress in Aerospace Sciences · doi:10.1016/s0376-0421(02)00005-2 · Cited by 1,150
Self-citationTheory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems
Zeigler, B. P., Praehofer, H., & Kim, T. G. (2000) · Cited by 1,098
(1958) · Medical Entomology and Zoology · Cited by 1,059
(2000) · The Handbooks of Fuzzy Sets Series · doi:10.1007/978-1-4615-4429-6 · Cited by 1,055
Water quality modeling: A review of the analysis of uncertainty
Beck, M. B. (1987) · Water Resources Research · doi:10.1029/wr023i008p01393 · Cited by 954
Stern, F., Wilson, R. V., Coleman, H. W., & Paterson, E. G. (2001) · Journal of Fluids Engineering · doi:10.1115/1.1412235 · Cited by 935
Duclos-Lassalle, G. (2000) · Érudit (Université de Montréal) · Cited by 846
Uncertainties in risk analysis: Six levels of treatment
Paté-Cornell, M. E. (1996) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(96)00067-1 · Cited by 811
Error and uncertainty in modeling and simulation
Oberkampf, W. L., Deland, S. M., Rutherford, B. M., Diegert, K. V., & Alvin, K. F. (2002) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(01)00120-x · Cited by 773
Self-citationCode Verification by the Method of Manufactured Solutions
Roache, P. J. (2002) · Journal of Fluids Engineering · doi:10.1115/1.1436090 · Cited by 760
Modeling and Simulation: Challenges and Best Practices for Industry
Dubois, G. (2018) · CRC Press eBooks · doi:10.1201/9781351241137 · Cited by 744
Helton, J. C. (1993) · Reliability Engineering & System Safety · doi:10.1016/0951-8320(93)90097-i · Cited by 735
Verification and validation of simulation models
Kleijnen, J. P. C. (1995) · European Journal of Operational Research · doi:10.1016/0377-2217(94)00016-6 · Cited by 730
Verification of Codes and Calculations
Roache, P. J. (1998) · AIAA Journal · doi:10.2514/2.457 · Cited by 679
Risk Modeling, Assessment, And Management
Haimes, Y. Y. (1999) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.1999.760577 · Cited by 625
Different methods are needed to propagate ignorance and variability
Ferson, S., & Ginzburg, L. R. (1996) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(96)00071-3 · Cited by 621
Probabilistic Risk Assessment and Management for Engineers and Scientists
Kumamoto, H., & Henley, E. J. (1996) · CERN Document Server (European Organization for Nuclear Research) · Cited by 618
Cullen, A. C., & Frey, H. C. (1999) · Cited by 612
Statistical tools for simulation practitioners
(1990) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(90)90147-b · Cited by 528
Uncertainty-Based Information: Elements of Generalized Information Theory
Klir, G. J., Wierman, M. J., & Kacprzyk, J. (1998) · Medical Entomology and Zoology · Cited by 494
Design and Analysis of Experiments
(1999) · Springer Texts in Statistics · doi:10.1007/b97673 · Cited by 491
Klir, G. J., & Wierman, M. J. (1999) · Studies in fuzziness and soft computing · doi:10.1007/978-3-7908-1869-7 · Cited by 488
Smithson, M. (1989) · Cognitive Science · doi:10.1007/978-1-4612-3628-3 · Cited by 476
Ignorance and Uncertainty: Emerging Paradigms
Smithson, M. J. (2012) · Medical Entomology and Zoology · Cited by 474
O’Regan, G. (2002) · A Practical Approach to Software Quality · doi:10.1007/978-0-387-22454-1_4 · Cited by 450
Predicting uncertainty in forecasts of weather and climate
Palmer, T. N. (2000) · Reports on Progress in Physics · doi:10.1088/0034-4885/63/2/201 · Cited by 446
Terminology for model credibility
(1979) · SIMULATION · doi:10.1177/003754977903200304 · Cited by 413
Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty
Helton, J. C. (1997) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659708811803 · Cited by 368
Terminology for model credibility
Abrahamson, G. (1980) · SIMULATION · doi:10.1177/003754978003500606 · Cited by 350
Code Verification by the Method of Manufactured Solutions
SALARI, K., & KNUPP, P. (2000) · doi:10.2172/759450 · Cited by 313
Laney, C. B. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9780511605604 · Cited by 288
Earman, J. (1992) · Cited by 255
Hora, S. C. (1996) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(96)00077-4 · Cited by 249
A Mathematical Theory of Hints
Kohlas, J., & Monney, P.-A. (1995) · Lecture notes in economics and mathematical systems · doi:10.1007/978-3-662-01674-9 · Cited by 241
Data Analysis: Statistical and Computational Methods for Scientists and Engineers
Carlton, M. A., & Brandt, S. (2000) · The American Statistician · doi:10.2307/2686041 · Cited by 213
Validation in Simulation: Various Positions in the Philosophy of Science
Kleindorfer, G. B., O'Neill, L., & Ganeshan, R. (1998) · Management Science · doi:10.1287/mnsc.44.8.1087 · Cited by 210
A Mathematical Theory of Hints: An Approach to the Dempster-Shafer Theory of Evidence
Kohlas, J., & Monney, P.-A. (1995) · Cited by 183
Almond, R. G. (1995) · Springer eBooks · doi:10.1007/978-1-4899-7106-7 · Cited by 171
Simulation Validation: A Confidence Assessment Methodology
Knepell, P. L., & Arangno, D. C. (1993) · Medical Entomology and Zoology · Cited by 150
Formal Methods and the Certification of Critical Systems
Rushby, J. (2004) · Cited by 139
Bayesian methodology for reliability model acceptance
Zhang, R., & Mahadevan, S. (2003) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(02)00269-7 · Cited by 124
Experimentation and Uncertainty Analysis for Engineers, 2nd Ed.
Wahl, T. L. (1999) · Journal of Hydraulic Engineering · doi:10.1061/(asce)0733-9429(1999)125:12(1320 · Cited by 114
The T experiments: errors in scientific software
Hatton, L. (1997) · IEEE Computational Science and Engineering · doi:10.1109/99.609829 · Cited by 110
Francez, N. (1992) · Cited by 81
Methods for the external validation of contiuous system simulation models:a review
smith, D. J. M. (1998) · Mathematical and Computer Modelling of Dynamical Systems · doi:10.1080/13873959808837066 · Cited by 78
Assessment of One- and Two-Equation Turbulence Models for Hypersonic Transitional Flows
Roy, C. J., & Blottner, F. G. (2001) · Journal of Spacecraft and Rockets · doi:10.2514/2.3755 · Cited by 73
Model uncertainty: theory and practical implications
Laskey, K. B. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.487959 · Cited by 71
Helton, J. C., Anderson, D. R., Baker, B. L., Bean, J. E., Berglund, J., Beyeler, W., Economy, K., Garner, J. W., Hora, S. C., Iuzzolino, H. J., et al. (1996) · Reliability Engineering & System Safety · doi:10.1016/0951-8320(95)00089-5 · Cited by 68
Model accreditation: A rationale and process for determining a numerical rating
Gass, S. I. (1993) · European Journal of Operational Research · doi:10.1016/0377-2217(93)90316-f · Cited by 68
Uncertainty and sensitivity analysis in performance assessment for the Waste Isolation Pilot Plant
Helton, J. C. (1999) · Computer Physics Communications · doi:10.1016/s0010-4655(98)00171-4 · Cited by 67
A review of fault management techniques used in safety-critical avionic systems
Johnson, D. (1996) · Progress in Aerospace Sciences · doi:10.1016/0376-0421(96)82785-0 · Cited by 53
Uncertainty Quantification for Multiscale Simulations1
DeVolder, B. G., Glimm, J. G., Grove, J. W., Kang, Y., Lee, Y., Pao, K., Sharp, D. H., & Ye, K. (2001) · Journal of Fluids Engineering · doi:10.1115/1.1445139 · Cited by 53
Frank, M. V. (1999) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(99)00002-2 · Cited by 43
Watson, A. I., Holle, R. L., & López, R. E. (1995) · Weather and Forecasting · doi:10.1175/1520-0434(1995)010<0620:apfcat>2.0.co;2 · Cited by 32
Stockman, C. T., Garner, J. W., Helton, J. C., Johnson, J. D., Shinta, A., & Smith, L. N. (2000) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(00)00039-9 · Cited by 28
Introductory Statistics and Random Phenomena
Denker, M., Woyczyński, W. A., & Ycart, B. (1998) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-2028-2 · Cited by 20
Ensemble Simulations of Explosive Cyclogenesis at Ranges of 2–5 Days
Sanders, F., Mullen, S. L., & Baumhefner, D. P. (2000) · Monthly Weather Review · doi:10.1175/1520-0493(2000)128<2920:esoeca>2.0.co;2 · Cited by 15
On Statistical Model Validation
Lee, L. H., & Poolla, K. R. (1996) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.2802308 · Cited by 14
Introductory Statistics and Random Phenomena
Lin, Y., Denker, M., & Woyczyński, W. A. (1999) · Technometrics · doi:10.1080/00401706.1999.10485688 · Cited by 13
Statistical validation for uncertainty models
Lee, L. H., & Poolla, K. R. (2005) · Springer eBooks · doi:10.1007/bfb0027675 · Cited by 8
Research works printed but not linked: 92
- Cohen ML, Rolph JE, and Steffey DL (eds) (1998), Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements, National Academy Press, Washington DC.
- IEEE (1984), IEEE Standard Dictionary of Electrical and Electronics Terms, ANSI/IEEE Std 100-1984, New York.
- IEEE (1991), IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990, New York.
- Oberkampf WL (1994), A proposed framework for computational fluid dynamics code calibration/validation, American Institute of Aeronautics and Astronautics, AIAA Paper No. 94-2540, 18th AIAA Aerospace Ground Testing Conf, Colorado Springs CO. · doi:10.2514/6.1994-2540
- Oberkampf WL and Trucano TG (2000), Validation methodology in computational fluid dynamics, Am Inst of Aeronaut and Astronaut, AIAA 2000-2549, Fluids 2000 Conf, Denver CO. · DOI
- Marvin JG (1995), Perspective on computational fluid dynamics validation, AIAA J. 33(10), 1778–1787.
- Cosner RR (1995), CFD validation requirements for technology transition, Am Inst of Aeronaut and Astronaut, AIAA Paper No. 95-2227, 26th AIAA Fluid Dynamics Conf, San Diego CA. · doi:10.2514/6.1995-2227
- Sindir MM, Barson SL, Chan DC, and Lin WH (1996), On the development and demonstration of a code validation process for industrial applications, Am Inst of Aeronaut and Astronaut, AIAA Paper No. 96-2032, 27th AIAA Fluid Dynamics Conf, New Orleans LA. · doi:10.2514/6.1996-2032
- Sindir MM and Lynch ED (1997), Overview of the state-of-practice of computational fluid dynamics in advanced propulsion system design, Am. Inst. Aeronaut. Astronaut, AIAA Paper No. 97-2124, 28th AIAA Fluid Dynamics Conf, Snowmass CO. · doi:10.2514/6.1997-2124
- Balci O (1997), Principles of simulation of model validation, verification, and testing, Trans. Soc. Comput. Simul. Int. 14, 3–12.
- Robinson S (1999), Simulation verification, validation, and confidence: A tutorial, Trans. Soc. Comput. Simul. Int. 16, 63–69.
- Pilch M, Trucano TG, Moya JL, Froehlich GK, Hodges AL, and Peercy DE (2001), Guidelines for Sandia ASCI Verification and Validation Plans-Content and Format: Version 2, Sandia National Laboratories, SAND2000-3101, Albuquerque NM. · doi:10.2172/750305
- Trucano TG, Pilch M, and Oberkampf WL (2002), General concepts for experimental validation of ASCI code applications, Sandia National Laboratories, SAND2002-0341, Albuquerque NM. · doi:10.2172/800777
- Easterling RG (2001), Measuring the predictive capability of computational models: principles and methods, issues and illustrations, Sandia National Laboratories, SAND2001-0243, Albuquerque NM. · DOI
- Oberkampf WL, DeLand SM, Rutherford BM, Diegert KV, and Alvin KF (2000), Estimation of total uncertainty in computational simulation, Sandia National Laboratories, SAND2000-0824, Albuquerque NM. · DOI
- Kramosil I (2001), Probabilistic Analysis of Belief Functions, Kluwer, New York. · doi:10.1007/978-1-4615-0587-7
- Kafka P (1994), Important issues using PSA technology for design of new systems and plants, Reliability Eng. Sys. Safety 45(1–2), 205–213. · doi:10.1016/0951-8320(94)90087-6
- Pate´-Cornell ME (1999), Conditional uncertainty analysis and implications for decision making: The case of WIPP, Risk Anal 19(5), 1003–1016. · doi:10.1111/j.1539-6924.1999.tb00457.x
- Helton JC , Anderson DR, Basabilvazo G, Jow H-N, and Marietta MG (2000), Conceptual structure of the 1996 performance assessment for the Waste Isolation Pilot Plant, Reliability Eng. Sys. Safety 69(1–3), 151–165. · DOI
- Mohanty S , Codell RB, Ahn TM, and Cragnolino GA (2000), An approach to the assessment of high-level radioactive waste containment II: Radionuclide releases from an engineered barrier system, Nucl. Eng. Des. 201, 307–325. · doi:10.1016/s0029-5493(00)00274-0
- Boyack BE , Catton I, Duffey RB, Griffith P, Katsma KR, Lellouche GS, Levy S, Rohatgi US, Wilson GE, Wulff W, and Zuber N (1990), Quantifying reactor safety margins, Part 1: An overview of the code scaling, applicability, and uncertainty evaluation methodology, Nucl. Eng. Des. 119, 1–15. · doi:10.1016/0029-5493(90)90071-5
- Wilson GE , Boyack BE, Catton I, Duffey RB, Griffith P, Katsma KR, Lellouche GS, Levy S, Rohatgi US, Wulff W, and Zuber N (1990), Quantifying reactor safety margins, Part 2: Characterization of important contributors to uncertainty, Nucl. Eng. Des. 119, 17–31. · doi:10.1016/0029-5493(90)90072-6
- Wulff W , Boyack BE, Catton I, Duffey RB, Griffith P, Katsma KR, Lellouche GS, Levy S, Rohatgi US, Wilson GE, and Zuber N (1990), Quantifying reactor safety margins, Part 3: Assessment and ranging of parameters, Nucl. Eng. Des. 119, 33–65. · doi:10.1016/0029-5493(90)90073-7
- Zuber N , Wilson GE, Ishii M, Wulff W, Boyack BE, Dukler AE, Griffith P, Healzer JM, Henry RE, Lehner JR, Levy S, and Moody FJ (1998), An integrated structure and scaling methodology for severe accident technical issue resolution: Development of methodology, Nucl. Eng. Des. 186(1–2), 1–21. · doi:10.1016/s0029-5493(98)00215-5
- Stevenson DE (1999), A critical look at quality in large-scale simulations, Comput. Sci. Eng. 1(3), 53–63.
- Gustafson J (1998), Computational verifiability and feasibility of the ASCI program, IEEE Comput. Sci. Eng. 5(1), 36–45.
- Shih TM (1985), A procedure to debug computer programs, Int. J. Numer. Methods Eng. 21(6), 1027–1037. · doi:10.1002/nme.1620210605
- Steinberg S , and Roache PJ (1985), Symbolic manipulation and computational fluid dynamics, J. Comput. Phys. 57(2), 251–284.
- Knupp P and Salari K (2002), Verification of Computer Codes in Computational Science and Engineering, Chapman & Hall/CRC, Boca Raton FL. · doi:10.1201/9781420035421
- Srivastava BN , Werle MJ, and Davis RT (1979), A finite difference technique involving discontinuous derivatives, Comput. Fluids 7(1), 69–74.
- Blottner FG (1982), Influence of boundary approximations and conditions on finite difference solutions, J. Comput. Phys. 48(2), 246–269. · doi:10.1016/0021-9991(82)90049-3
- Turkel E (1986), Accuracy of schemes with nonuniform meshes for compressible fluid-flows, Appl. Numer. Math. 2(6), 529–550. · doi:10.1016/0168-9274(86)90006-1
- Carpenter MH and Casper JH (1999), Accuracy of shock capturing in two spatial dimensions, AIAA J. 37(9), 1072–1079.
- Roy CJ, McWherter-Payne MA, and Oberkampf WL (2000), Verification and validation for laminar hypersonic flowfields, Am Inst of Aeronaut and Astronaut, AIAA2000-2550, Fluids 2000 Conf, Denver, CO. · doi:10.2514/6.2000-2550
- Botella O and Peyret R (2001), Computing singular solutions of the Navier-Stokes equations with the Chebyshev-Collocation method, Int. J. Numer. Methods Fluids 36(2), 125–163. · doi:10.1002/fld.121
- Haworth DC , Tahry SHE, and Huebler MS (1993), A global approach to error estimation and physical diagnostics in multidimensional computational fluid dynamics, Int. J. Numer. Methods Fluids 17(1), 75–97. · doi:10.1002/fld.1650170106
- Singhal AK (1998), Key elements of verification and validation of CFD software, American Institute of Aeronautics and Astronautics, AIAA 98-2639, 29th AIAA Fluid Dynamics Conf, Albuquerque NM. · doi:10.2514/6.1998-2639
- Wallace DR, Ippolito LM, and Cuthill BB (1996), Reference information for the software verification and validation process, Rept 500-234. · doi:10.6028/nist.sp.500-234
- Kaner C, Falk J, and Nguyen HQ (1999), Testing Computer Software, 2nd Edition, John Wiley, New York.
- NRC (1996), Statistical Software Engineering, Natl Res Council, Natl Academy Press, Washington DC.
- Partsch HA (1990), Specification and Transformation of Programs, Springer-Verlag, New York. · DOI
- Bowen JP and Hinchey MG (1995), 10-commandments of formal methods, Computer 28(4), 56–63. · doi:10.1109/2.375178
- Fairley RE (1985), Software Engineering Concepts, McGraw-Hill, New York.
- Baber R (1987), The Spine of Software: Designing Provably Correct Software: Theory and Practice, John Wiley, New York.
- Dyer M (1992), The Cleanroom Approach to Quality Software Development, John Wiley, New York.
- Lewis RO (1992), Independent Verification and Validation, 1st Edition, John Wiley, New York.
- Rook P (1990), Software Reliability Handbook, Elsevier Science Publ, New York.
- Jones C (1997), Applied Software Measurement, 2nd Edition, McGraw-Hill, New York.
- Hirsch C (1988), Numerical Computation of Internal and External Flows: Vol 1: Fundamentals of Numerical Discretization, John Wiley, New York.
- Oden JT (1993), Error estimation and control in computational fluid dynamics, The Mathematics of Finite Elements and Applications, JR Whiteman (ed), John Wiley, New York, 1–23. · doi:10.21236/ada266876
- Morton KW (1996), Numerical Solution of Convection-Diffusion Problems, CRC Press, Boca Raton FL.
- Babuska I and Strouboulis T (2001), The Finite Element Method and its Reliability, Oxford Univ Press, Oxford, UK. · doi:10.1093/oso/9780198502760.001.0001
- Roache PJ (1990), Need for control of numerical accuracy, J. Spacecr. Rockets 27(2), 98–102. · doi:10.2514/3.26113
- Roy CJ (2001), Grid convergence error analysis for mixed-order numerical schemes, Am Inst of Aeronaut and Astronaut, AIAA -2001-2606, AIAA Fluid Dynamics Conf, Anaheim CA. · doi:10.2514/6.2001-2606
- Cadafalch J , Perez-Segarra CC, Consul R, and Oliva A (2002), Verification of finite volume computations on steady state fluid flow and heat transfer, ASME J. Fluids Eng., 124(1), 11–21. · doi:10.1115/1.1436092
- Chen C-FA , Lotz RD, and Thompson BE (2002), Assessment of numerical uncertainty around shocks and corners on blunt trailing-edge supercritical airfoils, Comput. Fluids 31(1), 25–40.
- Venditti DA , and Darmofal DL (2000), Adjoint error estimation and grid adaptation for functional outputs: Application to quasi-one-dimensional flow, J. Comput. Phys. 164(1), 204–227.
- Jameson A , and Martinelli L (1998), Mesh refinement and modeling errors in flow simulation, AIAA J. 36(5), 676–686.
- Estep DJ , Larson MG, and Williams RD (2000), Estimating the error of numerical solutions of systems of reaction-diffusion equations, Mem. Am. Math. Soc. 146(696), 1–109.
- Babuska I , Ihlenburg F, Strouboulis T, and Gangaraj SK (1997), A posteriori error estimation for finite element solutions of Helmholtz’ equation, Part II: Estimation of the pollution error, Int. J. Numer. Methods Eng. 40, 3883–3900. · doi:10.1002/(sici)1097-0207(19971115)40:21<3883::aid-nme231>3.0.co;2-v
- Babuska I , Strouboulis T, Upadhyay CS, and Gangaraj SK (1995), A posteriori estimation and adaptive control of the pollution error in the h-version of the finite element method, Int. J. Numer. Methods Eng. 38, 4207–4235. · doi:10.1002/nme.1620382408
- Oden JT , Feng Y, and Prudhomme S (1998), Local and pollution error estimation for Stokesian flow, Int. J. Numer. Methods Fluids 27, 33–39. · doi:10.1002/(sici)1097-0363(199801)27:1/4<33::aid-fld648>3.0.co;2-3
- Zhang XD, Pelletier D, Trepanier JY, and Camarero R (2000), Verification of error estimators for the euler equations, Am Inst of Aeronaut and Astronau, AIAA-2000-1001, 38th AIAA Aerospace Sciences Meeting, Reno NV. · doi:10.2514/6.2000-1001
- Chorin AJ, Kast AP, and Kupferman R (1998), On the prediction of large-scale dynamics using unresolved computations, Lawrence Berkeley National Lab, LBNL-42283, Berkeley CA. · doi:10.1090/conm/238/03539
- Chorin AJ , Kast AP, and Kupferman R (1998), Optimal prediction of underresolved dynamics, Proc. Natl. Acad. Sci. U.S.A. 95, 4094–4098.
- Gunter BH (1993), How statistical design concepts can improve experimentation in the physical sciences, Comput. Phys. 7(3), 262–272.
- Bossel H (1994), Modeling and Simulation, 1st Edition, AK Peters, Wellesley MA. · doi:10.1007/978-3-663-10822-1_1
- Oberkampf WL and Aeschliman DP (1992), Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 1, Experimental results, AIAA J. 30(8), 2000–2009.
- Walker MA , and Oberkampf WL (1992), Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 2, Computational results, AIAA J. 30(8), 2010–2016.
- Oberkampf WL, Aeschliman DP, Henfling JF, and Larson DE (1995), Surface pressure measurements for CFD code validation in hypersonic flow, Am Inst of Aeronaut and Astronaut, AIAA Paper No 95-2273, 26th AIAA Fluid Dynamics Conf, San Diego CA. · doi:10.2514/6.1995-2273
- Aeschliman DP and Oberkampf WL (1998), Experimental methodology for computational fluid dynamics code validation, AIAA J. 36(5), 733–741.
- Oberkampf WL and Blottner FG (1998), Issues in computational fluid dynamics code verification and validation, AIAA J. 36(5), 687–695.
- Haldar A and Mahadevan S (2000), Probability, Reliability, and Statistical Methods in Engineering Design, John Wiley, New York.
- Du J , Mullen SL, and Sanders F (1997), Short-range ensemble forecasting of quantitative precipitation, Mon. Weather Rev. 125(10), 2427–2459. · doi:10.1175/1520-0493(1997)125<2427:srefoq>2.0.co;2
- Alapaty K , Raman S, and Niyogi DS (1997), Uncertainty in the specification of surface characteristics: A study of prediction errors in the boundary layer, Boundary-Layer Meteorol. 82(3), 473–500.
- Chlond A and Wolkau A (2000), Large-eddy simulation of a nocturnal stratocumulus-topped marine atmospheric boundary layer: An uncertainty analysis, Boundary-Layer Meteorol. 95(1), 31–55.
- Glimm J and Sharp D (1998), Stochastic methods for the prediction of complex multiscale phenomena, Q. Appl. Math. 56(4), 741–765. · doi:10.1090/qam/1668736
- Coleman HW and Stern F (1997), Uncertainties and CFD code validation, ASME J. Fluids Eng. 119, 795–803. · doi:10.1115/1.2819500
- Hanson KM (1999), A framework for assessing uncertainties in simulation predictions, Physica D 133, 179–188. · DOI
- Hills RG and Trucano TG (2002), Statistical validation of engineering and scientific models: A maximum likelihood based metric, Sandia Natl Labs, SAND2001-1783, Albuquerque NM. · doi:10.2172/791881
- Hills RG and Trucano TG (2001), Statistical validation of engineering and scientific models with application to CTH, Sandia Natl Labs, SAND2001-0312, Albuquerque NM. · doi:10.2172/782589
- Dowding K (2001), Quantitative validation of mathematical models, ASME Int Mech Eng Congress, New York. · doi:10.1115/imece2001/htd-24308
- Wilson RV , Stern F, Coleman HW, and Paterson EG (2001), Comprehensive approach to verification and validation of CFD simulations, Part 2: Application for RANS simulation of a cargo/container ship, ASME J. Fluids Eng. 123(4), 803–810. · doi:10.1115/1.1412236
- Singpurwalla ND and Wilson SP (1999), Statistical Methods in Software Engineering: Reliability and Risk, Springer-Verlag, Berlin. · doi:10.1007/978-1-4612-0565-4
- Yee HC , and Sweby PK (1998), Aspects of numerical uncertainties in time marching to steady-state numerical solutions, AIAA J. 36(5), 712–724.
- Glimm J , Hou SL, Kim HJ, Lee Y, Sharp DH, Ye K, and Zou WS (2001), Risk management for petroleum reservoir production: A simulation-based study of prediction, Comput. Geosci. 5(3), 173–197.
- Hills RG and Leslie IH (2003), Statistical validation of engineering and scientific models: Validation experiments to application, Sandia Natl Labs, SAND2003-0706, Albuquerque NM. · doi:10.2172/809991
- Hasselman TK (2001), Quantification of uncertainty in structural dynamic models, ASCE, J. Aerosp. Eng. 14(4), 158–165. · doi:10.1061/(asce)0893-1321(2001)14:4(158
- Logan RW and Nitta CK (2002), Verification & validation (V&V) methodology and quantitative reliability at confidence (QRC): Basis for an investment strategy, Lawrence Livermore Natl Lab, UCRL-ID-150874, Livermore CA. · doi:10.2172/15002506
- Hutton AG and Casey MV (2001), Quality and trust in industrial CFD—A European perspective, Am Inst of Aeronau and Astronaut, AIAA2001-0656, 39th AIAA Aerospace Sciences Meeting, Reno NV. · doi:10.2514/6.2001-656
- Levy DW, Zickuhr T, Vassberg J, Agrawal S, Wahls RW, Pirzadeh S, and Hemsch MJ (2002), Summary of data from the 1st AIAA CFD Drag Prediction Workshop, Am Inst of Aeronaut and Astronaut, AIAA-2002-0841, 40th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV. · doi:10.2514/6.2002-841
- Hemsch M (2002), Statistical analysis of CFD solutions from the Drag Prediction Workshop, Am Inst of Aeronaut and Astronaut, AIAA-2002-0842 40th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV. · doi:10.2514/6.2002-842
Cited by
Blocken, B. (2015) · Building and Environment · doi:10.1016/j.buildenv.2015.02.015 · Cited by 1,163
50 years of Computational Wind Engineering: Past, present and future
Blocken, B. (2014) · Journal of Wind Engineering and Industrial Aerodynamics · doi:10.1016/j.jweia.2014.03.008 · Cited by 825
(2016) · Systems Engineering for Commercial Aircraft · doi:10.4324/9781315611716-18 · Cited by 804
A review on the CFD analysis of urban microclimate
Toparlar, Y., Blocken, B., Maiheu, B., & van Heijst, G. J. F. (2017) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2017.05.248 · Cited by 759
LES over RANS in building simulation for outdoor and indoor applications: A foregone conclusion?
Blocken, B. (2018) · Building Simulation · doi:10.1007/s12273-018-0459-3 · Cited by 543
Metadata
- Authors
- Oberkampf, W. L., Trucano, T. G., & Hirsch, C.
- Venue
- Applied Mechanics Reviews
- Publisher
- ASME International
- Published
- 2004-09-01
- Volume
- 57
- Issue
- 5
- Pages
- 345-384
- ISSN
- 0003-6900 (ISSN-L)
- 2379-0407
- OpenAlex
- W2127951107
- Cited by
- 719
- References
- 80
- FWCI
- 38.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.