The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices
Open accessAbstract
Sensitivity analysis provides information on the relative importance of model input parameters and assumptions. It is distinct from uncertainty analysis, which addresses the question ‘How uncertain is the prediction?’ Uncertainty analysis needs to map what a model does when selected input assumptions and parameters are left free to vary over their range of existence, and this is equally true of a sensitivity analysis. Despite this, many uncertainty and sensitivity analyses still explore the input space moving along one-dimensional corridors leaving space of the input factors mostly unexplored. Our extensive systematic literature review shows that many highly cited papers (42% in the present analysis) fail the elementary requirement to properly explore the space of the input factors. The results, while discipline-dependent, point to a worrying lack of standards and recognized good practices. We end by exploring possible reasons for this problem, and suggest some guidelines for proper use of the methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Global Sensitivity Analysis. The Primer
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., & Tarantola, S. (2007) · doi:10.1002/9780470725184 · Cited by 6,430
Self-citationThe ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Factorial Sampling Plans for Preliminary Computational Experiments
Morris, M. D. (1991) · Technometrics · doi:10.1080/00401706.1991.10484804 · Cited by 4,943
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“The situation has worrying analogies with what we have witnessed in data analysis, where misuse of the p-value (Colquhoun, 2014) has been singled out as one of the reasons of the present reproducibility crisis affecting science (Ioannidis, 2005; Saltelli and Funtowicz, 2017).”
BackgroundVerification, Validation, and Confirmation of Numerical Models in the Earth Sciences
Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994) · Science · doi:10.1126/science.263.5147.641 · Cited by 3,010
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Overall, it is clear that the consequences of bad statistics can be dramatic – for example when wrong cures for cancer are identified at the pre-clinical stage of research, and are then passed on to the clinical trial phase (Begley and Ellis, 2012).”
BackgroundStructural optimization using sensitivity analysis and a level-set method
Allaire, G., Jouve, F., & Toader, A.-M. (2004) · Journal of Computational Physics · doi:10.1016/j.jcp.2003.09.032 · Cited by 2,759
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Sensitivity analysis of environmental models: A systematic review with practical workflow
Pianosi, F., Beven, K., Freer, J., Hall, J. W., Rougier, J., Stephenson, D. B., & Wagener, T. (2016) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2016.02.008 · Cited by 1,704
Statistics for experimenters: design, innovation, and discovery
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3429 · Cited by 1,644
Uncertainty and Quality in Science for Policy
Funtowicz, S. O., & Ravetz, J. R. (1990) · doi:10.1007/978-94-009-0621-1 · Cited by 1,575
Sensitivity Analysis for Importance Assessment
Saltelli, A. (2002) · Risk Analysis · doi:10.1111/0272-4332.00040 · Cited by 1,302
Self-citationTen iterative steps in development and evaluation of environmental models
Jakeman, A. J., Letcher, R. A., & Norton, J. P. (2006) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2006.01.004 · Cited by 1,111
Life itself: a comprehensive inquiry into the nature, origin, and fabrication of life
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4505 · Cited by 1,039
How to avoid a perfunctory sensitivity analysis
Saltelli, A., & Annoni, P. (2010) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.04.012 · Cited by 974
Self-citationDoubt is their product: how industry's assault on science threatens your health
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2701 · Cited by 653
Handbook of Uncertainty Quantification
(2016) · Springer eBooks · doi:10.1007/978-3-319-11259-6 · Cited by 581
(1985) · American Economic Review · Cited by 568
A framework for dealing with uncertainty due to model structure error
Refsgaard, J. C., van der Sluijs, J. P., Brown, J., & van der Keur, P. (2006) · Advances in Water Resources · doi:10.1016/j.advwatres.2005.11.013 · Cited by 535
Saltelli, A., & Tarantola, S. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502388618447 · Cited by 447
Self-citationDerivative based global sensitivity measures and their link with global sensitivity indices
Sobol’, I. M., & Kucherenko, S. (2009) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2009.01.023 · Cited by 445
Prediction: science, decision making, and the future of nature
(2001) · Choice Reviews Online · doi:10.5860/choice.38-2705 · Cited by 363
Useless arithmetic: why environmental scientists can't predict the future
(2007) · Choice Reviews Online · doi:10.5860/choice.44-6814 · Cited by 284
A methodology for meta-model based optimization in building energy models
Eisenhower, B., O’Neill, Z., Narayanan, S., Fonoberov, V. A., & Mezić, I. (2012) · Energy and Buildings · doi:10.1016/j.enbuild.2011.12.001 · Cited by 270
Sensitivity analyses would help
Fildes, R. (1986) · International Journal of Forecasting · doi:10.1016/0169-2070(86)90114-7 · Cited by 267
Tantalus on the Road to Asymptopia
Leamer, E. E. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.31 · Cited by 252
Trends in sensitivity analysis practice in the last decade
Ferretti, F., Saltelli, A., & Tarantola, S. (2016) · Science of The Total Environment · doi:10.1016/j.scitotenv.2016.02.133 · Cited by 241
Self-citationSurrogate Science: The Idol of a Universal Method for Scientific Inference
Gigerenzer, G., & Marewski, J. N. (2015) · Journal of Management · doi:10.1177/0149206314547522 · Cited by 215
Doubt Is Their Product: How Industry’s Assault on Science Threatens Your Health
Gilbert, S. G. (2009) · Environmental Health Perspectives · doi:10.1289/ehp.117-a218a · Cited by 214
Mathiness in the Theory of Economic Growth
Romer, P. M. (2015) · American Economic Review · doi:10.1257/aer.p20151066 · Cited by 201
Global sensitivity analysis with dependence measures
Da Veiga, S. (2015) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2014.945932 · Cited by 195
What is science’s crisis really about?
Saltelli, A., & Funtowicz, S. (2017) · Futures · doi:10.1016/j.futures.2017.05.010 · Cited by 187
Self-citationAn introduction to sensitivity assessment of simulation models
Norton, J. (2015) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2015.03.020 · Cited by 186
Shin, M.-J., Guillaume, J. H. A., Croke, B. F. W., & Jakeman, A. J. (2013) · Journal of Hydrology · doi:10.1016/j.jhydrol.2013.08.047 · Cited by 184
VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis
Razavi, S., Sheikholeslami, R., Gupta, H. V., & Haghnegahdar, A. (2018) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2018.10.005 · Cited by 112
On the General Theory of Skew Correlation and Non-Linear Regression
Pearson, K. (2010) · Medical Entomology and Zoology · Cited by 106
What do I make of your latinorumc Sensitivity auditing of mathematical modelling
Saltelli, A., Pereira, Â. G., van der Sluijs, J. P., & Funtowicz, S. (2013) · International Journal of Foresight and Innovation Policy · doi:10.1504/ijfip.2013.058610 · Cited by 96
Self-citationCargo-cult Statistics and Scientific Crisis
Stark, P. B., & Saltelli, A. (2018) · Significance · doi:10.1111/j.1740-9713.2018.01174.x · Cited by 84
Self-citationRomance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives?
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. F., & Puhlmann, L. M. C. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000116 · Cited by 67
Bayesian sensitivity analysis of a model of the aortic valve
Becker, W. E., Rowson, J., Oakley, J. E., Yoxall, A., Manson, G., & Worden, K. (2011) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2011.03.008 · Cited by 44
Self-citationBayesian sensitivity analysis of a nonlinear finite element model
Becker, W. E., Oakley, J. E., Surace, C., Gili, P. A., Rowson, J., & Worden, K. (2012) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2012.03.009 · Cited by 43
Self-citationEnsembles vs. information theory: supporting science under uncertainty
Nearing, G., & Gupta, H. V. (2018) · Frontiers of Earth Science · doi:10.1007/s11707-018-0709-9 · Cited by 41
Observations on the practice and profession of modeling and simulation: A survey approach
Padilla, J. J., Diallo, S. Y., Lynch, C. J., & Gore, R. J. (2017) · SIMULATION · doi:10.1177/0037549717737159 · Cited by 36
Reinert, E. S. (2000) · Journal of Economic Studies · doi:10.1108/01443580010341862 · Cited by 23
Cited Half-Life of the Journal LiteraturePreprint
Davis, P. M., & Cochran, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.07479 · Cited by 8
Cited by
The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., et al. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 628
Self-citationToward SALib 2.0: Advancing the accessibility and interpretability of global sensitivity analyses
Iwanaga, T., Usher, W., & Herman, J. (2022) · Socio-Environmental Systems Modelling · doi:10.18174/sesmo.18155 · Cited by 367
Sensitivity analysis methods in the biomedical sciences
Qian, G., & Mahdi, A. (2020) · Mathematical Biosciences · doi:10.1016/j.mbs.2020.108306 · Cited by 169
Exploratory modeling for analyzing coupled human-natural systems under uncertainty
Moallemi, E. A., Kwakkel, J., de Haan, F. J., & Bryan, B. A. (2020) · Global Environmental Change · doi:10.1016/j.gloenvcha.2020.102186 · Cited by 130
The sensitivity of simulated streamflow to individual hydrologic processes across North America
Mai, J., Craig, J. R., Tolson, B. A., & Arsenault, R. (2022) · Nature Communications · doi:10.1038/s41467-022-28010-7 · Cited by 63
Metadata
- Authors
- Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q.
- Venue
- Environmental Modelling & Software
- Publisher
- Elsevier BV
- Published
- 2019-04
- Volume
- 114
- Pages
- 29-39
- ISSN
- 1364-8152 (ISSN-L)
- OpenAlex
- W2962804139
- Cited by
- 813
- References
- 46
- FWCI
- 17.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.