The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
An overview of methods to evaluate uncertainty of deterministic models in decision support
Open accessAbstract
There is an increasing need for environmental management advice that is wide-scoped, covering various interlinked policies, and realistic about the uncertainties related to the possible management actions. To achieve this, efficient decision support integrates the results of pre-existing models. Many environmental models are deterministic, but the uncertainty of their outcomes needs to be estimated when they are utilized for decision support. We review various methods that have been or could be applied to evaluate the uncertainty related to deterministic models' outputs. We cover expert judgement, model emulation, sensitivity analysis, temporal and spatial variability in the model outputs, the use of multiple models, and statistical approaches, and evaluate when these methods are appropriate and what must be taken into account when utilizing them. The best way to evaluate the uncertainty depends on the definitions of the source models and the amount and quality of information available to the modeller.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dilemmas in a general theory of planning
Rittel, H. W. J., & Webber, M. M. (1973) · Policy Sciences · doi:10.1007/bf01405730 · Cited by 16,338
Factorial Sampling Plans for Preliminary Computational Experiments
Morris, M. D. (1991) · Technometrics · doi:10.1080/00401706.1991.10484804 · Cited by 4,943
Bayesian calibration of computer models
Kennedy, M. C., & O'Hagan, A. (2001) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00294 · Cited by 4,222
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118 · cited 1×
“Bayesian model averaging (Hoeting et al., 1999) can be used to combine and weigh several Bayesian models using their posterior probability as the weighting factor.”
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010) · Computer Physics Communications · doi:10.1016/j.cpc.2009.09.018 · Cited by 3,623
Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support
Walker, W. E., Harremoës, P., Rotmans, J., van der Sluijs, J. P., van Asselt, M. B. A., Janssen, P., & Krayer von Krauss, M. P. (2003) · Integrated Assessment · doi:10.1076/iaij.4.1.5.16466 · Cited by 2,067
An effective screening design for sensitivity analysis of large models
Campolongo, F., Cariboni, J., & Saltelli, A. (2007) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2006.10.004 · Cited by 2,011
Using Bayesian model averaging to calibrate forecast ensembles.
Raftery, A. E., Gneiting, T., Balabdaoui, F., & Polakowski, M. (2005) · Monthly Weather Review · doi:10.1175/mwr2906.1 · Cited by 1,915
The use of the multi-model ensemble in probabilistic climate projections
Tebaldi, C., & Knutti, R. (2007) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2007.2076 · Cited by 1,862
The ecological detective: confronting models with data
(1997) · Choice Reviews Online · doi:10.5860/choice.35-1501 · Cited by 1,726
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Characterising performance of environmental models
Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J., Marsili-Libelli, S., Newham, L. T. H., Norton, J. P., Perrin, C., et al. (2013) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.09.011 · Cited by 1,536
Uncertainty in the environmental modelling process – A framework and guidance
Refsgaard, J. C., van der Sluijs, J. P., Højberg, A. L., & Vanrolleghem, P. A. (2007) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2007.02.004 · Cited by 1,197
Application of multicriteria decision analysis in environmental decision making
Kiker, G. A., Bridges, T. S., Varghese, A., Seager, T. P., & Linkov, I. (2005) · Integrated Environmental Assessment and Management · doi:10.1897/ieam_2004a-015.1 · Cited by 914
Eliciting Expert Knowledge in Conservation Science
MARTIN, T. G., BURGMAN, M. A., FIDLER, F., KUHNERT, P. M., LOW‐CHOY, S., MCBRIDE, M., & MENGERSEN, K. (2012) · Conservation Biology · doi:10.1111/j.1523-1739.2011.01806.x · Cited by 829
A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY
Regan, H. M., Colyvan, M., & Burgman, M. A. (2002) · Ecological Applications · doi:10.1890/1051-0761(2002)012[0618:atatou]2.0.co;2 · Cited by 789
Bayesian analysis of computer code outputs: A tutorial
O’Hagan, A. (2006) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2005.11.025 · Cited by 755
Bayesian networks in environmental modelling
Aguilera, P. A., Fernández, A., Fernández, R., Rumí, R., & Salmerón, A. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2011.06.004 · Cited by 666
Sensitivity Analysis for Chemical Models
Saltelli, A., Ratto, M., Tarantola, S., & Campolongo, F. (2005) · Chemical Reviews · doi:10.1021/cr040659d · Cited by 639
The role of sensitivity analysis in ecological modelling
Cariboni, J., Gatelli, D., Liska, R., & Saltelli, A. (2007) · Ecological Modelling · doi:10.1016/j.ecolmodel.2005.10.045 · Cited by 563
Ecological risk assessment for the effects of fishing
Hobday, A. J., Smith, A. D. M., Stobutzki, I. C., Bulman, C., Daley, R., Dambacher, J. M., Deng, R. A., Dowdney, J., Fuller, M., Furlani, D., et al. (2011) · Fisheries Research · doi:10.1016/j.fishres.2011.01.013 · Cited by 563
Risk Assessment and Decision Analysis with Bayesian Networks
Fenton, N., & Neil, M. (2012) · CRC Press eBooks · doi:10.1201/9780367803018 · Cited by 531
A Bayesian network of eutrophication models for synthesis, prediction, and uncertainty analysis
Borsuk, M. E., Stow, C. A., & Reckhow, K. H. (2004) · Ecological Modelling · doi:10.1016/j.ecolmodel.2003.08.020 · Cited by 492
Integrated environmental modeling: A vision and roadmap for the future
Laniak, G. F., Olchin, G., Goodall, J., Voinov, A., Hill, M., Glynn, P., Whelan, G., Geller, G., Quinn, N., Blind, M., et al. (2013) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.09.006 · Cited by 478
Marcot, B. G., Holthausen, R. S., Raphael, M. G., Rowland, M. M., & Wisdom, M. J. (2001) · Forest Ecology and Management · doi:10.1016/s0378-1127(01)00452-2 · Cited by 417
The Role of Expert Opinion in Environmental Modelling
Krueger, T., Page, T., Hubacek, K., Smith, L., & Hiscock, K. (2012) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.01.011 · Cited by 413
Ensemble modelling of species distribution: the effects of geographical and environmental ranges
Grenouillet, G., Buisson, L., Casajus, N., & Lek, S. (2011) · Ecography · doi:10.1111/j.1600-0587.2010.06152.x · Cited by 409
(1967) · Management Science · doi:10.1287/mnsc.14.4.b141 · Cited by 402
Risk Assessment and Decision Analysis with Bayesian Networks
Fenton, N. (2012) · CRC Press eBooks · doi:10.1201/b13102 · Cited by 388
Scientific tools to support the practical implementation of ecosystem-based fisheries management
Smith, A. D. M., Fulton, E. J., Hobday, A. J., Smith, D. C., & Shoulder, P. (2007) · ICES Journal of Marine Science · doi:10.1093/icesjms/fsm041 · Cited by 386
Towards the characterization of streamflow simulation uncertainty through multimodel ensembles
Georgakakos, K. P., Seo, D.-J., Gupta, H., Schaake, J., & Butts, M. B. (2004) · Journal of Hydrology · doi:10.1016/j.jhydrol.2004.03.037 · Cited by 352
Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis
Yang, J. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.10.007 · Cited by 326
Environmental decision support systems (EDSS) development – Challenges and best practices
McIntosh, B. S., Ascough, J. C., Twery, M., Chew, J., Elmahdi, A., Haase, D., Harou, J. J., Hepting, D., Cuddy, S., Jakeman, A. J., et al. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2011.09.009 · Cited by 325
Limits for the Precision and Value of Information from Dependent Sources
Clemen, R. T., & Winkler, R. L. (1985) · Operations Research · doi:10.1287/opre.33.2.427 · Cited by 303
Evaluating uncertainty in integrated environmental models: A review of concepts and tools
Matott, L. S., Babendreier, J. E., & Purucker, S. T. (2009) · Water Resources Research · doi:10.1029/2008wr007301 · Cited by 286
Probabilistic Wind Vector Forecasting Using Ensembles and Bayesian Model Averaging
McLean Sloughter, J., Gneiting, T., & Raftery, A. E. (2013) · Monthly Weather Review · doi:10.1175/mwr-d-12-00002.1 · Cited by 210
Baroni, G., & Tarantola, S. (2014) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2013.09.022 · Cited by 173
Emulation of a complex global aerosol model to quantify sensitivity to uncertain parameters
Lee, L. A., Carslaw, K. S., Pringle, K. J., Mann, G. W., & Spracklen, D. V. (2011) · Atmospheric Chemistry and Physics · doi:10.5194/acp-11-12253-2011 · Cited by 164
Borsuk, M. E., Higdon, D., Stow, C. A., & Reckhow, K. H. (2001) · Ecological Modelling · doi:10.1016/s0304-3800(01)00328-3 · Cited by 154
A general framework for Dynamic Emulation Modelling in environmental problems
Castelletti, A. F., Galelli, S., Ratto, M., Soncini‐Sessa, R., & Young, P. C. (2012) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.01.002 · Cited by 150
Chu, M. L., Muñoz‐Carpena, R., Kiker, G. A., et al. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.12.003 · Cited by 149
Vialaneix, N., Follador, M., Ratto, M., & Leip, A. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2011.05.003 · Cited by 144
Case studies in Gaussian process modelling of computer codes
Kennedy, M. C., Anderson, C. W., Conti, S., & O’Hagan, A. (2006) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2005.11.028 · Cited by 131
O’Hagan, A. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2011.03.003 · Cited by 127
Bayesian decision analysis for environmental and resource management
Varis, O. (1997) · Environmental Modelling & Software · doi:10.1016/s1364-8152(97)00008-x · Cited by 119
An Introduction to the Practice of Ecological Modeling
Jackson, L. J., Trebitz, A. S., & Cottingham, K. L. (2000) · BioScience · doi:10.1641/0006-3568(2000)050[0694:aittpo]2.0.co;2 · Cited by 118
Does high forecast uncertainty preclude effective decision support?
Reichert, P., & Borsuk, M. E. (2005) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2004.10.005 · Cited by 102
Advancing projections of phytoplankton responses to climate change through ensemble modelling
Trolle, D., Elliott, J. A., Mooij, W. M., Janse, J. H., Bolding, K., Hamilton, D. P., & Jeppesen, E. (2014) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2014.01.032 · Cited by 100
Tomassini, L., Reichert, P., Knutti, R., Stocker, T. F., & Borsuk, M. E. (2007) · Journal of Climate · doi:10.1175/jcli4064.1 · Cited by 84
Ensemble modelling of nutrient loads and nutrient load partitioning in 17 European catchments
Kronvang, B., Behrendt, H., Andersen, H. E., Arheimer, B., Barr, A. D., Borgvang, S. A., Bouraoui, F., Granlund, K., Grizzetti, B., Groenendijk, P., et al. (2009) · Journal of Environmental Monitoring · doi:10.1039/b900101h · Cited by 79
Petropoulos, G. P., Wooster, M. J., Carlson, T. N., Kennedy, M. C., & Scholze, M. (2009) · Ecological Modelling · doi:10.1016/j.ecolmodel.2009.06.006 · Cited by 69
Estimation of Atlantic salmon smolt carrying capacity of rivers using expert knowledge
Uusitalo, L., Kuikka, S., & Romakkaniemi, A. (2005) · ICES Journal of Marine Science · doi:10.1016/j.icesjms.2005.02.005 · Cited by 67
Self-citationBormann, H., Breuer, L., Gräff, T., Huisman, J. A., & Croke, B. (2008) · Advances in Water Resources · doi:10.1016/j.advwatres.2008.01.002 · Cited by 64
Galatians, crucified, C., & Paul (2010) · Cited by 61
Quantifying Uncertainty in the Biospheric Carbon Flux for England and Wales
Kennedy, M. C., Anderson, C. W., O’Hagan, A., Lomas, M. R., Woodward, I., Gosling, J. P., & Heinemeyer, A. (2007) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2007.00489.x · Cited by 60
Gal, G., Makler-Pick, V., & Shachar, N. (2014) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2014.05.015 · Cited by 56
Kiirikki, M., Lehtoranta, J., Inkala, A., Pitkänen, H., Hietanen, S., Hall, P., Tengberg, A., Koponen, J., & Sarkkula, J. (2006) · Journal of Marine Systems · doi:10.1016/j.jmarsys.2006.02.008 · Cited by 55
Parry, H. R., Topping, C. J., Kennedy, M. C., Boatman, N. D., & Murray, A. W. A. (2012) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.08.006 · Cited by 52
Skinner, D. J. C., Rocks, S. A., Pollard, S., & Drew, G. H. (2013) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.2013.779899 · Cited by 44
Bridging uncertain and ambiguous knowledge with imprecise probabilities
Rinderknecht, S., Borsuk, M. E., & Reichert, P. (2011) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2011.07.022 · Cited by 44
A long-term sensitivity analysis of the denitrification and decomposition model
Qin, X., Wang, H., Li, Y., Li, Y., McConkey, B. G., Lemke, R., Li, C., Brandt, K., Gao, Q., Wan, Y., et al. (2013) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2013.01.005 · Cited by 42
Ensemble modelling of nitrogen fluxes: data fusion for a Swedish meso-scale catchment
Exbrayat, J., Viney, N. R., Seibert, J., Wrede, S., Frede, H. ‐., & Breuer, L. (2010) · Hydrology and earth system sciences · doi:10.5194/hess-14-2383-2010 · Cited by 30
Mäntyniemi, S., Uusitalo, L., Peltonen, H., Haapasaari, P., & Kuikka, S. (2013) · Canadian Journal of Fisheries and Aquatic Sciences · doi:10.1139/cjfas-2012-0315 · Cited by 29
Self-citationEnvironmental Risk Management Decision-Making in a Societal Context
Power, M. J., & McCarty, L. S. (2006) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807030500428538 · Cited by 17
Vanhatalo, J. P., Tuomi, L., Inkala, A., Helle, I., & Pitkänen, J. H. (2012) · Environmental Science & Technology · doi:10.1021/es302475v · Cited by 11
Self-citation
Cited by
A survey on missing data in machine learning
Emmanuel, T., Maupong, T., Mpoeleng, D., Semong, T., Mphago, B., & Tabona, O. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00516-9 · Cited by 1,067
Kreps, S. E., & Kriner, D. L. (2020) · Science Advances · doi:10.1126/sciadv.abd4563 · Cited by 328
“Models, therefore, are not “true/false but better/worse, and the possible discrepancy regarding the problem or its parts can have several plausible explanations” (5).”
BackgroundA review of approaches to uncertainty assessment in energy system optimization models
Yue, X., Pye, S., DeCarolis, J., Li, F. G. N., Rogan, F., & Gallachóir, B. Ó. (2018) · Energy Strategy Reviews · doi:10.1016/j.esr.2018.06.003 · Cited by 256
Sexton, J., Everingham, Y. L., & Inman-Bamber, G. (2017) · European Journal of Agronomy · doi:10.1016/j.eja.2015.11.009 · Cited by 53
A Bayesian Belief Network to assess rate of changes in coral reef ecosystems
Franco, C., Hepburn, L. A., Smith, D. J., Nimrod, S., & Tucker, A. (2016) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2016.02.029 · Cited by 25
Metadata
- Authors
- Uusitalo, L., Lehikoinen, A., Helle, I., & Myrberg, K.
- Venue
- Environmental Modelling & Software
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 63
- Pages
- 24-31
- ISSN
- 1364-8152 (ISSN-L)
- OpenAlex
- W2075966418
- Cited by
- 538
- References
- 65
- FWCI
- 28.51
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.