Cited by 387Hydrology and Watershed Management StudiesMeteorological Phenomena and SimulationsGroundwater flow and contamination studies

The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.

Appraisal of the generalized likelihood uncertainty estimation (GLUE) method

Open access

Stedinger, Jery R., Vogel, Richard M., Lee, Seung Uk, Batchelder, Rebecca·2008·Water Resources Research

DOI 10.1029/2008wr006822

Open this paper in CoChat — free account Open PDF

Abstract

Recent research documents that the widely accepted generalized likelihood uncertainty estimation (GLUE) method for describing forecasting precision and the impact of parameter uncertainty in rainfall/runoff watershed models fails to achieve the intended purpose when used with an informal likelihood measure. In particular, GLUE generally fails to produce intervals that capture the precision of estimated parameters, and the difference between predictions and future observations. This paper illustrates these problems with GLUE using a simple linear rainfall/runoff model so that model calibration is a linear regression problem for which exact expressions for prediction precision and parameter uncertainty are well known and understood. The simple regression example enables us to clearly and simply illustrate GLUE deficiencies. Beven and others have suggested that the choice of the likelihood measure used in a GLUE computation is subjective and may be selected to reflect the goals of the modeler. If an arbitrary likelihood is adopted that does not reasonably reflect the sampling distribution of the model errors, then GLUE generates arbitrary results without statistical validity that should not be used in scientific work. The traditional subjective likelihood measures that have been used with GLUE also fail to reflect the nonnormality, heteroscedasticity, and serial correlation among the residual errors generally found in real problems, and hence are poor metrics for even simple sensitivity analyses and model calibration. Most previous applications of GLUE only produce uncertainty intervals for the average model prediction, which by construction should not be expected to include future observations with the prescribed probability. We show how the GLUE methodology when properly implemented with a statistically valid likelihood function can provide prediction intervals for future observations which will agree with widely accepted and statistically valid analyses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1029/2008wr006822

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

River flow forecasting through conceptual models part I — A discussion of principles
Nash, J. E., & Sutcliffe, J. V. (1970) · Journal of Hydrology · doi:10.1016/0022-1694(70)90255-6 · Cited by 24,401
The Advanced Theory of Statistics.
Clarke, G. M., Kendall, M. G., & Stuart, A. (1978) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344782 · Cited by 6,727
The future of distributed models: model calibration and uncertainty prediction
Beven, K., & Binley, A. (1992) · Hydrological Processes · doi:10.1002/hyp.3360060305 · Cited by 4,527
A manifesto for the equifinality thesis
Beven, K. J. (2005) · Journal of Hydrology · doi:10.1016/j.jhydrol.2005.07.007 · Cited by 2,634
Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology
Beven, K., & Freer, J. (2001) · Journal of Hydrology · doi:10.1016/s0022-1694(01)00421-8 · Cited by 2,189
An Introduction to Bayesian Inference in Econometrics.
Sargan, J. D., & Zellner, A. (1974) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344559 · Cited by 1,938
An Introduction to Bayesian Inference in Econometrics.
Aigner, D. J., & Zellner, A. (1972) · Journal of the American Statistical Association · doi:10.2307/2284674 · Cited by 1,687
Rainfall-Runoff Modelling: The Primer
Beven, K. (2012) · Wiley eBooks · doi:10.1002/9781119951001 · Cited by 1,640
Dual state–parameter estimation of hydrological models using ensemble Kalman filter
Moradkhani, H., Sorooshian, S., Gupta, H. V., & Houser, P. R. (2004) · Advances in Water Resources · doi:10.1016/j.advwatres.2004.09.002 · Cited by 927
Time series analysis: forecasting and control.
Anderson, O. D. (1980) · Journal of the Franklin Institute · doi:10.1016/0016-0032(78)90015-7 · Cited by 927
Dynamically dimensioned search algorithm for computationally efficient watershed model calibration
Tolson, B. A., & Shoemaker, C. A. (2007) · Water Resources Research · doi:10.1029/2005wr004723 · Cited by 898
Uncertainty assessment of hydrologic model states and parameters: Sequential data assimilation using the particle filter
Moradkhani, H., Hsu, K., Gupta, H. V., & Sorooshian, S. (2005) · Water Resources Research · doi:10.1029/2004wr003604 · Cited by 811
Monte Carlo assessment of parameter uncertainty in conceptual catchment models: the Metropolis algorithm
Kuczera, G. A., & Parent, É. (1998) · Journal of Hydrology · doi:10.1016/s0022-1694(98)00198-x · Cited by 803
Bayesian Estimation of Uncertainty in Runoff Prediction and the Value of Data: An Application of the GLUE Approach
Freer, J. E., Beven, K., & Ambroise, B. (1996) · Water Resources Research · doi:10.1029/95wr03723 · Cited by 764
Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis
Spear, R. C. (1980) · Water Research · doi:10.1016/0043-1354(80)90040-8 · Cited by 621
Improved treatment of uncertainty in hydrologic modeling: Combining the strengths of global optimization and data assimilation
Vrugt, J. A., Diks, C. G. H., Gupta, H. V., Bouten, W., & Verstraten, J. M. (2005) · Water Resources Research · doi:10.1029/2004wr003059 · Cited by 613
An integrated hydrologic Bayesian multimodel combination framework: Confronting input, parameter, and model structural uncertainty in hydrologic prediction
Ajami, N. K., Duan, Q., & Sorooshian, S. (2007) · Water Resources Research · doi:10.1029/2005wr004745 · Cited by 597
Bayesian analysis of input uncertainty in hydrological modeling: 1. Theory
Kavetski, D., Kuczera, G. A., & Franks, S. W. (2006) · Water Resources Research · doi:10.1029/2005wr004368 · Cited by 582
Inverse problem in hydrogeology
Carrera, J., Alcolea, A., Medina, A., Hidalgo, J. J., & Slooten, L. J. (2005) · Hydrogeology Journal · doi:10.1007/s10040-004-0404-7 · Cited by 551
Uncertainty in the calibration of effective roughness parameters in HEC-RAS using inundation and downstream level observations
Pappenberger, F., Beven, K. J., Horritt, M. S., & Blazkova, S. D. (2004) · Journal of Hydrology · doi:10.1016/j.jhydrol.2004.06.036 · Cited by 516
Bayesian analysis of input uncertainty in hydrological modeling: 2. Application
Kavetski, D., Kuczera, G. A., & Franks, S. W. (2006) · Water Resources Research · doi:10.1029/2005wr004376 · Cited by 463
An Introduction to Bayesian Inference in Econometrics
Good, I. J. (1975) · Technometrics · doi:10.1080/00401706.1975.10489285 · Cited by 452
Bayesian recursive parameter estimation for hydrologic models
Thiemann, M., Trosset, M. W., Gupta, H. V., & Sorooshian, S. (2001) · Water Resources Research · doi:10.1029/2000wr900405 · Cited by 431
Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology
Mantovan, P., & Todini, E. (2006) · Journal of Hydrology · doi:10.1016/j.jhydrol.2006.04.046 · Cited by 419
Stochastic parameter estimation procedures for hydrologie rainfall‐runoff models: Correlated and heteroscedastic error cases
Sorooshian, S., & Dracup, J. A. (1980) · Water Resources Research · doi:10.1029/wr016i002p00430 · Cited by 396
Calibration of Watershed Models
Duan, Q. (2003) · Water science and application · doi:10.1029/ws006 · Cited by 345
Generalized likelihood uncertainty estimation (GLUE) using adaptive Markov Chain Monte Carlo sampling
Blasone, R.-S., Vrugt, J. A., Madsen, H., Rosbjerg, D., Robinson, B. A., & Zyvoloski, G. A. (2008) · Advances in Water Resources · doi:10.1016/j.advwatres.2007.12.003 · Cited by 338
Towards a Bayesian total error analysis of conceptual rainfall-runoff models: Characterising model error using storm-dependent parameters
Kuczera, G. A., Kavetski, D., Franks, S. W., & Thyer, M. (2006) · Journal of Hydrology · doi:10.1016/j.jhydrol.2006.05.010 · Cited by 335
Rainfall-Runoff Modelling
Minns, A. W., & Hall, M. (2004) · Taylor & Francis eBooks · doi:10.1201/9780203024119.ch9 · Cited by 334
Inference and relationship
Stuart, A., & Kendall, M. G. (1979) · Medical Entomology and Zoology · Cited by 269
So just why would a modeller choose to be incoherent?
Beven, K. J., Smith, P. J., & Freer, J. E. (2008) · Journal of Hydrology · doi:10.1016/j.jhydrol.2008.02.007 · Cited by 261
What do we mean by ‘uncertainty’? The need for a consistent wording about uncertainty assessment in hydrology
Montanari, A. (2007) · Hydrological Processes · doi:10.1002/hyp.6623 · Cited by 250
A comparative study of Markov chain Monte Carlo methods for conceptual rainfall‐runoff modeling
Marshall, L. A., Nott, D. J., & Sharma, A. (2004) · Water Resources Research · doi:10.1029/2003wr002378 · Cited by 243
Large sample behaviors of the generalized likelihood uncertainty estimation (GLUE) in assessing the uncertainty of rainfall‐runoff simulations
Montanari, A. (2005) · Water Resources Research · doi:10.1029/2004wr003826 · Cited by 231
Confronting input uncertainty in environmental modelling
Kavetski, D., Franks, S. W., & Kuczera, G. A. (2003) · Water science and application · doi:10.1029/ws006p0049 · Cited by 227
Uncertainty assessment of integrated distributed hydrological models using GLUE with Markov chain Monte Carlo sampling
Blasone, R.-S., Madsen, H., & Rosbjerg, D. (2008) · Journal of Hydrology · doi:10.1016/j.jhydrol.2007.12.026 · Cited by 199
Sensitivity analysis in model calibration: GSA-GLUE approach
Ratto, M., Tarantola, S., & Saltelli, A. (2001) · Computer Physics Communications · doi:10.1016/s0010-4655(01)00159-x · Cited by 188
On undermining the science?
Beven, K. J. (2006) · Hydrological Processes · doi:10.1002/hyp.6396 · Cited by 179
Degrees of Belief
Hendricks, V. F., Kuipers, T. A. F., Schmidt-Petri, C., Dalen, D., Hintikka, J., Seidenfeld, T., Woleński, J., Suppes, P., Symons, J., & Huber, F. (2008) · Synthese Library/Synthese library · doi:10.1007/978-1-4020-9198-8 · Cited by 151
Reply to comment regarding the ICE-hypothesis
Nielsen, S. B., Gallagher, K., Egholm, D. L., Clausen, O. R., & Summerfield, M. A. (2009) · Journal of Geodynamics · doi:10.1016/j.jog.2009.06.004 · Cited by 130
An Introduction to Bayesian Inference
Sørensen, D., & Gianola, D. (2002) · Statistics for biology and health · doi:10.1007/0-387-22764-4_5 · Cited by 127
Quantifying hydrological modeling errors through a mixture of normal distributions
Schaefli, B., Talamba, D. B., & Musy, A. (2006) · Journal of Hydrology · doi:10.1016/j.jhydrol.2006.07.005 · Cited by 114
Modelling the effect of fire-exclusion and prescribed fire on wildfire size in Mediterranean ecosystems
Piñol, J., Beven, K. J., & Viegas, D. X. (2004) · Ecological Modelling · doi:10.1016/j.ecolmodel.2004.09.001 · Cited by 110
Comment on “Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology” by Pietro Mantovan and Ezio Todini
Beven, K. J., Smith, P. J., & Freer, J. E. (2007) · Journal of Hydrology · doi:10.1016/j.jhydrol.2007.02.023 · Cited by 100
Including prior information in the estimation of effective soil parameters in unsaturated zone modelling
Mertens, J., Madsen, H., Feyen, L., Jacques, D., & Feyen, J. (2004) · Journal of Hydrology · doi:10.1016/j.jhydrol.2004.02.011 · Cited by 98
Dynamic real-time prediction of flood inundation probabilities
Romanowicz, R. J., & Beven, K. J. (1998) · Hydrological Sciences Journal · doi:10.1080/02626669809492117 · Cited by 98
Bayesian estimation of parameters in a regional hydrological model
Engeland, K., & Gottschalk, L. (2002) · Hydrology and earth system sciences · doi:10.5194/hess-6-883-2002 · Cited by 85
A Bayesian perspective on input uncertainty in model calibration: Application to hydrological model “abc”
Huard, D., & Mailhot, A. (2006) · Water Resources Research · doi:10.1029/2005wr004661 · Cited by 84
Assessing the impacts of parameter uncertainty for computationally expensive groundwater models
Mugunthan, P., & Shoemaker, C. A. (2006) · Water Resources Research · doi:10.1029/2005wr004640 · Cited by 83
Modeling the Effect of Algal Dynamics on Arsenic Speciation in Lake Biwa
Hellweger, F. L., & Lall, U. (2004) · Environmental Science & Technology · doi:10.1021/es049660k · Cited by 81
On the value of experimental data to reduce the prediction uncertainty of a process-oriented catchment model
Uhlenbrook, S., & Sieber, A. (2004) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2003.12.006 · Cited by 78
Efficient prediction uncertainty approximation in the calibration of environmental simulation models
Tolson, B. A., & Shoemaker, C. A. (2008) · Water Resources Research · doi:10.1029/2007wr005869 · Cited by 76
Unraveling uncertainties in hydrologic model calibration: Addressing the problem of compensatory parameters
Clark, M., & Vrugt, J. A. (2006) · Geophysical Research Letters · doi:10.1029/2005gl025604 · Cited by 72
Interflow
Beven, K. J. (1989) · Springer eBooks · doi:10.1007/978-94-009-2352-2_7 · Cited by 71
Uncertainty in the estimation of critical loads: A practical methodology
Zak, S. K., Beven, K. J., & Reynolds, B. (1997) · Water Air & Soil Pollution · doi:10.1007/bf02047040 · Cited by 70
Comparison of parameter estimation methods for crop models
Tremblay‐Franco, M., & Wallach, D. (2004) · Agronomie · doi:10.1051/agro:2004033 · Cited by 68
The use of generalised likelihood measures for uncertainty estimation in high order models of environmental systems
Beven, K. J., Freer, J. E., Hankin, B., & Schulz, K. (2000) · Bristol Research (University of Bristol) · Cited by 61
A multi-group model of Schistosoma japonicum transmission dynamics and control: model calibration and control prediction
Liang, S., Spear, R. C., Seto, E., Hubbard, A., & Qiu, D. (2005) · Tropical Medicine & International Health · doi:10.1111/j.1365-3156.2005.01386.x · Cited by 55
Multi-objective parameter conditioning of a three-source wheat canopy model
Mo, X., & Beven, K. J. (2003) · Agricultural and Forest Meteorology · doi:10.1016/j.agrformet.2003.09.009 · Cited by 53
Reply to comment by Keith Beven, Paul Smith and Jim Freer on “Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology”
Mantovan, P., Todini, E., & Martina, M. L. V. (2007) · Journal of Hydrology · doi:10.1016/j.jhydrol.2007.02.029 · Cited by 52
Identifying storm flow pathways in a rainforest catchment using hydrological and geochemical modelling
Kinner, D. A., & Stallard, R. F. (2004) · Hydrological Processes · doi:10.1002/hyp.1498 · Cited by 49
A maximum likelihood criterion for use with data collected at unequal time intervals
Duan, Q., Sorooshian, S., & Ibbitt, R. P. (1988) · Water Resources Research · doi:10.1029/wr024i007p01163 · Cited by 44
A synthetic groundwater modelling study of the accuracy of GLUE uncertainty intervals
Christensen, S. (2004) · Hydrology research · doi:10.2166/nh.2004.0004 · Cited by 43
Uncertainty Analysis for Watershed Modeling Using Generalized Likelihood Uncertainty Estimation with Multiple Calibration Measures
Jia, Y., & Culver, T. B. (2008) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)0733-9496(2008)134:2(97 · Cited by 41
Predictive uncertainty of a nearshore bed evolution model
Ruessink, G. (2005) · Continental Shelf Research · doi:10.1016/j.csr.2004.12.007 · Cited by 36
Stochastic Streamflow Simulation Using SAMS-2003
Salas, J. D., Sveinsson, Ó. G. B., Lane, W. L., & Frevert, D. K. (2006) · Journal of Irrigation and Drainage Engineering · doi:10.1061/(asce)0733-9437(2006)132:2(112 · Cited by 24
Uncertainty in modelled estimates of acid deposition across Wales: a GLUE approach
Page, T., Whyatt, D., Beven, K. J., & Metcalfe, S. E. (2004) · Atmospheric Environment · doi:10.1016/j.atmosenv.2004.01.029 · Cited by 23
Evaluation of two hybrid metric-conceptual models for simulating phosphorus transfer from agricultural land in the river enborne, a lowland UK catchment
Smith, R. M. S., Evans, D. J., & Wheater, H. S. (2004) · Journal of Hydrology · doi:10.1016/j.jhydrol.2004.07.046 · Cited by 20
Improved Prediction Methods for Wildfires Using High Performance Computing: A Comparison
Bianchini, G., Cortés, A., Margalef, T., & Emilio Luque, A. (2006) · Lecture notes in computer science · doi:10.1007/11758501_73 · Cited by 15

Research works printed but not linked: 5

10.1016/j.hydrol.2008.02.007 · DOI
P. J. Bickel K. A. Doksum 2001 Prentice Hall Upper Saddle River N. J.
Carrera J., 2005, The inverse problem in hydrology, J. Hydrol., 13, 206
Romanowicz R., 1994, Statistics for the Environment II:Water Related Issues, 297
Romanowicz R., 1996, Floodplain Processes, 333

Sorted by citations

A decade of Predictions in Ungauged Basins (PUB)—a review
Hrachowitz, M., Savenije, H. H. G., Blöschl, G., McDonnell, J. J., Sivapalan, M., Pomeroy, J. W., Arheimer, B., Blume, T., Clark, M., Ehret, U., et al. (2013) · Hydrological Sciences Journal · doi:10.1080/02626667.2013.803183 · Cited by 1,321
Understanding predictive uncertainty in hydrologic modeling: The challenge of identifying input and structural errors
Renard, B., Kavetski, D., Kuczera, G., Thyer, M., & Franks, S. W. (2010) · Water Resources Research · doi:10.1029/2009wr008328 · Cited by 935
What Role Does Hydrological Science Play in the Age of Machine Learning?
Nearing, G. S., Kratzert, F., Sampson, A. K., Pelissier, C. S., Klotz, D., Frame, J. M., Prieto, C., & Gupta, H. V. (2021) · Water Resources Research · doi:10.1029/2020wr028091 · Cited by 894
Pursuing the method of multiple working hypotheses for hydrological modeling
Clark, M., Kavetski, D., & Fenicia, F. (2011) · Water Resources Research · doi:10.1029/2010wr009827 · Cited by 602
A formal likelihood function for parameter and predictive inference of hydrologic models with correlated, heteroscedastic, and non‐Gaussian errors
Schoups, G., & Vrugt, J. A. (2010) · Water Resources Research · doi:10.1029/2009wr008933 · Cited by 571

See all 387

Authors: Stedinger, J. R., Vogel, R. M., Lee, S. U., & Batchelder, R.
Affiliations: Cornell University (US)
Tufts University (US)
ORCID: Stedinger, Jery R.
Vogel, Richard M.
Venue: Water Resources Research
Publisher: American Geophysical Union (AGU)
Published: 2008-11
Volume: 44
Issue: 12
ISSN: 0043-1397 (ISSN-L)
1944-7973
DOI: 10.1029/2008wr006822
OpenAlex: W2155633525
Cited by: 386
References: 69
Citations by year: 2026: 6 · 2025: 11 · 2024: 10 · 2023: 8 · 2022: 21 · 2021: 21 · 2020: 29 · 2019: 29 · 2018: 28 · 2017: 15 · 2016: 29 · 2015: 29 · 2014: 31 · 2013: 33 · 2012: 34
FWCI: 18.29
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations: Water Resources Research (publishedVersion)
Topics: Physical Sciences › Environmental Science › Water Science and Technology › Hydrology and Watershed Management Studies
Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
Physical Sciences › Environmental Science › Environmental Engineering › Groundwater flow and contamination studies
Keywords: generalized likelihood uncertainty estimation, adhesives, rainfall-runoff modeling, parameter uncertainty, prediction intervals, likelihood function, model calibration, linear regression, forecast uncertainty, heteroscedasticity, serial correlation, uncertainty intervals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Appraisal of the generalized likelihood uncertainty estimation (GLUE) method

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks