CoChat

Objective functions used as performance metrics for hydrological models: state-of-the-art and critical analysis

Ferreira, Paloma Mara de Lima, Paz, Adriano Rolim da, Bravo, Juan Martín · 2020 · RBRH
Cited by 21

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Abstract

Hydrological models (HMs) can be applied for different purposes, and a key step is model calibration using objective functions (OF) to quantify the agreement between observed and calculated discharges. Fully understanding the OF is important to properly take advantage of model calibration and interpret the results. This study evaluates 36 OF proposed in the literature, considering two watersheds of different hydrological regimes. Daily simulated streamflow time-series, using a distributed hydrological model (MGB-IPH), and ten daily streamflow synthetic time-series, generated from the observed and calculated streamflows, were used in the analysis of each watershed. These synthetic data were used to evaluate how does each metric evaluate hypothetical cases that present isolated very well known error behaviors. Despite of all NSE-derived (Nash-Sutcliffe efficiency) metrics that use the square of the residuals in their formulation have shown higher sensitivity to errors in high flows, the ones that use daily and monthly averages of flow rates in absolute terms were more stringent than the others to assess HMs performance. Low flow errors were better evaluated by metrics that use the flow logarithm. The constant presence of zero flow rates deteriorate them significantly, with the exception of the metrics TRMSE (Transformed root mean square error) did not demonstrate this problem. An observed limitation of the formulations of some metrics was that the errors of overestimation or underestimation are compensated. Our results reassert that each metric should be interpreted specifically thinking about the aspects it has been proposed for, and simultaneously taking into account a set of metrics would lead to a broader evaluation of HM ability (e.g. multiobjective model evaluation). We recommend that the use of synthetic time series as those proposed in this work could be useful as an auxiliary step towards better understanding the evaluation of a calibrated hydrological model for each study case, taking into account model capabilities and observed hydrologic regime characteristics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1590/2318-0331.252020190155

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 49

  1. Akhtar M., 2009, Use of regional climate model simulations as input for hydrological models for the Hindukush–Karakorum–Himalaya region, Hydrology and Earth System Sciences, 13, 1075, 10.5194/hess-13-1075-2009 · DOI
  2. Boyle D. P., 2000, Toward improved calibration of hydrologic models: combining the strengths of manual and automatic methods, Water Resources Research, 36, 3663, 10.1029/2000WR900207 · DOI · Paper page
  3. Bravo J. M., 2009, Incorporating forecasts of rainfall in two hydrologic models used for medium-range streamflow forecasting, Journal of Hydrologic Engineering, 14, 435, 10.1061/(ASCE)HE.1943-5584.0000014 · doi:10.1061/(asce)he.1943-5584.0000014
  4. Collischonn W., 2007, The MGB-IPH model for large-scale rainfall-runoff modelling, Hydrological Sciences Journal, 52, 878, 10.1623/hysj.52.5.878 · DOI
  5. Criss R. E., 2008, Do Nash values have value? Discussion and alternate proposals, Hydrological Processes, 22, 2723, 10.1002/hyp.7072 · DOI
  6. Dakhlaoui H., 2012, Toward a more efficient Calibration Schema for HBV rainfall-runoff model, Journal of Hydrology (Amsterdam), 444-445, 161, 10.1016/j.jhydrol.2012.04.015 · doi:10.1016/j.jhydrol.2012.04.015
  7. Efstratiadis A., 2010, One decade of multi-objective calibration approaches in hydrological modelling: a review Multi-objective calibration approaches in hydrological modelling, Hydrological Sciences Journal, 55, 58, 10.1080/02626660903526292 · DOI · Paper page
  8. Felix W. S., 2016, Representação dos processos hidrológicos em bacia hidrográfica do semiárido paraibano com modelagem hidrológica distribuída, Revista Brasileira de Recursos Hídricos- RBRH, 21, 556
  9. Fenícia F., 2007, Soft combination of local models in a multi-objective framework, Hydrology and Earth System Sciences, 11, 1797, 10.5194/hess-11-1797-2007 · DOI
  10. Fowler K., 2018, Improved rainfall-runoff calibration for drying climate: choice of objective function, Water Resources Research, 54, 3392, 10.1029/2017WR022466 · DOI
  11. Garcia F., 2017, Which objective function to calibrate rainfall–runoff models for low-flow index simulations?, Hydrological Sciences Journal, 62, 1149, 10.1080/02626667.2017.1308511 · DOI
  12. Gupta H. V., 1998, Toward improved calibration of hydrologic models: multiple and noncommensurable measures of information, Water Resources Research, 34, 751, 10.1029/97WR03495 · DOI · Paper page
  13. Gupta H. V., 1999, Status of automatic calibration for hydrologic models: comparison with multilevel expert calibration, Journal of Hydrologic Engineering, 4, 135, 10.1061/(ASCE)1084-0699(1999)4:2(135) · doi:10.1061/(asce)1084-0699(1999)4:2(135 · Paper page
  14. Gupta H. V., 2009, Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling, Journal of Hydrology (Amsterdam), 377, 80, 10.1016/j.jhydrol.2009.08.003 · DOI · Paper page
  15. Gutierrez J. C. T., 2019, A new stopping criterion for multi-objective evolutionary algorithms: application in the calibration of a hydrologic model, Computational Geosciences, 23, 1219, 10.1007/s10596-019-09870-3 · doi:10.1007/s10596-019-09870-3
  16. Herman M. R., 2018, Evaluating the role of evapotranspiration remote sensing data in improving hydrological modeling predictability, Journal of Hydrology (Amsterdam), 556, 39, 10.1016/j.jhydrol.2017.11.009 · DOI · Paper page
  17. Hogue T. S., 2000, A multistep automatic calibration scheme for river forecasting models, Journal of Hydrometeorology, 1, 524, 10.1175/1525-7541(2000)001<0524:AMACSF>2.0.CO;2 · DOI
  18. Janssen P. H. M., 1995, Calibration of process-oriented models, Ecological Modelling, 83, 55, 10.1016/0304-3800(95)00084-9 · DOI · Paper page
  19. Kollat J. B., 2012, When are multiobjective calibration trade-offs in hydrologic models meaningful?, Water Resources Research, 48, 1, 10.1029/2011WR011534 · DOI
  20. Krause P., 2005, Comparison of different efficiency criteria for hydrological model assessment, Advances in Geosciences, 5, 89, 10.5194/adgeo-5-89-2005 · DOI · Paper page
  21. Legates D. R., 1999, Evaluating the use of “goodness-of-fit” measures in hydrologicv and hydroclimatic model validation, Water Resources Research, 35, 233, 10.1029/1998WR900018 · DOI · Paper page
  22. Li C., 2010, Effect of calibration data series length on performance and optimal parameters of hydrological model, Water Science and Engineering, 3, 378
  23. Lindström G., 1997, Development and test of the distributed HBV-96 hydrological model, Journal of Hydrology (Amsterdam), 201, 272, 10.1016/S0022-1694(97)00041-3 · DOI
  24. Madsen H., 2000, Automatic calibration of a conceptual rainfall–runoff model using multiple objectives, Journal of Hydrology (Amsterdam), 235, 276, 10.1016/S0022-1694(00)00279-1 · DOI · Paper page
  25. Molina-Navarro E., 2017, The impact of the objective function in multi-site and multi-variable calibration of the SWAT model, Environmental Modelling & Software, 93, 255, 10.1016/j.envsoft.2017.03.018 · DOI
  26. Moriasi D. N., 2007, Model evaluation guidelines for systematic quatification of accuracy in watershed simulations, American Society of Agricultural and Biological Engineers, 50, 885
  27. Muleta M. K., 2012, Model performance sensitivity to objective function during automated calibrations, Journal of Hydrologic Engineering, 17, 756, 10.1061/(ASCE)HE.1943-5584.0000497 · DOI
  28. Nelson A. M., 2017, Impact of length of dataset on streamflow calibration parameters and performance of apex model, Journal of the American Water Resources Association, 53, 1164, 10.1111/1752-1688.12564 · DOI
  29. Oliveira R. F., 2018, Hydrological calibration and validation of the MGBIPH model for water resource management in the upper Teles Pires River basin in the Amazon-Cerrado ecotone in Brazil, Acta Amazonica, 49, 54, 10.1590/1809-4392201800812 · DOI
  30. Paiva R. C. D., 2013, Largescale hydrologic and hydrodynamic modeling of the Amazon River basin, Water Resources Research, 49, 1226, 10.1002/wrcr.20067 · DOI · Paper page
  31. Pappenberger F., 2004, Functional classification and evaluation of hydrographs based on Multicomponent Mapping (Mx), Intl. J. River Basin Management, 2, 89, 10.1080/15715124.2004.9635224 · DOI
  32. Parker S. R., 2019, Targeted hydrologic model calibration to improve prediction of ecologically-relevant flow metrics, Journal of Hydrology (Amsterdam), 573, 546, 10.1016/j.jhydrol.2019.03.081 · doi:10.1016/j.jhydrol.2019.03.081
  33. Pechlivanidis I. G., 2012, Using an informational entropy-based metric as a diagnostic of flow duration to drive model parameter identification, Journal Global NEST, 14, 325
  34. Pereira F. F., 2014, Implementation of a two-way coupled atmospherichydrological system for environmental modeling at regional scale, Hydrology Research, 45, 504, 10.2166/nh.2013.335 · doi:10.2166/nh.2013.335
  35. Pool S., 2018, Evaluating model performance:towards a non-parametric variant of the Kling-Gupta efficiency, Hydrological Sciences Journal, 63, 1941, 10.1080/02626667.2018.1552002 · DOI · Paper page
  36. Pushpalatha R., 2012, A review of efficiency criteria suitable for evaluating low-flow simulations, Journal of Hydrology (Amsterdam), 420-421, 171, 10.1016/j.jhydrol.2011.11.055 · DOI · Paper page
  37. Rajib M. A., 2016, YU; Z. Multi-objective calibration of a hydrologic model using spatially distributed emotely sensed/in-situ soil moisture, Journal of Hydrology (Amsterdam), 536, 192, 10.1016/j.jhydrol.2016.02.037 · DOI · Paper page
  38. Ribeiro Neto A., 2006, Hydrological modelling in Amazonia—use of the MGB-IPH model and alternative databases, 303, 246
  39. Rientjes T. H. M., 2013, Multi-variable calibration of a semi-distributed hydrological model using streamflow data and satellite-based evapotranspiration, Journal of Hydrology (Amsterdam), 505, 276, 10.1016/j.jhydrol.2013.10.006 · DOI
  40. Romanowicz R. J., 2013, On the choice of calibration periods and objective functions: a practical guide to model parameter identification, Acta Geophysica, 61, 1477, 10.2478/s11600-013-0157-6 · doi:10.2478/s11600-013-0157-6
  41. Rwetabula J., 2012, Simulation of hydrological processes in the Simiyu River, tributary of Lake Victoria, Tanzania, Water AS, 38, 623
  42. Schaefli B., 2007, Do Nash values have value?, Hydrological Processes, 21, 2075, 10.1002/hyp.6825 · DOI · Paper page
  43. Troin M., 2015, Performance and Uncertainty Evaluation of Snow Models on Snowmelt Flow Simulations over a Nordic Catchment (Mistassibi, Canada), Hydrology, 2, 289, 10.3390/hydrology2040289 · doi:10.3390/hydrology2040289
  44. Tucci C. E. M., 2008, Short- and long-term flow forecasting in the Rio Grande watershed (Brazil), Atmospheric Science Letters, 9, 53, 10.1002/asl.165 · doi:10.1002/asl.165
  45. Van Liew M. W., 2007, Suitability of SWAT for the Conservation Effects Assessment Project: Comparison on USDA Agricultural Research Service Watersheds, Journal of Hydrological Research, 12, 173, 10.1061/(ASCE)1084-0699(2007)12:2(173) · doi:10.1061/(asce)1084-0699(2007)12:2(173 · Paper page
  46. Westerberg I. K., 2011, Calibration of hydrological models using flow-duration curves, Hydrology and Earth System Sciences, 15, 2205, 10.5194/hess-15-2205-2011 · DOI · Paper page
  47. Wohling T., 2013, Evaluating multiple performance criteria to calibrate the distributed hydrological model of the upper Neckar catchment, Environmental Earth Sciences, 69, 453, 10.1007/s12665-013-2306-2 · DOI
  48. Zhang Y., 2016, Multi-metric calibration of hydrological model to capture overall flow regimes, Journal of Hydrology (Amsterdam), 539, 525, 10.1016/j.jhydrol.2016.05.053 · DOI
  49. Zink M., 2018, Conditioning a hydrologic model using patterns of remotely sensed land surface temperature, Water Resources Research, 54, 2976, 10.1002/2017WR021346 · DOI

Metadata

Authors
Ferreira, P. M. D. L., Paz, A. R. D., & Bravo, J. M.
Affiliations
  • Universidade Federal da Paraíba, Brasil
  • Universidade Federal do Rio Grande do Sul, Brasil
Venue
RBRH
Published
2020
Volume
25
ISSN
  • 2318-0331
  • 1414-381X
Cited by
21
References
1

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.