Cited by 7,178Meteorological Phenomena and SimulationsHydrology and Watershed Management StudiesForecasting Techniques and Applications

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.

Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature

Open access
Chai, T., Draxler, R. R.·2014·Geoscientific Model Development

Published version of a preprint

Abstract

Abstract. Both the root mean square error (RMSE) and the mean absolute error (MAE) are regularly employed in model evaluation studies. Willmott and Matsuura (2005) have suggested that the RMSE is not a good indicator of average model performance and might be a misleading indicator of average error, and thus the MAE would be a better metric for that purpose. While some concerns over using RMSE raised by Willmott and Matsuura (2005) and Willmott et al. (2009) are valid, the proposed avoidance of RMSE in favor of MAE is not the solution. Citing the aforementioned papers, many researchers chose MAE over RMSE to present their model evaluation statistics when presenting or adding the RMSE measures could be more beneficial. In this technical note, we demonstrate that the RMSE is not ambiguous in its meaning, contrary to what was claimed by Willmott et al. (2009). The RMSE is more appropriate to represent model performance than the MAE when the error distribution is expected to be Gaussian. In addition, we show that the RMSE satisfies the triangle inequality requirement for a distance metric, whereas Willmott et al. (2009) indicated that the sums-of-squares-based statistics do not satisfy this rule. In the end, we discussed some circumstances where using the RMSE will be more beneficial. However, we do not contend that the RMSE is superior over the MAE. Instead, a combination of metrics, including but certainly not limited to RMSEs and MAEs, are often required to assess model performance.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This technical note argues against the recommendation by Willmott and Matsuura (2005) and Willmott et al. (2009) to abandon RMSE in favor of MAE, showing that RMSE is not ambiguous in meaning and is more appropriate when errors are expected to be Gaussian. The authors also demonstrate that RMSE satisfies the triangle inequality required of a distance metric, contrary to the earlier claims, while concluding that a combination of metrics is often needed.

Key findings

  • RMSE is not ambiguous in its meaning, contrary to Willmott et al. (2009); it is more appropriate than MAE when the error distribution is expected to be Gaussian, whereas MAE suits uniformly distributed errors.
  • Using randomly generated zero-mean, unit-variance Gaussian pseudo-errors, RMSEs converge to the true standard deviation (SE = 1) once n ≥ 100, with standard deviation within 5% of truth; MAEs converge to 0.8, approximating the expectation of √(2/π). Neither metric is robust at n = 4 or n = 10.
  • RMSE satisfies the triangle inequality required for a distance metric: the example in Willmott et al. (2009) (d(a,c)=4, d(a,b)=2, d(b,c)=3) actually represents the MSE, not the RMSE, and the L2-norm proof in Eqs. (5)–(7) shows RMSE does satisfy it.
  • RMSE has practical advantages: it avoids absolute values (useful for gradients/sensitivities and data assimilation cost functions) and is generally better at discriminating among model results by weighting unfavorable conditions more heavily.

Methods

  • Technical note / commentary design: no new observational dataset; the argument is analytical, using norm theory (L1- and L2-norms linked to MAE and RMSE) and a proof of the triangle inequality for the L2-norm.
  • A Monte Carlo-style demonstration: five sets of pseudo-errors of size n = 4, 10, 100, 1000, 10 000, and 1000 000 drawn from a zero-mean, unit-variance Gaussian distribution with different random seeds, comparing RMSEs and MAEs (Table 1).
  • Reference to prior empirical example: Chai et al. (2009) reported RMSE-to-MAE ratios of 1.63 to 2.29 for model NO2 column predictions versus SCIAMACHY satellite observations.

Limitations

  • Stated by the authors: the analysis assumes unbiased errors and does not consider uncertainties from observation errors or the method used to compare model and observations; all statistics are less useful with limited sample sizes (e.g., n = 4 or 10), where presenting the errors themselves may be more appropriate; for non-normal distributions, additional moments (mean, variance, skewness, flatness) are needed; outliers several orders larger than other samples may justifiably be removed when sample numbers are limited, and systematic errors may need removal before computing RMSEs.
  • Evident from the text: the demonstration relies on synthetic Gaussian pseudo-errors rather than real model–observation comparisons, so the convergence results are conditional on the Gaussian assumption; the authors explicitly do not claim RMSE is superior to MAE, so the note offers guidance rather than a definitive metric selection rule.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.5194/gmd-7-1247-2014

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 Own interpretation · 6 No basis given · 4 Supported · 3 Sources not found · 3 Own result · 1 No full text of the sources · 1 Not checked yet

Show all 20 claims

  • Probably not a substantial claim. 10
  • It may state two separate claims as one. 2
  • The quote only partly states this claim. 1
12 of 12
  • The authors state this as an assumption.

    The errors are unbiased and follow a normal distribution when presenting the RMSE.

    “The underlying assumption when presenting the RMSE is that the errors are unbiased and follow a normal distribution.”
    “We also assume the error sample set is unbiased.”
  • The paper relies on this without support.

    Model errors are likely to have a normal distribution rather than a uniform distribution.

    “Because model errors are likely to have a normal distribution rather than a uniform distribution, the RMSE is a better metric to present than the MAE for such a type of data.”
  • The authors state this as an assumption.

    The uncertainties brought in by observation errors or the method used to compare model and observations are not considered.

    “The uncertainties brought in by observation errors or the method used to compare model and observations are not considered here.”
  • The paper relies on this without support.

    There are often hundreds of observations available to calculate model statistics.

    “Fortunately, there are often hundreds of observations available to calculate model statistics, unlike the examples with n = 4 (Willmott and Matsuura, 2005) and n = 10 ( Willmott et al., 2009).”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 12 references

Cited by

Sorted by citations

Items 5,101–5,200 of 7,174

Metadata

Authors
Chai, T., & Draxler, R. R.
Affiliations
National Oceanic and Atmospheric Administration (US)
Venue
Geoscientific Model Development
Published
2014-06-30
Volume
7
Issue
3
Pages
1247-1250
ISSN
  • 1991-959X (ISSN-L)
  • 1991-9603
OpenAlex
W2102148524
Cited by
7,178
References
12
Citations by year
2026: 1015 · 2025: 1120 · 2024: 980 · 2023: 873 · 2022: 751 · 2021: 683 · 2020: 538 · 2019: 411 · 2018: 324 · 2017: 235 · 2016: 175 · 2015: 59 · 2014: 13
FWCI
106.31
Type
journal article
Language
en
Open access
Locations
  • Geoscientific model development (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Oceanic and Atmospheric Administration (NA09NES4400006)
Topics
  • Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
  • Physical Sciences › Environmental Science › Water Science and Technology › Hydrology and Watershed Management Studies
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
Keywords
root mean square error, mean absolute error, model evaluation metrics, error distribution, Gaussian errors, triangle inequality, distance measures, model evaluation, hydrological model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review