More accurate tests for the statistical significance of result differences
Abstract
Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly used tests often underestimate the significance and so are less likely to detect differences that exist between different techniques. This underestimation comes from an independence assumption that is often violated. We point out some useful tests that do not make this assumption, including computationally-intensive randomization tests.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For evaluation measures such as F-score, BLEU, METEOR and ROUGE that do not compute means, the common practice is to assume that they are not normally distributed.
“For other evaluation measures, such as F-score, BLEU, METEOR and ROUGE that do not compute means, the common practice is to assume that they are not normally distributed (Yeh, 2000;Berg-Kirkpatrick et al., 2012).”
From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Yeh, 2000
Approximate randomization can be used for test sets of more than 20 points.
“To overcome this, approximate randomization can be used, as was done, e.g., by Yeh (2000) for test sets of more than 20 points.”
From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Yeh (2000)
Cited by
Sorted by citations
A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation
Goutte, C., & Gaussier, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_25 · Cited by 2,285 · see citing works
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539 · see citing works
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1128 · Cited by 384 · see citing works
Søgaard, A., Johannsen, A., Plank, B., Hovy, D., & Martínez Alonso, H. (2014) · Proceedings of the Eighteenth Conference on Computational Natural Language Learning · doi:10.3115/v1/w14-1601 · Cited by 39 · see citing works
- Authors
- Yeh, A.
- Affiliations
- Mitre (United States) (US)
- Corresponding author
- Yeh, Alexander
- Venue
- Proceedings of the 18th conference on Computational linguistics -
- Published
- 2000
- Volume
- 2
- Pages
- 947
- ISSN
- 1525-2477 (ISSN-L)
- OpenAlex
- W2057399676
- Cited by
- 378
- References
- 0
- Citations by year
- 2025: 5 · 2024: 12 · 2023: 11 · 2022: 9 · 2021: 15 · 2020: 27 · 2019: 27 · 2018: 20 · 2017: 27 · 2016: 20 · 2015: 33 · 2014: 32 · 2013: 35 · 2012: 22
- Type
- conference abstract
- Language
- en
- Open access
- closed
- Locations
- International Conference on Computational Linguistics (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- statistical significance testing, evaluation metrics, memory retrieval, precision, permutation tests, independence assumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
