A Structured Review of the Validity of BLEU
Open accessAbstract
The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique—in other words, whether BLEU scores correlate with real-world utility and user-satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for), but does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BLEU lacks diagnostic sensitivity and correlates poorly with human ratings at the segment level.
“As Reiter [8] highlighted, BLEU lacks diagnostic sensitivity and correlates poorly with human ratings at the segment level.”
From Metric Inversion in Generative AI Translation: Divergence Between Automated Evaluation Metrics and Human Expert Judgement in the LRM Era (2026) · cites this paper as Reiter [8]
The Results section reports poor BLEU–human correlations at the text level, supporting the segment-level part of the claim. But the Conclusion explicitly supports BLEU for diagnostic evaluation of machine-translation systems, so the claim that it lacks diagnostic sensitivity is not supported by this review. (LLM, conf=0.88)
Experimental — accuracy not yet measuredTraditional NLP metrics such as ROUGE, METEOR, and BLEU are known to poorly correlate with human evaluations of natural language generations.
“Traditional NLP metrics such as ROUGE, METEOR, and BLEU—which simply count n-gram overlap between generated and reference text—are known to poorly correlate with human evaluations of natural language generations83–85.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [83]
The Results section reports poor BLEU–human correlations for natural language generation, supporting that part of the claim. But the paper says its survey is limited to BLEU and does not assess metrics such as METEOR; it provides no finding here that ROUGE or METEOR correlate poorly with human evaluations. It also reports reasonable system-level BLEU–human correlations for machine translation, so the claim should not be generalized to every setting. (LLM, conf=0.92)
Experimental — accuracy not yet measuredComparing LLM-generated reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments.
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Reiter, 2018
The Introduction describes BLEU as measuring word-based overlap with a reference text, and the Results report poor BLEU–human-judgment correlations for natural language generation. The paper does not establish that human reference reviews are diverse or that their diversity causes this limitation; it also does not specifically assess LLM-generated reviews. (LLM, conf=0.86)
Experimental — accuracy not yet measuredBy chaining code explanation with code synthesis, they can evaluate explanation using execution-based pass@k, overcoming limitations of BLEU and other heuristics.
“By chaining code explanation with code synthesis, we can evaluate this task using the execution-based pass@k metric overcoming the major limitations of BLEU and other heuristics-based metrics (Reiter, 2018).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Reiter, 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Flagged first, then by citations
Research works printed but not linked: 14
- Banerjee, Satanjeev and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL-2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
- Belz, Anja and Albert Gatt. 2008. Intrinsic vs. extrinsic evaluation measures for referring expression generation. In Proceedings of ACL-2008, pages 197-200.
- Belz, Anja and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In Proceedings of EACL 2006, pages 313-320.
- Bojar, Ondřej, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016a. Results of the WMT16 metrics shared task. In Proceedings of WMT-2016, pages 199-231.
- Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016b. Findings of the 2016 conference on machine translation. In Proceedings of WMT-2016, pages 131-198.
- Bouamor, Houda, Hanan Alshikhabobakr, Behrang Mohit, and Kemal Oflazer. 2014. A human judgment corpus and a metric for Arabic MT evaluation. In Proceedings of EMNLP-2014, pages 207-213.
- Callison-Burch, Chris, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL-2006, pages 249-256.
- Doddington, George. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of HLT 2002, pages 138-145.
- Espinosa, Dominic, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta-evaluation of broad-coverage surface realization. In Proceedings of EMNLP-2010, pages 564-574.
- Graham, Yvette. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of EMNLP-2015, pages 128-137.
- Kilickaya, Mert, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proceedings of EACL-2017, pages 199-209.
- Novikova, Jekaterina, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of EMNLP-2017, pages 2241-2252.
- Prasad, Vinay, Chul Kim, Mauricio Burotto, and Andrae Vandross. 2015. The strength of association between surrogate end points and survival in oncology: A systematic review of trial-level meta-analyses. JAMA Internal Medicine, 175(8):1389-1398.
- Reiter, Ehud and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529-558.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
“Traditional NLP metrics such as ROUGE, METEOR, and BLEU — which simply count n-gram overlap between generated and reference text — are known to poorly correlate with human evaluations of natural language generations 83 – 85 .”
BackgroundBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“1 In light of this criticism, it has been argued that automated metrics are not suitable to assess linguistic properties (Scott and Moore, 2007), and Reiter (2018) discouraged the use of automatic metrics as a (primary) evaluation metric.”
BackgroundA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
Reassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106 · see citing works
Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers
Marie, B., Fujita, A., & Rubino, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.566 · Cited by 75 · see citing works
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50 · see citing works
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Thomson, C., & Reiter, E. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.22 · Cited by 38 · see citing works
Self-citationOctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“By chaining code explanation with code synthesis, we can evaluate this task using the execution-based pass@ k metric overcoming the major limitations of BLEU and other heuristics-based metrics (Reiter, 2018).”
MethodThe statistical advantage of automatic NLG metrics at the system level
Wei, J., & Jia, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.533 · Cited by 16 · see citing works
“Research in system-level metrics have a tradition of evaluating metric correlation to human judgment with the Pearson correlation coefficient (Reiter, 2018).”
BackgroundQAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Ji, T., Lyu, C., Jones, G., Zhou, L., & Graham, Y. (2022) · Entropy · doi:10.3390/e24111514 · Cited by 12 · see citing works
A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation
Boag, W., Kané, H., Rawat, S., Wei, J., & Goehler, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445909 · Cited by 5 · see citing works
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
“…from traditional natural language processing tasks [70, 73] to real-world [46, 20], open-ended response generation [65], which greatly limits the applicability of traditional n-gram matching methods (e.g., BLEU [50] and ROUGE [41]) [43, 53] or model-based evaluators [77, 81] for evaluation.”
Zhang, Y., & Zhang, Y. (2026) · Proceedings of the 3rd International Conference on Educational Digitalization and Computer Science in the Guangdong-Hong Kong-Macao Greater Bay Area · doi:10.1145/3828183.3828226 · Cited by 1 · see citing works
“As Reiter [8] highlighted, BLEU lacks diagnostic sensitivity and correlates poorly with human ratings at the segment level.”
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
- Authors
- Reiter, E.
- Affiliations
- University of Aberdeen (GB)
- Corresponding author
- Reiter, Ehud
- ORCID
- Reiter, Ehud
- Venue
- Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2018-09
- Volume
- 44
- Issue
- 3
- Pages
- 393-401
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W2806532810
- Cited by
- 386
- References
- 1
- Citations by year
- 2026: 63 · 2025: 63 · 2024: 60 · 2023: 38 · 2022: 40 · 2021: 56 · 2020: 36 · 2019: 24 · 2018: 6
- FWCI
- 16.94
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
- Keywords
- BLEU, evaluation metrics, machine translation, natural language generation, diagnostic assessment, hypothesis testing, text evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
