A Call for Clarity in Reporting BLEU Scores
Open accessAbstract
The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SacreBLEU, to facilitate this.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In conversational memory settings, the assumption that correctness can be determined by surface-level overlap does not hold, because multiple valid responses may satisfy the same memory constraint while receiving different scores.
“In conversational memory settings, this assumption does not hold. Multiple valid responses may satisfy the same memory constraint while receiving different scores.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
Section 2.2 explains that BLEU scores depend on n-gram matches against a reference, and the Introduction shows that changing the number of references changes a system’s score. The paper does not examine conversational memory or establish that multiple responses satisfying the same memory constraint receive different scores. (LLM, conf=0.80)
Experimental — accuracy not yet measuredConversational memory is commonly evaluated using generation-based metrics such as BLEU, ROUGE, exact match, and token-level F1.
“Conversational memory is commonly evaluated using generation-based metrics such as BLEU, ROUGE, exact match, and token-level F1”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
The Abstract identifies BLEU as a dominant metric for machine translation, supporting its use as an evaluation metric. The paper does not address conversational memory or establish that it is commonly evaluated with BLEU, ROUGE, exact match, and token-level F1. Its discussion of F1 in §3.1 concerns parsing, not conversational memory. (LLM, conf=0.78)
Experimental — accuracy not yet measuredCommon evaluation practices such as task-disclosed prompting and string-matching metrics can be systematically misleading, even for factual memory, by conflating memory fidelity with prompt adaptation and generation style.
“We further show that common evaluation practices-such as task-disclosed prompting and string-matching metrics-can be systematically misleading, even for factual memory, by conflating memory fidelity with prompt adaptation and generation style”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
Section 2.2 (page not available) shows that BLEU scores for machine translation can vary with reference preprocessing, supporting the narrower point that a string-matching metric can yield misleading comparisons. The paper does not address task-disclosed prompting, factual memory, or a conflation of memory fidelity with prompt adaptation and generation style. (LLM, conf=0.86)
Experimental — accuracy not yet measuredGeneration-based metrics such as BLEU, ROUGE, exact match, and token-level F1 poorly reflect semantic correctness and behavioral validity in open-ended generation.
“these metrics are known to poorly reflect semantic correctness and behavioral validity in open-ended generation”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
The Introduction says the paper is not concerned with BLEU’s shortcomings as a proxy for human evaluation; it examines how preprocessing and reporting make BLEU scores difficult to compare. Searches for semantic correctness, behavioral validity, ROUGE, exact match, and token-level F1 found no passage assessing whether these metrics reflect those qualities in open-ended generation. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 20
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. ArXiv e-prints, abs/1409.0473. · arXiv
- E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Gr- ishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A proce- dure for quantitatively comparing the syntactic cov- erage of english grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pa- cific Grove, California, February 19-22, 1991.
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, pages 169-214. Association for Computational Linguis- tics.
- Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in ma- chine translation research. In 11th Conference of the European Chapter of the Association for Computa- tional Linguistics.
- Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the iwslt 2017 evaluation campaign. In 14th International Workshop on Spoken Language Translation, pages 2-14, Tokyo, Japan.
- David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Pro- ceedings of the 43rd Annual Meeting of the Associa- tion for Computational Linguistics (ACL'05), pages 263-270. Association for Computational Linguis- tics.
- Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for op- timizer instability. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176-181. Association for Computational Linguis- tics.
- Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017. A convolutional encoder model for neural machine translation. In Proceed- ings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 123-135. Association for Computa- tional Linguistics.
- Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting better over time? In Proceedings of the 14th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 443-451. As- sociation for Computational Linguistics.
- Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1-10. Association for Computational Linguistics.
- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexan- dra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine transla- tion. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Com- panion Volume Proceedings of the Demo and Poster Sessions, pages 177-180. Association for Computa- tional Linguistics.
- Thang Luong, Hieu Pham, and Christopher D. Man- ning. 2015a. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pages 1412-1421. Asso- ciation for Computational Linguistics.
- Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Pro- ceedings of the 53rd Annual Meeting of the Associ- ation for Computational Linguistics and the 7th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 11-19. Association for Computational Linguistics.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics.
- Ines Rehbein and Josef van Genabith. 2007. Tree- bank annotation schemes and parser evaluation for german. In Proceedings of the 2007 Joint Confer- ence on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
- Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 0(0):393-401.
- Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexan- dra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine trans- lation. In Proceedings of the Software Demonstra- tions of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 65-68. Association for Computational Lin- guistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725. Association for Computational Linguistics.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv e-prints, abs/1706.03762. · arXiv
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv e- prints, abs/1609.08144. · arXiv
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05120 · Cited by 22 · see citing works
Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1150
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
- Authors
- Post, M.
- Affiliations
- Amazon (Germany) (DE)
- Corresponding author
- Post, Matt
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Pages
- 186-191
- arXiv
- 1804.08771
- OpenAlex
- W2798362442
- Cited by
- 2,111
- References
- 0
- Citations by year
- 2026: 138 · 2025: 191 · 2024: 192 · 2023: 452 · 2022: 306 · 2021: 393 · 2020: 292 · 2019: 126 · 2018: 21
- FWCI
- 85.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Machine Translation (WMT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Pennsylvania
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- BLEU, machine translation evaluation, parameter sensitivity, tokenization, normalization, test score comparability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
