A Call for Clarity in Reporting BLEU Scores
Open accessAbstract
The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric.Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters.These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared.I quantify this variation, finding differences as high as 1.8 between commonly used configurations.The main culprit is different tokenization and normalization schemes applied to the reference.Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for usersupplied reference processing, and provide a new tool, SACREBLEU, 1 to facilitate this.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Conventional string-matching metrics and explicit task-type prompting are misaligned with cognitive memory scenarios.
“We further show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
Section 2.2 describes BLEU’s n-gram matching and explains how reference preprocessing affects the comparability of machine-translation scores. It does not address cognitive memory scenarios or explicit task-type prompting. Searches for those concepts and for string-matching metrics in memory tasks found no passage supporting the claimed misalignment. (LLM, conf=0.76)
Experimental — accuracy not yet measuredMultiple valid responses may satisfy the same memory constraint while receiving different scores.
“Multiple valid responses may satisfy the same memory constraint while receiving different scores.”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Post, 2018
The Introduction and §2.2 report that BLEU scores vary with the number of references and with preprocessing, including for the same translation system. Our copy does not address whether multiple valid responses satisfy a shared memory constraint yet receive different scores; searches for memory constraints and valid responses found no such passage. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · see citing works
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. ArXiv e-prints, abs/1409.0473. · arXiv
- E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Gr- ishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A proce- dure for quantitatively comparing the syntactic cov- erage of english grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pa- cific Grove, California, February 19-22, 1991.
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, pages 169-214. Association for Computational Linguis- tics.
- Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in ma- chine translation research. In 11th Conference of the European Chapter of the Association for Computa- tional Linguistics.
- Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the iwslt 2017 evaluation campaign. In 14th International Workshop on Spoken Language Translation, pages 2-14, Tokyo, Japan.
- David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Pro- ceedings of the 43rd Annual Meeting of the Associa- tion for Computational Linguistics (ACL'05), pages 263-270. Association for Computational Linguis- tics.
- Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for op- timizer instability. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176-181. Association for Computational Linguis- tics.
- Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017. A convolutional encoder model for neural machine translation. In Proceed- ings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 123-135. Association for Computa- tional Linguistics.
- Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting better over time? In Proceedings of the 14th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 443-451. As- sociation for Computational Linguistics.
- Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1-10. Association for Computational Linguistics.
- Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Pro- ceedings of the 53rd Annual Meeting of the Associ- ation for Computational Linguistics and the 7th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 11-19. Association for Computational Linguistics.
- Ines Rehbein and Josef van Genabith. 2007. Tree- bank annotation schemes and parser evaluation for german. In Proceedings of the 2007 Joint Confer- ence on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
- Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 0(0):393-401.
- Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexan- dra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine trans- lation. In Proceedings of the Software Demonstra- tions of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 65-68. Association for Computational Lin- guistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725. Association for Computational Linguistics.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv e-prints, abs/1706.03762. · arXiv
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv e- prints, abs/1609.08144. · arXiv
Cited by
Sorted by citations
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · see citing works
“Prior research has attempted to generate effective prompts using separate models [19].”
MethodBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
“Evaluation: 1) human-targeted translation error rate (HTER) 8 , SacreBLEU (Post, 2018) and METEOR (Banerjee and Lavie, 2005) between the Marathi generated sentence compared to the human post-edited sentence, 2) HTER, Sacre-BLEU, METEOR, and semantic similarity score, i.e., BERTScore (Zhang* et al., 2020), between the…”
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“With BLEU, for instance, libraries often have their own BLEU score implementation, which may differ from one another, thus affecting the scores (this is recently addressed by Post, 2018).”
BackgroundA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
Unsupervised Domain Clusters in Pretrained Language Models
Aharoni, R., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.692 · Cited by 181 · see citing works
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
Thompson, B., & Post, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.8 · Cited by 128 · see citing works
Self-citationOn Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Wang, C., & Sennrich, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.326 · Cited by 118 · see citing works
DeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113 · see citing works
Reassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79 · see citing works
Imitation Attacks and Defenses for Black-box Machine Translation Systems
Wallace, E., Stern, M., & Song, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.446 · Cited by 75 · see citing works
Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers
Marie, B., Fujita, A., & Rubino, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.566 · Cited by 75 · see citing works
Multi-task Learning for Multilingual Neural Machine Translation
Wang, Y., Zhai, C., & Hassan, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.75 · Cited by 59 · see citing works
Training language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60 · see citing works
Uncertainty Estimation in Autoregressive Structured Prediction
Gales, M., & Malinin, A. (2021) · Apollo - University of Cambridge Repository · doi:10.17863/cam.63497 · Cited by 48 · see citing works
Cao, Y. T., & Daumé, H. (2021) · Computational Linguistics · doi:10.1162/coli_a_00413 · Cited by 47 · see citing works
Query-Key Normalization for Transformers
Henry, A., Dachapally, P. R., Pawar, S. S., & Chen, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.379 · Cited by 45
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.257 · Cited by 40 · see citing works
“Despite tokenized BLEU scores, we also report SacreBLEU 17 (Post, 2018) and COMET 18 (Rei et al., 2020) scores in Table 12 and 13 to provide a reference for future research.”
Data and Parameter Scaling Laws for Neural Machine Translation
Gordon, M. A., Duh, K., & Kaplan, J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.478 · Cited by 31 · see citing works
Accelerating Transformer Inference for Translation via Parallel Decoding
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., & Rodola, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.689 · Cited by 29 · see citing works
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.633 · Cited by 24 · see citing works
Automatic Product Copywriting for E-commerce
Zhang, X., Zou, Y., Zhang, H., Zhou, J., Diao, S., Chen, J., Ding, Z., He, Z., He, X., Xiao, Y., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21508 · Cited by 24 · see citing works
DEMETR: Diagnosing Evaluation Metrics for Translation
Karpinska, M., Raj, N., Thai, K., Song, Y., Gupta, A., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.649 · Cited by 22 · see citing works
Towards Making the Most of ChatGPT for Machine Translation
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · arXiv · doi:10.48550/arxiv.2303.13780 · Cited by 20 · see citing works
The statistical advantage of automatic NLG metrics at the system level
Wei, J., & Jia, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.533 · Cited by 16 · see citing works
“As objective, reproducible quantities, BLEU scores facilitate cross-paper comparisons (Post, 2018).”
BackgroundMethodCyberbully Detection Using BERT with Augmented Texts
Guo, X., Anjum, U., & Zhan, J. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020581 · Cited by 12 · see citing works
Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphs
Wang, Z., Li, Z., Jiang, Z., Tu, D., & Shi, W. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.281 · Cited by 12 · see citing works
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
Confidence-Modulated Speculative Decoding for Large Language Models
Sen, J., Dasgupta, S., & Waghela, H. (2025) · 2025 IEEE 6th India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon66021.2025.11254640
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
“(Bulian et al., 2022; Post, 2018) At the same time, modern LLMs exhibit diverse generation behaviors in terms of verbosity, reasoning style, and response length.”
InfluentialMesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“We follow the exact prompt setup of Brown et al. [17] evaluate with {0,1,5,10} -and 50-shots, and report the performance in Table 16 with respect to BLEU-sb [78] for models trained on 50B tokens.”
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation
Daheim, N., Dziri, N., Sachan, M., Gurevych, I., & Ponti, E. M. (2023) · arXiv · doi:10.48550/arxiv.2303.17574
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
- Authors
- Post, M.
- Affiliations
- Amazon (Germany) (DE)
- Corresponding author
- Post, Matt
- Venue
- Proceedings of the Third Conference on Machine Translation: Research Papers
- Published
- 2018
- Pages
- 186-191
- OpenAlex
- W2798362442
- Cited by
- 2,112
- References
- 3
- Citations by year
- 2026: 138 · 2025: 191 · 2024: 192 · 2023: 452 · 2022: 306 · 2021: 394 · 2020: 292 · 2019: 126 · 2018: 21
- FWCI
- 85.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Machine Translation (WMT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Pennsylvania
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- BLEU, machine translation evaluation, parameter sensitivity, tokenization, normalization, test score comparability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
