CoChat
Sign Up
PreprintCited by 1,005Topic ModelingNatural Language Processing TechniquesText Readability and Simplification

BLEURT: Learning Robust Metrics for Text Generation

Open access
Sellam, Thibault, Das, Dipanjan, Parikh, Ankur P.·2020·arXiv
DOI 10.48550/arxiv.2004.04696arXiv 2004.04696
Open this paper in CoChat — free accountOpen PDF

Abstract

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2004.04696

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • BLEURT effectively captures content-level deviations while remaining efficient for large-scale evaluation.

    “This metric effectively captures content-level deviations while remaining efficient for large-scale evaluation.”

    From LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge (2025) · cites this paper as [35]

    The Introduction (p. 2) describes BLEURT’s lexical- and semantic-level training signals, and the Conclusion (p. 8) reports strong agreement with human assessment; these support the claim that it captures more than surface-level differences. The Introduction (p. 1) says automatic metrics are cheap to compute, but our copy does not establish that BLEURT itself is efficient for large-scale evaluation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Other proposed metrics such as BERTScore, BLEURT, and NUBIA are computationally expensive, not widely used, and thus not readily comparable with prior work for benchmarking.

    “Other proposed metrics such as BERTScore [28,70], BLEURT [58], NU-BIA [37], are computationally expensive, not widely used and thus not readily comparable with prior work for benchmarking.”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [58]

    The Abstract (p. 1) identifies BLEURT as a proposed text-generation metric, and the training setup (Appendix B.1, p. 11) describes substantial training and accelerator use. Our copy does not establish that BLEURT is computationally expensive relative to other metrics, is not widely used, or is unsuitable for comparison with prior benchmarks; Section 5.1 (p. 5) instead reports comparisons with existing metrics. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • BLEURT is trained for scoring candidate-reference similarity.

    “BLEURT30, which is trained for scoring candidate-reference similarity.”

    From Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation (2023) · cites this paper as [30]

  • BERTScore and BLEURT are limited by their pre-trained embedding length, so they are only used for division-based evaluation.

    “Because BERTScore and BLEURT are limited by their pre-trained embedding length, we only use these evaluations for the division-based evaluation.”

    From Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation (2023) · cites this paper as [30]

    Sections 3 (p. 2) and 5.1 (p. 5) establish that BLEURT uses BERT and that the paper evaluates BERTScore. The passages do not address a pre-trained embedding length limit or division-based evaluation; searches for sequence length, truncation, and division-based evaluation found no supporting passage. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 49 · Cited by 1,005

Research works printed but not linked: 48

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR .
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL .
  3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR .
  4. Ondˇ rej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of WMT .
  5. Ondˇ rej Bojar, Yvette Graham, Amir Kamran, and Miloˇ s Stanojevi´ c. 2016. Results of the wmt16 metrics shared task. In Proceedings of WMT .
  6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of EMNLP .
  7. Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. Proceedings of ACL .
  8. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. Proceedings of ACL .
  9. Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of NAACL HLT .
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT .
  11. Ondˇ rej Duˇ sek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser. 2019. Automatic quality estimation for natural language generation: Ranting (jointly rating and ranking). Proceedings of INLG .
  12. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of NAACL HLT .
  13. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Doll´ ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of CVPR .
  14. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings NAACL HLT .
  15. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG .
  16. Ben Goodrich, Mohammad Ahmad Saleh, Peter Liu, and Vinay Rao. 2019. Assessing the factual accuracy of text generation. In Proceedings of ACM SIGKDD .
  17. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. Proceedings of NAACL HLT .
  18. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. Proceedings of EMNLP .
  19. Philipp Koehn. 2009. Statistical machine translation . Cambridge University Press.
  20. Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of ACL .
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out .
  22. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. Proceedings of EMNLP .
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692 . · arXiv
  24. Qingsong Ma, Ondˇ rej Bojar, and Yvette Graham. 2018. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of WMT .
  25. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment-casict-dcu submission to wmt17 metrics task. In Proceedings of WMT .
  26. Qingsong Ma, Johnny Wei, Ondˇ rej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of WMT .
  27. Inderjeet Mani. 1999. Advances in automatic text summarization . MIT press.
  28. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of ACL .
  29. Kathleen McKeown. 1992. Text generation . Cambridge University Press.
  30. Jekaterina Novikova, Ondˇ rej Duˇ sek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. Proceedings of EMNLP .
  31. Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL .
  32. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of ACL .
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. Proceedings of ACL .
  34. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of WMT .
  35. Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach . Oxford University Press.
  36. Milos Stanojevic and Khalil Simaan. 2014. Beer: Better evaluation as ranking. In Proceedings of WMT .
  37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS .
  38. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv:1910.08684 . · arXiv
  39. Gaurav Singh Tomar, Thyago Duque, Oscar T¨ ackstr¨ om, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural paraphrase identification of questions with noisy pretraining. Proceedings of the First Workshop on Subword and Character Level Models in NLP .
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS .
  41. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. Proceedings of ICML .
  42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR .
  43. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. Proceedings of ICLR .
  44. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of NAACL HLT .
  45. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. Proceedings of EMNLP .
  46. Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. Sumqe: a bert-based summary quality estimation model supplementary material. In Proceedings of EMNLP .
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Proceedings of ICLR .
  48. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Proceedings of EMNLP .

Authors
Sellam, T., Das, D., & Parikh, A. P.
Affiliations
Google (United States) (US)
ORCID
Das, Dipanjan
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
7881-7892
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2004.04696
arXiv
2004.04696
OpenAlex
W3016254928
Cited by
1,005
References
49
Citations by year
2026: 102 · 2025: 144 · 2024: 108 · 2023: 275 · 2022: 161 · 2021: 166 · 2020: 48 · 2019: 1
FWCI
62.06
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
BLEURT, text generation evaluation, BERT, human judgment correlation, WebNLG, out-of-distribution generalization, limited training data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review