CoChat
Sign Up
Cited by 490Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Open access
Zhao, Wei, Peyrard, Maxime, Liu, Fei, Gao, Yang, Meyer, Christian M., Eger, Steffen·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
DOI 10.18653/v1/d19-1053
Open this paper in CoChat — free accountOpen PDF

Abstract

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d19-1053

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MoverScore's combination of BERT's contextual embeddings and IDF weighting correlates highly with human judgment.

    “The combination of BERT's contextual embeddings and IDF weighting has been shown by Zhao et al. (2019) to correlate highly with human judgment.”

    From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Zhao et al. (2019)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 55 · Cited by 490

Research works printed but not linked: 46

  1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic proposi- tional image caption evaluation. In Computer Vision -ECCV 2016 -14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part V, pages 382-398. · DOI · Paper page
  2. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Bet- ter rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing, Hong Kong, China.
  3. Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Conference on Machine Trans- lation (WMT).
  4. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 643-653.
  5. Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic eval- uation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 2748-2760, Florence, Italy. Association for Computational Linguistics.
  6. Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, (5):603-619.
  7. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate im- age captioning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 5804-5812.
  8. Steffen Eger, Gözde Gül S ¸ahin, Andreas Rücklé, Ji- Ung Lee, Claudia Schulz, Mohsen Mesgar, Kr- ishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding NLP systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 1634-1647, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  9. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artifi- cial Intelligence Research (JAIR).
  10. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. NEWSROOM: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  11. Yinuo Guo, Chong Ruan, and Junfeng Hu. 2018. Me- teor++: Incorporating copy knowledge into machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 740-745.
  12. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In Proceedings of Neural Information Processing Systems (NIPS).
  13. Eduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization eval- uation with basic elements. In Proceedings of the Fifth Conference on Language Resources and Eval- uation (LREC 2006), pages 604-611.
  14. Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kil- ian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the Interna- tional Conference on Machine Learning (ICML).
  15. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objec- tive function for neural conversation models. In Pro- ceedings of the North American Chapter of the As- sociation for Computational Linguistics (NAACL).
  16. Chin-Yew Lin. 2004. ROUGE: A Package for Au- tomatic Evaluation of summaries. In Proceedings of ACL workshop on Text Summarization Branches Out, pages 74-81, Barcelona, Spain.
  17. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 740-755. Springer.
  18. Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. In Proceed- ings of the Conference on Empirical Methods in Nat- ural Language Processing (EMNLP).
  19. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguis- tic knowledge and transferability of contextual rep- resentations. arXiv preprint arXiv:1903.08855. · arXiv
  20. Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceed- ings of the Second Conference on Machine Transla- tion, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 589-597.
  21. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267- 300. · DOI
  22. Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task. In Pro- ceedings of the Third Conference on Machine Trans- lation (WMT).
  23. Franc ¸ois Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Pro- ceedings of the 48th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1552- 1561. Association for Computational Linguistics.
  24. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual em- beddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy. Association for Compu- tational Linguistics.
  25. Ani Nenkova and Rebecca J. Passonneau. 2004. Evalu- ating content selection in summarization: The pyra- mid method. In Proceedings of the 2004 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 145-152. Association for Computational Linguistics.
  26. Jun-Ping Ng and Viktoria Abrecht. 2015. Better sum- marization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing, pages 1925-1930, Lisbon, Portugal. Associa- tion for Computational Linguistics.
  27. Jekaterina Novikova, Ondřej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computa- tional Linguistics. · DOI · Paper page
  28. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the North American Chapter of the Association for Computational Lin- guistics (NAACL).
  29. Maxime Peyrard. 2019a. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1059-1073, Florence, Italy. Association for Computational Linguistics.
  30. Maxime Peyrard. 2019b. Studying summarization evaluation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5093-5100, Florence, Italy. Association for Compu- tational Linguistics.
  31. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization.
  32. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation (WMT).
  33. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393- 401. · DOI · Paper page
  34. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision.
  35. Andreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated power mean word embeddings as universal cross-lingual sen- tence representations. CoRR, abs/1803.01400. · arXiv
  36. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the Annual Meeting of the Association for Computational Lin- guistics (ACL).
  37. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. RUSE: Regressor using sentence embeddings for automatic machine translation eval- uation. In Proceedings of the Third Conference on Machine Translation (WMT).
  38. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipan- jan Das, et al. 2019. What do you learn from context? probing for sentence structure in con- textualized word representations. arXiv preprint arXiv:1905.06316. · arXiv
  39. Stephen Tratz and Eduard H Hovy. 2008. Summa- rization Evaluation Using Transformed Basic Ele- ments. In Proceedings of the text analysing confer- ence, (TAC 2008).
  40. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based Im- age Description Evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575. · DOI · Paper page
  41. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. · arXiv
  42. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural lan- guage generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745. · arXiv
  43. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2018. Learning neural templates for text gen- eration. In Proceedings of the Conference on Em- pirical Methods in Natural Language Processing (EMNLP).
  44. Suofei Zhang, Wei Zhao, Xiaofu Wu, and Quan Zhou. 2018. Fast dynamic routing based on weighted ker- nel density estimation. CoRR, abs/1805.10807. · arXiv
  45. Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, and Min Yang. 2019. Towards scalable and reli- able capsule networks for challenging NLP appli- cations. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 1549-1559, Florence, Italy. Association for Computational Linguistics.
  46. Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, and Zhou Zhao. 2018. Investigating cap- sule networks with dynamic routing for text classifi- cation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing. Association for Computational Linguistics. · DOI

Authors
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S.
Affiliations
  • Technische Universität Darmstadt (DE)
  • University of Central Florida (US)
ORCID
Venue
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Publisher
Association for Computational Linguistics
Published
2019
Pages
563-578
DOI
10.18653/v1/d19-1053
OpenAlex
W2970785793
Cited by
490
References
55
Citations by year
2026: 38 · 2025: 62 · 2024: 61 · 2023: 110 · 2022: 82 · 2021: 84 · 2020: 48 · 2019: 5
FWCI
20.95
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • TUbilio (Technical University of Darmstadt) (submittedVersion)
Funders
  • National Science Foundation (1909603)
  • Deutsche Forschungsgemeinschaft (GRK 1994, GRK 1994/1)
  • Technische Universität Darmstadt
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
text generation evaluation, contextual embeddings, earth mover distance, paraphrase detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review