CoChat
Sign Up
PreprintCited by 353Topic ModelingNatural Language Processing TechniquesText Readability and Simplification

BARTScore: Evaluating Generated Text as Text Generation

Open access
Yuan, Weizhe, Neubig, Graham, Liu, Pengfei·2021·arXiv
DOI 10.48550/arxiv.2106.11520arXiv 2106.11520
Open this paper in CoChat — free accountOpen PDF

Abstract

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.11520

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reference-free metrics like BARTScore and QRelScore overcome the reference limitation but often assign a single overall score, which is less interpretable and not comprehensive.

    “Reference-free metrics like BARTScore (Yuan et al., 2021) and QRelScore (Wang et al., 2022) overcome this limitation, but they often assign a single overall score as the evaluation result, which is less interpretable and not comprehensive.”

    From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Yuan et al., 2021

    Section 3.2 (p. 4) describes a source-to-hypothesis BARTScore variant that does not use a reference, alongside variants that do use one; §4.2.2 (p. 6) says the variant is selected for the evaluation task. The paper does not characterize BARTScore as less interpretable or generally not comprehensive; §5 (p. 10) instead reports evaluation from seven perspectives. This source does not establish the claim about QRelScore. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Uncertainty metrics such as token probability or entropy require access to token-level probability distributions, which may not be available for systems accessed through limited external APIs.

    “However, uncertainty metrics such as token probability or entropy require access to token-level probability distributions, information which may not be available to users for example when systems are accessed through limited external APIs.”

    From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Yuan et al., 2021

    Section 3.1 (p. 3) describes BARTScore as using a model’s generation probabilities for token sequences, supporting the probability-based part of the claim. The paper does not discuss entropy as an uncertainty metric or whether limited external APIs expose token-level probability distributions; searches for those terms found no relevant passage. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • Hybrid-LLM uses synthetic preference labels from MixInstruct based on BARTScore and relies on a single BERT-based router.

    “it uses synthetic preference labels from the MixInstruct dataset (Jiang et al., 2023) based on BARTScore (Yuan et al., 2021) and relies on a single BERT-based router.”

    From RouteLLM: Learning to Route LLMs with Preference Data (2024) · cites this paper as Yuan et al., 2021

  • Conditional language model scores have been used to evaluate properties of texts.

    “Additionally, conditional language model scores have been used to evaluate properties of texts (Yuan et al., 2021; Fu et al., 2023).”

    From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Yuan et al., 2021

  • Existing fact-checking approaches either require access to the output probability distribution or external databases interfaced via separate modules.

    “Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules.”

    From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Yuan et al., 2021

    The Introduction (p. 1) describes BARTScore’s use of generation probabilities, and §4.3.2 (p. 8) compares it with factuality metrics. Neither passage says that existing fact-checking approaches require access to a system’s output probability distribution or external databases connected through separate modules. Searches for those requirements found no supporting passage in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. [2][2] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 642-652. PMLR, 2020. -
  2. [3][3] Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, and Pengfei Liu. Metrics also disagree in the low scoring range: Revisiting summarization evaluation metrics. In Proceedings of the 28th International Conference on Computational Linguistics , pages 5702-5711, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. -
  3. [5][5] Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 675-686, Melbourne, Australia, July 2018. Association for Computational Linguistics. -
  4. [6][6] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. -
  5. [7][7] Hoa Trang Dang. Overview of duc 2005. In Proceedings of the document understanding conference , volume 2005, pages 1-12, 2005. -
  6. [8][8] Michael Denkowski and Alon Lavie. Extending the METEOR machine translation evaluation metric to the phrase level. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 250-253, Los Angeles, California, June 2010. Association for Computational Linguistics. -
  7. [11][11] Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. BanditSum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3739-3748, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. -
  8. [12][12] Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. GSum: A general framework for guided neural abstractive summarization. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) , Mexico City, June 2021. -
  9. [13][13] A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics , 9:391-409, 2021. -
  10. [14][14] Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2214-2220, 2019. -
  11. [15][15] David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York , 2007. -
  12. [16][16] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4098-4109, 2018. -
  13. [17][17] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4098-4109, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. -
  14. [18][18] Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 708-719, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. -
  15. [19][19] Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , volume 1, pages 708-719, 2018. -
  16. [20][20] K. Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In NIPS , 2015. -
  17. [21][21] Tsutomu Hirao, Manabu Okumura, Norihito Yasuda, and Hideki Isozaki. Supervised automatic evaluation for summarization with voted regression model. Information Processing & Management , 43(6):1521-1535, 2007. -
  18. [22][22] J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. Largescale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pages 44-54, Hong Kong, China, November 2019. Association for Computational Linguistics. -
  19. [23][23] Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. Scirex: A challenge dataset for document-level information extraction. arXiv preprint arXiv:2005.00512 , 2020. - · arXiv
  20. [25][25] Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation , 1972. -
  21. [28][28] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , pages 388-395, Barcelona, Spain, July 2004. Association for Computational Linguistics. -
  22. [29][29] Philipp Koehn. Statistical machine translation . Cambridge University Press, 2009. -
  23. [31][31] Teven Le Scao and Alexander Rush. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2627-2636, Online, June 2021. Association for Computational Linguistics. -
  24. [33][33] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871-7880, Online, July 2020. Association for Computational Linguistics. -
  25. [36][36] Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics , pages 150-157, 2003. -
  26. [37][37] Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7999-8009, 2020. -
  27. [38][38] Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaicheng Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. Explainaboard: An explainable leaderboard for nlp. arXiv preprint arXiv:2104.06387 , 2021. - · arXiv
  28. [40][40] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345 , 2019. - · arXiv
  29. [41][41] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3730-3740, Hong Kong, China, November 2019. Association for Computational Linguistics. -
  30. [42][42] Yixin Liu and Pengfei Liu. Simcls: A simple framework for contrastive learning of abstractive summarization. arXiv preprint arXiv:2106.01890 , 2021. - · arXiv
  31. [43][43] Qingsong Ma, Ondˇ rej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers , pages 671-688, Belgium, Brussels, October 2018. Association for Computational Linguistics. -
  32. [44][44] Qingsong Ma, Johnny Wei, Ondˇ rej Bojar, and Yvette Graham. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pages 62-90, Florence, Italy, August 2019. Association for Computational Linguistics. -
  33. [45][45] François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages 1552-1561, 2010. -
  34. [46][46] Chaitanya Malaviya, Graham Neubig, and Patrick Littell. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 2529-2535, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. -
  35. [47][47] Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4984-4997, Online, July 2020. Association for Computational Linguistics. -
  36. [49][49] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1747-1759, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. -
  37. [50][50] Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 , pages 145-152, Boston, Massachusetts, USA, May 2 - May 7 2004. Association for Computational Linguistics. -
  38. [52][52] Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization , pages 74-84, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. -
  39. [53][53] Maja Popovi´ c. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. -
  40. [54][54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
  41. [55][55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. -
  42. [57][57] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2685-2702, Online, November 2020. Association for Computational Linguistics. -
  43. [60][60] Timo Schick and Hinrich Schütze. Few-shot text generation with pattern-exploiting training. arXiv preprint arXiv:2012.11926 , 2020. - · arXiv
  44. [61][61] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118 , 2020. - · arXiv
  45. [62][62] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. -
  46. [63][63] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7881-7892, Online, July 2020. Association for Computational Linguistics. -
  47. [64][64] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 , 2020. - · arXiv
  48. [65][65] Miloš Stanojevi´ c and Khalil Sima'an. BEER: BEtter evaluation as ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation , pages 414-419, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. -
  49. [67][67] Alex Wang, Kyunghyun Cho, and M. Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In ACL , 2020. -
  50. [68][68] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. -
  51. [69][69] Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6209-6219, Online, July 2020. Association for Computational Linguistics. -
  52. [70][70] Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745 , 2015. - · arXiv
  53. [71][71] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. -
  54. [72][72] Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? arXiv preprint arXiv:2102.00176 , 2021. - · arXiv
  55. [74][74] Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pages 789-797, Hong Kong, China, November 2019. Association for Computational Linguistics. -
  56. [75][75] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning , pages 11328-11339. PMLR, 2020. -
  57. [77][77] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 563-578, Hong Kong, China, November 2019. Association for Computational Linguistics. -
  58. [78][78] Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. arXiv preprint arXiv:2004.08795 , 2020. - · arXiv
  59. [79][79] Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6197-6208, Online, July 2020. Association for Computational Linguistics. -
  60. [80][80] Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Searching for effective neural extractive summarization: What works and what's next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1049-1058, Florence, Italy, July 2019. Association for Computational Linguistics. -
  61. [81][81] Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 654-663, Melbourne, Australia, July 2018. Association for Computational Linguistics.

Cited by

Sorted by citations

Authors
Yuan, W., Neubig, G., & Liu, P.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.11520
arXiv
2106.11520
OpenAlex
W3176456866
Cited by
353
References
17
Citations by year
2026: 3 · 2025: 49 · 2024: 66 · 2023: 171 · 2022: 58 · 2021: 6
Type
posted content
Language
en
Open access
Licence
creativecommons.org/publicdomain/zero/1.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
text generation evaluation, BART, sequence-to-sequence models, informativeness, fluency, factuality, machine translation evaluation, meta-evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review