Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 58

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

TRUE: Re-evaluating Factual Consistency Evaluation

Honovich, Or, Aharoni, Roee, Herzig, Jonathan, Taitelbaum, Hagai, Kukliansy, Doron, Cohen, Vered, Scialom, Thomas, Szpektor, Idan, Hassidim, Avinatan, Matias, Yossi·2022·Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
DOI 10.18653/v1/2022.naacl-main.287
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.naacl-main.287

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 ROC AUC when splitting TRUE's data according to the grounding length.
Figure 1 ROC AUC when splitting TRUE's data according to the grounding length.CC-BY · From the PDF
Figure 2 ROC curves for the best performing methods.
Figure 2 ROC curves for the best performing methods.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 31 references

Research works printed but not linked: 35

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI · Paper page
  2. Gillian Brown and George Yule. 1983. Discourse Anal- ysis. Cambridge University Press.
  3. Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? adversarial meta- evaluation of factuality in summarization. In Find- ings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2082-2095, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Eval- uating Predictive Uncertainty, Visual Object Classi- fication, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg. Springer Berlin Hei- delberg.
  5. Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduc- tion, and creation: A unified framework for evaluat- ing natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  6. Emily Denton, Mark Díaz, Ian Kivlichan, Vinodku- mar Prabhakaran, and Rachel Rosen. 2021. Whose ground truth? accounting for individual and collec- tive identities underlying dataset annotation. arXiv preprint arXiv:2112.04554. · arXiv
  7. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. Towards question-answering as an automatic metric for evaluating the content quality of a sum- mary. Transactions of the Association for Computa- tional Linguistics, 9:774-789.
  8. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Confer- ence on Learning Representations (ICLR).
  9. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re- evaluating summarization evaluation. arXiv preprint arXiv:2007.12626. · arXiv
  10. C. Fillmore. 1976. Frame semantics and the nature of language *. Annals of the New York Academy of Sci- ences, 280.
  11. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tat- sunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mi- hir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan- Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Nat- ural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics. · DOI · Paper page
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems, volume 28.
  14. J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019. Large- scale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 44-54, Hong Kong, China. Associ- ation for Computational Linguistics. · DOI
  15. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv:1909.11942. · arXiv
  16. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. NeurIPS 2018 Work- shop on Interpretability and Robustness for Audio, Speech, and Language.
  17. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  19. Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. 2017. The RepEval 2017 shared task: Multi-genre natural language inference · DOI
  20. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An Annotated Cor- pus of Semantic Roles. Computational Linguistics, 31(1):71-106. · DOI · Paper page
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
  22. Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 751-762, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
  23. Libo Qin, Tianbao Xie, Shijue Huang, Qiguang Chen, Xiao Xu, and Wanxiang Che. 2021. Don't be contra- dicted with anything! CI-ToD: Towards benchmark- ing consistency for task-oriented dialogue system. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2357-2367, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
  25. Ehud Reiter and Craig Thomson. 2020. Shared task on evaluating accuracy. In Proceedings of the 13th In- ternational Conference on Natural Language Gener- ation, pages 227-231, Dublin, Ireland. Association for Computational Linguistics.
  26. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con- trastive evidence. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 624-643, Online. Asso- ciation for Computational Linguistics. · DOI · Paper page
  27. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summa- rization asks for fact-based evaluation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
  28. Thomas Scialom and Felix Hill. 2021. Beametrics: A benchmark for language generation evaluation eval- uation. arXiv preprint arXiv:2110.09147. · arXiv
  29. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebas- tian Gehrmann, Qijun Tan, Markus Freitag, Dipan- jan Das, and Ankur Parikh. 2020b. Learning to eval- uate translation beyond English: BLEURT submis- sions to the WMT metrics 2020 shared task. In Pro- ceedings of the Fifth Conference on Machine Trans- lation, pages 921-927, Online. Association for Com- putational Linguistics.
  30. Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics. · DOI · Paper page
  31. Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 100-110, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
  32. Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog eval- uation metrics. In The First Workshop on Evalua- tions and Assessments of Neural Conversation Sys- tems, pages 15-33, Online. Association for Compu- tational Linguistics.
  33. Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4913-4922, Online. Associa- tion for Computational Linguistics. · DOI
  34. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text gen- eration. In Advances in Neural Information Process- ing Systems.
  35. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI

Cited by

See all 58

Metadata

Authors
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y.
Venue
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Publisher
Association for Computational Linguistics
Published
2022
Pages
3905-3920
DOI
10.18653/v1/2022.naacl-main.287
Cited by
58
References
31

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: TRUE: Re-evaluating Factual Consistency Evaluation. https://app.cochat.ai/papers/10.18653/v1/2022.naacl-main.287
The paper
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022). TRUE: Re-evaluating Factual Consistency Evaluation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. https://doi.org/10.18653/v1/2022.naacl-main.287

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.