The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Evaluating the Factual Consistency of Abstractive Text Summarization
Open accessAbstract
Currently used metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents.We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary.Training data is generated by applying a series of rule-based transformations to the sentences of source documents.The factual consistency model is then trained jointly for three tasks: 1) identify whether sentences remain factually consistent after transformation, 2) extract a span in the source documents to support the consistency prediction, 3) extract a span in the summary sentence that is inconsistent if one exists.Transferring this model to summaries generated by several state-of-the art models reveals that this highly scalable approach substantially outperforms previous models, including those trained with strong supervision using standard datasets for natural language inference and fact checking.Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In most previous works, factuality has been defined as whether a claim in a text can be supported by evidence from a separate, trustworthy knowledge base.
“In most previous works, factuality has been defined as whether a claim in a text can be supported by evidence from a separate, trustworthy knowledge base, with applications in fact-checking (Thorne et al., 2018b) (where the knowledge base is a large source like Wikipedia) and summarization (Kryscinski et al., 2020) (where the knowledge base is an input document or documents).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Kryscinski et al., 2020
The Introduction defines a factually consistent summary as one whose statements are entailed by the source document. But the “Model generated claims” section explicitly says the source document is not guaranteed to be true, so it does not support describing that document as a trustworthy knowledge base. Page numbers are unavailable in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredHallucinations are observed over a wide range of tasks beyond pure fact verification.
“Additionally, hallucinations are observed over a wide range of tasks beyond pure fact verification (Kryscinski et al., 2020; Maynez et al., 2020).”
From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Kryscinski et al., 2020
The “Model generated claims” section gives examples of factually incorrect output from summarization models, supporting the observation of hallucination-like errors in text generation. The paper examines abstractive summarization, however, and does not establish that hallucinations occur across a wide range of tasks. (LLM, conf=0.84)
Experimental — accuracy not yet measuredResearch suggests that up to 30% of machine-generated abstractive summaries may contain factual inconsistencies.
“Research suggests that up to 30% of machine-generated abstractive summaries may contain factual inconsistencies [52, 53].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [53]
The performance of the FactCC model surpasses classifiers trained on the MNLI or FEVER datasets.
“Results show that the performance of their FactCC model surpasses the classifiers trained on the MNLI or FEVER datasets.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [126]
Prior synthetic data generation methods such as entity swapping and sentence negation were designed to target specific error types that occurred in claims from earlier summarization models.
“These methods, such as entity swapping and sentence negation (Kryscinski et al., 2020;Goyal and Durrett, 2021), were designed to target specific error types that occurred in claims from earlier summarization models.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Kryscinski et al., 2020
A surge of research explores leveraging data-augmentation methods to construct synthetical data for fine-tuning the classifier, either by rule-based perturbation or generation.
“In response to this challenge, a surge of research explores leveraging data-augmentation methods to construct synthetical data for fine-tuning the classifier, either by rule-based perturbation [79, 152, 266] or generation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [152]
Most prior hallucination detection benchmarks concentrated on task-specific hallucinations.
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [152]
QA prompting has been widely studied in the summarization literature and recent work in factual precision.
“This approach has been widely studied in the summarization literature and recent work in factual precision (Kryscinski et al., 2020; Wang et al., 2020; Gao et al., 2022; Manakul et al., 2023).”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Kryscinski et al., 2020
The Introduction and Methods discuss factual consistency in summarization and a document–sentence checking approach, but neither addresses QA prompting. Searches for question answering, question generation, and prompting found no passage establishing that QA prompting has been widely studied in summarization or factual-precision work. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2017) · Cited by 11,625
(2004) · Cited by 9,718
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,043
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,934
Nallapati, R. M., Zhai, F., & Zhou, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10958 · Cited by 1,278
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182
Self-citationA large annotated corpus for learning natural language inferencePreprint
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.05326 · Cited by 1,020
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 784
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Chen, Y.-C., & Bansal, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1063 · Cited by 633
New York Times Annotated Corpus
Sandhaus, E., & Linguistic Data Consortium (2012) · TIB Data Manager · doi:10.57702/qbrk4fuh · Cited by 594
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2×
“…(Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Vinyals et al., 2015; Vaswani et al., 2017), pre-training and transfer learning (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018), and availability of large-scale supervised datasets (Sandhaus, 2008; Nallapati et…”
BackgroundSelf-citationNewsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Grusky, M., Naaman, M., & Artzi, Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1065 · Cited by 538
Fine-tune BERT for Extractive SummarizationPreprint
Liu, Y. (2019) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2024.2 · Cited by 519
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 369
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335
Self-citationA Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
Hsu, W.-T., Lin, C.-K., Lee, M.-Y., Min, K., Tang, J. P., & Sun, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1013 · Cited by 276
Pretraining-Based Natural Language Generation for Text Summarization
Zhang, H., Cai, J., Xu, J., & Wang, J. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1074 · Cited by 267
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265
Dorr, B. J., Zajic, D. M., & Schwartz, R. M. (2003) · doi:10.3115/1119467.1119468 · Cited by 223
Improving Abstraction in Text Summarization
Kryściński, W., Paulus, R., Xiong, C., & Socher, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1207 · Cited by 174
Self-citationBIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
Sharma, E., Li, C., & Wang, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1212 · Cited by 173
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation
Guo, H., Pasunuru, R., & Bansal, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1064 · Cited by 151
Learning to Extract Coherent Summary via Deep Reinforcement Learning
Wu, Y., & Hu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11987 · Cited by 145
Bottom-Up Abstractive SummarizationPreprint
Gehrmann, S., Deng, Y., & Rush, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10792 · Cited by 115
Li, H., Zhu, J., Zhang, J., & Zong, C. (2018) · International Conference on Computational Linguistics · Cited by 85
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · arXiv · doi:10.48550/arxiv.1808.08745 · Cited by 39
Improving Neural Abstractive Document Summarization with Structural Regularization
Li, W., Xiao, X., Lyu, Y., & Wang, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1441 · Cited by 38
Closed-Book Training to Improve Summarization Encoder Memory
Jiang, Y., & Bansal, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1440 · Cited by 35
BanditSum: Extractive Summarization as a Contextual BanditPreprint
Dong, Y., Shen, Y., Crawford, E., van Hoof, H., & Cheung, J. C. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09672 · Cited by 34
On the Abstractiveness of Neural Document Summarization
Zhang, F., Yao, J.-G., & Yan, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1089 · Cited by 19
Flagged first, then by citations
Research works printed but not linked: 9
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguis- tics.
- Kyunghyun Cho, Bart van Merrienboer, C ¸aglar Gülc ¸ehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724- 1734.
- Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing. Association for Compu- tational Linguistics.
- Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. CoRR, abs/1808.09381. · arXiv
- Ramesh Nallapati, Bowen Zhou, C ¸aglar Gülc ¸ehre, Bing Xiang, et al. 2016. Abstractive text summa- rization using sequence-to-sequence rnns and be- yond. Proceedings of SIGNLL Conference on Com- putational Natural Language Learning.
- Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. CoRR, abs/1804.06451. · arXiv
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proc. of NAACL.
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. CoRR, abs/1803.05355. · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“…detection benchmarks, most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al.,…”
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587
Self-citationA comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 964
Self-citationBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“…detection benchmarks, most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al.,…”
GSum: A General Framework for Guided Neural Abstractive Summarization
Dou, Z.-Y., Liu, P., Hayashi, H., Jiang, Z., & Neubig, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.384 · Cited by 210
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset
Chen, Y., Liu, Y., Liang, C., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.449 · Cited by 183
A Survey of Automatic Text Summarization: Progress, Process and Challenges
Mridha, M. F., Lima, A. A., Nur, K. M., Das, S. C., Hasan, M., & Kabir, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3129786 · Cited by 173
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters
Wan, Y., Pu, G., Sun, J., Garimella, A., Chang, K.-W., & Peng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.243 · Cited by 280
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149
Deep reinforcement and transfer learning for abstractive text summarization: A review
Alomari, A., Idris, N. B., Sabri, A. Q. M., & Alsmadi, I. M. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101276 · Cited by 149
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.532 · Cited by 146
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
Dual emotion based fake news detection: A deep attention-weight update approach
Luvembe, A. M., Li, W., Li, S., Liu, F., & Xu, G. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103354 · Cited by 126
Enhancing Factual Consistency of Abstractive Summarization
Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X., & Jiang, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.58 · Cited by 127
Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization
Chintagunta, B., Katariya, N., Amatriain, X., & Kannan, A. (2021) · doi:10.18653/v1/2021.nlpmc-1.9 · Cited by 129
Logical Natural Language Generation from Open-Domain Tables
Chen, W., Chen, J., Su, Y., Chen, Z., & Wang, W. Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.708 · Cited by 142
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119
Self-citationEntity-level Factual Consistency of Abstractive Text Summarization
Nan, F., Nallapati, R. M., Wang, Z., dos Santos, C. N., Zhu, H., Zhang, D., McKeown, K. R., & Xiang, B. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.235 · Cited by 127
Towards a Unified Multi-Dimensional Evaluator for Text Generation
Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., & Han, J. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.131 · Cited by 117
An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics
Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2022) · ACM Computing Surveys · doi:10.1145/3545176 · Cited by 152
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
“, 2018) and in evaluating the factual consistency of generated summaries (Goodrich et al., 2019; Kryściński et al., 2019; Xu et al., 2020).”
BackgroundChen, C., Zhao, K., Leng, J., Liu, C., Fan, J., & Zheng, P. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.102982 · Cited by 118
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Planning with Learned Entity Prompts for Abstractive Summarization
Narayan, S., Zhao, Y., Maynez, J., Simões, G., Nikolaev, V., & McDonald, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00438 · Cited by 124
Multi-Fact Correction in Abstractive Text Summarization
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J. C. K., & Liu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.749 · Cited by 115
Annotating and Modeling Fine-grained Factuality in Summarization
Goyal, T., & Durrett, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.114 · Cited by 100
Reducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99
Luvembe, A. M., Li, W., Li, S.-Y., Liu, F., & Wu, X. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103653 · Cited by 99
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection
Chen, S., Zhang, F., Sone, K., & Roth, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.475 · Cited by 83
AlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81
Fill in the BLANC: Human-free quality estimation of document summaries
Vasilyev, O., Dharnidharka, V., & Bohannon, J. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.2 · Cited by 107
Improving Factual Consistency of Abstractive Summarization via Question Answering
Nan, F., Nogueira dos Santos, C., Zhu, H., Ng, P., McKeown, K., Nallapati, R., Zhang, D., Wang, Z., Arnold, A. O., & Xiang, B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.536 · Cited by 78
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
Cao, M., Dong, Y., & Cheung, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.236 · Cited by 87
Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures.
Joshi, A., Katariya, N., Amatriain, X., & Kannan, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.335 · Cited by 89
Extractive Summarization via ChatGPT for Faithful Summary Generation
Zhang, H., Liu, X., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.214 · Cited by 117
Abstractive text summarization: State of the art, challenges, and improvements
Shakil, H., Farooq, A., & Kalita, J. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128255 · Cited by 98
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70
Neural natural language processing for long texts: A survey on classification and summarization
Tsirmpas, D., Gkionis, I., Papadopoulos, G. T., & Mademlis, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108231 · Cited by 77
Toward Text Data Augmentation for Sentiment Analysis
Abonizio, H., Paraíso, E. C., & Barbon, S. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3114390 · Cited by 75
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity
Harkous, H., Groves, I., & Saffari, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.218 · Cited by 78
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 113
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
Huang, Y., Feng, X., Feng, X., & Qin, B. (2021) · arXiv · doi:10.48550/arxiv.2104.14839 · Cited by 62
A Comprehensive Survey of Abstractive Text Summarization Based on Deep Learning
Zhang, M., Zhou, G., Yu, W., Huang, N., & Liu, W. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7132226 · Cited by 75
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.42 · Cited by 75
Improving Abstractive Summarization of Legal Rulings through Textual Entailment
Feijo, D. D. V., & Moreira, V. P. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-021-09305-4 · Cited by 59
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
Yin, W., Radev, D., & Xiong, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.435 · Cited by 78
Self-citationSummIt: Iterative Text Summarization via ChatGPT
Zhang, H., Liu, X., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.714 · Cited by 96
August, T., Wang, L. L., Bragg, J., Hearst, M. A., Head, A., & Lo, K. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3589955 · Cited by 77
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Liu, Y., Fabbri, A. R., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.-S., Xiong, C., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.228 · Cited by 104
Self-citationTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Wan, D., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.74 · Cited by 58
Prompted Opinion Summarization with GPT-3.5
Bhaskar, A., Fabbri, A. R., & Durrett, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.591 · Cited by 74
ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics
Zhang, M., Li, C., Wan, M., Zhang, X., & Zhao, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121364 · Cited by 61
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Tang, L., Goyal, T., Fabbri, A. R., Laban, P., Xu, J., Yavuz, S., Kryściński, W., Rousseau, J. F., & Durrett, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.650 · Cited by 87
LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network
Zhong, W., Tang, D., Feng, Z., Duan, N., Zhou, M., Gong, M., Shou, L., Jiang, D., Wang, J., & Yin, J. P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.539 · Cited by 53
Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 60
Controllable Neural Dialogue Summarization with Personal Named Entity Planning
Liu, Z., & Chen, N. F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.8 · Cited by 55
Neural Deepfake Detection with Factual Structure of Text
Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.193 · Cited by 73
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50
Discourse-Aware Unsupervised Summarization for Long Scientific Documents
Dong, Y., Mircea, A., & Cheung, J. C. K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.93 · Cited by 71
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00506 · Cited by 79
Program Enhanced Fact Verification with Verbalization and Graph Attention Network
Yang, X., Nie, F., Feng, Y., Liu, Q., Chen, Z., & Zhu, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.628 · Cited by 52
Ladhak, F., Durmus, E., He, H., Cardie, C., & McKeown, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.100 · Cited by 46
Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes
Moro, G., Ragazzi, L., Valgimigli, L., Frisoni, G., Sartori, C., & Marfia, G. (2023) · Sensors · doi:10.3390/s23073542 · Cited by 51
Contrastive Data and Learning for Natural Language Processing
Zhang, R., Ji, Y., Zhang, Y., & Passonneau, R. J. (2022) · doi:10.18653/v1/2022.naacl-tutorials.6 · Cited by 44
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning
Tang, X., Nair, A., Wang, B., Wang, B., Desai, J., Wade, A., Li, H., Celikyilmaz, A., Mehdad, Y., & Radev, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.415 · Cited by 43
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation
Xie, Y., Sun, F., Deng, Y., Li, Y., & Ding, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.10 · Cited by 49
Factual consistency evaluation of summarization in the Era of large language models
Luo, Z., Xie, Q., & Ananiadou, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124456 · Cited by 59
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Mishra, A., Patel, D., Vijayakumar, A., Li, X. L., Kapanipathi, P., & Talamadupula, K. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.104 · Cited by 39
Improving Truthfulness of Headline Generation
Matsumaru, K., Takase, S., & Okazaki, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.123 · Cited by 42
Evaluating Factuality in Text Simplification
Devaraj, A., Sheffield, W., Wallace, B., & Li, J. J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.506 · Cited by 40
HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization
Cao, S., & Wang, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.58 · Cited by 43
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Ribeiro, L., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.236 · Cited by 53
MENLI: Robust Evaluation Metrics from Natural Language Inference
Chen, Y., & Eger, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00576 · Cited by 53
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 45
Understanding Neural Abstractive Summarization Models via Uncertainty
Xu, J., Desai, S., & Durrett, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.508 · Cited by 43
Metadata
- Authors
- Kryscinski, W., McCann, B., Xiong, C., & Socher, R.
- Affiliations
- Salesforce (United States) (US)
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 9332-9346
- OpenAlex
- W2981456979
- Cited by
- 669
- References
- 41
- Citations by year
- 2026: 94 · 2025: 86 · 2024: 56 · 2023: 171 · 2022: 106 · 2021: 118 · 2020: 37
- FWCI
- 41.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, factual consistency, weakly supervised learning, rule-based transformation, span extraction, natural language inference, fact-checking, summarization evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
