Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
Open accessAbstract
State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text.Despite being assumed incorrect, we find that much hallucinated content is factual, namely consistent with world knowledge.These factual hallucinations can be beneficial in a summary by providing useful background information.In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities.Our method utilizes an entity's prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively.Empirical results suggest that our approach outperforms five baselines and strongly correlates with human judgments.Furthermore, we show that our detector, when used as a reward signal in an off-line reinforcement learning (RL) algorithm, significantly improves the factuality of summaries while maintaining the level of abstractiveness. 1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing work focuses on investigating the causes of hallucination for specific tasks and small language models.
“Most exiting work mainly focuses on investigating the causes of hallucination for specific tasks and small language models (Cao et al., 2022;Zheng et al., 2023;Das et al., 2023).”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Cao et al., 2022
The Conclusion describes this paper as investigating hallucination in abstractive summarization, a specific task, and Appendix A.6 asks where the model learns to hallucinate. But the paper does not establish that most existing work investigates causes, or that it focuses on small language models; its Additional information section says the experiments use large BART models. Page numbers are unavailable in our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe paper uses hallucination patterns from prior work: four QA patterns (comprehension, factualness, specificity, inference), three dialogue patterns (extrinsic-soft, extrinsic-hard, extrinsic-grouped), and three summarization patterns (factual, non-factual, intrinsic).
“Specifically, we consider four types of hallucination patterns for question answering (i.e., comprehension, factualness, specificity, and inference) (Zheng et al., 2023), three types of hallucination patterns for knowledgegrounded dialogue (i.e., extrinsic-soft, extrinsichard, and extrinsic-grouped) (Das et al., 2023), and three types of hallucination patterns for text summarization (i.e., factual”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Cao et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · see citing works
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197 · see citing works
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132 · see citing works
Self-citationOn Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Wang, C., & Sennrich, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.326 · Cited by 118 · see citing works
Reducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99 · see citing works
Flagged first, then by citations
Research works printed but not linked: 19
- Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi- fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computa- tional Linguistics. · DOI
- Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 864-870, Online. As- sociation for Computational Linguistics. · DOI · Paper page
- Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 3592-3603, Online. Association for Computational Linguistics. · DOI · Paper page
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28, pages 1693-1701.
- Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High qual- ity word alignments without parallel training data using static and contextualized embeddings. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1627-1643, Online. Association for Computational Linguistics. · DOI
- Daniel Kang and Tatsunori Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 718-731, Online. Association for Computational Linguistics. · DOI
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity- level factual consistency of abstractive text summa- rization. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 2727-2733, Online. Association for Computational Linguistics.
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
- Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simoes, and Ryan McDonald. 2021. Planning with entity chains for abstractive summarization. arXiv preprint arXiv:2104.07606. · arXiv
- Ani Nenkova and Rebecca J. Passonneau. 2004. Evalu- ating content selection in summarization: The pyra- mid method. In Proceedings of the Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152.
- Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
- Richard Yuanzhe Pang and He He. 2021. Text gener- ation by learning from demonstrations. In Interna- tional Conference on Learning Representations.
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Re- thinking the inception architecture for computer vi- sion. In Proceedings of IEEE Conference on Com- puter Vision and Pattern Recognition,.
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics. · DOI
- Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020. Un- derstanding neural abstractive summarization models via uncertainty. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 6275-6281, Online. As- sociation for Computational Linguistics. · DOI
- Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339.
- PMLR.
- Chunting Zhou, Jiatao Gu, Mona Diab, Paco Guz- man, Luke Zettlemoyer, and Marjan Ghazvinine- jad. 2020. Detecting hallucinated content in con- ditional neural sequence generation. arXiv preprint arXiv:2011.02593. · arXiv
Cited by
Sorted by citations
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 288 · see citing works
“…desirable to lawyers if they resulted in generated language that, for example, removed unnecessary information from a given ar-gument (at the expense of being faithful to it) or invented a novel analogy never yet proposed (at the expense of being grounded in the lexicon) (Cao et al., 2022).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“, 2023), including abstractive summarization (Maynez et al., 2020; Cao et al., 2022), dialogue generation (Shuster et al.”
BackgroundLanguage models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30 · see citing works
Fine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14 · see citing works
“Other use of the term ‘hallucination’ refers to summarization or translation system outputs not supported by the reference text (Maynez et al., 2020; Zhang et al., 2020) even if they are factual (Cao et al., 2022).”
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Fang, B., Dai, X., & Karimi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.578 · Cited by 4 · see citing works
HallE-Control: Controlling Object Hallucination in Large Multimodal Models
Zhai, B., Yang, S., Xu, C., Shen, S., Keutzer, K., Li, C., & Li, M. (2023) · arXiv · doi:10.48550/arxiv.2310.01779 · Cited by 2 · see citing works
- Authors
- Cao, M., Dong, Y., & Cheung, J.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 3340-3354
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4206850841
- Cited by
- 87
- References
- 10
- Citations by year
- 2026: 15 · 2025: 19 · 2024: 12 · 2023: 32 · 2022: 9
- FWCI
- 9.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Compute Canada
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination detection, factual hallucination, abstractive summarization, masked language models, reward signals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
