Reducing Quantity Hallucinations in Abstractive Summarization
Open accessAbstract
It is well-known that abstractive summaries are subject to hallucination---including material that is not supported by the original text. While summaries can be made hallucination-free by limiting them to general phrases, such summaries would fail to be very informative. Alternatively, one can try to avoid hallucinations by verifying that any specific entities in the summary appear in the original text in a similar context. This is the approach taken by our system, Herman. The system learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.) in a beam-worth of abstractive summaries produced by state-of-the-art models, in order to up-rank those summaries whose quantity terms are supported by the original text. Experimental results demonstrate that the ROUGE scores of such up-ranked summaries have a higher Precision than summaries that have not been up-ranked, without a comparable loss in Recall, resulting in higher F$_1$. Preliminary human evaluation of up-ranked vs. original summaries shows people's preference for the former.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Autoregressive generation for abstractive compression is slow and may produce hallucinated content.
“However, this autoregressive generation process is slow and it may produce hallucinated content (Zhao et al., 2020).”
From LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024) · cites this paper as Zhao et al., 2020
The Abstract (p. 1) and Introduction (p. 1) state that abstractive summaries can contain hallucinated facts. The paper does not state that autoregressive generation is slow; searches for generation speed, latency, and inference time did not establish that part of the claim. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe correctness of numerals in generated text, such as date, quantities and scalars, is important for readers.
“However, the correctness of numerals in generated text, such as date, quantities and scalars are important for readers [246, 315, 321].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [321]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,953 · see citing works
(2004) · Cited by 9,718 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Open Information Extraction from the Web
Etzioni, O., Banko, M., Cafarella, M. J., Soderland, S., & Weld, D. S. (2007) · CACM · doi:10.1145/1409360.1409378 · Cited by 2,620 · see citing works
Text Chunking using Transformation-Based Learning
Ramshaw, L., & Marcus, M. P. (1995) · VLC@ACL · doi:10.1007/978-94-017-2390-9_10 · Cited by 2,031 · see citing works
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,024 · see citing works
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,779 · see citing works
Open information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · Cited by 1,410 · see citing works
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · see citing works
Nallapati, R. M., Zhai, F., & Zhou, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10958 · Cited by 1,262 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
Self-citationOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
New York Times Annotated Corpus
Sandhaus, E., & Linguistic Data Consortium (2012) · TIB Data Manager · doi:10.57702/qbrk4fuh · Cited by 624 · see citing works
A Class of Submodular Functions for Document Summarization
Lin, H. Z., & Bilmes, J. (2011) · Cited by 621 · see citing works
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Grusky, M., Naaman, M., & Artzi, Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1065 · Cited by 491 · see citing works
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 359 · see citing works
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335 · see citing works
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · see citing works
Global Inference for Sentence Compression: An Integer Linear Programming Approach
Clarke, J., & Lapata, M. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2433 · Cited by 303 · see citing works
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · see citing works
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171 · see citing works
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · see citing works
BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization
Wang, K., Quan, X., & Wang, R. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1207 · Cited by 59 · see citing works
Text specificity and impact on quality of news summaries
Louis, A., & Nenkova, A. (2012) · Cited by 20 · see citing works
(2015) · doi:10.4230/oasics.slate.2021.8 · Cited by 1 · see citing works
Flagged first, then by citations
3 of the 42 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015b. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
- Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointergenerator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 10731083, Vancouver, Canada. Association for Computational Linguistics.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
“However, the correctness of the numerals in the generated texts such as date, quantity and scalar are important for readers [168, 212, 216].”
BackgroundInfluentialHaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
AI Hallucinations: A Misnomer Worth Clarifying
Maleki, N., Padmanabhan, B., & Dutta, K. (2024) · 2024 IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00033 · Cited by 194 · see citing works
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3332837 · Cited by 145 · see citing works
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.532 · Cited by 135 · see citing works
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175 · see citing works
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
Cao, M., Dong, Y., & Cheung, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.236 · Cited by 87 · see citing works
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features
Rashkin, H., Reitter, D., Tomar, G. S., & Das, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.58 · Cited by 74 · see citing works
HILL: A Hallucination Identifier for Large Language Models
Leiser, F., Eckhardt, S., Leuthe, V., Knaeble, M., Maedche, A., Schwabe, G., & Sunyaev, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642428 · Cited by 72 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 75 · see citing works
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
Huang, Y., Feng, X., Feng, X., & Qin, B. (2021) · arXiv · doi:10.48550/arxiv.2104.14839 · Cited by 62 · see citing works
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.42 · Cited by 75 · see citing works
Improving Abstractive Summarization of Legal Rulings through Textual Entailment
Feijo, D. D. V., & Moreira, V. P. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-021-09305-4 · Cited by 59 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56 · see citing works
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Tang, L., Goyal, T., Fabbri, A. R., Laban, P., Xu, J., Yavuz, S., Kryściński, W., Rousseau, J. F., & Durrett, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.650 · Cited by 62 · see citing works
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.57 · Cited by 58 · see citing works
Controllable Neural Dialogue Summarization with Personal Named Entity Planning
Liu, Z., & Chen, N. F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.8 · Cited by 52 · see citing works
Searle, T., Ibrahim, Z. M., Teo, J., & Dobson, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104358 · Cited by 52 · see citing works
Ladhak, F., Durmus, E., He, H., Cardie, C., & McKeown, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.100 · Cited by 46 · see citing works
Scim: Intelligent Skimming Support for Scientific Papers
Fok, R., Kambhamettu, H., Soldaini, L., Bragg, J., Lo, K., Hearst, M. A., Head, A., & Weld, D. S. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584034 · Cited by 43 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
Bibliographic Research with ChatGPT may be Misleading: The Problem of Hallucination
Metze, K., Morandin-Reis, R. C., Lorand-Metze, I., & Florindo, J. B. (2024) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2023.08.018 · Cited by 40 · see citing works
Ando, K., Okumura, T., Komachi, M., Horiguchi, H., & Matsumoto, Y. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000158 · Cited by 36 · see citing works
Balachandran, V., Hajishirzi, H., Cohen, W. W., & Tsvetkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 32 · see citing works
Learning to Revise References for Faithful Summarization
Adams, G., Shing, H.-C., Sun, Q., Winestock, C., McKeown, K. R., & Elhadad, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.296 · Cited by 30 · see citing works
Multilingual Summarization with Factual Consistency Evaluation
Aharoni, R., Narayan, S., Maynez, J., Herzig, J., Clark, E., & Lapata, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.220 · Cited by 36 · see citing works
QuestEval: Summarization Asks for Fact-based EvaluationPreprint
Scialom, T., Dray, P.-A., Gallinari, P., Lamprier, S., Piwowarski, B., Staiano, J., & Wang, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12693 · Cited by 22 · see citing works
ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts
Mukherjee, R., Bohra, A., Banerjee, A., Sharma, S., Hegde, M., Shaikh, A., Shrivastava, S., Dasgupta, K., Ganguly, N., Ghosh, S., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.748 · Cited by 23 · see citing works
Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
Quevedo, E., Yero, J., Koerner, R., Rivas, P., & Cerny, T. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86623-4_13 · Cited by 32 · see citing works
Hierarchical Summarization for Longform Spoken Dialog
Li, D., Chen, T., Tung, A., & Chilton, L. B. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474771 · Cited by 19 · see citing works
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Chen, Y., Liu, P., & Qiu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.179 · Cited by 18 · see citing works
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E. M., & Cohen, S. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.551 · Cited by 21 · see citing works
Self-citationSEASum: Syntax-Enriched Abstractive Summarization
Liu, S., Yang, L., & Cai, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116819 · Cited by 17 · see citing works
Dong, Y., Wieting, J., & Verga, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.76 · Cited by 16 · see citing works
Semantic Similarity Based Evaluation for Abstractive News Summarization
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.3 · Cited by 17 · see citing works
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 18 · see citing works
Self-citationTowards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking
Elaraby, M., Zhong, Y., & Litman, D. J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.481 · Cited by 14 · see citing works
X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization
Chaudhury, S., Swaminathan, S., Gunasekara, C., Crouse, M., Ravishankar, S., Kimura, D., Murugesan, K., Astudillo, R. F., Naseem, T., Kapanipathi, P., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.478 · Cited by 14 · see citing works
A Comprehensive Survey on Legal Summarization: Challenges and Future Directions
Akter, M., Çano, E., Weber, E., Dobler, D., & Habernal, I. (2025) · ACM Computing Surveys · doi:10.1145/3776586 · Cited by 13 · see citing works
Abstractive Summarizers Become Emotional on News Summarization
Ahuir, V., González, J. Á., Hurtado, L.-F., & Segarra, E. (2024) · Applied Sciences · doi:10.3390/app14020713 · Cited by 15 · see citing works
Abstractive summarization with deep reinforcement learning using semantic similarity rewards
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000505 · Cited by 12 · see citing works
Factual Error Correction for Abstractive Summaries Using Entity Retrieval
Lee, H., Park, C., Yoon, S., Bui, T., Dernoncourt, F., Kim, J., & Jung, K. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.41 · Cited by 10 · see citing works
Token-Level Fact Correction in Abstractive Summarization
Shin, J., Park, S.-B., & Song, H.-J. (2023) · IEEE Access · doi:10.1109/access.2022.3233854 · Cited by 10 · see citing works
Learning with Rejection for Abstractive Text Summarization
Cao, M., Dong, Y., He, J., & Cheung, J. C. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.663 · Cited by 10 · see citing works
PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India
Urlana, A., Chen, P., Zhao, Z., Cohen, S. B., Shrivastava, M., & Haddow, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.777 · Cited by 12 · see citing works
Self-citationGenerating EDU Extracts for Plan-Guided Summary Re-Ranking
Adams, G., Fabbri, A. R., Ladhak, F., Elhadad, N., & McKeown, K. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.151 · Cited by 9 · see citing works
Improving abstractive summarization with energy-based re-ranking
Pernes, D., Mendes, A., & Martins, A. F. T. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.1 · Cited by 9 · see citing works
Efficient Fine-Tuning Large Language Models for Knowledge-Aware Response Planning
Nguyen, M., Kishan, K. C., Nguyen, T., Chadha, A., & Vu, T. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43415-0_35 · Cited by 8 · see citing works
On the Trade-off between Redundancy and Cohesiveness in Extractive Summarization
Cardenas, R., Gallé, M., & Cohen, S. B. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15191 · Cited by 8 · see citing works
Self-citationEnhanced ICD-10 code assignment of clinical texts: A summarization-based approach
Sun, Y., Sang, L., Wu, D., He, S., Chen, Y., Duan, H., Chen, H., & Lü, X. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102967 · Cited by 8 · see citing works
Cl2sum: abstractive summarization via contrastive prompt constructed by LLMs hallucination
Huang, X., Nong, Q., Wang, X., Zhang, H., Du, K., Yin, C., Li, Y., Yan, B., & Zhang, X. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01795-y · Cited by 7 · see citing works
Summary Explorer: Visualizing the State of the Art in Text Summarization
Syed, S., Yousef, T., Al‐Khatib, K., Jänicke, S., & Potthast, M. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.22 · Cited by 7 · see citing works
Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.76 · Cited by 7 · see citing works
A Step-by-Step Gradient Penalty with Similarity Calculation for Text Summary Generation
Zhao, S., Li, Q., He, T., & Wen, J. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-11031-0 · Cited by 7 · see citing works
Expository Text Generation: Imitate, Retrieve, Paraphrase
Balepur, N., Huang, J., & Chang, K. C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.729 · Cited by 12 · see citing works
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Liu, Y. H., Cao, M., Blodgett, S. L., Cheung, J. C. K., Olteanu, A., & Trischler, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.413 · Cited by 6 · see citing works
Reducing extrinsic hallucination in multimodal abstractive summaries with post-processing technique
Rafi, S. M., Laitonjam, L., & Das, R. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10895-8 · Cited by 6 · see citing works
Inspecting the Factuality of Hallucinated Entities in Abstractive Summarization.Preprint
Cao, M., Dong, Y., & Cheung, J. C. K. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization
Xu, X., Dušek, O., Narayan, S., Rieser, V., & Konstas, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.133 · Cited by 7 · see citing works
Improving Faithfulness by Augmenting Negative Summaries from Fake Documents
Wang, T., Ladhak, F., Durmus, E., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.816 · Cited by 5 · see citing works
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Kamoi, R., Goyal, T., & Durrett, G. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.11 · Cited by 5 · see citing works
Entity-based De-noising Modeling for Controllable Dialogue Summarization
Liu, Z., & Chen, N. F. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.40 · Cited by 4 · see citing works
Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
Ando, K., Komachi, M., Okumura, T., Horiguchi, H., & Matsumoto, Y. (2022) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai57707.2022.00034 · Cited by 4 · see citing works
q2d: Turning Questions into Dialogs to Teach Models How to Search
Bitton, Y., Cohen-Ganor, S., Hakimi, I., Lewenberg, Y., Aharoni, R., & Weinreb, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.843 · Cited by 5 · see citing works
Fok, R., Soldaini, L., Trier, C., Bransom, E., MacMillan, K., Cheng, E. Y.-Y., Kambhamettu, H., Bragg, J., Lo, K., Hearst, M. A., et al. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3665648 · Cited by 3 · see citing works
Asynchronous Event Driven Brain Teaser Using Node.js
Subbulakshmi, N., Begum, S. A., Vignesh, R. V., & Chandru, R. (2024) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc60852.2024.10689905 · Cited by 3 · see citing works
Farr, D. T., Ng, L. H. X., Cruickshank, I. J., Manzonelli, N., Clark, N., Starbird, K., Bastian, N. D., & West, J. D. (2025) · IEEE Access · doi:10.1109/access.2025.3585414 · Cited by 3 · see citing works
FineSum: Target-Oriented, Fine-Grained Opinion Summarization
Ge, S., Huang, J., Yu, M., & Han, J. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570397 · Cited by 3 · see citing works
Lan, Z., Li, W., Su, J., Xiao, X., Liu, J., Wu, W., & Lyu, Y. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14267 · Cited by 3 · see citing works
How Optimal is Greedy Decoding for Extractive Question Answering?Preprint
Castel, O., Ram, O., Efrat, A., & Levy, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05857 · Cited by 4 · see citing works
STRONG – Structure Controllable Legal Opinion Summary Generation
Zhong, Y., & Litman, D. J. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.37 · Cited by 4 · see citing works
CARE: A Framework for Correcting Numerical Hallucinations in LLM-Generated Financial Texts
Kim, J., & Jung, W. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00018 · Cited by 2 · see citing works
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 4 · see citing works
Mitigating Media Bias through Neutral Article GenerationPreprint
Lee, N., Bang, Y., Madotto, A., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00336 · Cited by 6 · see citing works
Prompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2022) · doi:10.36227/techrxiv.16652392.v2 · Cited by 2 · see citing works
Exploring the Influence of Dialog Input Format for Unsupervised Clinical Questionnaire Filling
Toudeshki, F. G., Liednikova, A., Jolivet, P., & Gardent, C. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.1 · Cited by 1 · see citing works
Liu, J., Du, W., Li, Q., Wang, X., Zhou, Z., & Liu, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448276 · Cited by 1 · see citing works
Motghare, M., & Agrawal, A. (2024) · International Conference on Artificial Intelligence and Quantum Computation-Based Sensor Application (ICAIQSA) · doi:10.1109/icaiqsa64000.2024.10882315 · Cited by 1 · see citing works
LexiSem: A re-ranker balancing lexical and semantic quality for enhanced abstractive summarization
Aloraini, E., Kassab, H., Hamdi, A., & Shaban, K. B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130816 · Cited by 1 · see citing works
IILAP: Interactive Information Literacy Assessment Platform
Dadic, P., & Магдановна, Е. Л. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-06136-2_28 · Cited by 1 · see citing works
Impact of Domain Dependent Text Preprocessing on Summarization of Indian Legal Documents
Nikita, Rana, D. P., & Mehta, R. G. (2026) · SN Computer Science · doi:10.1007/s42979-026-04735-w · Cited by 1 · see citing works
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
Zhu, R., Qi, J., & Lau, J. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.377 · Cited by 4 · see citing works
Guo, B., Feng, C., Liu, F., Li, X., & Wang, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7894-6_11 · Cited by 1 · see citing works
Improving Consistency for Text Summarization with Energy Functions
Zeng, Q., Yin, Q., Li, Z., Gao, Y., Nag, S., Wang, Z., Yin, B., Ji, H., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.798 · Cited by 1 · see citing works
Who Are All The Stochastic Parrots Imitating? They Should Tell Us!
Shaier, S., Hunter, L., & Kann, K. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.13 · Cited by 2 · see citing works
A Study on Deep Learning Frameworks for Opinion Summarization
Ramakrishnan, S., & Babu, L. D. D. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-981-19-4182-5_18 · Cited by 1 · see citing works
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable FeaturesPreprint
Rashkin, H., Reitter, D., Tomar, G. S., & Das, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06963 · Cited by 1 · see citing works
Prompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2021) · doi:10.36227/techrxiv.16652392.v1 · Cited by 1 · see citing works
Prompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2021) · doi:10.36227/techrxiv.16652392 · Cited by 1 · see citing works
Hierarchical Summarization for Longform Spoken DialogPreprint
Li, D., Chen, T., Tung, A., & Chilton, L. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09597
Summary Explorer: Visualizing the State of the Art in Text SummarizationPreprint
Syed, S., Yousef, T., Al‐Khatib, K., Jänicke, S., & Potthast, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01879
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09209 · Cited by 14 · see citing works
Factual Error Correction in Summarization with Retriever-Reader Pipeline
Li, W., Liu, J., & Gao, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30105-6_33
Nong, Q., Zhang, X., Jin, Z. G., Gao, C., Shang, W., Du, K., Ding, Z., & Ma, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4600240
Yu, H. Q., Scanlon, B., & Reiff‐Marganiec, S. (2025) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose67019.2025.00005
Sebire, I. G., Barale, C., & Lapata, M. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769237
Human-Machine Dialogue in the Medical Field. Using Dialog to Collect Important Patient Information
Liednikova, A. (2022) · theses.fr (ABES) · doi:10.70675/b572f8cbz5876z4683z828bzfb09085ddd78
Enhancing online reading experiences for scientific documents: a study on layout usability
Saeed, Z., Faisal, C. M. N., Aunsa, A. D. H., & Iqbal, M. Z. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2671392
HIVE: Understanding Post-hallucination Reasoning in Vision Language Models
He, F., Wang, Z., Wang, Q., Guan, Q., Liu, D., Tang, R., & Li, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37271-0_34
- Authors
- Zhao, Z., Cohen, S. B., & Webber, B.
- Affiliations
- University of Edinburgh (GB)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Published
- 2020
- Pages
- 2237-2249
- OpenAlex
- W3099474967
- Cited by
- 99
- References
- 42
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 10 · 2023: 28 · 2022: 25 · 2021: 22
- FWCI
- 7.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination mitigation, abstractive summarization, beam search, human evaluation of summaries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
