Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Despite the remarkable performance of generative large language models (LLMs) on abstractive summarization, they face two significant challenges: their considerable size and tendency to hallucinate. Hallucinations are concerning because they erode reliability and raise safety issues. Pruning is a technique that reduces model size by removing redundant weights, enabling more efficient sparse inference. Pruned models yield downstream task performance comparable to the original, making them ideal alternatives when operating on a limited budget. However, the effect that pruning has upon hallucinations in abstractive summarization with LLMs has yet to be explored. In this paper, we provide an extensive empirical study across five summarization datasets, two state-of-the-art pruning methods, and five instruction-tuned LLMs. Surprisingly, we find that hallucinations are less prevalent from pruned LLMs than the original models. Our analysis suggests that pruned models tend to depend more on the source document for summary generation. This leads to a higher lexical overlap between the generated summary and the source document, which could be a reason for the reduction in hallucination risk.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is crucial as both a matter of principle and to mitigate the well-documented tendency of LLMs to produce”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Chrysostomou et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“…work observed that models tend to generate hallucinatory content that is not based on or cannot be entailed from the source document (Vinyals and Le, 2015; Rohrbach et al., 2018; Cao et al., 2018; Maynez et al., 2020; Raunak et al., 2021; Falke et al., 2019; Maynez et al., 2020; Chen et al., 2022).”
Background
Research works printed but not linked: 49
- Adams, 2023, From sparse to dense: GPT-4 summarization with chain of density prompting, 68, 10.18653/v1/2023.newsum-1.7 · DOI
- Almazrouei, 2023, The Falcon series of open language models, arXiv preprint, arXiv:2311.16867 · arXiv
- Cao, 2018, Faithful to the original: Fact-aware neural abstractive summarization, 10.1609/aaai.v32i1.11912 · DOI · Paper page
- Choubey, 2023, CaPE: Contrastive parameter ensembling for reducing hallucination in abstractive summarization, 10755, 10.18653/v1/2023.findings-acl.685 · DOI
- Chrysostomou, 2022, An empirical study on explanations in out-of-domain settings, 6920, 10.18653/v1/2022.acl-long.477 · DOI
- Ciosici, 2021, Perhaps PTLMs should go to school – a task to assess open book and closed book QA, 6104, 10.18653/v1/2021.emnlp-main.493 · DOI
- Cohn, 2008, Sentence compression beyond word deletion, 137, 10.3115/1599081.1599099 · DOI · Paper page
- Dettmers, 2022, GPT3.int8(): 8-bit matrix multiplication for transformers at scale
- Dun, 2023, Sweeping heterogeneity with smart mops: Mixture of prompts for LLM task adaptation, arXiv preprint
- Fierro, 2022, Factual consistency of multilingual pretrained language models, 3046, 10.18653/v1/2022.findings-acl.240 · DOI
- Frantar, 2022, Optimal brain compression: A framework for accurate post-training quantization and pruning, 4475
- Frantar, 2023, OPTQ: Accurate quantization for generative pre-trained transformers
- Guerreiro, 2023, Optimal transport for unsupervised hallucination detection in neural machine translation, 13766, 10.18653/v1/2023.acl-long.770 · DOI
- Guzmán, 2019, The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English, 6098, 10.18653/v1/D19-1632 · DOI · Paper page
- Hagiwara, 1994, A simple and effective method for removal of hidden units and weights, Neurocomputing, 6, 207, 10.1016/0925-2312(94)90055-8 · DOI
- Han, 2015, Learning both weights and connections for efficient neural network
- Hasan, 2024, Pruning for protection: Increasing jailbreak resistance in aligned LLMs without fine-tuning, arXiv preprint
- Hassibi, 1993, Optimal brain surgeon and general network pruning, 293, 10.1109/ICNN.1993.298572 · DOI · Paper page
- Huang, 2020, What have we achieved on text summarization?, 446, 10.18653/v1/2020.emnlp-main.33 · DOI · Paper page
- Srinivasan Iyer , XiVictoria Lin, RamakanthPasunuru, TodorMihaylov, DanielSimig, PingYu, KurtShuster, TianluWang, QingLiu, Punit SinghKoura, XianLi, BrianO’Horo, GabrielPereyra, JeffWang, ChristopherDewan, AsliCelikyilmaz, LukeZettlemoyer, and VesStoyanov. 2023. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint, arXiv:2212.12017. · arXiv
- Jaiswal, 2024, Compressing LLMs: The truth is rarely pure and never simple
- Ji, 2023, Towards mitigating LLM hallucination via self reflection, 1827, 10.18653/v1/2023.findings-emnlp.123 · DOI · Paper page
- Jiang, 2023, Mistral 7B, arXiv preprint
- King, 2022, Don’t say what you don’t know: Improving the consistency of abstractive summarization by constraining beam search, 555, 10.18653/v1/2022.gem-1.51 · DOI
- Laban, 2023, SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization, 9662, 10.18653/v1/2023.emnlp-main.600 · DOI
- Lango, 2023, Critic-driven decoding for mitigating hallucinations in data-to-text generation, 2853, 10.18653/v1/2023.emnlp-main.172 · DOI
- Md, 2023, Building real-world meeting summarization systems using large language models: A practical perspective, 343, 10.18653/v1/2023.emnlp-industry.33 · DOI
- LeCun, 1989, Optimal brain damage
- Lin, 2004, ROUGE: A package for automatic evaluation of summaries, 74
- Ma, 2023, LLM-Pruner: On the structural pruning of large language models
- Manor, 2019, Plain English summarization of contracts, 1, 10.18653/v1/W19-2201 · DOI
- Merity, 2017, Pointer sentinel mixture models
- Milintsevich, 2023, Calvados at MEDIQA-chat 2023: Improving clinical note generation with multi-task instruction finetuning, 529, 10.18653/v1/2023.clinicalnlp-1.56 · DOI
- Namburi, 2023, The cost of compression: Investigating the impact of compression on parametric knowledge in language models, 5255, 10.18653/v1/2023.findings-emnlp.349 · DOI
- OpenAI, JoshAchiam, StevenAdler, SandhiniAgarwal, LamaAhmad, IlgeAkkaya, Florencia LeoniAleman, DiogoAlmeida, JankoAltenschmidt, SamAltman, ShyamalAnadkat, RedAvila, IgorBabuschkin, SuchirBalaji, ValerieBalcom, PaulBaltescu, HaimingBao, MohammadBavarian, JeffBelgum, IrwanBello, JakeBerdine, GabrielBernadett-Shapiro, ChristopherBerner, LennyBogdonoff, OlegBoiko, MadelaineBoyd, Anna-LuisaBrakman, GregBrockman, TimBrooks, MilesBrundage, KevinButton, TrevorCai, RosieCampbell, AndrewCann, BrittanyCarey, ChelseaCarlson, RoryCarmichael, BrookeChan, CheChang, FotisChantzis, DerekChen, SullyChen, RubyChen, JasonChen, MarkChen, BenChess, ChesterCho, CaseyChu, Hyung WonChung, DaveCummings, JeremiahCurrier, YunxingDai, CoryDecareaux, ThomasDegry, NoahDeutsch, DamienDeville, ArkaDhar, DavidDohan, SteveDowling, SheilaDunning, AdrienEcoffet, AttyEleti, TynaEloundou, DavidFarhi, LiamFedus, NikoFelix, Simón PosadaFishman, JustonForte, IsabellaFulford, LeoGao, ElieGeorges, ChristianGibson, VikGoel, TarunGogineni, GabrielGoh, RaphaGontijo-Lopes, JonathanGordon, MorganGrafstein, ScottGray, RyanGreene, JoshuaGross, Shixiang ShaneGu, YufeiGuo, ChrisHallacy, JesseHan, JeffHarris, YuchenHe, MikeHeaton, JohannesHeidecke, ChrisHesse, AlanHickey, WadeHickey, PeterHoeschele, BrandonHoughton, KennyHsu, ShengliHu, XinHu, JoostHuizinga, ShantanuJain, ShawnJain, JoanneJang, AngelaJiang, RogerJiang, HaozhunJin, DennyJin, ShinoJomoto, BillieJonn, HeewooJun, TomerKaftan, ŁukaszKaiser, AliKamali, IngmarKanitscheider, Nitish ShirishKeskar, TabarakKhan, LoganKilpatrick, Jong WookKim, ChristinaKim, YongjikKim, Jan HendrikKirchner, JamieKiros, MattKnight, DanielKokotajlo, ŁukaszKondraciuk, AndrewKondrich, ArisKonstantinidis, KyleKosic, GretchenKrueger, VishalKuo, MichaelLampe, IkaiLan, TeddyLee, JanLeike, JadeLeung, DanielLevy, Chak MingLi, RachelLim, MollyLin, StephanieLin, MateuszLitwin, TheresaLopez, RyanLowe, PatriciaLue, AnnaMakanju, KimMalfacini, SamManning, TodorMarkov, YanivMarkovski, BiancaMartin, KatieMayer, AndrewMayne, BobMcGrew, Scott MayerMcKinney, ChristineMcLeavey, PaulMcMillan, JakeMcNeil, DavidMedina, AalokMehta, JacobMenick, LukeMetz, AndreyMishchenko, PamelaMishkin, VinnieMonaco, EvanMorikawa, DanielMossing, TongMu, MiraMurati, OlegMurk, DavidMély, AshvinNair, ReiichiroNakano, RajeevNayak, ArvindNeelakantan, RichardNgo, HyeonwooNoh, LongOuyang, CullenO’Keefe, JakubPachocki, AlexPaino, JoePalermo, AshleyPantuliano, GiambattistaParascandolo, JoelParish, EmyParparita, AlexPassos, MikhailPavlov, AndrewPeng, AdamPerelman, Filipede Avila Belbute Peres, MichaelPetrov, Henrique Pondede Oliveira Pinto, MichaelPokorny, MichellePokrass, Vitchyr H.Pong, TollyPowell, AletheaPower, BorisPower, ElizabethProehl, RaulPuri, AlecRadford, JackRae, AdityaRamesh, CameronRaymond, FrancisReal, KendraRimbach, CarlRoss, BobRotsted, HenriRoussez, NickRyder, MarioSaltarelli, TedSanders, ShibaniSanturkar, GirishSastry, HeatherSchmidt, DavidSchnurr, JohnSchulman, DanielSelsam, KylaSheppard, TokiSherbakov, JessicaShieh, SarahShoker, PranavShyam, SzymonSidor, EricSigler, MaddieSimens, JordanSitkin, KatarinaSlama, IanSohl, BenjaminSokolowsky, YangSong, NatalieStaudacher, Felipe PetroskiSuch, NatalieSummers, IlyaSutskever, JieTang, NikolasTezak, Madeleine B.Thompson, PhilTillet, AminTootoonchian, ElizabethTseng, PrestonTuggle, NickTurley, JerryTworek, Juan Felipe CerónUribe, AndreaVallone, ArunVijayvergiya, ChelseaVoss, CarrollWainwright, Justin JayWang, AlvinWang, BenWang, JonathanWard, JasonWei, C. J.Weinmann, AkilaWelihinda, PeterWelinder, JiayiWeng, LilianWeng, MattWiethoff, DaveWillner, ClemensWinter, SamuelWolrich, HannahWong, LaurenWorkman, SherwinWu, JeffWu, MichaelWu, KaiXiao, TaoXu, SarahYoo, KevinYu, QimingYuan, WojciechZaremba, RowanZellers, ChongZhang, MarvinZhang, ShengjiaZhao, TianhaoZheng, JuntangZhuang, WilliamZhuk, and BarretZoph. 2024. GPT-4 technical report. arXiv preprint, arXiv:2303.08774. · arXiv
- Saggion, 2013, Automatic Text Summarization: Past, Present and Future, 3, 10.1007/978-3-642-28569-1_1 · DOI · Paper page
- Son, 2022, HaRiM +: Evaluating summary quality with hallucination risk, 895
- Tang, 2023, Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors, 11626, 10.18653/v1/2023.acl-long.650 · DOI
- Tang, 2023, In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis, 56, 10.18653/v1/2023.newsum-1.6 · DOI
- Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint
- Wallace, 2021, Generating (factual?) narrative summaries of RCTs: Experiments with neural multi-document summarization, AMIA Summits on Translational Science Proceedings, 2021, 605
- Williams, 2023, How does calibration data affect the post-training pruning and quantization of large language models?, arXiv preprint
- Wolf, 2020, Transformers: State-of-the-art natural language processing, 38, 10.18653/v1/2020.emnlp-demos.6 · DOI · Paper page
- Guangxuan, 2022, Can model compression improve NLP fairness, arXiv preprint, arXiv:2201.08542 · arXiv
- Weijia, 2023, Understanding and detecting hallucinations in neural machine translation via model introspection, Transactions of the Association for Computational Linguistics, 11, 546, 10.1162/tacl_a_00563 · DOI · Paper page
- Zhao, 2023, Incorporating attribution importance for improving faithfulness metrics, 4732, 10.18653/v1/2023.acl-long.261 · DOI
- Zhao, 2022, On the impact of temporal concept drift on model explanations, 4039, 10.18653/v1/2022.findings-emnlp.298 · DOI
- Zhao, 2024, Reagent: A model-agnostic feature attribution method for generative language models, arXiv preprint, arXiv:2402.00794, 10.22541/au.170709121.16176681/v1 · DOI
- Zhao, 2022, Utilizing subjectivity level to mitigate identity term bias in toxic comments classification, Online Social Networks and Media, 29, 100205, 10.1016/j.osnem.2022.100205 · DOI
Cited by
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13
Metadata
- Authors
- Chrysostomou, G., Zhao, Z., Williams, M., & Aletras, N.
- Affiliations
- AstraZeneca, UK
- University of Sheffield, UK. zhixue.zhao@sheffield.ac.uk
- University of Sheffield, UK. mwilliams15@sheffield.ac.uk
- University of Sheffield, UK. n.aletras@sheffield.ac.uk
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 1163-1181
- ISSN
- 2307-387X
- Cited by
- 18
- References
- 42
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization. https://app.cochat.ai/papers/10.1162/tacl_a_00695
- The paper
- Chrysostomou, G., Zhao, Z., Williams, M., & Aletras, N. (2024). Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00695
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.