Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Despite the remarkable performance of generative large language models (LLMs) on abstractive summarization, they face two significant challenges: their considerable size and tendency to hallucinate. Hallucinations are concerning because they erode reliability and raise safety issues. Pruning is a technique that reduces model size by removing redundant weights, enabling more efficient sparse inference. Pruned models yield downstream task performance comparable to the original, making them ideal alternatives when operating on a limited budget. However, the effect that pruning has upon hallucinations in abstractive summarization with LLMs has yet to be explored. In this paper, we provide an extensive empirical study across five summarization datasets, two state-of-the-art pruning methods, and five instruction-tuned LLMs. Surprisingly, we find that hallucinations are less prevalent from pruned LLMs than the original models. Our analysis suggests that pruned models tend to depend more on the source document for summary generation. This leads to a higher lexical overlap between the generated summary and the source document, which could be a reason for the reduction in hallucination risk.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is crucial as both a matter of principle and to mitigate the well-documented tendency of LLMs to produce”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Chrysostomou et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“…work observed that models tend to generate hallucinatory content that is not based on or cannot be entailed from the source document (Vinyals and Le, 2015; Rohrbach et al., 2018; Cao et al., 2018; Maynez et al., 2020; Raunak et al., 2021; Falke et al., 2019; Maynez et al., 2020; Chen et al., 2022).”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · cited 1×
“To mitigate the effect of prompt variability, we summarize each document using three distinct prompt templates (Table 1).”
MethodWhat is the State of Neural Network Pruning?
Blalock, D., Ortiz, J. J. G., Frankle, J., & Guttag, J. (2020) · arXiv · doi:10.48550/arxiv.2003.03033 · Cited by 560 · cited 1×
“Weights may be removed individually (un-structured pruning), according to defined blocks (semi-structured pruning), or in relation to model components (structured pruning) (Blalock et al., 2020; Mishra et al., 2021; Ma et al., 2023).”
BackgroundObject Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420 · cited 1×
“…work observed that models tend to generate hallucinatory content that is not based on or cannot be entailed from the source document (Vinyals and Le, 2015; Rohrbach et al., 2018; Cao et al., 2018; Maynez et al., 2020; Raunak et al., 2021; Falke et al., 2019; Maynez et al., 2020; Chen et al., 2022).”
BackgroundBenchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00632 · Cited by 415
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2301.13848 · Cited by 415
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356 · cited 1×
“QA methods decompose the task to a question answering problem (Wang et al., 2020; Deutsch et al., 2021; Durmus et al., 2020).”
MethodBARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · cited 2×
“For example, Falke et al. (2019) found that 25% of the model generated summaries contain hallucinated content.”
BackgroundUp or Down? Adaptive Rounding for Post-Training Quantization
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., & Blankevoort, T. (2020) · arXiv · doi:10.48550/arxiv.2004.10568 · Cited by 252
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · cited 1×
“LLMs are known to be sensitive to prompt design (Petroni et al., 2019; Elazar et al., 2021; Fierro and Søgaard, 2022).”
BackgroundCompressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · arXiv · doi:10.48550/arxiv.2104.06683 · Cited by 142 · cited 1×
“In abstractive summarization, model hallucinations are a thoroughly studied subject (Cao et al., 2020; Durmus et al., 2020; Raunak et al., 2021; Narayan et al., 2023; Laban et al., 2023).”
BackgroundFactual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
Frantar, E., & Alistarh, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00774 · Cited by 131 · cited 6×
“…update procedure, instead relying upon only the weight magnitudes and norm of the input activations: Sparsity Level Following previous work (Frantar and Alistarh, 2023; Sun et al., 2023), we evaluate our pruning methods across both semi-structured and unstructured settings: • 2:4…”
BackgroundMethodInfluentialDetecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122
Abstractive Summarization: A Survey of the State of the Art
Lin, H., & Ng, V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019815 · Cited by 120 · cited 1×
“Abstractive summarization is the task of distilling the key information from a given document, to generate a summary consisting of text that might not appear in the original document (Cohn and Lap-ata, 2008; Saggion and Poibeau, 2013; Lin and Ng, 2019).”
BackgroundReducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99
Self-citationAccelerating Sparse Deep Neural Networks
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., & Micikevicius, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08378 · Cited by 97
A Simple and Effective Pruning Approach for Large Language Models
Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2023) · arXiv · doi:10.48550/arxiv.2306.11695 · Cited by 92
On hallucination and predictive uncertainty in conditional language generation
Xiao, Y., & Wang, W. Y. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.236 · Cited by 79
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
Deutsch, D., Bedrax-Weiss, T., & Roth, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00397 · Cited by 75
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · arXiv · doi:10.48550/arxiv.2111.09525 · Cited by 38
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
Towards Improving Faithfulness in Abstractive Summarization
Chen, X., Li, M., Gao, X., & Zhang, X. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1780 · Cited by 21
Towards Improving Faithfulness in Abstractive Summarization
Chen, X., Li, M., Gao, X., & Zhang, X. (2022) · arXiv · doi:10.48550/arxiv.2210.01877 · Cited by 20
On the Impact of Calibration Data in Post-training Quantization and Pruning
Williams, M., & Aletras, N. (2023) · arXiv · doi:10.48550/arxiv.2311.09755 · Cited by 10
Self-citation
Research works printed but not linked: 49
- Adams, 2023, From sparse to dense: GPT-4 summarization with chain of density prompting, 68, 10.18653/v1/2023.newsum-1.7 · DOI
- Almazrouei, 2023, The Falcon series of open language models, arXiv preprint, arXiv:2311.16867 · arXiv
- Cao, 2018, Faithful to the original: Fact-aware neural abstractive summarization, 10.1609/aaai.v32i1.11912 · DOI · Paper page
- Choubey, 2023, CaPE: Contrastive parameter ensembling for reducing hallucination in abstractive summarization, 10755, 10.18653/v1/2023.findings-acl.685 · DOI
- Chrysostomou, 2022, An empirical study on explanations in out-of-domain settings, 6920, 10.18653/v1/2022.acl-long.477 · DOI
- Ciosici, 2021, Perhaps PTLMs should go to school – a task to assess open book and closed book QA, 6104, 10.18653/v1/2021.emnlp-main.493 · DOI
- Cohn, 2008, Sentence compression beyond word deletion, 137, 10.3115/1599081.1599099 · DOI · Paper page
- Dettmers, 2022, GPT3.int8(): 8-bit matrix multiplication for transformers at scale
- Dun, 2023, Sweeping heterogeneity with smart mops: Mixture of prompts for LLM task adaptation, arXiv preprint
- Fierro, 2022, Factual consistency of multilingual pretrained language models, 3046, 10.18653/v1/2022.findings-acl.240 · DOI
- Frantar, 2022, Optimal brain compression: A framework for accurate post-training quantization and pruning, 4475
- Frantar, 2023, OPTQ: Accurate quantization for generative pre-trained transformers
- Guerreiro, 2023, Optimal transport for unsupervised hallucination detection in neural machine translation, 13766, 10.18653/v1/2023.acl-long.770 · DOI
- Guzmán, 2019, The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English, 6098, 10.18653/v1/D19-1632 · DOI · Paper page
- Hagiwara, 1994, A simple and effective method for removal of hidden units and weights, Neurocomputing, 6, 207, 10.1016/0925-2312(94)90055-8 · DOI
- Han, 2015, Learning both weights and connections for efficient neural network
- Hasan, 2024, Pruning for protection: Increasing jailbreak resistance in aligned LLMs without fine-tuning, arXiv preprint
- Hassibi, 1993, Optimal brain surgeon and general network pruning, 293, 10.1109/ICNN.1993.298572 · DOI · Paper page
- Huang, 2020, What have we achieved on text summarization?, 446, 10.18653/v1/2020.emnlp-main.33 · DOI · Paper page
- Srinivasan Iyer , XiVictoria Lin, RamakanthPasunuru, TodorMihaylov, DanielSimig, PingYu, KurtShuster, TianluWang, QingLiu, Punit SinghKoura, XianLi, BrianO’Horo, GabrielPereyra, JeffWang, ChristopherDewan, AsliCelikyilmaz, LukeZettlemoyer, and VesStoyanov. 2023. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint, arXiv:2212.12017. · arXiv
- Jaiswal, 2024, Compressing LLMs: The truth is rarely pure and never simple
- Ji, 2023, Towards mitigating LLM hallucination via self reflection, 1827, 10.18653/v1/2023.findings-emnlp.123 · DOI · Paper page
- Jiang, 2023, Mistral 7B, arXiv preprint
- King, 2022, Don’t say what you don’t know: Improving the consistency of abstractive summarization by constraining beam search, 555, 10.18653/v1/2022.gem-1.51 · DOI
- Laban, 2023, SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization, 9662, 10.18653/v1/2023.emnlp-main.600 · DOI
- Lango, 2023, Critic-driven decoding for mitigating hallucinations in data-to-text generation, 2853, 10.18653/v1/2023.emnlp-main.172 · DOI
- Md, 2023, Building real-world meeting summarization systems using large language models: A practical perspective, 343, 10.18653/v1/2023.emnlp-industry.33 · DOI
- LeCun, 1989, Optimal brain damage
- Lin, 2004, ROUGE: A package for automatic evaluation of summaries, 74
- Ma, 2023, LLM-Pruner: On the structural pruning of large language models
- Manor, 2019, Plain English summarization of contracts, 1, 10.18653/v1/W19-2201 · DOI
- Merity, 2017, Pointer sentinel mixture models
- Milintsevich, 2023, Calvados at MEDIQA-chat 2023: Improving clinical note generation with multi-task instruction finetuning, 529, 10.18653/v1/2023.clinicalnlp-1.56 · DOI
- Namburi, 2023, The cost of compression: Investigating the impact of compression on parametric knowledge in language models, 5255, 10.18653/v1/2023.findings-emnlp.349 · DOI
- OpenAI, JoshAchiam, StevenAdler, SandhiniAgarwal, LamaAhmad, IlgeAkkaya, Florencia LeoniAleman, DiogoAlmeida, JankoAltenschmidt, SamAltman, ShyamalAnadkat, RedAvila, IgorBabuschkin, SuchirBalaji, ValerieBalcom, PaulBaltescu, HaimingBao, MohammadBavarian, JeffBelgum, IrwanBello, JakeBerdine, GabrielBernadett-Shapiro, ChristopherBerner, LennyBogdonoff, OlegBoiko, MadelaineBoyd, Anna-LuisaBrakman, GregBrockman, TimBrooks, MilesBrundage, KevinButton, TrevorCai, RosieCampbell, AndrewCann, BrittanyCarey, ChelseaCarlson, RoryCarmichael, BrookeChan, CheChang, FotisChantzis, DerekChen, SullyChen, RubyChen, JasonChen, MarkChen, BenChess, ChesterCho, CaseyChu, Hyung WonChung, DaveCummings, JeremiahCurrier, YunxingDai, CoryDecareaux, ThomasDegry, NoahDeutsch, DamienDeville, ArkaDhar, DavidDohan, SteveDowling, SheilaDunning, AdrienEcoffet, AttyEleti, TynaEloundou, DavidFarhi, LiamFedus, NikoFelix, Simón PosadaFishman, JustonForte, IsabellaFulford, LeoGao, ElieGeorges, ChristianGibson, VikGoel, TarunGogineni, GabrielGoh, RaphaGontijo-Lopes, JonathanGordon, MorganGrafstein, ScottGray, RyanGreene, JoshuaGross, Shixiang ShaneGu, YufeiGuo, ChrisHallacy, JesseHan, JeffHarris, YuchenHe, MikeHeaton, JohannesHeidecke, ChrisHesse, AlanHickey, WadeHickey, PeterHoeschele, BrandonHoughton, KennyHsu, ShengliHu, XinHu, JoostHuizinga, ShantanuJain, ShawnJain, JoanneJang, AngelaJiang, RogerJiang, HaozhunJin, DennyJin, ShinoJomoto, BillieJonn, HeewooJun, TomerKaftan, ŁukaszKaiser, AliKamali, IngmarKanitscheider, Nitish ShirishKeskar, TabarakKhan, LoganKilpatrick, Jong WookKim, ChristinaKim, YongjikKim, Jan HendrikKirchner, JamieKiros, MattKnight, DanielKokotajlo, ŁukaszKondraciuk, AndrewKondrich, ArisKonstantinidis, KyleKosic, GretchenKrueger, VishalKuo, MichaelLampe, IkaiLan, TeddyLee, JanLeike, JadeLeung, DanielLevy, Chak MingLi, RachelLim, MollyLin, StephanieLin, MateuszLitwin, TheresaLopez, RyanLowe, PatriciaLue, AnnaMakanju, KimMalfacini, SamManning, TodorMarkov, YanivMarkovski, BiancaMartin, KatieMayer, AndrewMayne, BobMcGrew, Scott MayerMcKinney, ChristineMcLeavey, PaulMcMillan, JakeMcNeil, DavidMedina, AalokMehta, JacobMenick, LukeMetz, AndreyMishchenko, PamelaMishkin, VinnieMonaco, EvanMorikawa, DanielMossing, TongMu, MiraMurati, OlegMurk, DavidMély, AshvinNair, ReiichiroNakano, RajeevNayak, ArvindNeelakantan, RichardNgo, HyeonwooNoh, LongOuyang, CullenO’Keefe, JakubPachocki, AlexPaino, JoePalermo, AshleyPantuliano, GiambattistaParascandolo, JoelParish, EmyParparita, AlexPassos, MikhailPavlov, AndrewPeng, AdamPerelman, Filipede Avila Belbute Peres, MichaelPetrov, Henrique Pondede Oliveira Pinto, MichaelPokorny, MichellePokrass, Vitchyr H.Pong, TollyPowell, AletheaPower, BorisPower, ElizabethProehl, RaulPuri, AlecRadford, JackRae, AdityaRamesh, CameronRaymond, FrancisReal, KendraRimbach, CarlRoss, BobRotsted, HenriRoussez, NickRyder, MarioSaltarelli, TedSanders, ShibaniSanturkar, GirishSastry, HeatherSchmidt, DavidSchnurr, JohnSchulman, DanielSelsam, KylaSheppard, TokiSherbakov, JessicaShieh, SarahShoker, PranavShyam, SzymonSidor, EricSigler, MaddieSimens, JordanSitkin, KatarinaSlama, IanSohl, BenjaminSokolowsky, YangSong, NatalieStaudacher, Felipe PetroskiSuch, NatalieSummers, IlyaSutskever, JieTang, NikolasTezak, Madeleine B.Thompson, PhilTillet, AminTootoonchian, ElizabethTseng, PrestonTuggle, NickTurley, JerryTworek, Juan Felipe CerónUribe, AndreaVallone, ArunVijayvergiya, ChelseaVoss, CarrollWainwright, Justin JayWang, AlvinWang, BenWang, JonathanWard, JasonWei, C. J.Weinmann, AkilaWelihinda, PeterWelinder, JiayiWeng, LilianWeng, MattWiethoff, DaveWillner, ClemensWinter, SamuelWolrich, HannahWong, LaurenWorkman, SherwinWu, JeffWu, MichaelWu, KaiXiao, TaoXu, SarahYoo, KevinYu, QimingYuan, WojciechZaremba, RowanZellers, ChongZhang, MarvinZhang, ShengjiaZhao, TianhaoZheng, JuntangZhuang, WilliamZhuk, and BarretZoph. 2024. GPT-4 technical report. arXiv preprint, arXiv:2303.08774. · arXiv
- Saggion, 2013, Automatic Text Summarization: Past, Present and Future, 3, 10.1007/978-3-642-28569-1_1 · DOI · Paper page
- Son, 2022, HaRiM +: Evaluating summary quality with hallucination risk, 895
- Tang, 2023, Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors, 11626, 10.18653/v1/2023.acl-long.650 · DOI
- Tang, 2023, In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis, 56, 10.18653/v1/2023.newsum-1.6 · DOI
- Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint
- Wallace, 2021, Generating (factual?) narrative summaries of RCTs: Experiments with neural multi-document summarization, AMIA Summits on Translational Science Proceedings, 2021, 605
- Williams, 2023, How does calibration data affect the post-training pruning and quantization of large language models?, arXiv preprint
- Wolf, 2020, Transformers: State-of-the-art natural language processing, 38, 10.18653/v1/2020.emnlp-demos.6 · DOI · Paper page
- Guangxuan, 2022, Can model compression improve NLP fairness, arXiv preprint, arXiv:2201.08542 · arXiv
- Weijia, 2023, Understanding and detecting hallucinations in neural machine translation via model introspection, Transactions of the Association for Computational Linguistics, 11, 546, 10.1162/tacl_a_00563 · DOI · Paper page
- Zhao, 2023, Incorporating attribution importance for improving faithfulness metrics, 4732, 10.18653/v1/2023.acl-long.261 · DOI
- Zhao, 2022, On the impact of temporal concept drift on model explanations, 4039, 10.18653/v1/2022.findings-emnlp.298 · DOI
- Zhao, 2024, Reagent: A model-agnostic feature attribution method for generative language models, arXiv preprint, arXiv:2402.00794, 10.22541/au.170709121.16176681/v1 · DOI
- Zhao, 2022, Utilizing subjectivity level to mitigate identity term bias in toxic comments classification, Online Social Networks and Media, 29, 100205, 10.1016/j.osnem.2022.100205 · DOI
Cited by
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13
Metadata
- Authors
- Chrysostomou, G., Zhao, Z., Williams, M., & Aletras, N.
- Affiliations
- AstraZeneca, UK
- University of Sheffield, UK. zhixue.zhao@sheffield.ac.uk
- University of Sheffield, UK. mwilliams15@sheffield.ac.uk
- University of Sheffield, UK. n.aletras@sheffield.ac.uk
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 1163-1181
- ISSN
- 2307-387X
- Cited by
- 18
- References
- 42
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.