Benchmarking Large Language Models for News Summarization
Open accessThe DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [70], human evaluation results indicate that humans tend to prefer the results generated by LLMs [42,138] compared to that of fine-tuned models.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [138]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are known to produce errors in many settings, from document summarisation7, to general reasoning tasks as well as more clinically relevant tasks8.”
From A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“LLMs (Bommasani et al., 2021; Chowdhery et al., 2022; Brown et al., 2020) have two distinctive features over previous pretrained models.”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
Method(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method(2005) · Cited by 4,152
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method
Research works printed but not linked: 1
- Conroy, 2006, Back to basics: Classy 2006, 10.12968/sece.2006.11.755s · doi:10.12968/sece.2006.11.755s
Cited by
Items 101–200 of 427
Proceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
Murugaraj, K., Lamsiyah, S., & Schommer, C. (2025) · IEEE Access · doi:10.1109/access.2025.3528733 · Cited by 16
“Furthermore, given the success of large language models (LLMs) in various NLP tasks, including text summarization [24], [25], [53], [54], we further evaluate the performance of recent LLMs on historical text summarization.”
Cao, Y., Chen, Z., Kumar, P., Pei, Q., Yu, Y., Li, H., Dimino, F., Ausiello, L., Subbalakshmi, K. P., & Ndiaye, P. M. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00107 · Cited by 17
“Equipped with vast knowledge bases and advanced zero-shot learning capabilities, LLMs can perform a wide range of text-related tasks, such as summarization [32], question answering [24], and sentiment analysis [30], without requiring task-specific training.”
A Review on Optimization-Based Automatic Text Summarization Approach
Wahab, M. H. H., Ali, N. H., Hamid, N. A. W. A., Subramaniam, S. K., Latip, R. B., & Othman, М. (2023) · IEEE Access · doi:10.1109/access.2023.3348075 · Cited by 19
“The works demonstrated by Tang et al. [51] and Zhang et al. [52] challenge the prevailing sentiment surrounding deep learning-based ATS, showcasing promising summary results.”
BackgroundThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries
Huang, K., Shi, Y., Ding, D., Li, Y., Yang, F., Lakshmanan, L. V. S., & Xiao, X. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749702 · Cited by 39
Kurland, D. B., Alber, D. A., Palla, A., de Souza, D. N., Lau, D., Laufer, I., Frempong-Boadu, A. K., Kondziolka, D. S., & Oermann, E. K. (2025) · Neurosurgery · doi:10.1227/neu.0000000000003354 · Cited by 10
Beyond the surface: stylometric analysis of GPT-4o’s capacity for literary style imitation
Mikros, G. K. (2025) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf035 · Cited by 28
ChatGPT vs. Human Journalists: Analyzing News Summaries Through BERTScore and Moderation Standards
Kim, H.-S., Kang, J.-W., & Choi, S. (2025) · Electronics · doi:10.3390/electronics14112115 · Cited by 13
ECAMP: Entity-centered Context-aware Medical Vision Language Pre-training
Wang, R., Yao, Q., Jiang, Z., Lai, H., He, Z., Tao, X., & Zhou, S. K. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103690 · Cited by 22
Kim, H., Lee, S., You, S. C., Huh, S., Kim, J.-E., Kim, S.-T., Ko, D.-R., Kim, J. H., Lee, J. H., Lim, J. S., et al. (2025) · JMIR Medical Informatics · doi:10.2196/76848 · Cited by 18
“Although smaller models may have limitations in processing lengthy contexts and complex medical information, proper instruction-tuning enables them to perform specific tasks on par with larger models [31].”
MMSFT: Multilingual Multimodal Summarization by Fine-Tuning Transformers
Phani, S., Abdul, A., Prasad, M. K. S., & Sarma, H. K. D. (2024) · IEEE Access · doi:10.1109/access.2024.3454382 · Cited by 11
Assessing Usability of Large Language Models in EducationPreprint
Huovinen, L. (2024) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/p54gk · Cited by 19
PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India
Urlana, A., Chen, P., Zhao, Z., Cohen, S. B., Shrivastava, M., & Haddow, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.777 · Cited by 23
LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models
Yang, X., Zhao, H., Phung, D., Buntine, W., & Du, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00744 · Cited by 30
BETA: A Bit-Grained Transformer Attention Accelerator With Efficient Early Termination
Wang, H., Wang, H., Yue, Z., Liu, J., Wei, T., Wei, S., Hu, Y., & Yin, S. (2025) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2025.3596228 · Cited by 11
“L ARGE language models (LLMs) have demonstrated remarkable proficiency across a broad spectrum of tasks, ranging from content creation [1], [2] to chatbots [3], [4].”
Using large language models to analyze political texts through natural language understanding
Benoit, K. R., de Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
Unsupervised Extractive Summarization with Learnable Length Control Strategies
Jie, R., Meng, X., Jiang, X., & Liu, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29797 · Cited by 10
Generating EDU Extracts for Plan-Guided Summary Re-Ranking
Adams, G., Fabbri, A. R., Ladhak, F., Elhadad, N., & McKeown, K. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.151 · Cited by 18
Self-citationPuertas, E., Mariscal-Vivas, G., & Martínez-Requejo, S. (2023) · International Conference on Education and E-Learning · doi:10.1145/3637989.3637998 · Cited by 8
“Therefore, although they can be useful as sources of general information, if we really want to exploit the full potential of these new technologies in education to promote more personalized learning, it is necessary to solve these problems of context and specificity in the answers [10].”
BackgroundOn the Role of Large Language Models in Crowdsourcing Misinformation Assessment
Xu, J., Han, L., Sadiq, S. W., & Demartini, G. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31417 · Cited by 14
“While significant work has been conducted to understand the effectiveness of these models on a large number of human tasks (Zhang et al. 2023), in this work, we instead focus on how these LLMs may impact human decision-making processes when human decision-makers are exposed to LLM output.”
BackgroundGenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
Hamid, A., Samidi, H. R., Pappachan, P., Finin, T., & Yus, R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0081 · Cited by 27
Dubey, A., Feng, Z., Kidambi, R., Mehta, A., & Wang, D. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672022 · Cited by 65
“LLMs are very effective in generating summaries that are preferred by humans (Liu et al., 2022; Zhang et al., 2024).”
BackgroundUsing Large Language Models for Goal-Oriented Dialogue Systems
Legashev, L. V., Shukhman, A., Badikov, V. R., & Kurynov, V. (2025) · Applied Sciences · doi:10.3390/app15094687 · Cited by 11
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
Machine Advisors: Integrating Large Language Models Into Democratic Assemblies
Špecián, P. (2024) · Social Epistemology · doi:10.1080/02691728.2024.2379271 · Cited by 11
Multi-Label Classification with Generative Large Language Models
Niraula, N. B., Ayhan, S., Chidambaram, B., & Whyatt, D. (2024) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc62030.2024.10748883 · Cited by 16
“The rapid rise of generative large language models (LLMs) enables new approaches to many Natural Language Processing (NLP) tasks such as content generation [1], summarization [2], machine translation [3], problem solving [4], software code generation [5] and chat [6].”
Mapping the evolution of generative AI: insights from bibliometric research
Pathak, S., & Pallasena, R. K. (2025) · Journal of Decision System · doi:10.1080/12460125.2024.2438612 · Cited by 13
The Viability of Crowdsourcing for RAG Evaluation
Gienapp, L., Hagen, T., Fröbe, M., Hagen, M., Stein, B., Potthast, M., & Scells, H. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730093 · Cited by 14
“RAG Response Judgment Topics (TRECRAG’24[32]) 301 Topics (TRECRAG’24[32]) 65 RAG responses 1,806 Quality criteria [”
InfluentialEmpowered stock market forecasting using Large Language Model on social media content
Sperlí, G., & Sichinolfi, M. A. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112727 · Cited by 13
Progressive Document-level Text Simplification via Large Language Models
Fang, D., Qiang, J., Zhu, Y. Q., Yuan, Y., Li, W., & Liu, Y. (2025) · New Generation Computing · doi:10.1007/s00354-025-00309-1 · Cited by 21
A large language model-empowered agent for reliable and robust structural analysis
Liu, J., Geng, Z., Cao, R., Cheng, L., Bocchini, P., & Cheng, M. (2026) · Structure and Infrastructure Engineering · doi:10.1080/15732479.2026.2630123 · Cited by 33
Zheng, J., Li, C., Wu, P., & Pan, L. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104689 · Cited by 8
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Jia, Q., Ren, S., Liu, Y., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.679 · Cited by 35
Chain-of-event prompting for multi-document summarization by large language models
Bao, S., Li, T., & Cao, B. (2024) · International Journal of Web Information Systems · doi:10.1108/ijwis-12-2023-0249 · Cited by 17
Evaluating Top-k RAG-based approach for Game Review Generation
Chauhan, P., Sahani, R. K., Datta, S., Qadir, A., Raj, M., & Ali, M. M. (2024) · IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT) · doi:10.1109/ic2pct60090.2024.10486273 · Cited by 13
“Benchmarking Large Language Models for News Sum-marization[5] .”
MethodChen, X., Zhou, W., Hoda, R., Li, A., Bain, C., & Poon, P. (2024) · Digital Health · doi:10.1177/20552076241293932 · Cited by 12
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, S. W.-T. (2023) · arXiv · doi:10.48550/arxiv.2305.14739 · Cited by 6
“Following prior work (Zhang et al., 2023; Zhou et al., 2023), we evaluate the models using prompting.”
MethodAutomated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
Zhang, Y., Su, Y., Liu, Y., Wang, X., Burgess, J., Sui, E., Wang, C., Aklilu, J. G., Lozano, A., Wei, A., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02754 · Cited by 45
“The primary issue lies in accurately and robustly measuring the semantic similarity between model-generated answers and ground-truth answers, a long-standing challenge in nat-ural language processing [12, 62, 64].”
Pataranutaporn, P., Archiwaranguprok, C., Chan, S., Loftus, E. F., & Maes, P. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712112 · Cited by 17
“With the rapid proliferation of LLMs, everyday people [78, 93], legal professionals [15, 36], medical practitioners [60, 86], and experts in numerous fields now routinely rely on AI-powered tools to distill and interpret complex information.”
GPT VS. HUMAN FOR SCIENTIFIC REVIEWS: A DUAL SOURCE REVIEW ON APPLICATIONS OF CHATGPT IN SCIENCE
Wu, C., Varghese, A. J., Oommen, V., & Karniadakis, G. E. (2024) · Journal of Machine Learning for Modeling and Computing · doi:10.1615/jmachlearnmodelcomput.2024052432 · Cited by 17
Cross-Document Distillation via Graph-Based Summarization of Extracted Essential Knowledge
Ragazzi, L., Moro, G., Valgimigli, L., & Fiorani, R. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3490375 · Cited by 18
“While decoder-only architectures driven by large language models (LLMs) have gained popularity for news summarization [85], [86], their application in MDS remains unexplored, and recent research proves that encode.decoder networks offer superior summarization performance [87].”
Burstein, R., Mafuta, E. M., & Proctor, J. L. (2025) · International Health · doi:10.1093/inthealth/ihaf015 · Cited by 9
“7 –10 For example, OpenAI’s generative pretrained model GPT-4 and other recently released LLMs can surpass human benchmarks on visual commonsense reasoning, multitask language understanding and competition-level mathematics.”
A multi-stage interactive writing task for the assessment of English language writing proficiency
Runge, A., Goodwin, S., Attali, Y., Poe, M., Mulcaire, P., Lo, K.-L., & LaFlair, G. T. (2025) · Language Testing · doi:10.1177/02655322251349908 · Cited by 7
“…explored how LLMs can elicit metalinguistic feedback on writing (Beguš et al., 2023), detect stance in and support learning of argumentative writing (Eguchi & Kyle, 2023), and support student writing by identifying main points in a text and suggesting areas for elaboration (Zhang et al., 2024).”
Questions about Contracts: Prompt Templates for Structured Answer Generation
Roegiest, A., Chitta, R., Donnelly, J., Lash, M., Vtyurina, A., & Longtin, F. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.8 · Cited by 16
How Do Large Language Models Perform on the Question Answering Tasks?
Fischer, K., Fürst, D., Steindl, S., Lindner, J., & Schäfer, U. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-92605-1_12 · Cited by 11
Using language models to label clusters of scientific documents
Murray, D., Ni, C., Gu, W., & Hubbard, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05445-5 · Cited by 8
“…have already demonstrated utility in a variety of tasks relevant to social science research (Ziems et al., 2024), including text summarization (Zhang et al., 2024; Liu et al., 2023; Pu, Gao, & Wan, 2023; Liu et al., 2024) and text annotation (Gilardi, Alizadeh, & Kubli, 2023; Kuzman,…”
Zhang, M., Zhao, N., Qin, J., Xu, Q., Pan, K., & Luo, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-30306-9 · Cited by 6
SumBlogger: Abstractive Summarization of Large Collections of Scientific Articles
Zakkas, P., Verberne, S., & Zavrel, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_23 · Cited by 16
PALLM: Evaluating and Enhancing Palliative Care Conversations with Large Language Models
Wang, Z., Yuan, F., LeBaron, V. T., Flickinger, T., & Barnes, L. E. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3712300 · Cited by 16
“Your refractory severe hypoxia means that your body isn't adequately exchanging gases, which can lead to this sensation, nothing unusual.”
Kowalchuk, P., Grotte, A., Brandsberg‐Dahl, S., Sabharwal, V., & Jensen, U. V. (2025) · Offshore Technology Conference · doi:10.4043/35607-ms · Cited by 5
IOAgent: Democratizing Trustworthy HPC I/O Performance Diagnosis Capability via LLMs
Egersdoerfer, C., Sareen, A., Bez, J. L., Byna, S., Xu, D., & Dai, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00036 · Cited by 11
Improving Multi-Stage Long Document Summarization with Enhanced Coarse Summarizer
Lim, J., & Song, H.-J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.13 · Cited by 6
“…generated by large language models (LLMs) consistently outperform those of fine-tuned summarization methods in short document summarization (Zhang et al., 2023; Pu et al., 2023b), they typically fall short in the context of long document summarization (Yang et al., 2023; Nijkamp et al.,…”
BackgroundInvestigating the Pre-Training Bias in Low-Resource Abstractive Summarization
Chernyshev, D., & Dobrov, B. V. (2024) · IEEE Access · doi:10.1109/access.2024.3379139 · Cited by 10
“Amid the rise of universal NLP problem-solving models (e.g. ChatGPT or GPT-4 [27]) capable of producing summaries of quality comparable to exclusively fine-tuned models [40], [46] these observations call into question the reasonableness of task-specific pre-training in abstractive summarization.”
BackgroundHint-Enhanced In-Context Learning Wakes Large Language Models Up For Knowledge-Intensive Tasks
Wang, Y., Guo, Q., Ni, X., Shi, C., Liu, L., Jiang, H., & Yang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447527 · Cited by 9
“Hence, we propose a prompt-based knowledge-extracting scheme to extract valuable information from selected examples, leveraging LLMs’ reasoning and summarizing ability [14, 15].”
MethodWho Knows the Answer? Finding the Best Model and Prompt for Each Query Using Confidence-Based Search
Gerych, W., Rizk, Y., Isahagian, V., Muthusamy, V., Duesterwald, E., & Venkateswaran, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29763 · Cited by 9
“Large language models (LLMs) have demonstrated impressive performance on a range of tasks such as question answering (Robinson, Rytting, and Wingate 2022), text summarization (Zhang et al. 2023), and search (Spatharioti et al. 2023).”
BackgroundDisentangling Likes and Dislikes in Personalized Generative Explainable Recommendation
Shimizu, R., Wada, T., Wang, Y., Kruse, J., O’Brien, S., HtaungKham, S., Song, L., Yoshikawa, Y., Saito, Y., Tsung, F., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714583 · Cited by 12
Fernández-Becerra, L., González-Santamarta, M. Á., Guerrero‐Higueras, Á. M., Rodríguez-Lera, F. J., & Olivera, V. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4773675 · Cited by 14
Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries
Gero, Z., Singh, C., Xie, Y., Zhang, S., Subramanian, P., Vozila, P., Naumann, T., Gao, J., & Poon, H. (2024) · arXiv · doi:10.48550/arxiv.2403.01002 · Cited by 5
“Meanwhile, pre-trained large language models (LLMs) have shown impressive proficiency in a range of complex nat-ural language processing tasks [1–3], including in text summarization [4, 5].”
BackgroundZhang, C., Zhu, S., Yang, X., Tseng, Y.-C., Jiang, S., & Rzeszotarski, J. M. (2025) · Proceedings of the 7th ACM Conference on Conversational User Interfaces · doi:10.1145/3719160.3736618 · Cited by 4
Houweling, J., Arras, M. M. L., Willighagen, E., Jennen, D. G. J., Evelo, C. T., & Kienhuis, A. S. (2026) · Evidence-Based Toxicology · doi:10.1080/2833373x.2026.2638036 · Cited by 6
Context Is King: Large Language Models’ Interpretability in Divergent Knowledge Scenarios
Piñeiro-Martín, A., Santos-Criado, F.-J., García-Mateo, C., Docío-Fernández, L., & López-Pérez, M. D. C. (2025) · Applied Sciences · doi:10.3390/app15031192 · Cited by 4
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
Yuan, J., He, Z., Bai, H., Leng, J., & Jiang, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889583 · Cited by 13
“Recently, the decoder-only transformer has emerged as the leading architecture for large language models (LLMs) [1]– [3], showcasing exceptional effectiveness across various application areas [4]–[6].”
Cao, B., Guan, X., Bao, S., Wu, J., & Fan, J. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3563357 · Cited by 6
“Unsupervised methods for MDS typically rely on decoder-only Transformer architectures, such as the GPT series, which leverage the generative capabilities of large language models (LLMs) to produce summaries [31].”
Ethical, robust and accurate use of AI in animal research
Vasquez, B., DeRicco, J. S., Yates, B. J., & Cunningham, R. K. (2025) · Lab Animal · doi:10.1038/s41684-025-01649-7 · Cited by 3
Shi, Z., Wu, B., Hu, B., Zhong, J., Li, Z., Zhang, F., Chen, Z., Yang, C., Guo, B., Xu, Q., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66910-6 · Cited by 5
“…and face critical challenges, such as clinical domain errors, which include hallucinations 19 and omissions from medical document summarization 20,21 ; temporal instability, which indicates that short-and long-term temporal variations in LLM output exist 22 ; and validation gaps, which…”
A Zero-Shot Strategy for Knowledge Graph Engineering Using GPT-3.5
Carta, S., Giuliani, A., Manca, M. M., Piano, L., Podda, A. S., Pompianu, L., & Tiddia, S. G. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.573 · Cited by 6
Sarmento, M. A., & Oliveira, H. (2024) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2024.245395 · Cited by 4
GraphAide: Advanced Graph-Assisted Query and Reasoning System
Purohit, S., Chin, G., Mackey, P., & Cottam, J. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825705 · Cited by 5
“The development of LLM-based applications remains a dynamic area of research, offering superior performance in tasks such as summarization, correlation, and inference across various input sources Zhang et al. (2024).”
AllHands :Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models
Zhang, C., Ma, Z., Wu, Y., He, S., Qin, S., Ma, M., Qin, X., Kang, Y., Liang, Y., Gou, X., et al. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00011 · Cited by 19
DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4
Hu, Y., Song, K., Cho, S., Wang, X., Foroosh, H., & Liu, F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.519 · Cited by 23
“There are around 1k Reddit posts and 0.6k CNN/DM articles, each having multiple annotated summaries.”
BackgroundInfluential3A-COT: an attend-arrange-abstract chain-of-thought for multi-document summarization
Zhang, Y., Gao, S., Huang, Y., Yu, Z., & Tan, K. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02225-0 · Cited by 8
“However, there is currently a lack of LLMs-based research on MDS.”
Lu, T., Hu, J., & Chen, P. (2024) · doi:10.36227/techrxiv.171742386.68305769/v1 · Cited by 33
Verifying Robustness of Neural Networks in Vision-Based End-to-End Autonomous Driving
Bernardeschi, C., Lami, G., Merola, F., & Rossi, F. (2025) · IEEE Access · doi:10.1109/access.2025.3563120 · Cited by 4
“The components are trained jointly and share the same objective function (and consequently loss and backpropagation), usually represented by the planning performance [22], [23], [24].”
Zhu, Y., Li, Y., Li, L., Hu, D., Hao, X., Chen, D., Zhang, X., Zhang, Z., Zhou, Y., Litvak, M., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3567744 · Cited by 6
“[51] indicates that when LLMs serve as summarization models, their generated summaries contain more subtle and challenging factual errors.”
A Knowledge Graph-based RAG for Cross-Document Information Extraction
Suryawanshi, S., Waghmode, S., Sawant, R., & Gupta, M. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11035854 · Cited by 5
“…Models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks like Information Extraction and Summarization [1], [2]; however, their limited context window and difficulty in linking Cross-Document concepts restrict their effectiveness in processing long-form…”
LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System
Li, F., Li, Y., Liu, Y., Zhou, C., Wang, Y., Deng, X., Xue, W., Liu, D., Xiao, L., Gu, H., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3750601.3750602 · Cited by 14
Understanding Online Discussion Experiences of Blind Screen Reader Users
Ferdous, M. J., Nayak, A. K., Prakash, Y., Venkatraman, N., Jayarathna, S., Lee, H.-N., & Ashok, V. G. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2589865 · Cited by 4
“Such summarization support can easily be provided using LLMs, which have been shown to generate accurate and creative summaries for a wide range of natural language tasks [39, 223].”
The Effect of Document Summarization on LLM-Based Relevance Judgments
Mohtadi, S., Roitero, K., Mizzaro, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_5 · Cited by 7
CivilSum: A Dataset for Abstractive Summarization of Indian Court Decisions
Malik, M., Zhao, Z., Fonseca, M., Rao, S., & Cohen, S. B. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657859 · Cited by 4
“Parikh et al. [33] has introduced a large dataset of 10,764 Indian legal documents and employed a weakly supervised approach to tackle the challenge of summarizing legal documents.”
Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning
Bin, Y., Liao, J., Ding, Y., Li, H., Yang, Y., Ng, S.-K., & Shen, H. T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681677 · Cited by 3
“…Intelligence (AI) models, which not only enhance the decision making by providing insights into the relationships and dependencies between different pieces of information [37], but also enable systems to understand context better [62], Sentence Set to Order 'their dad is there ring leader .”
BackgroundWahab, M. H. H., Hamid, N. A. W. A., Subramaniam, S. K., Latip, R. B., & Othman, М. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125951 · Cited by 3
An Automatic Deep Learning Approach for Trailer Generation through Large Language Models
Balestri, R., Cascarano, P., Esposti, M. D., & Pescatore, G. (2024) · International Conference on Frontiers of Signal Processing (ICFSP) · doi:10.1109/icfsp62546.2024.10785516 · Cited by 10
“This task has been largely investigated exploring different techniques [28] such as learning-based paradigms [29]–[31] and, very recently, focusing on the adoption of LLMs [32], [33].”
Schuerkamp, R., Ahlstrom, H., & Giabbanelli, P. J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113151 · Cited by 4
Towards Large Language Model Organization: A Case Study on Abstractive Summarization
Boros, K., & Oyamada, M. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386199 · Cited by 5
“Following the suggestion of [6], we exclude fluency from the evaluation.”
BackgroundMethodCan Large Language Models Support Medical Facilitation Work? A Speculative Analysis
Abdulhamid, N. G., Ochieng, M., Bali, K., Ankrah, E. A., Karusala, N., Ronen, K., & O’Neill, J. (2023) · African Human Computer Interaction Conference (AfriCHI) · doi:10.1145/3628096.3628752 · Cited by 8
“LLMs are highly effective at summarising real-world text [59, 71].”
BackgroundThe Performance of GPT-3.5 in Summarizing Scientific and News Articles
Arshad, S., Yaqoob, M., & Mehmood, T. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-0293-0_4 · Cited by 6
Bridging the Integrity Gap: Towards AI-assisted Design Research
Oksanen, J. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647962 · Cited by 7
“Recent advances in language models have significantly expedited research efforts in the domain of automatic summarisation, extending its capabilities to challenging conversational contexts [20] and in some tasks to human levels of performance [25].”
Less is More: Stress Detection through Condensed Social Media Contents
Alghamdi, Z., Kumarage, T., Agrawal, G., Liu, H., & Bernard, H. R. (2024) · European Conference on Social Media · doi:10.34190/ecsm.11.1.2273 · Cited by 6
Classifying AI-generated summaries And Human Summaries Based on Statistical Features
Mathews, D., Varghese, J. P., & Samuel, L. C. (2024) · International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT) · doi:10.1109/tqcebt59414.2024.10545131 · Cited by 6
“AI-generated summaries, in particular, have seen a surge in applications across diverse domains, from news aggregation to content recommendation and automated reporting[3].”
BackgroundNa batida do funk: modelagem de tópicos combinando LLM, Engenharia de Prompt e BERTopic
Yepez, J., Tavares, B., Peres, F. D. C. L., & Becker, K. E. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2024.243148 · Cited by 4
“Dada a configurac¸˜ao correta dos parˆametros e prompts , LLMs tˆem demostrado capacidade de sumarizac¸˜ao de textos compar´aveis `as feitas por humanos [Zhang et al. 2024].”
BackgroundEvaluating LLMs in Persian News Summarization
VarastehNezhad, A., Tavasoli, R., Masumi, M., Majd, S. S., & Shamsfard, M. (2024) · International Conference on Information and Knowledge Technology (IKT) · doi:10.1109/ikt65497.2024.10892758 · Cited by 4
“They particularly shine in specialized or rare scenarios, effectively adapting the summary length to align with the input text and adeptly capturing multiple topics within a document [16, 17].”
Extractive summarization of Indian legal judgements using unsupervised role labelling
Shreyasi, S., Bandyopadhyay, A., & Chakrabarti, P. P. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09453-x · Cited by 5
Terminators: Terms of Service Parsing and Auditing Agents
Mridul, M. A., Kang, I., & Seneviratne, O. W. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3720554.3736183 · Cited by 4
“In particular, these models have shown great strength in handling unstructured documents, such as the ability to parse verbose documents to automate summarization and analysis of the content [14, 17, 20, 22].”
TaxonGPT: Taxonomic Classification Using Generative Artificial IntelligencePreprint
Huang, H., Li, T., Wang, Z., Seldon, D. S., & Rodrigo, A. G. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.28.618575 · Cited by 4
Kim, H., Lee, S., You, S. C., Huh, S., Kim, J.-E., Kim, S.-T., Ko, D.-R., Kim, J. H., Lee, J. H., Lim, J. S., et al. (2025) · medRxiv · doi:10.1101/2025.04.03.25325003 · Cited by 3
Rhodes, J. H., Nteta, T. M., Eichen, A., Rice, D. R., & Wall, G. (2025) · The Conversation · doi:10.64628/aai.3h5v3wfqh · Cited by 3
AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and InsightsPreprint
Xu, J., Li, G., & Jiang, J. Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5417394 · Cited by 3
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Que, H., Duan, F., He, L., Mou, Y., Zhou, W., Liu, J., Rong, W., Wang, Z. M., Yang, J., Zhang, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.16191 · Cited by 3
Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs
Osés Grijalba, J., Ureña-López, L. A., Martínez Cámara, E., & Camacho-Collados, J. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/4wrzvsf2shaz · Cited by 3
“…the standard machine learning training workflow enables the usage of task-agnostic architectures to resolve a wide range of tasks such as sentiment analysis (Deng et al., 2023; Zhang et al., 2023c), machine translation (Jiao et al., 2023) or text summarisation (Zhang et al., 2023b), to name a few.”
BackgroundEvaluating the Adaptability of Large Language Models for Knowledge-aware Question and Answering
Thakkar, J., Kolekar, S., Gite, S., Pradhan, B., & Alamri, A. M. (2024) · International Journal on Smart Sensing and Intelligent Systems · doi:10.2478/ijssis-2024-0021 · Cited by 3
Metadata
- Authors
- Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- Corresponding author
- Zhang, Tianyi
- Ladhak, Faisal
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 39-57
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391494845
- Cited by
- 415
- References
- 79
- Citations by year
- 2026: 108 · 2025: 165 · 2024: 96
- FWCI
- 54.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, news summarization, automatic summarization, human evaluation, instruction tuning, zero-shot summarization, fine-tuning, model size, human-written summaries, paraphrase, freelance writers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.