Benchmarking Large Language Models for News Summarization
Open accessThe DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [70], human evaluation results indicate that humans tend to prefer the results generated by LLMs [42,138] compared to that of fine-tuned models.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [138]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are known to produce errors in many settings, from document summarisation7, to general reasoning tasks as well as more clinically relevant tasks8.”
From A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“LLMs (Bommasani et al., 2021; Chowdhery et al., 2022; Brown et al., 2020) have two distinctive features over previous pretrained models.”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
Method(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method(2005) · Cited by 4,152
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method
Research works printed but not linked: 1
- Conroy, 2006, Back to basics: Classy 2006, 10.12968/sece.2006.11.755s · doi:10.12968/sece.2006.11.755s
Cited by
Items 201–300 of 427
ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction
Zhang, Y., Li, N., Gan, Q., Zhang, W., Wipf, D., & Wang, M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679153 · Cited by 4
The role of GPT as an adaptive technology in climate change journalism
Jeng, J. H., Kasangu, G., Starke, A. D., Seddik, K. M., & Trattner, C. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728332 · Cited by 3
Design of Mutation Operators in Fireworks Algorithm Assisted by Large Language Models
Liu, Y., & Tan, Y. (2025) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec65147.2025.11043064 · Cited by 2
TacTalk: Personalizing Haptics Through Conversation
Mishra, A., & Schneider, O. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736638 · Cited by 5
AI ‐powered nursing handoffs: Introducing and evaluating the patient report template
Vald, G., Sermet, Y., Chi, N., & Demir, İ. (2025) · Learning Health Systems · doi:10.1002/lrh2.70026 · Cited by 5
Constituency Parsing Using LLMs
Bai, X., Wu, J., Chen, Y., Wang, Z., Chen, K., Zhang, M., & Zhang, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3600867 · Cited by 31
“Recently,largelanguagemodels(LLMs)haveachievedre-markableperformanceonmanyNLPtasks,suchastextclas-sification[10],[11],commonsensereasoning[12],[13],text summarization[14],[15],[16],machinetranslation[17]anddi-aloguesystems[18],usingpromptingtechnologies.”
HEAL-Summ: a lightweight and ethical framework for accessible summarization of health information
Fisher, A., Srinivasan, K. K., Hillier, S. A., & Mago, V. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1619274 · Cited by 2
“Lightweight, open-source approaches have shown promise in related areas to address this concern, including question summarization (16) and text summarization tasks where smaller LLMs have been demonstrated to align closely with human-authored content (17).”
MATA: A Memory-Efficient Attention Accelerator for LLMs Exploiting Look-Back KV Cache Pruning
Zhu, Z., Li, G., Liu, T., Mo, Z., Liang, X., & Cheng, J. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3626150 · Cited by 4
“LLMs have been widely used in various natural language processing applications, such as dialog systems [27], [34], code completion [21], [31] and document summarization [6], [38], many of which necessitate low-latency and energy-efficient inference.”
Nunes, D. A. P., Furrer, D., Berger, S., Cecchi, G., Ferreira‐Gomes, J., Neto, F. L. M., de Matos, D. M., Apkarian, A. V., & Branco, P. (2025) · European Journal of Pain · doi:10.1002/ejp.70184 · Cited by 4
“LLMs have been noted for their summarization capabilities and have further shown that model prompting is the key for better results in similar tasks, as opposed to model size, justifying the use of a cheaper and “smaller” 8 billion parameter model (Zhang et al. 2024).”
Ray, P. P., & Pradhan, M. P. (2025) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2025.100249 · Cited by 4
RLV: LLM-based vulnerability detection by retrieving and refining contextual information
Qiu, F., Liu, Z., Hu, B., Cai, Z., Bao, L., & Wang, X. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112756 · Cited by 4
Detecting Training Data For Large Language Models: A Survey
Yang, C., Li, J., Shulin, L. A. N., Wang, Y., Du, H., Gong, C., Yao, X., Niyato, D. T., & Zhu, L. (2026) · ACM Computing Surveys · doi:10.1145/3779430 · Cited by 3
“…models (LLMs) have shown outstanding performance across various natural language processing (NLP) tasks [1, 13, 21, 98, 112, 116, 117, 135] such as question answering [47, 129], machine translation [69], text summarization [136], even using for scientific research [111] and code generation [91].”
Sutriawan, S., Rustad, S., Shidik, G. F., & Pujiono (2026) · Array · doi:10.1016/j.array.2025.100671 · Cited by 3
Carou‐Senra, P., Rodríguez‐Pombo, L., Alvarez‐Lorenzo, C., & Goyanes, Á. (2026) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2026.126587 · Cited by 3
Using LLMs to Enhance Democracy
Lazar, S., & Manuali, L. (2026) · Minds and Machines · doi:10.1007/s11023-026-09767-y · Cited by 28
Wu, M., Liu, G., Wang, B., & Ma, Z. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2633738 · Cited by 3
Disentangled graph recommendation via dynamic long and short-term intent modeling
Wang, J., Wang, J., & Jin, D. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104786 · Cited by 2
Han, R., An, L., Zhou, W., Chen, M., & Li, G. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104808 · Cited by 2
From Transformers Come Themes: Evaluating BERTopic for Qualitative Analysis of Social Media Data
Kaur, A., & Wallace, J. (2025) · International Conference on Human-Engaged Computing (ICHEC) · doi:10.1145/3786995.3786997 · Cited by 4
Zheng, J., Wu, P., & Pan, L. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104966 · Cited by 2
Comparison of automatic summarization of texts in Russian
Дагаев, А. Е., & Popov, D. I. (2024) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2024.4.69474 · Cited by 4
Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis
Ahad, J. I., Sultan, R. M., Kaikobad, A., Rahman, F. A., Amin, M. R., Mohammed, N., & Rahman, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825310 · Cited by 11
“After fine-tuning the models, we then tested our fine-tuned model on the benchmark Open-i dataset [38], writer summaries [27], and the large scientific document dataset CL-SciSumm [39] to compare performance, shown in Table II.”
A Review of Text Generation Based on Large Language Model
刘, 津. (2025) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2025.141019 · Cited by 2
Wiktor, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641555.3705044 · Cited by 2
LLAMA-2 Acceleration Using the ARM Scalable Vector Extension
Rossi, F., Cococcioni, M., & Saponara, S. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-3-031-84100-2_31 · Cited by 2
Sun, D., Lyu, Y., Li, J., Chen, Y., Wang, T., Kimura, T., & Abdelzaher, T. (2025) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp65954.2025.00062 · Cited by 4
“LLMs excel in tasks that require understanding, summarizing, and generating texts, achieving remarkable success in applications ranging from question answering to conversation generation [2]–[4].”
Yu, F., An, W., Wang, H., & Yin, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05397-w · Cited by 4
“In text classification tasks, contrastive learning significantly enhances model performance by comparing the differences between positive and negative samples (Zhang et al., 2024a, 2024b, 2024c, 2024d).”
InfluentialGiabbanelli, P. J., Daumas, C., Flandre, N. Y. J.-P., Pitkar, A., & Vazquez-Estrada, J. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2536663 · Cited by 9
AccLLM: Accelerating Long-Context LLM Inference via Algorithm-Hardware Co-Design
Liang, Y., Shi, H., Shao, H., & Wang, Z. (2026) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2026.3658524 · Cited by 12
“L ARGE language models (LLMs) [1], [2], [3], [4] have revolutionized natural language processing (NLP) with their outstanding capabilities, enabling diverse applications [5], including document summarization [6], chatbots [1], and question answering [7].”
SumREN: Summarizing Reported Speech about Events in News
Reddy, R. G., Elfardy, H., Chan, H. P., Small, K., & Ji, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26506 · Cited by 6
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.56 · Cited by 4
“Methods that could be considered for both subtasks include Large Language Models (LLMs) (Zhang et al., 2023), or retrieval-based methods (An et al., 2021).”
MethodThe current state of summarization
Retkowski, F. (2023) · AI critique · doi:10.1515/9783839467664-016 · Cited by 11
Improve Conversational Search with Multi-document Information
He, S., Zhang, S., Zhang, X., & Feng, Z. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8178-6_1 · Cited by 2
Overview and Trend of Large-scale Model Deployment Mode
Zheng, Q., Wang, J., & Yun, S. (2024) · IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) · doi:10.1109/bmsb62888.2024.10608242 · Cited by 4
Using Generative AI to Support Standardization Work – the Case of 3GPP
Staron, M., Ström, J., Karlsson, A., & Meding, W. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00038 · Cited by 2
Burstein, R., Mafuta, E. M., & Proctor, J. L. (2024) · medRxiv · doi:10.1101/2024.11.14.24317253 · Cited by 2
Aspect-Driven Structuring of Historical Dutch Newspaper Archives
Kroll, H., Kreutz, C. K., Cuper, M., Thang, B. M., & Balke, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43849-3_4 · Cited by 3
Less Hallucination and More Factuality: Human Values Alignment in Text Summarization
Qiu, Z., & Xu, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9671-7_14 · Cited by 2
Inferring Relationships between Tabular Data and Topics using LLM for a Dataset Search Task
Fujita, Y., Hayashi, T., & Kuwahara, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826033 · Cited by 1
“Recent rapid advances in LLM enabled us to achieve a high-performance in several tasks such as text summarization [10], coding support [11], and translation [12], and annotation is one of such tasks [13].”
Shan, H., & Lü, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130124 · Cited by 1
Transformer-Based Approach for Detecting LLM-Generated Scientific Text
Bader, A., & Alhijawi, B. (2025) · International Conference on New Trends in Computing Sciences (ICTCS) · doi:10.1109/ictcs65341.2025.10989343 · Cited by 4
“For instance, Zhang et al. [6] demonstrated that LLMs have been effectively used in news summarization.”
ETL: Ensemble–at–Training for Low-cost Open-source Large Language Models
Zaki, A. M., Khalil, M. I., & Abbas, H. M. (2025) · doi:10.1109/iceeng64546.2025.11031386 · Cited by 1
“Large Language Models (LLMs) have demonstrated the potential to generate high-quality summaries [1]–[3].”
Amari, A., & Ammar, M. A. B. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96231-8_29 · Cited by 1
“Large language models (LLMs) [8] have excelled, particularly in abstractive summarization, and are driving significant advancements in this area [9].”
Integrating AI in historical research: a new method to analyzing national narratives
Yao, N. (2025) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf075 · Cited by 3
Automating Legal Tasks: LLMs, Legal Documents, and the AI Act
Liepiņa, R., Lagioia, F., Lippi, M., Pałka, P., Micklitz, H., & Sartor, G. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009492553.029 · Cited by 2
Leveraging LLMs for Smart Cities Qualitative Data Analysis
Covato, E., Soomro, K., Khan, Z., Bilal, M., Kirby, S., & Yiangou, T. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-87639-4_16 · Cited by 1
Cavalcanti, R., Casini, G., Assis, G., Real, L., Vianna, D., Mann, P., & Paes, A. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37825 · Cited by 2
“Diversos estudos j´a mostraram as habilidades dos LLMs em tarefas como gerac¸˜ao de resumos, escrita de artigos jornal´ısticos e redac¸˜oes acadˆemicas de n´ıvel b´asico e intermedi´ario [Zhang et al. 2024, Kaliterna et al. 2024].”
Pan, R., Bernal-Beltrán, T., Salas‐Zárate, M. D. P., Paredes‐Valverde, M. A., García-Díaz, J. A., & Valencia‐García, R. (2025) · Applied Sciences · doi:10.3390/app152111834 · Cited by 1
“[20] conducted a large-scale human evaluation of LLMs for news summarization.”
Zhang, X., Dörig, V., Cui, P., Zouhar, V., Netland, T. H., & Sachan, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00515-4 · Cited by 2
“The conventional process of textbook reading may also change, given the capabilities of LLMs to provide high-quality summaries (Zhang et al., 2023, 2024 ; Cohan et al., 2018), retrieve information (Lewis et al., 2020; Zhu et al., 2023), and answer questions (Ko et al., 2022; Guu et al., 2020 ).”
A Comparative Evaluation of Transformer-Based Models for Abstractive Turkish News Summarization
Gülsoy, M., & Akbulut, A. (2025) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap68205.2025.11222238 · Cited by 1
“In this context, the summarization performance of 10 different LLMs was compared, and a detailed analysis was presented [3].”
Confidence-weighted integration of human and machine judgments for superior decision-making
Yáñez, F., Luo, X., Minero, O. V., & Love, B. C. (2025) · Patterns · doi:10.1016/j.patter.2025.101423 · Cited by 3
Legal lay summarization: exploring methods and data generation with large language models
Moro, G., Magnani, L. D. M., & Ragazzi, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11392-7 · Cited by 7
“Despite advancements in text summarization technologies (Moro and Ragazzi 2022; Zhang et al. 2024), legal lay summarization (LLS), namely the task of summarizing technical legal texts in simpler terms, remains an unresolved issue.”
Kumar, S., & Singh, S. R. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3623667 · Cited by 1
“However, pretrained LLMs can generate higher coherence, relevance, and factual, accurate summaries, surpassing human-written summaries [49].”
Benchmarking In-Context Learning Strategies of Large Language Models for Math Reasoning Tasks
Rong, Y., Seßler, K., Gözlüklü, E., & Kasneci, E. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3630117 · Cited by 24
“…architecture [5] based on the self-attention mechanism, which superseded recurrent neural networks for natural language processing and led to a trend of extremely large and powerful models, mastering many text-based tasks, such as summarizing [6], [7], translating [8] or question answering [9].”
Arslan, M., & Munawar, S. (2026) · Green Technologies and Sustainability · doi:10.1016/j.grets.2026.100352 · Cited by 2
Günay, E. E., Chern, W., Elhabashy, A. E., Kremer, P. A., Haapala, K. R., Kim, K.-Y. J., & Kremer, G. E. O. (2026) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4071183 · Cited by 3
Doropoulos, S., Vologiannidis, S. D., & Magnisalis, I. D. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108078 · Cited by 2
A Companion Robot Platform for Exploring Technical and Ethical Aspects in Elderly Care
Boi, L., Massa, S. M., Recupero, D. R., Riboni, D., Alonso, R., Cardinali, M., Ricci, E. D., & Pirni, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-10561-5_17 · Cited by 1
Khalid, S. M., Wölker, T., Molano, L.-A. G., Graf, S., & Keller, A. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag089 · Cited by 1
Knowledge-enhanced AI models for domain-specific application: a survey
Li, X.-Y., Wang, Y., Liang, J., Wang, X., Zhou, P., Zhao, Q., Zhang, Y., & Song, Q. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4635-0 · Cited by 1
“Summarization [69] distills lengthy documents into concise outputs, enhancing efficiency in news aggregation [70] and medical documentation [71].”
I/O Analysis is All You Need: An I/O Analysis for Long-Sequence Attention
Lu, X., Long, B., Chen, X., Han, Y., & Sun, X. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790174 · Cited by 2
BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination
Wang, H., Wang, H., Wei, S., Hu, Y., & Yin, S. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420780 · Cited by 4
“Large language models (LLMs) now underpin a wide range of applications, from automated content generation [1], [2] to conversational assistants [3], [4].”
LLMs can pass the CCNA certification exam but do they understand networking?
Manocchio, L. D., Layeghy, S., Houssel, P. R. B., & Portmann, M. (2026) · Computer Networks · doi:10.1016/j.comnet.2026.112236 · Cited by 1
Specializing LLMs to Low-Documented Domains with RAG: An Analysis Across Models and Retrieval Depths
Mereu, J., Carcangiu, A., Artizzu, V., Cau, F. M., & Spano, L. D. (2026) · SN Computer Science · doi:10.1007/s42979-026-04844-6 · Cited by 2
When AI Rewrites the News: How Sentiment, Framing, and LLM Disclosure Shape Perceptions
Khatiwada, P., Pappu, V., Bagozzi, B. E., & Mauriello, M. L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791527 · Cited by 6
“Already, newsrooms and media platforms are experimenting with LLMs for tasks ranging from headline generation [29, 76] and article summarization to complete editorial drafting [103, 119].”
Large-Scale Metadata Processing for 3D Cultural Heritage Objects
Münster, S. (2026) · Land · doi:10.3390/land15050751 · Cited by 3
Liu, C., Ren, X., & Xue, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132644 · Cited by 1
From prompt engineering to prompt design: Research strategies for visual generative AI
Colombo, G., Niederer, S., & Gaetano, C. D. (2026) · Big Data & Society · doi:10.1177/20539517261451462 · Cited by 3
“…by researchers to automatically tag social media posts ’ alignment on charged topics, such as the war in Ukraine (Rogers and Zhang, 2024), identify hate-ful, offensive, and toxic language (Li et al., 2024), and perform summarization tasks with notable success (Pu et al., 2023; Zhang et al., 2024).”
Automated data synthesis and retrieval-augmented generation for legal large language models
Ren, W., Shen, L., Zhu, J., Li, S., Hong, Y., Wang, J., & Cao, D. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116267 · Cited by 3
Turning hallucinations into knowledge: Towards identifying clickbait using LLM-generated fallacies
Zhang, C., Wang, Z., Zhang, Z., Zhu, Y., Qiang, J., & Huang, Y. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104905 · Cited by 3
Lertyosbordin, C., & Iamruttanawong, K. (2026) · Applied Sciences · doi:10.3390/app16125754 · Cited by 1
Beyond Quantity: Research with Subsymbolic AI
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 14
Linguistic Compression in Single-Sentence Human-Written Summaries
Yin, F., & van Schijndel, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.532 · Cited by 1
Background Summarization of Event Timelines
Pratapa, A., Small, K., & Dreyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.505 · Cited by 4
Andako, O. O. (2023) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2023.20.3.1450 · Cited by 5
“Upwork An online freelancing platform that connects businesses and individuals with freelancers offering a wide range of skills and services [28].”
Evaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing
Li, Z., Li, Z., Xiao, K., & Li, X. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429345 · Cited by 2
“…formatted data and unformatted data and are capable of performing a wide variety of natural language processing tasks, including text generation [6, 7], translation [1, 8], summarization [9, 10], question answering [11, 12], and code generation and comprehension [13, 14]. text generation [6, 7].”
BackgroundS3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document
Shaik, K., Wang, D., Zheng, W., Cao, Q., Fan, H., Schwartz, P., & Feng, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63759-9_27 · Cited by 5
A Study on Methods of Synchronization Between Gestural-Visual and Audiovisual Communication
Costa, R. R. A., Kulesza, R., Nobrega, R. M. T., França, D. C., de Araújo, T. M. U., Costa, R. E. O., & Filho, G. L. D. S. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70439-0_9 · Cited by 1
Beyond the Bar: Generative AI as a Transformative Component in Legal Document Review
Yang, E., Omrani, R., Curtin, E., Emory, T. S., Gray, L., Pickens, J., Reff, N., Traylor, C., Underwood, S., Lewis, D. D., et al. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826089 · Cited by 7
Balancing Efficiency and Quality in LLM-Based Entity Resolution on Structured Data
Nananukul, N., & Kekriwal, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78548-1_21 · Cited by 5
A Framework for Analyzing and Summarizing News and Articles using Large Language Model
Muthukumar, D. S., Karthik, B., Pavani, K., Kiran, K. U., & Reddy, C. A. (2024) · International Conference on Ubiquitous Computing and Intelligent Information Systems (ICUIS) · doi:10.1109/icuis64676.2024.10866258 · Cited by 3
ChatGPT struggles to recognize reproducible science
Chang, J. R., & Nordling, T. E. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02428-z · Cited by 3
DocSum: Domain-Adaptive Pre-training for Document Abstractive Summarization
Chau, P. P. M., Bakkali, S., & Doucet, A. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00144 · Cited by 1
“Most recently, prompt-based PLMs have become a promising approach because of their strong understanding and capabilities to generate coherent and factual output across different inputs, as shown by [2,7,26].”
InfluentialMesnage, C. S., Wang, X., Dong, H., & Aishwaryaprajna, N. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-89366-7_3 · Cited by 1
Trustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2
Alwahhab, A. B. A., Sabeeh, V., Belhaj, H., Sibaee, S., Ahmad, S., Khurfan, I., & Alharbi, A. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.51 · Cited by 2
“They effectively utilize vast knowledge sources and deeply comprehend complex language details (Alyafeai et al., 2023; Zhang et al., 2023).”
BackgroundLORE: A Literature Semantics Framework for Evidenced Disease-Gene Pathogenicity Prediction at ScalePreprint
Li, P.-H., Sun, Y.-Y., Juan, H., Chen, C., Tsai, H., & Huang, J. (2024) · medRxiv · doi:10.1101/2024.08.10.24311801 · Cited by 1
“Indeed, as much smaller and open-248 source LLMs have been shown competent for article-level comprehension 33,34 , the methodology is not constrained to enterprise LLMs. Finally, while our framework demonstrates remarkable efficacy in capturing disease-gene relationships through unsupervised…”
BackgroundFrom RAGs to (feature) Riches -An Efficient Pipeline for Exploratory Text Analysis *Preprint
Roesti, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5331899 · Cited by 1
CMap: a database for mapping job titles, sector specialization, and promotions across 24 sectors
Subhani, S., Memon, S. A., & AlShebli, B. (2025) · Scientific Data · doi:10.1038/s41597-025-05526-3 · Cited by 1
“5 and GPT-4 excel at a wide range of natural language tasks, including question answering 25 , entity recognition 26 , relation extraction 27 , summarization 28 , machine translation 29 , paraphrasing 30 , and even various data wrangling tasks 31 .”
An, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., & Wang, J. (2026) · arXiv · doi:10.48550/arxiv.2605.08840
“These models have enabled breakthroughs in various tasks, such as document summarization(Zhang et al., 2024a), multi-turn dialogues (Du et al., 2021), retrieval augmentation (Yao et al., 2022), and code generation (Roziere et al., 2023).”
潘, 琪. (2023) · Overseas English Testing Pedagogy and Research · doi:10.12677/oetpr.2023.54018
Data Selection Curriculum for Abstractive Text Summarization
Sun, S., Yuan, R., He, J., Cao, Z., Li, W., & Jia, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.537
“Recent studies (Liu et al., 2023b; Zhang et al., 2023; Liu et al., 2023a) have shown that Large Language Models (LLMs), including GPT-3.”
EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media
Kolagar, Z., Steindl, S., & Zarcone, A. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.4 · Cited by 2
LLMusic: Modelagem de tópicos em letras de músicas combinando LLM, Engenharia de Prompt e BERTopic
Rojas, J. R., & Becker, K. E. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd_estendido.2024.243767
A Novel Summarization Framework based on Reference-Free Evaluation of Multiple Large Language Models
Feng, W., Zhao, H., Zhang, M., Yang, H., & Tang, W. (2024) · IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom) · doi:10.1109/metacom62920.2024.00047 · Cited by 2
“There is no doubt that LLMs have propelled the field of summarization into a new era [23, 24, 25].”
BackgroundTiwari, P., Shrivastava, A., & Chowdary, C. R. (2024) · Research Square · doi:10.21203/rs.3.rs-5340698/v1
Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMsPreprint
Fong, H., & Gui, G. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5053346 · Cited by 2
LLM-Powered Reading Aid for Visually Impaired Online Learners
Li, Z., Shen, S., & Xie, Z. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_19 · Cited by 1
A Comprehensive Analysis of Pre-trained NLP Models
Kumari, S., Verma, N., Kamble, S. D., & Barekar, P. V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-8537-7_13
Song, M. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-97-9327-3_7
Metadata
- Authors
- Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- Corresponding author
- Zhang, Tianyi
- Ladhak, Faisal
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 39-57
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391494845
- Cited by
- 415
- References
- 79
- Citations by year
- 2026: 108 · 2025: 165 · 2024: 96
- FWCI
- 54.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, news summarization, automatic summarization, human evaluation, instruction tuning, zero-shot summarization, fine-tuning, model size, human-written summaries, paraphrase, freelance writers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.