The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Benchmarking Large Language Models for News Summarization
Open accessAbstract
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [70], human evaluation results indicate that humans tend to prefer the results generated by LLMs [42,138] compared to that of fine-tuned models.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [138]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are known to produce errors in many settings, from document summarisation7, to general reasoning tasks as well as more clinically relevant tasks8.”
From A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“LLMs (Bommasani et al., 2021; Chowdhery et al., 2022; Brown et al., 2020) have two distinctive features over previous pretrained models.”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
Method(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method(2005) · Cited by 4,152
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method
Research works printed but not linked: 1
- Conroy, 2006, Back to basics: Classy 2006, 10.12968/sece.2006.11.755s · doi:10.12968/sece.2006.11.755s
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“…language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023a; Chu et al., 2023).”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
“We use the dataset collected in Zhang et al. (2023), where they first ask freelance writers to write high-quality summaries for news articles, and then ask annotators to compare human-written summaries and LLM-3 Figure 2: Averaged G-E VAL -4’s scores for human-written summaries and GPT-3.”
MethodG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
“We use the dataset collected in Zhang et al. (2023), where they first ask freelance writers to write high-quality summaries for news articles, and then ask annotators to compare human-written summaries and LLM-3 Figure 2: Averaged G-E VAL -4’s scores for human-written summaries and GPT-3.”
MethodHarnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation
Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01670-7 · Cited by 423
“Structured prompts (Experiments 3, 7, 8, and 4) .”
Large language models empowered agent-based modeling and simulation: a survey and perspectives
Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., & Li, Y. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03611-3 · Cited by 294
“…parameters and extensive pre-training corpus, LLMs have shown astonishing abilities not only in natural language processing tasks (Li et al., 2023a; Zhang et al., 2024c) such as text generation, summarization, translation, etc., but also in complex reasoning and planning tasks, such as solving…”
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“…language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023a; Chu et al., 2023).”
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Beware of botshit: How to manage the epistemic risks of generative chatbots
Hannigan, T. R., McCarthy, I. P., & Spicer, A. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.03.001 · Cited by 217
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 378
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145
Sources of Hallucination by Large Language Models on Inference Tasks
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.182 · Cited by 142
Using natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140
“LLMs excel in various tasks, such as annotating text with psychological constructs 21,40,91,92 , answering questions 93 , summarizing content 94 and machine translation 95 , often without specific training.”
Large Language Model‐Based Chatbots in Higher Education
Yigci, D., Eryilmaz, M., Yetisen, A. K., Tasoglu, S., & Ozcan, A. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400429 · Cited by 113
“[99,100] Similarly, ChatGPT appeared superior to traditional fi ne-tuning methods on query-based summary generation tasks across diverse domains including stories, dialogue meetings, news, and Reddit posts.”
Tool learning with large language models: a survey
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40678-2 · Cited by 491
Mustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 117
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
“Automatic validation metrics, however, lack contextual understanding and therefore cannot reliably determine relevance, correlate poorly with human judgment, and are considered underinformative [67,70].”
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 240
“Some previous works [21], [22] propose to establish complete benchmarks for LLMs.”
Extractive Summarization via ChatGPT for Faithful Summary Generation
Zhang, H., Liu, X., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.214 · Cited by 162
A Survey of Text Watermarking in the Era of Large Language Models
Liu, A., Pan, L., Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3691626 · Cited by 299
“The watermark message, denoted as 𝑤 , can be a zero-bit watermark, signifying merely its presence or absence, or a multi-bit watermark, embedding detailed, customized information.”
Graph Neural Prompting with Large Language Models
Tian, Y., Song, H., Wang, Z., Wang, H., Hu, Z., Wang, F., Chawla, N. V., & Xu, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29875 · Cited by 196
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63
“…documents (U.S. news-weekly articles from 1980 to 2012), which are relatively understudied in the social science literature on classi fi cation analysis with LLMs despite the known strengths of LLMs in thematically and holistically summarizing large corpora (Karell et al. 2024; Zhang et al. 2024).”
Exploring the Impact of Temperature on Large Language Models: Hot or Cold?
Li, L., Sleem, L., Gentile, N., Nichil, G., & State, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.135 · Cited by 62
Me-LLaMA: Foundation Large Language Models for Medical ApplicationsPreprint
Xie, Q., Chen, Q., Chen, A., Cheng, P., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V. K., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4240043/v1 · Cited by 103
“42”
BackgroundSummIt: Iterative Text Summarization via ChatGPT
Zhang, H., Liu, X., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.714 · Cited by 139
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 73
From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
Adams, G., Fabbri, A. R., Ladhak, F., Lehman, E., & Elhadad, N. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.7 · Cited by 128
“Zhang et al. (2023) find that zeroshot GPT-3 summaries perform on par with humans by soliciting high-quality summaries from freelance writers.”
BackgroundSelf-citationWang, Y., Zhang, Z., & Wang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.482 · Cited by 160
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Large language models can help boost food production, but be mindful of their risks
Clercq, D. D., Nehring, E., Mayne, H., & Mahdi, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1326153 · Cited by 76
“LLMs are proficient in parsing through vast quantities of news text data and distilling them into succinct, relevant summaries that are on par with human-written summaries [69].”
BackgroundLarge language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 56
Applicability of large language models and generative models for legal case judgement summarization
Deroy, A., Ghosh, K., & Ghosh, S. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09411-z · Cited by 93
“In fact, LLMs have been used recently for news document summarization [11].”
BackgroundMethodLarge language models meet NLP: a survey
Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., & Yu, P. S. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50472-3 · Cited by 238
The homogenizing effect of large language models on human expression and thought
Sourati, Z., Ziabari, A. S., & Dehghani, M. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.01.003 · Cited by 126
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 63
LLM-TAKE: Theme-Aware Keyword Extraction Using Large Language Models
Maragheh, R. Y., Fang, C., Irugu, C. C., Parikh, P., Cho, J. H. D., Xu, J., Sukumar, S., Patel, M., Körpeoğlu, E., Kumar, S., et al. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386476 · Cited by 69
“In particular, they demonstrated beating state-of-the-art models in other language tasks such as summarization and sentiment analysis [10]–[12].”
BackgroundHallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38
“Recently, the emergence of large language models (LLMs) [29, 81, 85, 99, 132] has dominated a wide range of tasks in natural language processing (NLP), achieving unprecedented progress in language understanding [39, 47], generation [128, 140] and reasoning [20, 58, 87, 107, 115].”
Gin, B. C., O’Sullivan, P., Hauer, K. E., Abdulnour, R.-E. E., Mackenzie, M. E., ten Cate, O., & Boscardin, C. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005930 · Cited by 56
Scaling Up Video Summarization Pretraining with Large Language Models
Argaw, D. M., Yoon, S., Heilbron, F. C., Deilamsalehy, H., Bui, T., Wang, Z., Dernoncourt, F., & Chung, J. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00796 · Cited by 63
Sottana, A., Liang, B., Zou, K., & Yuan, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.543 · Cited by 76
Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., & Lan, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.263 · Cited by 77
Fine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review
Yu, Y., Rong, G., Shen, H., Zhang, H., Shao, D., Wang, M., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695993 · Cited by 80
“LLMs have demonstrated impressive performance in summarization tasks [14, 57, 62] and annotation tasks [12, 42].”
Shi, Y., Ren, P., Wang, J., Han, B., ValizadehAslani, T., Agbavor, F., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104533 · Cited by 40
Hu, L., He, H., Wang, D., Zhao, Z., Shao, Y., & Nie, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29782 · Cited by 77
“…LLMs in this task, previous studies have demonstrated that LLMs exhibit strong language abilities, such as text comprehension, summarization, and sentiment analysis (Zhang et al. 2023; Wang et al. 2023b; Hsieh et al. 2023), which can be used to distill useful knowledge for enhancing small models.”
MethodLLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
Gürcan, Ö. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240190 · Cited by 81
Generative AI in Financial Reporting
Blankespoor, E., deHaan, E., & Li, Q. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70050 · Cited by 53
MeetingBank: A Benchmark Dataset for Meeting Summarization
Hu, Y., Ganter, T., Deilamsalehy, H., Dernoncourt, F., Foroosh, H., & Liu, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.906 · Cited by 148
Vizniuk, A., Diachenko, G., Laktionov, І., Siwocha, A., Xiao, M., & Smoląg, J. (2025) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2025-0007 · Cited by 42
McKinney, S. (2024) · Journal of Deliberative Democracy · doi:10.16997/jdd.1556 · Cited by 57
Fine-tuning large language models for interdisciplinary environmental challenges
Zhang, Y., Lin, S., Xiong, Y., Li, N., Zhong, L., Ding, L., & Hu, Q. (2025) · Environmental Science and Ecotechnology · doi:10.1016/j.ese.2025.100608 · Cited by 31
Angioni, S., Consoli, S., Dessí, D., Osborne, F., Recupero, D., & Salatino, A. (2024) · IEEE Access · doi:10.1109/access.2024.3407188 · Cited by 24
“For instance, large language models have been employed to summarize news [36].”
BackgroundXie, T., Kuang, Y., Tang, Y., Liao, J., & Yang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126371 · Cited by 39
Evaluating gender bias in large language models in long-term care
Rickman, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03118-0 · Cited by 40
“However, other work (e.g. Zhang et al. [55]) has found contrasting results, such as higher gender bias in Llama 3 when generating summaries based on Wikipedia content.”
Úbeda, P. L., Martín‐Noguerol, T., Díaz-Angulo, C., & Luna, A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105443 · Cited by 30
A Systematic Literature Review of Hallucinations in Large Language Models
Woesle, C., Fischer-Brandies, L., & Buettner, R. (2025) · IEEE Access · doi:10.1109/access.2025.3601206 · Cited by 40
“Large Language Models (LLMs) such as GPT-3, GPT-4, and PaLM have demonstrated remarkable capabilities in natural language understanding and generation [1], [2], question answering [3], code generation [4], powering advancements in dialogue systems [5], [6], and summarization [7], [8].”
A comprehensive survey on automatic text summarization with exploration of LLM-based methods
Zhang, Y., Jin, H., Meng, D., Wang, J., & Tan, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131928 · Cited by 321
Probing LLMs for hate speech detection: strengths and vulnerabilities
Roy, S., Harshvardhan, A. V., Mukherjee, A., & Saha, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.407 · Cited by 49
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 41
“Such three perspectives are notoriously known as vital factors that affect the trustworthiness of LLMs. 1) Out-of-Distribution (OOD) Detection: A fundamental premise of machine learning is the similarity in distribution be-tween training and future unseen test data.”
BackgroundEnhancing abstractive summarization of scientific papers using structure information
Bao, T., Zhang, H., & Zhang, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125529 · Cited by 24
“Zhang et al. (2024b) found that LLMs achieve high-quality news summarization through instruction tuning, and their evaluations revealed that LLM-generated summaries are comparable to those written by humans.”
Harašta, J., Novotná, T., & Šavelka, J. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09422-w · Cited by 47
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20
“For some domains, LLMs have already been shown to be effective at summarizing textual and numeric information, in some cases judged as on par with human written summaries (Zhang et al., 2023; Singh et al., 2023).”
BackgroundSelf-citationA comprehensive survey for automatic text summarization: Techniques, approaches and perspectives
Luo, M., Xue, B., & Niu, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128280 · Cited by 34
Li, S., Tan, W., Zhang, C., Li, J., Ren, H., Guo, Y., Jia, J., Liu, Y., Pan, X., Guo, J., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125920 · Cited by 24
BLIP: Facilitating the Exploration of Undesirable Consequences of Digital Technologies
Pang, R. Y., Santy, S., Just, R., & Reinecke, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642054 · Cited by 30
A large language model framework for literature-based disease–gene association prediction
Li, P.-H., Sun, Y.-Y., Juan, H., Chen, C., Tsai, H., & Huang, J. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf070 · Cited by 29
“Indeed, small and open-source LLMs have been demonstrated to be competent for article-level comprehension [35, 36], hence the methodology is not constrained to enterprise LLMs. Finally, our framework demonstrates remarkable efficacy in capturing disease–gene relationships through unsupervised…”
Bridging Generations using AI-Supported Co-Creative Activities
Kim, C. Y., Sato, A. J., White, N. T., Ho, H.-R., Lee, C. P., Hwang, Y., & Mutlu, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713718 · Cited by 35
“The domain in which the LLM-based tools are used varies from healthcare [29, 30, 32] to law [19, 24], and also span from simple text summarization tasks [55, 95], to complex and creative tasks, such as brainstorming ideas [62, 72] or creating new content ( i.e., co-creation) [53, 81, 97].”
LLMs outperform outsourced human coders on complex textual analysis
Bermejo, V. J., Gago, A., Gálvez, R. H., & Harari, N. (2025) · Scientific Reports · doi:10.1038/s41598-025-23798-y · Cited by 37
“In particular, we do not examine higher-order capabilities such as multi-sentence summarization, natural language inference (e.g., contradiction or entailment), or causal reasoning—tasks increasingly used to probe deeper comprehension and abstraction in LLMs 73,74 .”
Li, S., & Xu, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103467 · Cited by 19
Knowledge-Enhanced Graph Topic Transformer for Explainable Biomedical Text Summarization
Xie, Q., Tiwari, P., & Ananiadou, S. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3308064 · Cited by 33
“It’scrucialtonotethatalthoughChatGPT’sper-formance,asindicatedbytheROUGEscore,maynotout-performtheresultsfromfine-tunedmethods,theappropriate-nessofusingROUGEasabenchmarkforLLMshasrecently beenquestioned[55].”
Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers
Subbiah, M., Zhang, S., Chilton, L. B., & McKeown, K. R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00702 · Cited by 53
“…tasks (Tang et al., 2024; Zhang et al., 2023a; Tang et al., 2022; Jahan et al., 2024; Liu et al., 2023; Pu et al., 2023), including ones (Zhang et al., 2023b; Goyal et al., 2023) that have shown performance is saturated on the commonly used CNN/DM news summarization dataset (Hermann et…”
MethodSelf-citationA Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 155
Kaniwa, Y., Kuribayashi, M., Kayukawa, S., Sato, D., Takagi, H., Asakawa, C., & Morishima, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3676492 · Cited by 25
“Research shows that LLMs excel in downstream tasks like text summarization [80] and text classifcation [71].”
Ding, Z., Smith, A., Zhang, W., Tetreault, J., & Jaimes, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.217 · Cited by 45
Lee, S. Y.-T., Bahukhandi, A., Liu, D., & Ma, K. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456398 · Cited by 30
“…text exhibits statistical differences in certain linguistic features, such as part-of-speech (POS) tags, we conducted experiments on a dataset [73] that contains both human-written and LLM-generated summaries to select differentiable linguistic features and use them to compute the naturalness…”
MethodInvestigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 45
Weinz, M., Zannone, N., Allodi, L., & Apruzzese, G. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736195 · Cited by 44
“Importantly, LLM-based tools not only are effective at (i) imitating human writing to create persuasive texts [38], but they can also (ii) facilitate the summarization of unstructured information [108], such as that acquired via open-source intelligence (OSINT).”
UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models
Liu, Q., He, Y., Xu, T., Lian, D., Liu, C., Zheng, Z., & Chen, E. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679793 · Cited by 36
Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
Azarafza, M., Nayyeri, M., Steinmetz, C., Staab, S., & Rettberg, A. (2024) · International Conference on Control, Mechatronics and Automation (ICCMA) · doi:10.1109/iccma63715.2024.10843921 · Cited by 25
“Large language models have showcased remarkable performance in natural language processing tasks such as news summarization[8, 9], sentiment analysis[10, 11], Question-answering[12] and lexicalizing of arbitrary knowledge graph triples[13].”
Rautaray, J., Panigrahi, S., Nayak, A. K., Sahu, P., & Mishra, K. (2025) · IEEE Access · doi:10.1109/access.2025.3538169 · Cited by 15
Enhancing food safety review classification with large language models and label embedding
Xiong, S., Tian, W., Wang, B., Zhang, Y., Zhang, X., Che, Y., Shi, L., & Si, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127550 · Cited by 14
An evaluation of estimative uncertainty in large language models
Tang, Z., Shen, K., & Kejriwal, M. (2026) · npj Complexity · doi:10.1038/s44260-026-00070-6 · Cited by 34
“LLMs are now increasingly tasked with high-stakes communication, from summarizing scienti fi c literature 27,28 to serving as conversational assistants 29,30 and customer service agents 31,32 .”
Fatemi, B., Rabbi, F., & Opdahl, A. L. (2023) · IEEE Access · doi:10.1109/access.2023.3345414 · Cited by 18
“…their proficiency in addressing a multitude of natural language processing (NLP) tasks such as named entity recognition [24], text summarization [28] and sentiment analysis [12], and researchers from different domains such as computer science, medicine and social science, have recognized the…”
A topic modeling‐based bibliometric exploration of automatic summarization research
Chen, X., Xie, H., Tao, X., Xu, L., Wang, J., Dai, H., & Wang, F. L. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1540 · Cited by 24
“By leveraging knowledge from other related domains or topics, zero/few-shot learning can help the model generate informative summaries with limited training data (Zhang, Ladhak, et al., 2024).”
The potential of Large Language Models for social robots in special education
Voultsiou, E., Vrochidou, Ε., Moussiades, L., & Papakostas, G. A. (2025) · Progress in Artificial Intelligence · doi:10.1007/s13748-025-00363-2 · Cited by 23
Hila, A. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02426-3 · Cited by 14
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13
Sha, Y., Jiang, M., Luo, G., Meng, W., Zhai, X., Pan, H., Li, J., Yan, Y., Qiao, Y., Yang, W., et al. (2024) · Phytochemical Analysis · doi:10.1002/pca.3437 · Cited by 14
Ali, S. J., Reinhartz-Berger, I., & Bork, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75872-0_14 · Cited by 14
SKEL-Betweener: a Neural Motion Rig for Interactive Motion Authoring
Agrawal, D., Buhmann, J., Borer, D., Sumner, R. W., & Guay, M. (2024) · ACM Transactions on Graphics · doi:10.1145/3687941 · Cited by 28
“Recent motion models [Qin et al. 2022; Tevet et al. 2023] inspired by language models [Ranathunga et al. 2023; Zhang et al. 2024] and use transformers [Vaswani et al. 2017] with global attention, best suited for long range dependencies.”
On Large Language Models in National Security Applications
Caballero, W. N., & Jenkins, P. R. (2025) · Stat · doi:10.1002/sta4.70057 · Cited by 51
“Zhang et al. [90] study LLM performance on news article summarization, finding that a variety of LLMs performed on par with their human counterparts.”
Understanding the effects of human-written paraphrases in LLM-generated text detection
Lau, H. T., & Zubiaga, A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100151 · Cited by 13
Hallucinations in medical devices
Granstedt, J., Prabhat, K. C., Deshpande, R., Garcia, V. L., & Badano, A. G. (2025) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2025.100145 · Cited by 27
Early prediction of sepsis using chatGPT-generated summaries and structured data
Li, Q., Ma, H., Song, D., Bai, Y., Zhao, L., & Xie, K. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18378-7 · Cited by 16
Investigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Bai, Y., Li, J., Shen, J., & Zhao, L. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3464560 · Cited by 18
“Forinstance,LLMs canbeemployedfordocumentsummarization[13],[14],[15], [16],[17],allowinguserstoextractkeyinformationquicklyfrom largevolumesoftext.”
Configtrans: Network Configuration Translation Based on Large Language Models and Constraint Solving
Zheng, N., Li, F., Li, Z., Yang, Y., Hao, Y., Liu, C., & Wang, X. (2024) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp61940.2024.10858572 · Cited by 16
“Given the excellent performance of LLMs in traditional natural language processing (NLP) tasks such as translation [6], summarization [7], and named entity recognition [8], as well as their unique contributions in network operations [9], [10], particularly in network configuration generation [11],…”
Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study
Cui, Y., & Liang, M. (2024) · Applied Sciences · doi:10.3390/app14051925 · Cited by 16
Metadata
- Authors
- Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- Corresponding author
- Zhang, Tianyi
- Ladhak, Faisal
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 39-57
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391494845
- Cited by
- 415
- References
- 79
- Citations by year
- 2026: 108 · 2025: 165 · 2024: 96
- FWCI
- 54.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, news summarization, automatic summarization, human evaluation, instruction tuning, zero-shot summarization, fine-tuning, model size, human-written summaries, paraphrase, freelance writers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.