The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Benchmarking Large Language Models for News Summarization
Open accessAbstract
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [70], human evaluation results indicate that humans tend to prefer the results generated by LLMs [42,138] compared to that of fine-tuned models.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [138]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are known to produce errors in many settings, from document summarisation7, to general reasoning tasks as well as more clinically relevant tasks8.”
From A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“LLMs (Bommasani et al., 2021; Chowdhery et al., 2022; Brown et al., 2020) have two distinctive features over previous pretrained models.”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
Method(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
Method(2005) · Cited by 4,152
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“In this approach, a diverse set of natural language processing tasks are reformulated into the prompting format and the LLM’s parameters are updated for these tasks either through supervised finetuning or reinforcement learning.”
MethodBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 3×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodSelf-citationOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656 · cited 1×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“To benchmark LLMs, we evaluated on the standard CNN/DM (Hermann et al., 2015) and XSUM datasets (Narayan et al., 2018), but found that low-quality reference summaries caused several issues.”
MethodLexRank: Graph-based Lexical Centrality as Salience in Text Summarization
Erkan, G., & Radev, D. R. (2004) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1523 · Cited by 1,700
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,553 · cited 1×
“To contextualize the performance of LLMs, we mainly compare to previous state-of-the-art approaches that leveraged supervised finetuning (Liu and Lapata, 2019; Lewis et al., 2019; Zhang et al., 2020; Liu et al., 2022b).”
MethodUnified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodNarayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · cited 2×
“To benchmark LLMs, we evaluated on the standard CNN/DM (Hermann et al., 2015) and XSUM datasets (Narayan et al., 2018), but found that low-quality reference summaries caused several issues.”
MethodPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 4×
“On XSUM, the metrics have very low correlation with faithfulness and relevance but it is also because the reference summaries are terrible in these aspects (Table 3; also see Maynez et al., 2020).”
BackgroundMethodInfluentialBLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
MethodGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
Using lexical chains for text summarization
Barzilay, R., & Elhadad, M. (1997) · doi:10.7916/d85b09vz · Cited by 855 · cited 1×
“These two benchmarks contain large scale data in the order of houndreds of thousands summaries but are created via “incidental supervi-son”.”
BackgroundAdvances in automatic text summarization
(2000) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(00)90017-8 · Cited by 681
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 1×
“…language instruction (e.g., “summa-rize this news article in three sentences”) and so-licit outputs by having LLMs generate answers di-Model Recently, instruction-tuning has emerged as an effective way to improve LLM prompting performance (Sanh et al., 2021; Wang et al., 2022; Ouyang et al., 2022).”
BackgroundFast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Chen, Y.-C., & Bansal, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1063 · Cited by 595 · cited 2×
“In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
BackgroundMethodSummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · cited 3×
“Such a level of correlation is comparable to that reported in Fabbri et al. (2020), which measures the correlation of automatic metrics on evaluating finetuned LMs and even earlier neural summarization systems.”
MethodResultLearning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550 · cited 1×
“While the exact training details are unknown, the success of Instruct Davinci might be credited to “learning from human feedback” (LHF; Stiennon et al., 2020; Ziegler et al., 2019).”
BackgroundIntroduction to the Special Issue on Summarization
Radev, D. R., Hovy, E., & McKeown, K. (2002) · Computational Linguistics · doi:10.1162/089120102762671927 · Cited by 539 · cited 1×
“…of producing a concise paragraph that captures the main points of a news article and has been a core problem within the field of automatic summarization (Radev et al., 2002; Rush et al., 2015; Nal-1 llm_summarization lapati et al., 2016; See et al., 2017; Chen and Bansal, 2018; Dong et al., 2019).”
BackgroundSelf-citationAutomatic text structuring and summarization
Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997) · Information Processing & Management · doi:10.1016/s0306-4573(96)00062-3 · Cited by 495 · cited 1×
“In this work, we benchmark LLMs on news summarization to understand their potential for automatic summarization and focus on two popular news summarization benchmarks, CNN/DM (Hermann et al., 2015) and XSUM (Narayan et al., 2018).”
Summarization beyond sentence extraction: A probabilistic approach to sentence compression
Knight, K., & Marcu, D. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00222-9 · Cited by 482 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Self-citationNewsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Grusky, M., Naaman, M., & Artzi, Y. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1065 · Cited by 479
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1×
“Large language models (LLMs) have shown promising results in zero-/few-shot tasks across a wide range of domains (Chowdhery et al., 2022; Bai et al., 2022; Brown et al., 2020; Zhang et al., 2022) and raised significant interest for their potential for automatic summarization (Goyal et al., 2022; Liu…”
BackgroundSentence Fusion for Multidocument News Summarization
Barzilay, R., & McKeown, K. R. (2005) · Computational Linguistics · doi:10.1162/089120105774321091 · Cited by 429 · cited 1×
“Summarization evaluation is another active area of research.”
BackgroundSelf-citationFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · cited 1×
“While the exact training details are unknown, the success of Instruct Davinci might be credited to “learning from human feedback” (LHF; Stiennon et al., 2020; Ziegler et al., 2019).”
BackgroundOpt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405
Information fusion in the context of multi-document summarization
Barzilay, R., McKeown, K. R., & Elhadad, M. (1999) · Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics - · doi:10.3115/1034678.1034760 · Cited by 385 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundSelf-citationAutomated Text Summarization in SUMMARIST
Hovy, E. H., & Lin, C.-Y. (1997) · Cited by 356
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353 · cited 1×
“The list of metrics we evaluate are: Rouge-L (Lin, 2004), METEOR (Baner-jee and Lavie, 2005), BertScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021). on CNN/DM and XSUM so we discuss them separately in the following paragraphs.”
MethodDurmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306
Self-citationNenkova, A., & McKeown, K. (2011) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000015 · Cited by 301 · cited 1×
“…of producing a concise paragraph that captures the main points of a news article and has been a core problem within the field of automatic summarization (Radev et al., 2002; Rush et al., 2015; Nal-1 llm_summarization lapati et al., 2016; See et al., 2017; Chen and Bansal, 2018; Dong et al., 2019).”
BackgroundSelf-citationSentence reduction for automatic text summarization
Jing, H. (2000) · Proceedings of the sixth conference on Applied natural language processing - · doi:10.3115/974147.974190 · Cited by 279 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundFrom discourse structures to text summaries
Marcu, D. (1997) · Cited by 262
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Self-citationSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238 · cited 1×
“The list of metrics we evaluate are: Rouge-L (Lin, 2004), METEOR (Baner-jee and Lavie, 2005), BertScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021). on CNN/DM and XSUM so we discuss them separately in the following paragraphs.”
MethodFrom single to multi-document summarization: a prototype system and its evaluation
Lin, C.-Y., & Hovy, E. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073160 · Cited by 229 · cited 1×
“In this work, we benchmark LLMs on news summarization to understand their potential for automatic summarization and focus on two popular news summarization benchmarks, CNN/DM (Hermann et al., 2015) and XSUM (Narayan et al., 2018).”
Cut and paste based text summarization
Jing, H., & McKeown, K. R. (2000) · Cited by 228
Self-citationBRIO: Bringing Order to Abstractive Summarization
Liu, Y., Liu, P., Radev, D., & Neubig, G. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.207 · Cited by 212
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Liu, Y., Fabbri, A. R., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.-S., Xiong, C., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.228 · Cited by 212
Nenkova, A., Vanderwende, L., & McKeown, K. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148269 · Cited by 206 · cited 1×
“In this work, we benchmark LLMs on news summarization to understand their potential for automatic summarization and focus on two popular news summarization benchmarks, CNN/DM (Hermann et al., 2015) and XSUM (Narayan et al., 2018).”
Self-citationSilber, H. G., & McCoy, K. F. (2002) · Computational Linguistics · doi:10.1162/089120102762671954 · Cited by 205 · cited 1×
“CNN/DM includes articles from the CNN and DailyMail websites as the source article and adapt the bullet point highlights that come with the website articles as reference summaries.”
BackgroundBLEU might be Guilty but References are not Innocent
Freitag, M., Grangier, D., & Caswell, I. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.5 · Cited by 201
Summarizing Similarities and Differences Among Related Documents
Mani, I., & Bloedorn, E. (1999) · Information Retrieval · doi:10.1023/a:1009930203452 · Cited by 200 · cited 1×
“In this work, we benchmark LLMs on news summarization to understand their potential for automatic summarization and focus on two popular news summarization benchmarks, CNN/DM (Hermann et al., 2015) and XSUM (Narayan et al., 2018).”
News Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · cited 3×
“On XSUM, the metrics have very low correlation with faithfulness and relevance but it is also because the reference summaries are terrible in these aspects (Table 3; also see Maynez et al., 2020).”
BackgroundSentence compression beyond word deletion
Cohn, T., & Lapata, M. (2008) · Proceedings of the 22nd International Conference on Computational Linguistics - COLING '08 · doi:10.3115/1599081.1599099 · Cited by 169 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundFill in the BLANC: Human-free quality estimation of document summaries
Vasilyev, O., Dharnidharka, V., & Bohannon, J. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.2 · Cited by 169
Two uses of anaphora resolution in summarization
Steinberger, J., Poesio, M., Kabadjov, M. A., & Ježek, K. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.01.010 · Cited by 160 · cited 1×
“CNN/DM includes articles from the CNN and DailyMail websites as the source article and adapt the bullet point highlights that come with the website articles as reference summaries.”
BackgroundThe decomposition of human-written summary sentences
Jing, H., & McKeown, K. R. (1999) · Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/312624.312666 · Cited by 159 · cited 1×
“As a result, the reference summaries in these datasets are known to have quality issues (Maynez et al., 2020; Kang and Hashimoto, 2020), motivating us to addresses these defects to improve LLM evaluation.”
BackgroundSelf-citationDiscriminative Sentence Compression with Soft Syntactic Evidence
McDonald, R. (2006) · Conference of the European Chapter of the Association for Computational Linguistics · Cited by 159
Radev, D., Jing, H., & Budzikowska, M. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0005020 · Cited by 141
Sentence fusion via dependency graph compression
Filippova, K., & Strube, M. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613741 · Cited by 124 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundQAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119
Fair Work: Crowd Work Minimum Wage with One Line of Code
Whiting, M. E., Hugh, G., & Bernstein, M. S. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5283 · Cited by 115 · cited 2×
“Therefore, we pay our writers $4 for every article they summarize to following the recommended practice (Whiting et al., 2019).”
BackgroundMethodInfluentialImproved Natural Language Generation via Loss Truncation
Kang, D. D., & Hashimoto, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.66 · Cited by 115
Self-citationEvaluating Factuality in Generation with Dependency-level Entailment
Goyal, T., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.322 · Cited by 93
Explorations in Sentence Fusion
Marsi, E., & Krahmer, E. (2005) · Research portal (Tilburg University) · Cited by 85
11th Conference of the European Chapter of the Association for Computational Linguistics
McCarthy, D., & Wintner, S. (2006) · Conference of the European Chapter of the Association for Computational Linguistics · Cited by 80
Recursively Summarizing Books with Human Feedback
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., & Christiano, P. (2021) · arXiv · doi:10.48550/arxiv.2109.10862 · Cited by 76 · cited 2×
“3 following prior work (Wu et al., 2021).”
MethodSuper-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksPreprint
Wang, Y., Mishra, S. R., Alipoormolabashi, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07705 · Cited by 46
Supervised Sentence Fusion with Single-Stage Inference
Thadani, K., & McKeown, K. R. (2013) · International Joint Conference on Natural Language Processing · Cited by 43
Self-citationDeveloping reproducible and comprehensible computational models
Lane, P. C. R., & Gobet, F. (2003) · University of Hertfordshire Research Archive (University of Hertfordshire) · doi:10.1016/s0004-3702 · Cited by 42
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · arXiv · doi:10.48550/arxiv.2111.09525 · Cited by 38 · cited 1×
“The list of metrics we evaluate are: Rouge-L (Lin, 2004), METEOR (Baner-jee and Lavie, 2005), BertScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021). on CNN/DM and XSUM so we discuss them separately in the following paragraphs.”
MethodDeutsch, D., & Roth, D. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.24 · Cited by 37 · cited 1×
“Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
BackgroundKrahmer, E., Marsi, E., & van Pelt, P. (2008) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1557690.1557745 · Cited by 32
Multi-Document Summarization with Iterative Graph-based Algorithms
Mihalcea, R. F., & Tarau, P. (2005) · University of North Texas Digital Library (University of North Texas) · Cited by 18
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
Self-citationIntroduction to Large Language Models
Singh, B. M. (2024) · Apress eBooks · doi:10.1007/979-8-8688-0569-1_1 · Cited by 8
NAACL-ANLP 2000 Workshop on Automatic summarization -
(2000) · doi:10.3115/1117575 · Cited by 7
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 1×
“…(LLMs) have shown promising results in zero-/few-shot tasks across a wide range of domains (Chowdhery et al., 2022; Bai et al., 2022; Brown et al., 2020; Zhang et al., 2022) and raised significant interest for their potential for automatic summarization (Goyal et al., 2022; Liu et al., 2022a).”
Background
Research works printed but not linked: 1
- Conroy, 2006, Back to basics: Classy 2006, 10.12968/sece.2006.11.755s · doi:10.12968/sece.2006.11.755s
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“…language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023a; Chu et al., 2023).”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
“We use the dataset collected in Zhang et al. (2023), where they first ask freelance writers to write high-quality summaries for news articles, and then ask annotators to compare human-written summaries and LLM-3 Figure 2: Averaged G-E VAL -4’s scores for human-written summaries and GPT-3.”
MethodG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
“We use the dataset collected in Zhang et al. (2023), where they first ask freelance writers to write high-quality summaries for news articles, and then ask annotators to compare human-written summaries and LLM-3 Figure 2: Averaged G-E VAL -4’s scores for human-written summaries and GPT-3.”
MethodHarnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Metadata
- Authors
- Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- Corresponding author
- Zhang, Tianyi
- Ladhak, Faisal
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 39-57
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391494845
- Cited by
- 415
- References
- 79
- Citations by year
- 2026: 108 · 2025: 165 · 2024: 96
- FWCI
- 54.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, news summarization, automatic summarization, human evaluation, instruction tuning, zero-shot summarization, fine-tuning, model size, human-written summaries, paraphrase, freelance writers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.