A Field Guide to Automatic Evaluation of LLM-Generated Summaries
Open accessAbstract
Large Language models (LLMs) are rapidly being adopted for tasks such as text summarization, in a wide range of industries. This has driven the need for scalable, automatic, reliable, and cost-effective methods to evaluate the quality of LLM-generated text. What is meant by evaluating an LLM is not yet well defined and there are widely different expectations about what kind of information evaluation will produce. Evaluation methods that were developed for traditional Natural Language Processing (NLP) tasks (before the rise of LLMs) remain applicable but are not sufficient for capturing high-level semantic qualities of summaries. Emerging evaluation methods that use LLMs to evaluate LLM-output, appear to be powerful but lacking in reliability. New elements of LLM generated text that were not an element of previous NLP tasks, such as the artifacts of hallucination, need to be considered. We outline the different types of LLM evaluation currently used in the literature but focus on offline, system-level evaluation of the text generated by LLMs. Evaluating LLM-generated summaries is a complex and fast-evolving area, and we propose strategies for applying evaluation methods to avoid common pitfalls. Despite having promising strategies for evaluating LLM summaries, we highlight some open challenges that remain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inherent LLM biases impact the performance of LLM-as-a-judge, leading to biased assessments, preference for redundancy at the expense of accuracy, overestimation of decision quality, and similar issues.
“Therefore, biased assessments, preference for redundancy at the expense of accuracy, overestimation of the decision quality, and similar issues [24] prevent LLM judges from being considered reliable, autonomous evaluation tools, especially in the highly sensitive medical field.”
From Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study (2025) · cites this paper as [24]
Our previous survey on ransomware revealed concerns about the self-benchmarking practices in many anti-ransomware studies, which often used diverse ransomware samples and methodologies and based their claims of superiority merely on higher detection rates.
“Moreover, within the context of cybersecurity, our previous survey on ransomware revealed concerns about the self-benchmarking practices in many anti-ransomware studies, which often used diverse ransomware samples and methodologies and based their claims of superiority merely on higher detection rates [24].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [24]
Cites
(2004) · Cited by 9,718 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 4× · see citing works
“LLMs have known issues of hallucination, knowledge recency, reasoning inconsistency, difficulty in computational reasoning and more [1], which result in unpredictable errors.”
InfluentialA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1× · see citing works
“Model evaluation is often performed with benchmarks like HellaSwag, TruthfulQA and MMLU [9] Benchmarks are used to compare competing LLMs, using a fixed, standard dataset, and fixed, standard language tasks (like summarization).”
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335 · see citing works
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197 · see citing works
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175 · see citing works
Re-evaluating Evaluation in Text Summarization
Bhandari, M., Gour, P. N., Ashfaq, A., Liu, P., & Neubig, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.751 · Cited by 148 · see citing works
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Scialom, T., Lamprier, S., Piwowarski, B., & Staiano, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1320 · Cited by 121 · see citing works
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119 · see citing works
Fill in the BLANC: Human-free quality estimation of document summaries
Vasilyev, O., Dharnidharka, V., & Bohannon, J. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.2 · Cited by 84 · see citing works
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., & Gan, C. (2023) · arXiv · doi:10.48550/arxiv.2305.03047 · Cited by 81 · see citing works
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., & Li, H. (2023) · arXiv · doi:10.48550/arxiv.2308.05374 · Cited by 79 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 71 · see citing works
Josifoski, M., Šakota, M., Peyrard, M., & West, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.96 · Cited by 68 · see citing works
MasakhaNER: Named entity recognition for African languages
(2021) · HAL · doi:10.1162/tacl · Cited by 61 · see citing works
Bias and Fairness in Large Language Models: A SurveyPreprint
Gallegos, I. O., Rossi, R. A., Barrow, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.00770 · Cited by 58 · see citing works
Veselovsky, V., Ribeiro, M. H., Arora, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.15041 · Cited by 49 · see citing works
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · arXiv · doi:10.48550/arxiv.2111.09525 · Cited by 38 · see citing works
On the Limitations of Reference-Free Evaluations of Generated Text
Deutsch, D., Dror, R., & Roth, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.753 · Cited by 32 · see citing works
ROUGE-C: A fully automated evaluation method for multi-document summarization
He, T., Chen, J., Ma, L., et al. (2008) · IEEE International Conference on Granular Computing (GrC) · doi:10.1109/grc.2008.4664680 · Cited by 25 · see citing works
QuestEval: Summarization Asks for Fact-based EvaluationPreprint
Scialom, T., Dray, P.-A., Gallinari, P., Lamprier, S., Piwowarski, B., Staiano, J., & Wang, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12693 · Cited by 22 · see citing works
Building Guardrails for Large Language ModelsPreprint
Dong, Y., Mu, R., Jin, G., Qi, Y., Hu, J., Zhao, X., Meng, J., Ruan, W., & Huang, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01822 · Cited by 22 · see citing works
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document SummarizationPreprint
Gao, Y., Zhao, W., & Eger, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03724 · Cited by 14 · see citing works
How Well do Offline Metrics Predict Online Performance of Product Ranking Models?
Wang, X., Gao, R., Jain, A., Edge, G., & Ahuja, S. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591865 · Cited by 9 · see citing works
Josifoski, M., Šakota, M., Peyrard, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.04132 · Cited by 6 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive SummarizationPreprint
Shen, C., Cheng, L., Nguyen, X.-P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13091 · Cited by 6 · see citing works
Evaluating Factual Consistency of Texts with Semantic Role Labeling
Fan, J., Aumiller, D., & Gertz, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.9 · Cited by 2 · see citing works
Evaluating Factual Consistency of Texts with Semantic Role LabelingPreprint
Fan, J., Aumiller, D., & Gertz, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13309 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“Unlike traditional benchmarks with clear numerical metrics, LLM benchmarking often involves subjective judgments and complex evaluation criteria [13], [25].”
BackgroundVasilev, Y., Raznitsyna, I., Pamova, A., Burtsev, T., Bobrovskaya, T., Kosov, P., Vladzymyrskyy, A., Omelyanskaya, O., & Arzamasov, K. (2025) · Diagnostics · doi:10.3390/diagnostics16010003 · Cited by 5 · see citing works
“Therefore, biased assessments, preference for redundancy at the expense of accuracy, overestimation of the decision quality, and similar issues [24] prevent LLM judges from being considered reliable, autonomous evaluation tools, especially in the highly sensitive medical field.”
Zhang, S., Yin, M., Zhang, J., Liu, J., Han, Z., Zhang, J., Li, B., Wang, C., Wang, H., Chen, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.00212 · Cited by 1
- Authors
- van Schaik, T. A., & Pugh, B.
- Affiliations
- Microsoft, Redmond, USA
- Microsoft (United States) (US)
- Venue
- Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 2832-2836
- OpenAlex
- W4400530529
- Cited by
- 30
- References
- 38
- Citations by year
- 2026: 6 · 2025: 21 · 2024: 3
- FWCI
- 8.66
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- LLM-generated summaries, automated assessment, large language models, text summarization, hallucinations, system-level evaluation, evaluation pitfalls
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
