SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“Unlike previous studies that predominantly focused on specific tasks such as summarization (Fabbri et al., 2021; Tang et al., 2022)or particular domains such as world knowledge (Miao et al., 2023), this benchmark assesses factuality across five domains: world knowledge, science and technology,…”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
“SummEval (Fabbri et al., 2021) is a benchmark that compares different evaluation methods for summarization.”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
“SummEval (Fabbri et al., 2021) is a benchmark that compares different evaluation methods for summarization.”
BackgroundCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
“SummEval (Fabbri et al., 2020) Summaries generated by recent summarization models trained on CNN or DailyMail alongside human annotations.”
MethodGPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446
Evaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423
Evaluating Large Language Models on Medical Evidence Summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J., et al. (2023) · doi:10.1101/2023.04.22.23288967 · Cited by 423
“To obtain a comprehensive understanding of the summarization capabilities of LLMs, we conducted an extensive human evaluation of the model-generated summaries, which goes beyond the capabilities of automatic metrics [14, 15].”
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00632 · Cited by 415
“Such a level of correlation is comparable to that reported in Fabbri et al. (2020), which measures the correlation of automatic metrics on evaluating finetuned LMs and even earlier neural summarization systems.”
MethodResultBenchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2301.13848 · Cited by 415
“Such a level of correlation is comparable to that reported in Fabbri et al. (2020), which measures the correlation of automatic metrics on evaluating finetuned LMs and even earlier neural summarization systems.”
MethodResultPre-Trained Language Models for Text Generation: A Survey
Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2024) · ACM Computing Surveys · doi:10.1145/3649449 · Cited by 195
“Repre-sentative benchmarks include WMT Metrics Shared Tasks [60] for translation and SummEval [51] for summarization.”
BackgroundBiases in Large Language Models: Origins, Inventory, and Discussion
Navigli, R., Conia, S., & Ross, B. (2023) · Journal of Data and Information Quality · doi:10.1145/3597307 · Cited by 364
“Translation [24], Question Answering [94, 128], Information Retrieval [52, 130], Text Summarization [47, 49], Word Sense Disambiguation [6, 7, 14, 34, 86], Entity Linking [8, 25, 112], Semantic Role Labeling [18, 32, 33, 35, 105], Semantic Parsing [13, 85], and Natural Language Inference [90, 129], inter alia.”
BackgroundBARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353
“We can observe that: (1) Simply vanilla BARTS CORE can outperform BERTScore and MoverScore by a large margin on 8 settings except the I NFO perspective on SummEval.”
BackgroundInfluentialSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
“Currently we have a few benchmarks that are present for MT (MOSES scorer, 4 MultEval [24]), AS metrics [40] and for a few metrics on DG [117].”
BackgroundIs chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235
LLM-based NLG Evaluation: Current Status and Challenges
Gao, M., Hu, X., Yin, X., Ruan, J., Pu, X., & Wan, X. (2025) · Computational Linguistics · doi:10.1162/coli_a_00561 · Cited by 318
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“Unlike previous studies that predominantly focused on specific tasks such as summarization (Fabbri et al., 2021; Tang et al., 2022)or particular domains such as world knowledge (Miao et al., 2023), this benchmark assesses factuality across five domains: world knowledge, science and technology,…”
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207
“We also apply quantitative text-similarity methods 18 – 21 to compare the similarity of the LLM output to expert-generated responses.”
MethodUnderstanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197
A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models
Zhang, H., Yu, P. S., & Zhang, J. (2025) · ACM Computing Surveys · doi:10.1145/3731445 · Cited by 321
“As discussed in Section 2.3, researchers have explored reference-based and reference-free automatic metrics, but the correlation with human judgment is still limited and cannot reliably evaluate summary quality [56].”
InfluentialMMTEB: Massive Multilingual Text Embedding Benchmark
Enevoldsen, K., Chung, I., Kerboua, I. L., Kardos, M., Mathur, A., Stap, D., Gala, J. P., Siblini, W., Krzemiński, D., Winata, G. I., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.13595 · Cited by 317
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.-T., Smith, N. A., Zettlemoyer, L., & Yu, T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.71 · Cited by 131
BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization
Kryscinski, W., Rajani, N., Agarwal, D., Xiong, C., & Radev, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.488 · Cited by 88
Self-citationQuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175
“To evaluate QuestEval , we measure its correlation with human judgments on different datasets: 2020), it is one of the largest human-annotated datasets for summarization.”
MethodResultInfluentialMulti-document Summarization via Deep Learning Techniques: A Survey
Ma, C., Zhang, W. E., Guo, M., Wang, H., & Sheng, Q. Z. (2022) · ACM Computing Surveys · doi:10.1145/3529754 · Cited by 227
“However, current evaluation metrics [43] still have several obvious defects.”
BackgroundThe GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159
CTRLsum: Towards Generic Controllable Text Summarization
He, J., Kryscinski, W., McCann, B., Rajani, N., & Xiong, C. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.396 · Cited by 100
Self-citationEvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y., & Kim, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642216 · Cited by 204
“In the interface, the user composes pairs of prompts and a set of evaluation criteria, and then employs LLMs to both generate outputs with the prompts and evaluate the outputs on the criteria.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.532 · Cited by 132
“Alongside improving factuality, we have also witnessed growing interests in automated factu-ality evaluation, since popular word-matching-based metrics, e.g., ROUGE, correlate poorly with human-rated factual consistency levels (Gabriel et al., 2021; Fabbri et al., 2021).”
BackgroundRobustness Gym: Unifying the NLP Evaluation Landscape
Goel, K., Rajani, N. F., Vig, J., Taschdjian, Z., Bansal, M., & Ré, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations · doi:10.18653/v1/2021.naacl-demos.6 · Cited by 98
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
“We considered news article summaries produced by summarization models from SummEval (Fabbri et al., 2021) (e.”
MethodChen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023) · Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings) · doi:10.18653/v1/2023.findings-ijcnlp.32 · Cited by 68
“Furthermore, different types of text necessitate evaluation of distinct aspects, e.g. coherence, fluency, and consistency (Fabbri et al., 2021a; Mehri and Eskenazi, 2020a; Wang et al., 2023b), which makes it hard to design metrics for each type of text and dimension separately.”
BackgroundChen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023) · arXiv · doi:10.48550/arxiv.2304.00723 · Cited by 140
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119
Self-citationKoçak, B., Klontzas, M. E., Stanzione, A., Meddeb, A., Demircioğlu, A., Blüthgen, C., Bressem, K. K., Ugga, L., Mercaldo, N. D., Díaz, O., et al. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100030 · Cited by 144
Towards a Unified Multi-Dimensional Evaluator for Text Generation
Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., & Han, J. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.131 · Cited by 117
“It aims to evaluate the model output from multiple explainable dimensions and has been the dominant paradigm in human evaluation (Fabbri et al., 2021).”
BackgroundTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.287 · Cited by 58
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Ravaut, M., Joty, S., & Chen, N. F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.309 · Cited by 152
Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions
Omar, M., Choi, S., Nyang, D., & Mohaisen, D. (2022) · IEEE Access · doi:10.1109/access.2022.3197769 · Cited by 85
“As a case in point, Amazon builtan NLP-based recruiting tool that was deemed as a failure (and was eventually scraped) because the NLP-powered model demonstrated bias against female applicants [30].”
BackgroundRevisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Liu, Y., Fabbri, A., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.-S., Xiong, C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.228 · Cited by 55
Self-citationAnnotating and Modeling Fine-grained Factuality in Summarization
Goyal, T., & Durrett, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.114 · Cited by 100
A Closer Look into Using Large Language Models for Automatic Evaluation
Chiang, C.-H., & Lee, H.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.599 · Cited by 204
GeneAgent: self-verification language agent for gene-set analysis using domain databases
Wang, Z., Jin, Q., Wei, C.-H., Tian, S., Lai, P., Zhu, Q., Day, C., Ross, C., Leaman, R., & Lu, Z. (2025) · Nature Methods · doi:10.1038/s41592-025-02748-6 · Cited by 159
“…of different outputs between GeneAgent and GPT-4, we established four criteria following the existing studies on the evaluation of LLMs 57,58 . a. Relevance : Assess whether the content about genes pertinently reflects their functions, providing value to biologists. b. Readability :…”
Zhang, G., Jin, Q., Zhou, Y., Wang, S., Idnay, B., Luo, Y., Park, E., Nestor, J. G., Spotnitz, M. E., Soroush, A., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01239-w · Cited by 78
“PICO metrics NLG metrics are known to be inadequate for evaluating factual completeness and consistency 33 .”
Large Language Models are Diverse Role-Players for Summarization Evaluation
Wu, N., Gong, M., Shou, L., Liang, S., & Jiang, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_54 · Cited by 148
A Survey on Multi-modal Summarization
Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023) · ACM Computing Surveys · doi:10.1145/3584700 · Cited by 116
“[40] also illustrated how ROUGE is not capable of gauging the quality of generated summaries by doing an n-gram overlap with human-written reference summaries.”
BackgroundInfluentialUnderstanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Tang, L., Goyal, T., Fabbri, A. R., Laban, P., Xu, J., Yavuz, S., Kryściński, W., Rousseau, J. F., & Durrett, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.650 · Cited by 177
Self-citationFabbri, A. R., Han, S., Li, H., Li, H., Ghazvininejad, M., Joty, S., Radev, D., & Mehdad, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.57 · Cited by 117
Self-citationTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · arXiv · doi:10.48550/arxiv.2204.04991 · Cited by 84
“SummEval SummEval (Fabbri et al., 2020) is a comprehensive study of evaluation metrics for text summarization.”
BackgroundResultInfluentialScientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
“%) studies mentioned following a specific protocol for human evaluation: Chien et al [39] applied the approach proposed by Goldstein et al [52], and Lopez et al [31] and Searle et al [26] followed the protocol SummEval proposed by Fabbri et al [53].”
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y.-H., & Kim, J. (2023) · arXiv · doi:10.48550/arxiv.2309.13633 · Cited by 82
“In the interface, the user composes pairs of prompts and a set of evaluation criteria, and then employs LLMs to both generate outputs with the prompts and evaluate the outputs on the criteria.”
AlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81
Zero-Shot Cross-Lingual Summarization via Large Language Models
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., & Zhou, J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.2 · Cited by 115
“The instruction of each aspect (marked in purple in Figure 4) is inspired by SummEval (a widely-used summarization meta-evaluation benchmark dataset) (Fabbri et al., 2021).”
MethodBiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems
Asyrofi, M. H., Yang, Z., Yusuf, I. N. B., Kang, H. J., Thung, F., & Lo, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3136169 · Cited by 99
“problems, capitalization errors or obvious grammatical issues that make the text difficult to read [60].”
BackgroundEvaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00506 · Cited by 47
“, 2019), summarization (Bhandari et al., 2020; Pagnoni et al., 2021; Durmus et al., 2020; Gabriel et al., 2021; Fabbri et al., 2021; Durmus et al., 2022), and dialogue generation (Yeh et al.”
BackgroundWang, Y., Zhang, Z., & Wang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.482 · Cited by 162
From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
Adams, G., Fabbri, A. R., Ladhak, F., Lehman, E., & Elhadad, N. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.7 · Cited by 129
“The definitions of Informa-tive , Quality , and Attributable come from Aharoni et al. (2023), while Coherence comes from Fabbri et al. (2021) 3 .”
BackgroundSelf-citationTowards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
Lin, H., Luo, Z., Gao, W. G., Ma, J., Wang, B., & Yang, R. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645381 · Cited by 148
Neural natural language processing for long texts: A survey on classification and summarization
Tsirmpas, D., Gkionis, I., Papadopoulos, G. T., & Mademlis, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108231 · Cited by 87
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.42 · Cited by 75
“Prior work concerning evaluation of automatic metrics and human evaluation for NLG systems has mainly focused on general analysis of output quality or coherence and fluency (Callison-Burch et al., 2007; Graham, 2015; Fabbri et al., 2021), rather than factuality.”
BackgroundResultInfluentialTowards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
Deutsch, D., Bedrax-Weiss, T., & Roth, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00397 · Cited by 75
“The annotations provided by Fabbri et al. (2020) on the single-document summarization CNN/DM dataset score the outputs of 16 models across 100 instances.”
BackgroundMethodInfluentialSottana, A., Liang, B., Zou, K., & Yuan, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.543 · Cited by 119
Prompted Opinion Summarization with GPT-3.5
Bhaskar, A., Fabbri, A. R., & Durrett, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.591 · Cited by 124
“ROUGE and BERTScore have been critiqued in prior work as inaccurate indicators of summary quality (Fabbri et al., 2021; Liu and Liu, 2008; Cohan and Gohar-ian, 2016), particularly as the fluency and coherence of the outputs increase to near-human levels (Goyal et al., 2022).”
BackgroundResultSelf-citationEnhancing Uncertainty-Based Hallucination Detection with Stronger Focus
Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., & Fu, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.58 · Cited by 158
Overview of the MEDIQA 2021 Shared Task on Summarization in the Medical Domain
Ben Abacha, A., Mrabet, Y., Zhang, Y., Shivade, C., Langlotz, C., & Demner-Fushman, D. (2021) · Proceedings of the 20th Workshop on Biomedical Language Processing · doi:10.18653/v1/2021.bionlp-1.8 · Cited by 68
“To understand these metrics in the context of summarization, Fabbri et al. (2020) have compared 34 traditional and recent model-based metrics on a manually annotated sub-set from the CNN-DM dataset.”
MethodFELM: Benchmarking Factuality Evaluation of Large Language Models
Chen, S., Zhao, Y., Zhang, J., Chern, I.-C., Gao, S., Liu, P., & He, J. (2023) · neural information processing systems · doi:10.52202/075280-1927 · Cited by 198
Searle, T., Ibrahim, Z. M., Teo, J., & Dobson, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104358 · Cited by 83
“These measures have been used and defined in prior work for large scale qualitative assessment of summarisation texts(Fabbri et al., 2021).”
MethodGENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.787 · Cited by 81
“This has motivated adoption of human evaluation in recent works (Celikyilmaz et al., 2020; Fabbri et al., 2021), even though it poses The G ENIE architecture for evaluating text generation tasks, with a summarization example.”
BackgroundMethodRetrieval augmentation of large language models for lay language generation
Guo, Y., Qiu, W., Leroy, G., Wang, S., & Cohen, T. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104580 · Cited by 48
Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
Bagdasaryan, E., & Shmatikov, V. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833572 · Cited by 36
“Neither ROUGE, nor BLEU scores measure truthfulness, coherence, or other attributes [20, 22].”
BackgroundFactually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Roit, P., Ferret, J., Shani, L., Aharoni, R., Cideron, G., Dadashi, R., Geist, M., Girgin, S., Hussenot, L., Keller, O., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.344 · Cited by 126
ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics
Zhang, M., Li, C., Wan, M., Zhang, X., & Zhao, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121364 · Cited by 77
Deng, M., Tan, B., Liu, Z., Xing, E., & Hu, Z. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 47
“Content Selection, Importance (Nenkova and Passonneau, 2004) (Peyrard, 2019) (Fabbri et al., 2021), etc. et al., 2018) article.”
MethodInfluentialFactual consistency evaluation of summarization in the Era of large language models
Luo, Z., Xie, Q., & Ananiadou, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124456 · Cited by 80
Evaluating Generative Ad Hoc Information Retrieval
Gienapp, L., Scells, H., Deckers, N., Bevendorff, J., Wang, S., Kiesel, J., Syed, S., Fröbe, M., Zuccon, G., Stein, B., et al. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657849 · Cited by 72
“…take the general form 𝑘𝑖 = 1 ( 𝑖 ) · 𝑘𝑗 = 𝑖 𝑝 ( 𝑗 ) [17], where 𝑘 is the evaluation depth, or in our case, response length; 𝑔 ( 𝑑 𝑖 ) is the utility of the statement at position 𝑖 ; and 𝑝 ( 𝑗 ) is the probability of the user aborting their search immediately after position 𝑗 .”
Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success)
Shaib, C., Li, M., Joseph, S., Marshall, I., Li, J. J., & Wallace, B. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.119 · Cited by 44
What’s in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization
Adams, G., Alsentzer, E., Ketenci, M., Zucker, J., & Elhadad, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.382 · Cited by 44
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
“SummEval SummEval (Fabbri et al., 2020) is a comprehensive study of evaluation metrics for text summarization.”
BackgroundResultInfluentialAn Empirical Study of Clinical Note Generation from Doctor-Patient Encounters
Ben Abacha, A., Yim, W.-W., Fan, Y., & Lin, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.168 · Cited by 54
“Despite recurrent efforts in summarization evaluation, automatic evaluation of generated summaries still has several limitations and biases (Hardy et al., 2019; Fabbri et al., 2021; Ben Abacha et al., 2021).”
BackgroundA Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods
Deutsch, D., Dror, R., & Roth, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00417 · Cited by 46
“The standard method for calculating a CI for a correlation is the Fisher transformation (Fisher, 1992).”
MethodMeetingBank: A Benchmark Dataset for Meeting Summarization
Hu, Y., Ganter, T., Deilamsalehy, H., Dernoncourt, F., Foroosh, H., & Liu, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.906 · Cited by 148
Mutual Information Alleviates Hallucinations in Abstractive Summarization
van der Poel, L., Cotterell, R., & Meister, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.399 · Cited by 105
de Hond, A., Leeuwenberg, T., Bartels, R., van Buchem, M., Kant, I., Moons, K. G., & van Smeden, M. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00111-0 · Cited by 53
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Ribeiro, L., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.236 · Cited by 98
MENLI: Robust Evaluation Metrics from Natural Language Inference
Chen, Y., & Eger, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00576 · Cited by 113
“For instance, in ref-based setup, NLI-D with max aggregation beats all metrics except for DiscoScore with mean on SummEval and both NLI metrics highly correlate with system-level human ratings in RealSumm (above 0.8 Pearson), where most standard metrics obtain only 0.5–0.7 Pearson correlations.”
BackgroundMethodInfluentialControllable Abstractive Dialogue Summarization with Sketch Supervision
Wu, C.-S., Liu, L., Liu, W., Stenetorp, P., & Xiong, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.454 · Cited by 64
Self-citationFabbri, A. R., Rahman, F., Rizvi, I., Wang, B., Li, H., Mehdad, Y., & Radev, D. (2021) · doi:10.18653/v1/2021.acl-long.535 · Cited by 93
“Furthermore, the use of large self-supervised pretrained models such as BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2019) has achieved state-of-the-art performance across summarization tasks and strong performance in zero and few-shot settings (Fabbri et al., 2020a).”
BackgroundMethodResultInfluentialSelf-citationFactual Consistency Evaluation for Text Summarization via Counterfactual Estimation
Xie, Y., Sun, F., Deng, Y., Li, Y., & Ding, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.10 · Cited by 41
Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions
Murugadoss, B., Poelitz, C., Drosos, I., Le, V., McKenna, N., Negreanu, C., Parnin, C., & Sarkar, A. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i18.34157 · Cited by 72
“Firstly, we leverage two of the most prominently used datasets for coarse-grained NLG-quality evaluations: The SummEval (Fabbri et al. 2021) dataset contains news article summaries generated by different models together with human annotations for 4 different quality criteria e.g., fluency; and the TopicalChat (Gopalak…”
BackgroundMethodThe scales of justitia: A comprehensive survey on safety evaluation of LLMs
Liu, S., Li, C., Qiu, J., Huang, F., Chen, Z., Zhang, L., Hei, Y., Zhu, J., & Zhang, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104579 · Cited by 59
“Combined with natural language inference (NLI) models, GraphEval significantly improves balanced accuracy in hallucination detection, boosting NLI performance by approximately 6.2 points on benchmarks like SummEval [155] and QAGS-C [156], especially for longer outputs.”
InfluentialAn Investigation of Evaluation Methods in Automatic Medical Note Generation
Abacha, A. B., Yim, W., Michalopoulos, G., & Lin, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.161 · Cited by 85
(2021) · arXiv (Cornell University)
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2105.00071 · Cited by 47
“, 2019), summarization (Bhandari et al., 2020; Pagnoni et al., 2021; Durmus et al., 2020; Gabriel et al., 2021; Fabbri et al., 2021; Durmus et al., 2022), and dialogue generation (Yeh et al.”
BackgroundLadhak, F., Durmus, E., He, H., Cardie, C., & McKeown, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.100 · Cited by 46
Hallucination Detection and Hallucination Mitigation: An InvestigationPreprint
Luo, J., Li, T., Wu, D., Jenkin, M. R. M., Liu, S., & Dudek, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08358 · Cited by 72
“To evaluate its performance for summarization, the authors compare the proposed metric with various baselines on three annotated datasets: FRANK [36], QAGs [50] and SummEval [10], where Harim+ shows state-of-the-art correlation to human judgment.”
MethodOn the Limitations of Reference-Free Evaluations of Generated Text
Deutsch, D., Dror, R., & Roth, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.753 · Cited by 92
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.