Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Open accessAbstract
Evaluation of natural language generation (NLG) is complex and multi-dimensional.Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest.Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets.In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets.Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency.We then analyze the effects of factors such as the selection and number of incontext examples on performance.Finally, we study the efficacy of in-context learningbased evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.Our code is available at https: //github.com/JainSameer06/ICE
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
By providing LLMs with concrete evaluation guidelines and feeding them both the model-generated and source content, they can effectively assess faithfulness.
“By providing LLMs with concrete evaluation guidelines and feeding them both the model-generated and source content, they can effectively assess faithfulness.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [133]
Section 2.2 says the LLM receives a generated summary and its source text to score consistency; Section 2.1 defines consistency as factual correctness relative to the source. The paper also reports strong agreement with human judgments on that dimension. But its prompts use scored in-context examples, not the concrete evaluation guidelines described in the claim. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
(2004) · Cited by 9,718 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · see citing works
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · see citing works
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353 · see citing works
Self-citationNews Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · see citing works
Annotating and Modeling Fine-grained Factuality in Summarization
Goyal, T., & Durrett, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.114 · Cited by 100 · see citing works
Evaluating Coherence in Dialogue Systems using Entailment
Dziri, N., Kamalloo, E., Mathewson, K. W., & Zai͏̈ane, O. R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1381 · Cited by 91 · see citing works
GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems
Huang, L., Ye, Z., Qin, J., Lin, L., & Liang, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.742 · Cited by 81 · see citing works
Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 49 · see citing works
ExplainaBoard: An Explainable Leaderboard for NLP
Liu, P., Fu, J., Xiao, Y., Yuan, W., Chang, S., Dai, J., Liu, Y., Ye, Z., & Neubig, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.34 · Cited by 35 · see citing works
Self-citationPrompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · arXiv · doi:10.48550/arxiv.2102.07350
Flagged first, then by citations
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
“Exploiting this capability, researchers have ventured into novel paradigms for assessing the faithfulness of model-generated content (Luo et al., 2023b; Laban et al., 2023; Adlakha et al., 2023; Gao et al., 2023b; Jain et al., 2023).”
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“Exploiting this capability, researchers have ventured into novel paradigms for assessing the faithfulness of model-generated content (Luo et al., 2023b; Laban et al., 2023; Adlakha et al., 2023; Gao et al., 2023b; Jain et al., 2023).”
LLM-based NLG Evaluation: Current Status and Challenges
Gao, M., Hu, X., Yin, X., Ruan, J., Pu, X., & Wan, X. (2025) · Computational Linguistics · doi:10.1162/coli_a_00561 · Cited by 155 · see citing works
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A. F. T., Neubig, G., Garg, A., Clark, J. H., Freitag, M., & Fırat, O. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.100 · Cited by 46 · see citing works
Self-citationMembership Inference Attacks Against In-Context Learning
Wen, R., Li, Z., Michael Backes, A., & Zhang, Y. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690306 · Cited by 22 · see citing works
Maity, S., Deroy, A., & Sarkar, S. (2024) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3734947.3734949 · Cited by 22 · see citing works
Ferreira, A. A., da Rocha, L. C. D., da Cunha, W. L. M., Machado, A. C., Campos, J. M., Jallais, G. F., Viana, A. C. F., Tuler, E., Araujo, I. B., Macul, V. C., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-13560-9 · Cited by 5 · see citing works
Reference-Free Summarization Evaluation with Large Language Models
Akkasi, A., Fraser, K., & Komeili, M. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.16 · Cited by 5 · see citing works
Walking a Tightrope – Evaluating Large Language Models in High-Risk Domains
Hung, C.-C., Rim, W. B., Frost, L., Bruckner, L., & Lawrence, C. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.8 · Cited by 5 · see citing works
Linking Surface Facts to Large-Scale Knowledge Graphs
Radevski, G., Gashteovski, K., Hung, C.-C., Lawrence, C., & Glavaš, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.445 · Cited by 3 · see citing works
Can LLMs Beat Humans on Discerning Human-written and LLM-generated Science News?
Soós, D., Jiang, M., & Wu, J. F. (2025) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3720553.3746674 · Cited by 1 · see citing works
Jain, S., Keh, S. S., Chhetri, S., Dewan, K., Izquierdo, P., Prüssmann, J., Shrestha, P., Suárez, C. F., Shi, Z. R., Li, L., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30218 · Cited by 1 · see citing works
Self-citationECR: An Expertise-Enriched Conclude-Then-Refine Summarization Framework for Professional Articles
Liang, Z., Xie, K., Lu, S., Shi, Y., Yeerpan, T., & Wang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_10
Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation
Feng, Y., Zhao, S., Li, Y., Xiao, L., Wu, X., & Luu, A. T. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228609
AutoEER: Large Language Model Agent Framework for Automated English Exam Report
Xie, D., Li, X., Xiong, Q., Wang, S., Jiang, Y., Liang, Y., & Jin, A. (2025) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst68792.2025.11415007
- Authors
- Jain, S., Keshava, V., Mysore Sathyendra, S., Fernandes, P., Liu, P., Neubig, G., & Zhou, C.
- Affiliations
- Carnegie Mellon University (US)
- Instituto Superior Técnico (PT)
- Meta (United States) (US)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2023
- Published
- 2023
- Pages
- 8487-8495
- OpenAlex
- W4385571776
- Cited by
- 15
- References
- 17
- Citations by year
- 2025: 5 · 2024: 5 · 2023: 5
- FWCI
- 2.66
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- text summarization, multidimensional assessment, in-context learning, factual consistency, relevance, zero-shot summarization, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
