FineSurE: Fine-grained Summarization Evaluation using LLMs
Open accessAbstract
Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and timeconsuming nature of human evaluation.Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likertscale scores.This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations.To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs).It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment.We compare various open-source and proprietary LLMs as backbones for FineSurE.In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions.The code is available at https://github.com/ DISL-Lab/FineSurE-ACL24.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The LLM-as-Judge evaluation strategy is widely adopted and has been demonstrated to align well with human preferences.
“This LLM-as-Judge evaluation strategy is widely adopted in both NLP and software engineering domains [35], [36], as it has been demonstrated to align well with human preferences and provides more nuanced evaluation compared to traditional metrics [37], [38].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [36]
The LLM-based Evaluator section describes growing use of LLMs as evaluators, and Agreement with Human Score reports strong correlations with human judgments for some models. The paper studies summarization evaluation; it does not establish widespread adoption in software engineering or uniformly strong human agreement across LLMs. The supplied copy gives no page numbers for these passages. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 43
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical re- port. arXiv preprint arXiv:2303.08774. · arXiv
- Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq, Pengfei Liu, and Graham Neubig. 2020. Re- evaluating evaluation in text summarization. In EMNLP.
- Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large lan- guage models for reference-free text quality evalua- tion: A preliminary empirical study. arXiv preprint arXiv:2304.00723. · arXiv
- Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? adversarial meta- evaluation of factuality in summarization. In EMNLP.
- Alexander R Fabbri, Wojciech Kryściński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
- Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Im- proved qa-based factual consistency evaluation for summarization. In NAACL.
- Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023. FactKB: Generaliz- able factuality evaluation using language models en- hanced with factual knowledge. In EMNLP.
- Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. · arXiv
- Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint arXiv:2304.02554. · arXiv
- Mingqi Gao and Xiaojun Wan. 2022. Dialsummeval: Revisiting summarization evaluation for dialogues. In NAACL.
- Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. TrueTeacher: Learning factual consistency evaluation with large language models. In EMNLP.
- Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment. In EMNLP.
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS.
- Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In NAACL.
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
- Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. · arXiv
- Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In EMNLP.
- Philippe Laban, Wojciech Kryściński, Divyansh Agar- wal, Alexander Richard Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring llm ability at factual reasoning through the lens of summarization. In EMNLP.
- Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. SummaC: Re-visiting nli- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:163-177.
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In ACL.
- Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language mod- els. arXiv preprint arXiv:2305.13711. · arXiv
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factual- ity in abstractive summarization. In ACL.
- Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In EMNLP.
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. ArXiv, abs/1808.08745. · arXiv
- Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metrics. In NAACL.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL.
- Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference la- beling by large language models. arXiv preprint arXiv:2310.10076. · arXiv
- Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Sta- iano, and Alex Wang. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In EMNLP.
- Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language models are not yet human-level evaluators for abstractive summarization. In EMNLP.
- Yuchen Shen and Xiaojun Wan. 2023. Opinsummeval: Revisiting automated evaluation for opinion summa- rization. arXiv preprint arXiv:2310.18122. · arXiv
- Xiaoming Shi, Jie Xu, Jinru Ding, Jiali Pang, Sichen Liu, Shuqing Luo, Xingwei Peng, Lu Lu, Haihong Yang, Mingtao Hu, et al. 2023. Llm-mini-cex: Automatic evaluation of large language model for diagnostic conversation. arXiv preprint arXiv:2308.07635. · arXiv
- Hwanjun Song, Igor Shalyminov, Hang Su, Singh Siffi, Kaisheng Yao, and Saab Mansour. 2023. Enhanc- ing abstractiveness of summarization models through calibrated distillation. In EMNLP.
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- PA Utama, J Bambrick, NS Moosavi, and I Gurevych. 2022. Falsesum: generating document-level nli ex- amples for recognizing factual inconsistency in sum- marization. In NAACL.
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In ACL.
- Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. In NeurIPS.
- Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jia- jun Chen. 2023. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959. · arXiv
- Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text gener- ation. In NeurIPS.
- Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. BERTScore: Evaluat- ing text generation with bert. In ICLR.
- Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In EMNLP.
- Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In EMNLP.
Cited by
Sorted by citations
LLM-LADE: Large language model-based log anomaly detection with explanation
Zhang, Z., Li, S., Zhang, L., Ye, J., Hu, C., & Yan, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114064 · Cited by 42 · see citing works
Large Language Models for Knowledge Graph Embedding: A Survey
Liu, B., Fang, Y., Xu, N., Hou, S., Li, X., & Li, Q. (2025) · Mathematics · doi:10.3390/math13142244 · Cited by 29 · see citing works
Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Models
Pan, S., & Wu, D. J. (2025) · International Conference on Computer Vision and Data Mining (ICCVDM) · doi:10.1109/iccvdm66874.2025.11290689 · Cited by 29 · see citing works
Chen, C. L. P., Li, L., Wang, T., Wang, Z. Z., Leng, J., Shao, H., & Cheng, L. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.02.008 · Cited by 23 · see citing works
Controllable Abstraction in Summary Generation for Large Language Models via Prompt Engineering
Song, X., Liu, Y., Luan, Y., Guo, J., & Guo, X. (2025) · International Conference on Information Science, Parallel and Distributed Systems (ISPDS) · doi:10.1109/ispds67367.2025.11391214 · Cited by 17 · see citing works
Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language ModelsPreprint
Pan, S., & Wu, D. J. (2025) · Research Square · doi:10.21203/rs.3.rs-7739054/v1 · Cited by 14 · see citing works
Reasoning or not? A comprehensive evaluation of reasoning LLMs for dialogue summarization
Jin, K., Wang, Y., Santos, L., Fang, T., Yang, X., Im, S., & Oliveira, H. G. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129831 · Cited by 12 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMs
Yuan, H., Hong, S., & Zhang, H. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.192 · Cited by 6 · see citing works
Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendation
Shimizu, R., Wada, T., Wang, Y., Kruse, J., O’Brien, S., HtaungKham, S., Song, L., Yoshikawa, Y., Saito, Y., Tsung, F., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714583 · Cited by 5 · see citing works
From Retrieval to Response: Tracing the Impact of Embedding Quality in RAG Systems
Amaral, O., Guo, Y., & Tang, Q. (2025) · IEEE Access · doi:10.1109/access.2025.3644595 · Cited by 4 · see citing works
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3 · see citing works
The Effect of Document Summarization on LLM-Based Relevance Judgments
Mohtadi, S., Roitero, K., Mizzaro, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_5 · Cited by 3 · see citing works
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation
Koupaee, M., Vincent, J. W., Mansour, S., Shalyminov, I., He, H., Song, H., Shu, R., He, J., Nian, Y., Wong, A. W., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.609 · Cited by 3 · see citing works
Self-citationLFOSum: Summarizing Long-form Opinions with Large Language ModelsPreprint
Nayeem, M. T., & Rafiei, D. (2024) · Qeios · doi:10.32388/d1mvb5 · Cited by 3 · see citing works
CASESUMM: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court OpinionsPreprint
Heddaya, M., MacMillan, K., Malani, A., Mei, H., & Tan, C. (2025) · Qeios · doi:10.32388/ej9r29 · Cited by 3 · see citing works
Automatic Summarization Evaluation: Methods and Practices
Menéndez, H. D., & Dakhama, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80889-0_12 · Cited by 2 · see citing works
Li, S., Li, J., Qu, Y., Shi, X., Guo, Y., He, Z., Wang, Y., & Tan, W. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.477 · Cited by 2 · see citing works
Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Alymova, Y. A., Альфредовна, А. Д., Федоровна, Ш. Ю., Kodenko, M. R., Blokhin, I. A., & Reshetnikov, R. V. (2025) · AI · doi:10.3390/ai6110287 · Cited by 2 · see citing works
MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs
Zhang, M., Wang, R., Xia, B., Sun, Y., & Zhao, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-5123-8_24 · Cited by 1 · see citing works
Enhancing the Undetectability of AI-Generated Text: A Semantics-Preserving Evasion Technique
Luo, B. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227223 · Cited by 1 · see citing works
Quantum-Inspired Cross-Attention Alignment for Turkish Scientific Abstractive Summarization
ALTAY, G., & Küçüksille, E. U. (2025) · Electronics · doi:10.3390/electronics14224474 · Cited by 1 · see citing works
Large Language Models in the Travel Domain: An Industrial Experience
Meglio, S. D., Somma, A., Starace, L. L. L., Scippacercola, F., Sperlí, G., & Martino, S. D. (2025) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2025-022 · Cited by 1 · see citing works
Lie Algebra–Based Semantic Flow for Incompleteness Detection in Summarization
Ravikiran, M., Elluru, V., Iyer, K., Pendse, P., & Mohanty, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4575-9_5 · Cited by 1 · see citing works
Shah, A. M. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2718320 · Cited by 2 · see citing works
Bello-Orgaz, G., Dakhama, A., Ramirez‐Atencia, C., & Menéndez, H. D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5593064
Elatiky, A. F., Hamad, A. M., Khaled, H., & Fayez, M. (2026) · Algorithms · doi:10.3390/a19020096
Lei, Y., Zheng, S., Wang, R., Pang, L., He, L., Chen, S., Wang, Y., Shen, H., Cheng, X., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_21
Automating Documentation of Complex Data Processing Flows with Large Language Models
Mahya, P., Martínez-Gil, J., Winterer, M., Neumüller, C., & Krump, M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-24216-7_7
FREEsum: A Conceptual Framework for Evaluating Text Summarization Approaches
Alves, L. V., & Rocha, C. (2026) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2026.248662
Evas: a unified deep learning framework for multi-dimensional evaluation of text summaries
G, K. K. T., Shreyas, J., Nema, V. R., Radhakrishna, S., Kashyap, K., & Koolagudi, S. G. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02827-w
Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation
Botcazou, I., Amghar, T., Lamprier, S., & Saubion, F. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.286
Learning to Control Summaries with Score Ranking
Liu, H., Ding, L., & Henao, R. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1762
Anchoring the Cache: Mitigating Contextual Hallucination in KV-Compressed Long-Context Summarization
Fu, Y., Luo, C., Valvoda, J., Zhang, X., Lei, X., Pan, X., Liu, H., & Dong, Y. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1542
Aldashnan, B., Alothaim, A., & Alsanad, A. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3838729
The role of diversity in few-shot learning for multi-label classification of legal documents
Adhikary, S., Ganguly, D., Roy, D., Guha, S. K., & Ghosh, K. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09530-9
FineSurE-VN: Semantic-Based Evaluation of Vietnamese Text Summarization
Bui, T. M., Vu, A. M., Trinh, K. N., & Mai, D.-T. (2026) · International Conference on Intelligent Information Technology (ICIIT) · doi:10.1145/3805862.3805894
Contextual Learning Strategies for Dialogue Summarization Using Large Language Models
Penyasov, D. E., & Turalchuk, K. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700252
Yu, C., Huang, Q., Shen, X., Wan, Q., Dai, Z., Sun, J., & Liang, R. (2026) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3773078.3831792
Roy, A., Sunaira, F. H., Orpa, T., Yussouf, F. S., Naheen, I. T., & Khan, R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0357890
Disassembly-Augmented Root Cause Analysis for C/C++ Segmentation Faults in SAP HANA
Jung, J., You, C., Hong, J., Bach, T., & An, G. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834461
Feedback as a Guide: Enhancing LLM Comprehension of Modern Chinese Poetry
Wang, S., Wong, D. F., Luo, J., Yao, J., & Chao, L. S. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.475
- Authors
- Song, H., Su, H., Shalyminov, I., Cai, J., & Mansour, S.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Amazon (United States) (US)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 906-922
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402667059
- Cited by
- 32
- References
- 0
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 1
- FWCI
- 9.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Research Foundation
- National Research Foundation of Korea
- Ministry of Science and ICT, South Korea
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination detection, faithfulness, completeness, brevity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
