The DOI matches its Crossref record, with a small difference in title or year; 6 of the 19 references it cites were found in a registry or an index, and 13 are not checked yet.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Open accessAbstract
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful.Existing automatic metrics do not capture such mistakes effectively.We tackle the problem of evaluating faithfulness of a generated summary given its source document.We first collected human annotations of faithfulness for outputs from numerous models on two datasets.We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful.Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, 1 which leverages recent advances in reading comprehension.Given questionanswer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary.Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.* Most of the work is done while the authors were at Amazon Web Services AI.1 Faithfulness Evaluation with Question Answering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A similar phenomenon has been shown in other NLG tasks, where it has been discovered that traditional metrics have a poor correlation with human judgment in terms of the hallucination problem [45, 54, 101, 125].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The results show that these reference-free metrics have substantially higher correlations with human judgments of faithfulness than the baseline metrics.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“The extracted answers that do not match the provided answers indicate unfaithful information in the summary.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to classifier-based metrics, QA-based metrics [77, 119, 271, 310] have recently garnered attention for their enhanced ability to capture information overlap between the model's generation and its source.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [77]
The cited source says what the claim attributes to it.
In the citing paper
“Their approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precisionoriented.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“The guidelines are based on the insights provided byDurmus et al. (2020) regarding annotating faithfulness.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. D. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219885 · Cited by 3,028
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,376
Research works printed but not linked: 32
- Z. Cao, F. Wei, W. Li, and S. Li. 2018. Faithful to the original: Fact aware neural abstractive summariza- tion. In Association for the Advancement of Artifi- cial Intelligence (AAAI).
- Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University.
- P. Chen, F. Wu, T. Wang, and W. Ding. 2018. A se- mantic QA-based approach for text summarization evaluation. In Association for the Advancement of Artificial Intelligence (AAAI).
- Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In ACL.
- Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922. · arXiv
- M. Eyal, T. Baumel, and M. Elhadad. 2019. Ques- tion answering as an automatic evaluation metric for news article summarization. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL).
- T. Falke, L. F. R. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging appli- cation for natural language inference. In Association for Computational Linguistics (ACL).
- M. Gardner, J. Berant, H. Hajishirzi, A. Talmor, and S. Min. 2019. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291. · arXiv
- B. Goodrich, V. Rao, P. J. Liu, and M. Saleh. 2019. Assessing the factual accuracy of generated text. In International Conference on Knowledge Discovery and Data Mining (KDD).
- M. Grusky, M. Naaman, , and Y. Artzi. 2018. News- room: A dataset of 1.3 million summaries with di- verse extractive strategies. In North American Asso- ciation for Computational Linguistics (NAACL).
- T. Hashimoto, H. Zhang, and P. Liang. 2019. Unify- ing human and statistical evaluation for natural lan- guage generation. In North American Association for Computational Linguistics (NAACL).
- Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In Proceedings of the 28th International Conference on Neural Infor- mation Processing Systems -Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
- A. Holtzman, J. Buys, M. Forbes, and Y. Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- W. Kryściński, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. 2019. Neural text summarization: A critical evaluation. In Empirical Methods in Natural Language Processing (EMNLP).
- L. Lebanoff, J. Muchovej, F. Dernoncourt, D. S. Kim, S. Kim, W. Chang, and F. Liu. 2019. Analyzing sentence fusion in abstractive summarization. arXiv preprint arXiv:1910.00203. · arXiv
- J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective func- tion for neural conversation models. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110-119.
- M. Li, S. Roller, I. Kulikov, S. Welleck, Y. Boureau, K. Cho, and J. Weston. 2019. Don't say that! mak- ing inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911.03860. · arXiv
- C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Char- lin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsuper- vised evaluation metrics for dialogue response gen- eration. In Empirical Methods in Natural Language Processing (EMNLP).
- Y. Liu and M. Lapata. 2019. Text summarization with pretrained encoders. In Empirical Methods in Natu- ral Language Processing (EMNLP).
- R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard- Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic turing test: Learning to evaluate dia- logue responses. In Association for Computational Linguistics (ACL).
- Mani, G. Klein, L. Hirschman, T. Firmin, D. House, and B. Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In European Associ- ation for Computational Linguistics (EACL).
- R. T. McCoy, E. Pavlick, and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. arXiv preprint arXiv:1902.01007. · arXiv
- Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
- T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano. 2019. Answers unite! unsupervised metrics for re- inforced summarization models. In Empirical Meth- ods in Natural Language Processing (EMNLP).
- A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguis- tics (NAACL).
- Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. 2017. Neu- ral machine translation with reconstruction. In Asso- ciation for the Advancement of Artificial Intelligence (AAAI).
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700.
- Curran Associates, Inc. S. Wiseman, S. M. Shieber, and A. M. Rush. 2017. Challenges in data-to-document generation. In Em- pirical Methods in Natural Language Processing (EMNLP).
- F. Zhang, J. Yao, and R. Yan1. 2018. On the ab- stractiveness of neural document summarization. In Empirical Methods in Natural Language Processing (EMNLP).
- T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019a. BERTSCORE: Evaluat- ing text generation with BERT. arXiv preprint arXiv:1904.09675. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. · arXiv
Cited by
Sorted by citations
Items 201–300 of 318
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning
Wu, W., Li, W., Xiao, X., Liu, J., Li, S., & Lyu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.18 · Cited by 11
Exploring In-Context Learning for Knowledge Grounded Dialog Generation
Chen, Q., Wu, W., & Li, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.675 · Cited by 4
Probing Factually Grounded Content Transfer with Factual Ablation
West, P., Quirk, C., Galley, M., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.294 · Cited by 6
Building a Knowledge-Based Dialogue System with Text Infilling
Xue, Q., Takiguchi, T., & Ariki, Y. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.25 · Cited by 4
Improving Factual Consistency of Abstractive Summarization via Question AnsweringPreprint
Nan, F., dos Santos, C. N., Zhu, H., Ng, P., McKeown, K. R., Nallapati, R. M., Zhang, D., Wang, Z., Arnold, A. O., & Xiang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04623 · Cited by 4
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Fang, B., Dai, X., & Karimi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.578 · Cited by 4
Zhu, Y., Li, Y., Li, L., Hu, D., Hao, X., Chen, D., Zhang, X., Zhang, Z., Zhou, Y., Litvak, M., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3567744 · Cited by 4
Unsupervised Token-level Hallucination Detection from Summary Generation By-products
Marfurt, A., & Henderson, J. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.21 · Cited by 5
SimQA: Detecting Simultaneous MT Errors through Word-by-Word Question Answering
Han, H., Carpuat, M. J., & Boyd-Graber, J. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.378 · Cited by 5
Reducing named entity hallucination risk to ensure faithful summary generation
Akani, E., Favre, B., Béchet, F., & Gemignani, R. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.33 · Cited by 6
Chen, W., Wu, C.-K., Chen, H., & Chen, C.-C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.54 · Cited by 4
Atri, Y. K., Iyer, A. S., Chakraborty, T., & Goyal, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.133 · Cited by 4
A Model-Based Evaluation Metric for Question Answering Systems
Bakır, D., Aktaş, M. S., & Yildiz, B. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500032 · Cited by 3
Khosla, S., Lovelace, J., Dutt, R., & Pratapa, A. (2021) · doi:10.18653/v1/2021.dialdoc-1.13 · Cited by 3
Li, S., & Xu, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191247 · Cited by 3
Xia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 3
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14302 · Cited by 31
Pilgram, L., Kababji, S. E., Liu, D., & Emam, K. E. (2025) · Journal of Medical Internet Research · doi:10.2196/77893 · Cited by 4
A Review of Question-Answering Systems Using Deep Learning in the Arabic Language
Aloqla, A., Khalifa, R. M. S., & Alghamdi, W. (2025) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.14229 · Cited by 3
Wang, X., Wang, J., Xu, B., Lin, H., Zhang, B., & Yang, Z. (2022) · JMIR Medical Informatics · doi:10.2196/38052 · Cited by 2
What Makes a Good and Useful Summary? Incorporating Users in Automatic Summarization Research
ter Hoeve, M., Kiseleva, J., & de Rijke, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.4 · Cited by 4
Improving Factual Consistency of Dialogue Summarization with Fact-Augmentation Mechanism
Li, W., Zhou, X., Bai, X., & Pan, S. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891885 · Cited by 2
Evaluating Factuality in Cross-lingual Summarization
Gao, M., Wang, W., Wan, X., & Xu, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.786 · Cited by 2
Zhang, M., You, D., & Wang, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302104 · Cited by 2
ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
Vasilyev, O., & Bohannon, J. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.10 · Cited by 3
The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation
Chang, E., Shen, X., Marin, A., & Demberg, V. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.36 · Cited by 2
Altakrori, M. H., Scialom, T., Fung, B. C. M., & Cheung, J. C. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.153 · Cited by 2
Evaluating Factual Consistency of Texts with Semantic Role Labeling
Fan, J., Aumiller, D., & Gertz, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.9 · Cited by 4
Ujlayan, A., & Sharma, M. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-82322-1_5 · Cited by 2
Scirè, A., Ghonim, K., & Navigli, R. (2024) · arXiv · doi:10.48550/arxiv.2403.02270 · Cited by 2
Zhang, Y., Zhou, D., Wang, Z., & Lai, Y. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101570 · Cited by 1
Liu, J., Du, W., Li, Q., Wang, X., Zhou, Z., & Liu, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448276 · Cited by 1
Ok, C., Lee, E., Ju, H., Lee, J., & Oh, D. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115961 · Cited by 1
Factual Accuracy Checking by Validating Numerical Data in Financial Summaries
Bera, A., & Goel, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825697 · Cited by 1
Multi-Agent Based Casual Triple Extraction For Factuality Evaluation Using Large Language Models
Zhang, J., Xu, Z., Liu, L., Shen, Z., Cui, Y., & Zhang, Y. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00024 · Cited by 1
Lie Algebra–Based Semantic Flow for Incompleteness Detection in Summarization
Ravikiran, M., Elluru, V., Iyer, K., Pendse, P., & Mohanty, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4575-9_5 · Cited by 1
Exploring question answering: metric analysis and evaluation framework for enhanced interpretability
Navarro, L. C., Mucciaccia, S. S., Mutz, F. W., Paixão, T. M., Badue, C. S., Souza, A. F. D., & Oliveira-Santos, T. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12053-8 · Cited by 1
Liu, X., Pabolu, P., Jiang, Y., Fraser, T., Chen, R., & Gao, H. O. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133833 · Cited by 1
A Unified Summarization Model with Semantic Guide and Keyword Coverage Mechanism
Lin, W., Li, J., Yi, Z., Ji, B., Li, S., Yu, J., & Ma, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86383-8_27 · Cited by 1
Xiong, X., & Wei, W. (2022) · International Conference on Edge Computing and Applications (ICECAA) · doi:10.1109/icecaa55415.2022.9936453 · Cited by 1
Query-Focused Multi-document Summarization
Gao, J., Xiong, C., Bennett, P. N., & Craswell, N. (2023) · The information retrieval series · doi:10.1007/978-3-031-23080-6_4 · Cited by 1
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
Zhu, R., Qi, J., & Lau, J. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.377 · Cited by 7
UMSE: Unified Multi-scenario Summarization Evaluation
Gao, S., Yao, Z., Tao, C., Chen, X., Ren, P., Ren, Z., & Chen, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.236 · Cited by 2
Lotfi, E., Bruyn, M. D., Buhmann, J., & Daelemans, W. M. P. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.47 · Cited by 2
Contrastive Error Attribution for Finetuned Language Models
Ladhak, F., Durmus, E., & Hashimoto, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.643 · Cited by 5
Self-citationMulti-View Source Ablation for Faithful Summarization
Cao, S., Ma, L., Lu, D., Logan, R. L., Tetreault, J., & Jaimes, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.151 · Cited by 1
Content Richness Evaluation Method For Abstractive Summarization
Zhang, X., Hu, D., Li, B., Qin, Y. X., & Li, L. (2023) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis59572.2023.10263021 · Cited by 1
Guo, B., Feng, C., Liu, F., Li, X., & Wang, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7894-6_11 · Cited by 1
Hallucination Detection for Grounded Instruction Generation
Zhao, L., Nguyen, K., & Daumé, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.266 · Cited by 2
Improving Consistency for Text Summarization with Energy Functions
Zeng, Q., Yin, Q., Li, Z., Gao, Y., Nag, S., Wang, Z., Yin, B., Ji, H., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.798 · Cited by 1
Holistic Multi-layered System Design for Human-Centered Dialog Systems
Oruche, R., Akula, R., Goruganthu, S. K., & Calyam, P. P. (2024) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms59971.2024.10555807 · Cited by 1
Dutta, T., & Liu, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650810 · Cited by 1
Consecutive Question Generation via Dynamic Multitask Learning
Li, Y., Li, S., & Shi, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.493 · Cited by 1
Generating Coherent and Diverse Slogans with Sequence-to-Sequence Transformer.Preprint
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2021) · arXiv (Cornell University) · Cited by 1
EmailSum: Abstractive Email Thread SummarizationPreprint
Zhang, S., Çelikyılmaz, A., Gao, J., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14691 · Cited by 8
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of SummariesPreprint
Tang, X., Fabbri, A. R., Li, H., Mao, Z., Adams, G., Wang, B., Celikyilmaz, A., Mehdad, Y., & Radev, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09195 · Cited by 1
GIER: Gap-Driven Self-Refinement for Large Language Models
Dewri, R. (2025) · arXiv · doi:10.48550/arxiv.2509.00325
“…proposed fine-grained evaluation dimensions such as factual consistency, completeness, relevance, and helpfulness to assess LLM outputs across tasks like summarization and QA (Askell et al., 2021; Durmus et al., 2020; Fabbri et al., 2022; Tam et al., 2023; Wang et al., 2024; Zhang et al., 2024).”
Coreference Reasoning in Machine Reading Comprehension
Wu, M., Moosavi, N. S., Roth, D., & Gurevych, I. (2020) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2012.15573
Estimation of Summary-to-Text Inconsistency by Mismatched EmbeddingsPreprint
Vasilyev, O., & Bohannon, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05156
Towards Human-Free Automatic Quality Evaluation of German SummarizationPreprint
Iskender, N., Vasilyev, O., Polzehl, T., Bohannon, J., & Möller, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06027
Enriching Transformers with Structured Tensor-Product Representations for Abstractive SummarizationPreprint
Jiang, Y., Çelikyılmaz, A., Smolensky, P., Soulos, P., Rao, S., Palangi, H., Fernandez, R., Smith, C., Bansal, M., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01317
Investigating Crowdsourcing Protocols for Evaluatingthe Factual Consistency of Summaries
Tang, X., Fabbri, A. R., Mao, Z., Adams, G., Wang, B., Li, H., Mehdad, Y., & Radev, D. (2021)
Summary Explorer: Visualizing the State of the Art in Text SummarizationPreprint
Syed, S., Yousef, T., Al‐Khatib, K., Jänicke, S., & Potthast, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01879
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09209 · Cited by 15
RDF-to-Text Generation with Reinforcement Learning Based Graph-augmented Structural Neural Encoders.Preprint
Gao, H., Wu, L., Hu, P., Wei, Z., Xu, F., & Long, B. (2021) · arXiv (Cornell University)
Wang, X., Wang, J., Xu, B., Lin, H., Zhang, B., & Yang, Z. (2022) · doi:10.2196/preprints.38052
Image Captioning with External Knowledge
Nikiforova, S. (2023) · Utrecht University Repository (Utrecht University) · doi:10.33540/1709
Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generation
Brahman, F., Peng, B., Galley, M., Rao, S., Dolan, B., Chaturvedi, S., & Gao, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.547
BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics
Ma, L., Cao, S., Logan, R. L., Lu, D., Ran, S., Zhang, K., Tetreault, J., & Jaimes, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.716 · Cited by 3
Boosting Factual Consistency and High Coverage in Unsupervised Abstractive Summarization
Huang, Y.-H., Kuo, C.-T., & Chen, Y.-S. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.074
Zhao, G. L., Wu, T., Wang, M., Jiang, S., Zhang, Y., & Chu, J. (2023) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme60234.2023.00081
ATCoPE - Abstractive text comparison using prompt engineering
Chakraborty, S., & Pakray, P. (2024) · AIP conference proceedings · doi:10.1063/5.0208416
A Multi-Strategy QA Method Combining with Embedded User Feedback
Hu, Y., Zheng, X., & Zong, P. (2024) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd62003.2024.10604499
A Multi-Hop Reasoning Knowledge Selection Module for Dialogue Generation
Ma, Z., Liu, J., Xu, B., Lv, K., & Guo, S. (2024) · Electronics · doi:10.3390/electronics13163275
ECR: An Expertise-Enriched Conclude-Then-Refine Summarization Framework for Professional Articles
Liang, Z., Xie, K., Lu, S., Shi, Y., Yeerpan, T., & Wang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_10
The Chain of Narrative Framework: Advancing Controllable Generation of Financial Themes
Liu, S., Jia, G., Gu, D., Chen, L., & Yang, Q. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00022
Enhancing Generative Models for Scientific Text Simplification
Vendeville, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_39 · Cited by 1
Chen, D., Hu, D., & Li, L. (2024) · International Conference on Communication and Information Processing (ICCIP) · doi:10.1145/3708657.3708772
Alarfaj, F. K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-1758-6_9
RAG Based Product Review Summarization and Faithfulness Evaluation
Subbapurmath, N. R., & Kaur, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8197-6_25
Methods to Evaluate and Improve the Quality of Text Summarization
Nguyen, H. (2025) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2443390
Chen, D., Hu, D., & Li, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_33
Politically Unbiased News Report Generation
Bharne, A., Parveez, M., Prashanth, V., Singh, O., & Narayan, S. (2025) · doi:10.1109/icnlp65360.2025.11108576
FARE: Factual Alignment for Reliable Evaluation in Text Summarization
Wang, Z., Zhou, C., Zhang, J., & Hu, D. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-94898-5_54
Is it Enough to Ask Questions? Dialogue Evaluation through Question Answering and Generation
Vilaça, L., & Viana, P. (2025) · doi:10.1145/3746274.3760399
How Faithful Are Your Summaries? A Study of NLI-Based Verification in Portuguese
Paula, F. S. F., Westhelle, M., Corrêa, M. C. M., Bencke, L. R., & Moreira, V. P. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37834
MIRAGE: A Metrics lIbrary for Rating hAllucinations in Generated tExt
Vendeville, B., Магдановна, Е. Л., Loor, P. D., & Kamps, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761644 · Cited by 2
Hallucinations in Abstractive Text Summarization with Large Language Models
Delpisheh, N., & Chali, Y. (2025) · International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11231924 · Cited by 1
Triple-based Factual Correctness Evaluation of AI-Generated Summaries
Latipov, I.-A., Holenderski, M., & Meratnia, N. (2025) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2025.12.107
Elewa, M. S. A. H., Hamdi, A., Kassab, H., & Shaban, K. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315358 · Cited by 2
Localizing Factual Inconsistencies in Attributable Text Generation
Cattan, A., Roit, P., Zhang, S., Wan, D., Aharoni, R., Szpektor, I., Bansal, M., & Dagan, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.598 · Cited by 11
Katwe, P., Balabantaray, R. C., & Vittala, K. P. (2026) · SN Computer Science · doi:10.1007/s42979-025-04684-w
Baumgärtel, F., Bono, E., Fillinger, L., Galou, L., Kęska-Izworska, K., Walter, S. M., Andorfer, P., Kratochwill, K., Perco, P., & Ley, M. (2026) · bioRxiv · doi:10.64898/2026.01.09.697335
Improving Cloze Distractor Generation Through Retrieval-based Example Selection
Dong, J., Bai, J., Zhang, J., Li, C., Lin, X., & Rong, W. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00193
ConvHistQA: exploring comprehensive history selection in conversational question answering
Khan, I., Kumar, A., & Khare, B. K. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02723-3
Accelerating Personalization Signal Learning via Synthetic Data
Parveen, D., Kang, D., Paruchuri, A., Kayal, D., & Mallapragada, P. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_17
Metadata
- Authors
- Durmus, E., He, H., & Diab, M.
- Affiliations
- Cornell University (US)
- George Washington University (US)
- ORCID
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 5055-5070
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3022685184
- Cited by
- 306
- References
- 1
- Citations by year
- 2026: 20 · 2025: 28 · 2024: 28 · 2023: 75 · 2022: 68 · 2021: 71 · 2020: 16
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Amazon Web Services
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, lexical overlap, embedding similarity, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
