The DOI matches its Crossref record, with a small difference in title or year; 6 of the 19 references it cites were found in a registry or an index, and 13 are not checked yet.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Open accessAbstract
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful.Existing automatic metrics do not capture such mistakes effectively.We tackle the problem of evaluating faithfulness of a generated summary given its source document.We first collected human annotations of faithfulness for outputs from numerous models on two datasets.We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful.Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, 1 which leverages recent advances in reading comprehension.Given questionanswer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary.Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.* Most of the work is done while the authors were at Amazon Web Services AI.1 Faithfulness Evaluation with Question Answering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A similar phenomenon has been shown in other NLG tasks, where it has been discovered that traditional metrics have a poor correlation with human judgment in terms of the hallucination problem [45, 54, 101, 125].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The results show that these reference-free metrics have substantially higher correlations with human judgments of faithfulness than the baseline metrics.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“The extracted answers that do not match the provided answers indicate unfaithful information in the summary.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to classifier-based metrics, QA-based metrics [77, 119, 271, 310] have recently garnered attention for their enhanced ability to capture information overlap between the model's generation and its source.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [77]
The cited source says what the claim attributes to it.
In the citing paper
“Their approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precisionoriented.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“The guidelines are based on the insights provided byDurmus et al. (2020) regarding annotating faithfulness.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. D. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219885 · Cited by 3,028
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,376
Research works printed but not linked: 32
- Z. Cao, F. Wei, W. Li, and S. Li. 2018. Faithful to the original: Fact aware neural abstractive summariza- tion. In Association for the Advancement of Artifi- cial Intelligence (AAAI).
- Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University.
- P. Chen, F. Wu, T. Wang, and W. Ding. 2018. A se- mantic QA-based approach for text summarization evaluation. In Association for the Advancement of Artificial Intelligence (AAAI).
- Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In ACL.
- Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922. · arXiv
- M. Eyal, T. Baumel, and M. Elhadad. 2019. Ques- tion answering as an automatic evaluation metric for news article summarization. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL).
- T. Falke, L. F. R. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging appli- cation for natural language inference. In Association for Computational Linguistics (ACL).
- M. Gardner, J. Berant, H. Hajishirzi, A. Talmor, and S. Min. 2019. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291. · arXiv
- B. Goodrich, V. Rao, P. J. Liu, and M. Saleh. 2019. Assessing the factual accuracy of generated text. In International Conference on Knowledge Discovery and Data Mining (KDD).
- M. Grusky, M. Naaman, , and Y. Artzi. 2018. News- room: A dataset of 1.3 million summaries with di- verse extractive strategies. In North American Asso- ciation for Computational Linguistics (NAACL).
- T. Hashimoto, H. Zhang, and P. Liang. 2019. Unify- ing human and statistical evaluation for natural lan- guage generation. In North American Association for Computational Linguistics (NAACL).
- Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In Proceedings of the 28th International Conference on Neural Infor- mation Processing Systems -Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
- A. Holtzman, J. Buys, M. Forbes, and Y. Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- W. Kryściński, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. 2019. Neural text summarization: A critical evaluation. In Empirical Methods in Natural Language Processing (EMNLP).
- L. Lebanoff, J. Muchovej, F. Dernoncourt, D. S. Kim, S. Kim, W. Chang, and F. Liu. 2019. Analyzing sentence fusion in abstractive summarization. arXiv preprint arXiv:1910.00203. · arXiv
- J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective func- tion for neural conversation models. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110-119.
- M. Li, S. Roller, I. Kulikov, S. Welleck, Y. Boureau, K. Cho, and J. Weston. 2019. Don't say that! mak- ing inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911.03860. · arXiv
- C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Char- lin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsuper- vised evaluation metrics for dialogue response gen- eration. In Empirical Methods in Natural Language Processing (EMNLP).
- Y. Liu and M. Lapata. 2019. Text summarization with pretrained encoders. In Empirical Methods in Natu- ral Language Processing (EMNLP).
- R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard- Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic turing test: Learning to evaluate dia- logue responses. In Association for Computational Linguistics (ACL).
- Mani, G. Klein, L. Hirschman, T. Firmin, D. House, and B. Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In European Associ- ation for Computational Linguistics (EACL).
- R. T. McCoy, E. Pavlick, and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. arXiv preprint arXiv:1902.01007. · arXiv
- Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
- T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano. 2019. Answers unite! unsupervised metrics for re- inforced summarization models. In Empirical Meth- ods in Natural Language Processing (EMNLP).
- A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguis- tics (NAACL).
- Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. 2017. Neu- ral machine translation with reconstruction. In Asso- ciation for the Advancement of Artificial Intelligence (AAAI).
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700.
- Curran Associates, Inc. S. Wiseman, S. M. Shieber, and A. M. Rush. 2017. Challenges in data-to-document generation. In Em- pirical Methods in Natural Language Processing (EMNLP).
- F. Zhang, J. Yao, and R. Yan1. 2018. On the ab- stractiveness of neural document summarization. In Empirical Methods in Natural Language Processing (EMNLP).
- T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019a. BERTSCORE: Evaluat- ing text generation with BERT. arXiv preprint arXiv:1904.09675. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587
Metadata
- Authors
- Durmus, E., He, H., & Diab, M.
- Affiliations
- Cornell University (US)
- George Washington University (US)
- ORCID
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 5055-5070
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3022685184
- Cited by
- 306
- References
- 1
- Citations by year
- 2026: 20 · 2025: 28 · 2024: 28 · 2023: 75 · 2022: 68 · 2021: 71 · 2020: 16
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Amazon Web Services
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, lexical overlap, embedding similarity, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
