The DOI matches its Crossref record, with a small difference in title or year; 6 of the 19 references it cites were found in a registry or an index, and 13 are not checked yet.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Open accessAbstract
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful.Existing automatic metrics do not capture such mistakes effectively.We tackle the problem of evaluating faithfulness of a generated summary given its source document.We first collected human annotations of faithfulness for outputs from numerous models on two datasets.We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful.Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, 1 which leverages recent advances in reading comprehension.Given questionanswer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary.Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.* Most of the work is done while the authors were at Amazon Web Services AI.1 Faithfulness Evaluation with Question Answering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A similar phenomenon has been shown in other NLG tasks, where it has been discovered that traditional metrics have a poor correlation with human judgment in terms of the hallucination problem [45, 54, 101, 125].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The results show that these reference-free metrics have substantially higher correlations with human judgments of faithfulness than the baseline metrics.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“The extracted answers that do not match the provided answers indicate unfaithful information in the summary.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to classifier-based metrics, QA-based metrics [77, 119, 271, 310] have recently garnered attention for their enhanced ability to capture information overlap between the model's generation and its source.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [77]
The cited source says what the claim attributes to it.
In the citing paper
“Their approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precisionoriented.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“The guidelines are based on the insights provided byDurmus et al. (2020) regarding annotating faithfulness.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Durmus et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. D. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219885 · Cited by 3,028
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,376
Flagged first, then by citations
Research works printed but not linked: 32
- Z. Cao, F. Wei, W. Li, and S. Li. 2018. Faithful to the original: Fact aware neural abstractive summariza- tion. In Association for the Advancement of Artifi- cial Intelligence (AAAI).
- Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University.
- P. Chen, F. Wu, T. Wang, and W. Ding. 2018. A se- mantic QA-based approach for text summarization evaluation. In Association for the Advancement of Artificial Intelligence (AAAI).
- Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In ACL.
- Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922. · arXiv
- M. Eyal, T. Baumel, and M. Elhadad. 2019. Ques- tion answering as an automatic evaluation metric for news article summarization. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL).
- T. Falke, L. F. R. Ribeiro, P. A. Utama, I. Dagan, and I. Gurevych. 2019a. Ranking generated summaries by correctness: An interesting but challenging appli- cation for natural language inference. In Association for Computational Linguistics (ACL).
- M. Gardner, J. Berant, H. Hajishirzi, A. Talmor, and S. Min. 2019. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291. · arXiv
- B. Goodrich, V. Rao, P. J. Liu, and M. Saleh. 2019. Assessing the factual accuracy of generated text. In International Conference on Knowledge Discovery and Data Mining (KDD).
- M. Grusky, M. Naaman, , and Y. Artzi. 2018. News- room: A dataset of 1.3 million summaries with di- verse extractive strategies. In North American Asso- ciation for Computational Linguistics (NAACL).
- T. Hashimoto, H. Zhang, and P. Liang. 2019. Unify- ing human and statistical evaluation for natural lan- guage generation. In North American Association for Computational Linguistics (NAACL).
- Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In Proceedings of the 28th International Conference on Neural Infor- mation Processing Systems -Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
- A. Holtzman, J. Buys, M. Forbes, and Y. Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- W. Kryściński, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. 2019. Neural text summarization: A critical evaluation. In Empirical Methods in Natural Language Processing (EMNLP).
- L. Lebanoff, J. Muchovej, F. Dernoncourt, D. S. Kim, S. Kim, W. Chang, and F. Liu. 2019. Analyzing sentence fusion in abstractive summarization. arXiv preprint arXiv:1910.00203. · arXiv
- J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective func- tion for neural conversation models. In Human Lan- guage Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110-119.
- M. Li, S. Roller, I. Kulikov, S. Welleck, Y. Boureau, K. Cho, and J. Weston. 2019. Don't say that! mak- ing inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:1911.03860. · arXiv
- C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Char- lin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsuper- vised evaluation metrics for dialogue response gen- eration. In Empirical Methods in Natural Language Processing (EMNLP).
- Y. Liu and M. Lapata. 2019. Text summarization with pretrained encoders. In Empirical Methods in Natu- ral Language Processing (EMNLP).
- R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard- Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic turing test: Learning to evaluate dia- logue responses. In Association for Computational Linguistics (ACL).
- Mani, G. Klein, L. Hirschman, T. Firmin, D. House, and B. Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In European Associ- ation for Computational Linguistics (EACL).
- R. T. McCoy, E. Pavlick, and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. arXiv preprint arXiv:1902.01007. · arXiv
- Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
- Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
- T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano. 2019. Answers unite! unsupervised metrics for re- inforced summarization models. In Empirical Meth- ods in Natural Language Processing (EMNLP).
- A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguis- tics (NAACL).
- Z. Tu, Y. Liu, L. Shang, X. Liu, and H. Li. 2017. Neu- ral machine translation with reconstruction. In Asso- ciation for the Advancement of Artificial Intelligence (AAAI).
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700.
- Curran Associates, Inc. S. Wiseman, S. M. Shieber, and A. M. Rush. 2017. Challenges in data-to-document generation. In Em- pirical Methods in Natural Language Processing (EMNLP).
- F. Zhang, J. Yao, and R. Yan1. 2018. On the ab- stractiveness of neural document summarization. In Empirical Methods in Natural Language Processing (EMNLP).
- T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019a. BERTSCORE: Evaluat- ing text generation with BERT. arXiv preprint arXiv:1904.09675. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. · arXiv
Cited by
Sorted by citations
Items 101–200 of 318
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V. A., Lamm, M., Aroyo, L. M., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00490 · Cited by 16
Analyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.325 · Cited by 25
Dong, Y., Wieting, J., & Verga, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.76 · Cited by 16
Learning to Fuse Sentences with Transformers for Summarization
Lebanoff, L., Dernoncourt, F., Kim, D. S., Wang, L., Chang, W. H., & Liu, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.338 · Cited by 15
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
Self-citationCaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization
Choubey, P. K., Fabbri, A. R., Vig, J., Wu, C.-S., Liu, W., & Rajani, N. F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.685 · Cited by 26
On Hallucination and Predictive Uncertainty in Conditional Language GenerationPreprint
Xiao, Y., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15025 · Cited by 16
Semantic Similarity Based Evaluation for Abstractive News Summarization
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.3 · Cited by 18
Finding a Balanced Degree of Automation for Summary Evaluation
Zhang, S., & Bansal, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.531 · Cited by 24
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 24
A New Approach to Overgenerating and Scoring Abstractive Summaries
Song, K., Wang, B., Feng, Z., & Liu, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.110 · Cited by 14
Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 42
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Tang, X., Fabbri, A. R., Li, H., Mao, Z., Adams, G., Wang, B., Çelikyılmaz, A., Mehdad, Y., & Radev, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.417 · Cited by 21
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report GenerationPreprint
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2010.10042 · Cited by 264
Warm-Starting for Improving the Novelty of Abstractive Summarization
Alomari, A., Al‐Shamayleh, A. S., Idris, N. B., Sabri, A. Q. M., Alsmadi, I. M., & Omary, D. (2023) · IEEE Access · doi:10.1109/access.2023.3322226 · Cited by 17
Improving Factuality of Abstractive Summarization via Contrastive Reward Learning
Chern, I.-C., Wang, Z., Das, S. K., Sharma, B., Liu, P., & Neubig, G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.6 · Cited by 20
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.4 · Cited by 14
Truth or Error? Towards systematic analysis of factual errors in abstractive summaries
Lux, K.-M., Sappelli, M., & Larson, M. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.1 · Cited by 14
Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension
Agrawal, S., & Carpuat, M. J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00653 · Cited by 17
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
Jiang, Y., Çelikyılmaz, A., Smolensky, P., Soulos, P., Rao, S., Palangi, H., Fernandez, R., Smith, C., Bansal, M., & Gao, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.381 · Cited by 18
Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio
Liu, Y., Jia, Q. F., & Zhu, K. Q. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.153 · Cited by 19
(2021) · arXiv (Cornell University) · Cited by 13
MedKA: A knowledge graph-augmented approach to improve factuality in medical Large Language Models
Deng, Y., Zhao, S., Miao, Y., Zhu, J., & Li, J. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104871 · Cited by 18
Data Augmentation for Low-Resource Dialogue Summarization
Liu, Y., Maynez, J., Simões, G., & Narayan, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.53 · Cited by 13
Efficient Few-Shot Fine-Tuning for Opinion Summarization
Bražinskas, A., Nallapati, R. M., Bansal, M., & Dreyer, M. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.113 · Cited by 17
Multi-Document Summarization Using Selective Attention Span and Reinforcement Learning
Atri, Y. K., Goyal, V., & Chakraborty, T. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3316459 · Cited by 15
Enhancing Factual Consistency of Abstractive SummarizationPreprint
Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X., & Jiang, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08612 · Cited by 12
Automated Text Generation and Summarization for Academic Writing
Benites, F., Benites, A. D., & Anson, C. M. (2023) · Springer eBooks · doi:10.1007/978-3-031-36033-6_18 · Cited by 16
AgreeSum: Agreement-Oriented Multi-Document Summarization
Pang, R. Y., Lelkes, A. D., Tran, V., & Yu, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.299 · Cited by 13
Faithful Abstractive Summarization via Fact-aware Consistency-constrained Transformer
Lyu, Y., Zhu, C., Xu, T., Yin, Z., & Chen, E. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557319 · Cited by 11
Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistency
Wen, X., & Carenini, G. (2023) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2023.codi-1.9 · Cited by 14
Fact-Driven Abstractive Summarization by Utilizing Multi-Granular Multi-Relational Knowledge
Mao, Q., Li, J., Peng, H. M., He, S., Wang, L., Yu, P. S., & Wang, Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3161157 · Cited by 11
Efficient Attentions for Long Document SummarizationPreprint
Huang, L., Cao, S., Parulian, N. N., Ji, H., & Wang, L. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.02112 · Cited by 507
Abstractive summarization with deep reinforcement learning using semantic similarity rewards
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000505 · Cited by 13
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 22
From Chaos to Clarity: Claim Normalization to Empower Fact-Checking
Sundriyal, M., Chakraborty, T., & Nakov, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.439 · Cited by 24
Give the Truth: Incorporate Semantic Slot into Abstractive Dialogue Summarization
Zhao, L., Zeng, W., Xu, W., & Guo, J. H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.209 · Cited by 10
SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder
Lin, W., Li, S., Zhang, C., Ji, B., Yu, J., Ma, J., & Yi, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25198-6_6 · Cited by 11
Development of system for generating questions, answers, distractors using transformers
Barlybayev, A., & Matkarimov, B. T. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i2.pp1851-1863 · Cited by 17
Explainable Fact-Checking Through Question Answering
Yang, J., Vega‐Oliveros, D. A., Seibt, T., & Rocha, A. D. R. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747214 · Cited by 10
Learning with Rejection for Abstractive Text Summarization
Cao, M., Dong, Y., He, J., & Cheung, J. C. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.663 · Cited by 11
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Ravichander, A., Ghela, S., Wadden, D., & Choi, Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.71 · Cited by 13
SteLLA: A Structured Grading System Using LLMs with RAG
Qiu, H., White, B. F., Ding, A., da Costa, R. F., Hachem, A., Ding, W., & Chen, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825385 · Cited by 17
To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text
Wilber, M., Timkey, W., & van Schijndel, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.298 · Cited by 9
Fine-grained Factual Consistency Assessment for Abstractive Summarization Models
Zhang, S., Niu, J., & Wei, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.9 · Cited by 9
Improving abstractive summarization with energy-based re-ranking
Pernes, D., Mendes, A., & Martins, A. F. T. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.1 · Cited by 9
Analyzing the Abstractiveness-Factuality Tradeoff With Nonlinear Abstractiveness Constraints.Preprint
Dreyer, M., Liu, M., Feng, N., Atluri, S., & Ravi, S. (2021) · arXiv (Cornell University) · Cited by 9
Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization
Dreyer, M., Liu, M., Nan, F., Atluri, S., & Ravi, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.156 · Cited by 16
Xiong, J., Pan, L., Liu, Y., Zhu, L., Zhang, L., & Tan, S. (2025) · Applied Sciences · doi:10.3390/app15073850 · Cited by 12
Learning Cluster Patterns for Abstractive Summarization
Jo, S.-G., Park, S.-H., Kim, J.-J., & On, B.-W. (2023) · IEEE Access · doi:10.1109/access.2023.3346911 · Cited by 9
OntoFact: Unveiling Fantastic Fact-Skeleton of LLMs via Ontology-Driven Reinforcement Learning
Shang, Z., Ke, W., Xiu, N., Wang, P., Liu, J., Li, Y., Luo, Z., & Ji, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29859 · Cited by 10
The Effect of Pretraining on Extractive Summarization for Scientific Documents
Gupta, Y., Ammanamanchi, P. S., Bordia, S., Manoharan, A., Mittal, D., Pasunuru, R., Shrivastava, M., Singh, M., Bansal, M., & Jyothi, P. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.9 · Cited by 8
Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization
Chan, H. P., Zeng, Q., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.402 · Cited by 8
TODSum: Task-Oriented Dialogue Summarization with State TrackingPreprint
Zhao, L., Zheng, F., He, K., Zeng, W., Lei, Y., Jiang, H., Wu, W., Xu, W., Guo, J. Q., & Meng, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12680 · Cited by 19
Implementing AI Vertical use cases ‐ Scenario 1
Cagnoni, S., Emiliani, V., Lombardo, G., Alkema, W. B. L., Hooijmans, C. R., Alkema, S. G., Novotny, T., Hair, K., Nič, M., Macleod, M. R., et al. (2023) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2023.en-8223 · Cited by 8
Trends and Prospects of Retrieval-Augmented Generation (RAG) for Generative AI
Yoon, Y. C., & Kim, S. K. (2025) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2025.28.2.007 · Cited by 10
A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian
Oro, E., Granata, F. M., & Ruffolo, M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050141 · Cited by 12
Guide and interact: scene-graph based generation and control of video captions
Lu, X., & Gao, Y. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01012-7 · Cited by 7
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Jia, Q., Ren, S., Liu, Y., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.679 · Cited by 20
Exam Assessor Tool: An Automated System for Efficient Answer Sheet Evaluation
Singh, V. S., Verma, A., Srivastava, G., & Kumar, S. (2024) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2024/v17i6455 · Cited by 7
Benchmarking and Assessing Transformations Between Text and Causal Maps via Large Language Models
Giabbanelli, P. J., Gandee, T. J., Agrawal, A., & Hosseinichimeh, N. (2025) · Applied Ontology · doi:10.1177/15705838241304102 · Cited by 8
Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection
Vendeville, B., Магдановна, Е. Л., & Loor, P. D. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730304 · Cited by 7
Is Human Scoring the Best Criteria for Summary Evaluation?
Vasilyev, O., & Bohannon, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.192 · Cited by 8
Summary Explorer: Visualizing the State of the Art in Text Summarization
Syed, S., Yousef, T., Al‐Khatib, K., Jänicke, S., & Potthast, M. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.22 · Cited by 8
Leveraging speaker-aware structure and factual knowledge for faithful dialogue summarization
Zhao, L., Xu, W., Zhang, C., & Guo, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108550 · Cited by 7
Deutsch, D., & Roth, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.42 · Cited by 8
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Newman, B., Soldaini, L., Fok, R., Cohan, A., & Lo, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.193 · Cited by 14
Extracting Summary Knowledge Graphs from Long DocumentsPreprint
Wu, Z., Koncel-Kedziorski, R., Mari Ostendorf, E., & Hajishirzi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09162 · Cited by 11
RealEdit: Reddit Edits As a Large-scale Empirical Dataset for Image Transformations
Sushko, P., Bharadwaj, A., Lim, Z. Y., Ilin, V., Caffee, B., Chen, D., Salehi, M., Hsieh, C.-Y., & Krishna, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01251 · Cited by 11
A Question-Answering Approach to Evaluating Legal Summaries
Xu, H., & Ashley, K. D. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230977 · Cited by 6
Reducing extrinsic hallucination in multimodal abstractive summaries with post-processing technique
Rafi, S. M., Laitonjam, L., & Das, R. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10895-8 · Cited by 6
Language Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 10
NonFactS: NonFactual Summary Generation for Factuality Evaluation in Document Summarization
Soleimani, A., Monz, C., & Worring, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.400 · Cited by 7
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
Guo, Y., Clavel, C., Eddine, M. K., & Vazirgiannis, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.386 · Cited by 9
Siriwardhana, S., Gupta, C., Kaluarachchi, T., Dissanayake, V., Ellawela, S., & Nanayakkara, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2286090 · Cited by 6
Models See Hallucinations: Evaluating the Factuality in Video Captioning
Liu, H., & Wan, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.723 · Cited by 6
Fast and Accurate Factual Inconsistency Detection Over Long Documents
Lattimer, B. M., Chen, P. M., Zhang, X., & Yang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.105 · Cited by 15
Evaluating and Improving Factuality in Multimodal Abstractive Summarization
Wan, D., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.654 · Cited by 12
Inspecting the Factuality of Hallucinated Entities in Abstractive Summarization.Preprint
Cao, M., Dong, Y., & Cheung, J. C. K. (2021) · arXiv (Cornell University) · Cited by 6
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 32
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization
Zeng, Z., Chen, J., Xu, W., & Li, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.337 · Cited by 5
EchoGen: Generating Conclusions from Echocardiogram Notes
Tang, L., Kooragayalu, S., Wang, Y., Ding, Y., Durrett, G., Rousseau, J. F., & Peng, Y. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.35 · Cited by 5
Wang, X., & Zhao, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.180 · Cited by 5
Improving Faithfulness by Augmenting Negative Summaries from Fake Documents
Wang, T., Ladhak, F., Durmus, E., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.816 · Cited by 5
Self-citationGiabbanelli, P. J., Padilla, J. J., & Agrawal, A. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838754 · Cited by 6
Towards improving coherence and diversity of slogan generation
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000474 · Cited by 6
Towards Reference-free Text Simplification Evaluation with a BERT Siamese Network Architecture
Zhao, X., Durmus, E., & Yeung, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.838 · Cited by 5
Self-citationShortcomings of Question Answering Based Factuality Frameworks for Error Localization
Kamoi, R., Goyal, T., & Durrett, G. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.11 · Cited by 8
Coreference Reasoning in Machine Reading Comprehension
Wu, M., Moosavi, N. S., Roth, D., & Gurevych, I. (2021) · doi:10.18653/v1/2021.acl-long.448 · Cited by 7
Multi-Fact Correction in Abstractive Text SummarizationPreprint
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J. C. K., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02443 · Cited by 5
What Makes a Good Summary? Reconsidering the Focus of Automatic Summarization.Preprint
ter Hoeve, M., Kiseleva, J., & de Rijke, M. (2020) · arXiv (Cornell University) · Cited by 5
MoFE: Mixture of Factual Experts for Controlling Hallucinations in Abstractive Summarization.Preprint
Choubey, P. K., Vig, J., Liu, W., & Rajani, N. F. (2021) · arXiv (Cornell University) · Cited by 5
Li, Y., Li, L., Hu, D., Li, Y., Zhou, Y., Litvak, M., & Vanetik, N. (2023) · Research Square · doi:10.21203/rs.3.rs-3695932/v1 · Cited by 7
Unified Multi-Scenario Summarization Evaluation and Explanation
Shang, S., Yao, Z., Fu, H., Tao, C., Chen, X., Wang, F., Wang, Y., Ren, Z., & Gao, S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3509715 · Cited by 4
Deutsch, D., & Roth, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.296 · Cited by 5
A Gentle Introduction to Deep Nets and Opportunities for the Future
Church, K., Kordoni, V., Marcus, G., Davis, E., Ma, Y., & Chen, Z. (2022) · doi:10.18653/v1/2022.acl-tutorials.1 · Cited by 4
Gao, M., Wan, X., Su, J., Wang, Z., & Huai, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.779 · Cited by 5
QASem Parsing: Text-to-text Modeling of QA-based Semantics
Klein, A., Hirsch, E., Eliav, R., Pyatkin, V., Caciularu, A., & Dagan, I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.528 · Cited by 6
Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection
Maharaj, K., Saxena, A., Kumar, R., Mishra, A., & Bhattacharyya, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.764 · Cited by 6
Metadata
- Authors
- Durmus, E., He, H., & Diab, M.
- Affiliations
- Cornell University (US)
- George Washington University (US)
- ORCID
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 5055-5070
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3022685184
- Cited by
- 306
- References
- 1
- Citations by year
- 2026: 20 · 2025: 28 · 2024: 28 · 2023: 75 · 2022: 68 · 2021: 71 · 2020: 16
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Amazon Web Services
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, lexical overlap, embedding similarity, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
