Measuring Attribution in Natural Language Generation Models
Open accessAbstract
With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world. In this work, we present a new evaluation framework entitled Attributable to Identified Sources (AIS) for assessing the output of natural language generation models, when such output pertains to the external world. We first define AIS and introduce a two-stage annotation pipeline for allowing annotators to appropriately evaluate model output according to AIS guidelines. We empirically validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset) via human evaluation studies that suggest that AIS could serve as a common framework for measuring whether model-generated statements are supported by underlying sources. We release guidelines for the human evaluation studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work in model-based evaluation mainly reports aggregated scores because the goal is comparison between different systems.
“Prior work in model-based evaluation mainly reports aggregated scores since the goal is a comparison between different systems being evaluated (Zhang et al., 2020; Rashkin et al., 2021; Gao et al., 2022)”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Rashkin et al., 2021
Section 5.3 (p. 17) reports aggregate AIS scores for several systems, and Section 5.5.4 (p. 21) describes using AIS to rank generative models. But the paper describes a human evaluation study, not model-based evaluation, and does not establish that prior work mainly reports aggregate scores because its goal is system comparison. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe Attributable to Identified Sources (AIS) benchmark assesses whether the source documents support the output of text generation models.
“the Attributable to Identified Sources (AIS) benchmark (Rashkin et al., 2021) assesses whether the source documents support the output of text generation models.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Rashkin et al., 2021
Prior work in model-based evaluation mainly reports aggregated scores because the goal is comparison between different systems being evaluated.
“Prior work in model-based evaluation mainly reports aggregated scores since the goal is a comparison between different systems being evaluated (Zhang et al., 2020; Rashkin et al., 2021; Gao et al., 2022) while we report both to see the relationship between two types of metrics.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Rashkin et al., 2021
Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution.
“Previously, Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution.”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Rashkin et al. (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 37
- Adiwardana, Daniel, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 . · arXiv
- Anantha, Raviteja, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of NAACL .
- Belz, Anja, Simon Mille, and David M Howcroft. 2020. Disentangling the properties of human evaluation methods: Aclassification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation , pages 183-194.
- Carston, Robyn. 1988. Implicature, explicature, and truth-theoretic semantics. In Ruth Kempson, editor, Mental Representations: The Interface Between Language and Reality . Cambridge University Press, pages 155-181.
- Choi, Eunsol, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of EMNLP .
- Choi, Eunsol, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making Sentences Stand-Alone. Transactions of the Association for Computational Linguistics , 9:447-461.
- Dalton, Jeffrey, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. Cast-19: A dataset for conversational information seeking. Proceedings of SIGIR . Dinan, Emily, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of ICLR .
- Durmus, Esin, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of ACL .
- Dziri, Nouha, Hannah Rashkin, Tal Linzen, and David Reitter. 2021. Evaluating groundedness in dialogue systems: The BEGIN benchmark. ArXiv , abs/2105.00071. · arXiv
- Gehrmann, Sebastian, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondˇ rej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics .
- Gopnik, Alison and Henry M. Wellman. 1992. Why the child's theory of mind really is a theory. Mind & Language , 7(1-2):145-171.
- Grice, Herbert P. 1975. Logic and conversation. In Speech acts . Brill, pages 41-58.
- ArXiv , abs/2110.08222. Harrington, Leo A, Michael D Morley, AŠcedrov, and Stephen G Simpson. 1985. Harvey Friedman's research on the foundations of mathematics . Elsevier. · arXiv
- Honovich, Or, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of EMNLP .
- Howcroft, David M, Anja Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation , pages 169-182.
- Kwiatkowski, Tom, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:452-466.
- Ladhak, Faisal, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1410-1421, Association for Computational Linguistics, Dublin, Ireland.
- Maynez, Joshua, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of ACL .
- Mehri, Shikib and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of ACL , Online.
- Nallapati, Ramesh, Bowen Zhou, Cicero dos Santos, Ça˘ glar G ˙ ulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of CoNLL .
- McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In Proceedings of ACL .
- Nie, Yixin, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of AAAI .
- Parikh, Ankur, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of EMNLP .
- Pavlick, Ellie and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics , 7:677-694.
- Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:140:1-140:67.
- Rashkin, Hannah, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of ACL .
- Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 . · arXiv
- Santhanam, Sashank, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation. arXiv preprint arXiv:2110.05456 . · arXiv
- See, Abigail, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1073-1083, Association for Computational Linguistics, Vancouver, Canada.
- Thorne, James, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. CoRR , abs/2104.00640. · arXiv
- Thorne, James and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of COLING .
- Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of NAACL .
- Wang, Alex, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of ACL .
- Welleck, Sean, Jason Weston, Arthur D. Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of ACL .
- Wiseman, Sam, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of EMNLP .
- Zaheer, Manzil, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems , volume 33, pages 17283-17297, Curran Associates, Inc.
- Zhong, Ming, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6197-6208, Association for Computational Linguistics, Online.
Cited by
Sorted by citations
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
Evaluating Open-Domain Question Answering in the Era of Large Language Models
Kamalloo, E., Dziri, N., Clarke, C. L. A., & Rafiei, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.307 · Cited by 144 · see citing works
Lyu, Y., li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., & Chen, E. (2024) · ACM Transactions on Information Systems · doi:10.1145/3701228 · Cited by 144 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143 · see citing works
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141 · see citing works
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 75 · see citing works
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74 · see citing works
Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Koopman, B., & Zuccon, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.928 · Cited by 73 · see citing works
“In addition, we did not ask to attribute such claims to sources (Bohnet et al., 2022; Menick et al., 2022; Rashkin et al., 2021): correct attribution appears to be a critical requirement to improve the quality, interpretability and trustworthiness of these methods.”
BackgroundAn automated framework for assessing how well LLMs cite relevant medical references
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025) · Nature Communications · doi:10.1038/s41467-025-58551-6 · Cited by 72 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00506 · Cited by 67 · see citing works
Self-citationRetrieving Supporting Evidence for Generative Question Answering
Huo, S., Arabzadeh, N., & Clarke, C. L. A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625336 · Cited by 65 · see citing works
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 62 · see citing works
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60 · see citing works
Large language models present new questions for decision support
Handler, A., Larsen, K. R. T., & Hackathorn, R. D. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102811 · Cited by 60 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56 · see citing works
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591687 · Cited by 51 · see citing works
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., & Abend, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.559 · Cited by 41 · see citing works
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Roit, P., Ferret, J., Shani, L., Aharoni, R., Cideron, G., Dadashi, R., Geist, M., Girgin, S., Hussenot, L., Keller, O., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.344 · Cited by 39 · see citing works
Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution
Li, X., Cao, Y., Pan, L., Ma, Y., & Sun, A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.28 · Cited by 39 · see citing works
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36 · see citing works
Multilingual Summarization with Factual Consistency Evaluation
Aharoni, R., Narayan, S., Maynez, J., Herzig, J., Clark, E., & Lapata, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.220 · Cited by 36 · see citing works
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data
Weller, O., Marone, M., Weir, N., Lawrie, D., Khashabi, D., & Durme, B. V. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.140 · Cited by 34 · see citing works
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33 · see citing works
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R. R., Parvez, M. R., & Feng, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.59 · Cited by 32 · see citing works
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29 · see citing works
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28 · see citing works
Diving Deep into Modes of Fact Hallucinations in Dialogue Systems
Das, S., Saha, S., & Srihari, R. K. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.48 · Cited by 27 · see citing works
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 26 · see citing works
Correctness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21 · see citing works
Chiesurin, S., Dimakopoulos, D., Cabezudo, M. A. S., Eshghi, A., Papaioannou, I. V., Rieser, V., & Konstas, I. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.60 · Cited by 21 · see citing works
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Malaviya, C., Shaw, P., Chang, M.-W., Lee, K., & Toutanova, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.784 · Cited by 20 · see citing works
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 20 · see citing works
Query Refinement Prompts for Closed-Book Long-Form QA
Amplayo, R. K., Webster, K., Collins, M., Das, D., & Narayan, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.444 · Cited by 16 · see citing works
Self-citationFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 14 · see citing works
Attributable and Scalable Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.473 · Cited by 14 · see citing works
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13 · see citing works
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
Lu, X., Brahman, F., West, P., Jung, J., Chandu, K. R., Ravichander, A., Ammanabrolu, P., Jiang, L., Ramnath, S., Dziri, N., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.424 · Cited by 12 · see citing works
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Clark, E., Rijhwani, S., Gehrmann, S., Maynez, J., Aharoni, R., Nikolaev, V. A., Sellam, T., Siddhant, A., Das, D., & Parikh, A. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.584 · Cited by 12 · see citing works
Self-citationAttributionBench: How Hard is Automatic Attribution Evaluation?
Li, Y., Yue, X., Liao, Z., & Sun, H. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.886 · Cited by 11 · see citing works
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 10 · see citing works
Evaluating Synthetic Data Generation from User Generated Text
Chim, J., Ive, J., & Liakata, M. (2024) · Computational Linguistics · doi:10.1162/coli_a_00540 · Cited by 10 · see citing works
Chen, Z., Qiu, Y., Yang, L., Liao, B., & Cao, D. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.104795 · Cited by 10 · see citing works
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 10 · see citing works
Guo, S., Zhong, Y., Feng, Z., Kang, S., & Chen, J. (2025) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2025.12.005 · Cited by 8 · see citing works
GINGER: Grounded Information Nugget-Based Generation of Responses
Łajewska, W., & Balog, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730166 · Cited by 7 · see citing works
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 7 · see citing works
Evaluating and Modeling Attribution for Cross-Lingual Question Answering
Müller, B., Wieting, J., Clark, J. H., Kwiatkowski, T., Ruder, S., Soares, L. B., Aharoni, R., Herzig, J., & Wang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.10 · Cited by 7 · see citing works
TaTA: A Multilingual Table-to-Text Dataset for African Languages
Gehrmann, S., Ruder, S., Nikolaev, V. A., Botha, J., Chavinda, M., Parikh, A. P., & Rivera, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.118 · Cited by 7 · see citing works
Self-citationEnhancing Retrieval-Augmented Generation via Evidence Tree Search
Sun, H., Cai, H., Li, Y., Fan, X., Wei, X., Wang, S., Zhang, Y., & Yin, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1175 · Cited by 6 · see citing works
Hierarchical Indexing for Retrieval-Augmented Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00703 · Cited by 6 · see citing works
ir_explain: A Python Library of Explainable IR Methods
Saha, S., Agarwal, H., Venktesh, V., Anand, A., Mohanty, S., Majumdar, D., & Mitra, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730343 · Cited by 6 · see citing works
Lu, X., Mahesh, A., Shen, Z., Dudley, M., Sano, L., & Wang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_28 · Cited by 6 · see citing works
1-PAGER: One Pass Answer Generation and Evidence Retrieval
Jain, P., Soares, L. B., & Kwiatkowski, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.967 · Cited by 5 · see citing works
Nandwani, Y., Kumar, V., Raghu, D., Joshi, S., & Lastras, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.639 · Cited by 5 · see citing works
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 4 · see citing works
Quantifying Factual Divergence in Generative Models: SHAP-LIME Based Hallucination Score for LLMs
Haq, I., Saqib, M., Zhang, Y., & Khan, I. A. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02150-4 · Cited by 4 · see citing works
Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models
Foley, M., Rawat, A., Lee, T., Hou, Y., Picco, G., & Zizzo, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.410 · Cited by 4 · see citing works
From RAG to Riches: Retrieval Interlaced with Sequence Generation
Jain, P., Soares, L. B., & Kwiatkowski, T. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.502 · Cited by 4 · see citing works
Enhancing LLM’s Reliability by Iterative Verification Attributions with Keyword Fronting
Sui, Y., Ren, J., Tan, H., Chen, H., Li, Z., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70365-2_15 · Cited by 3 · see citing works
On the Capacity of Citation Generation by Large Language Models
Qian, H., Fan, Y., Zhang, R., & Guo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_9 · Cited by 3 · see citing works
LXR: Learning to eXplain Recommendations
Gurevitch, L., Bogina, V., Barkan, O., Schein, Y., Elisha, Y., & Koenigstein, N. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3732292 · Cited by 3 · see citing works
Localizing Factual Inconsistencies in Attributable Text Generation
Cattan, A., Roit, P., Zhang, S., Wan, D., Aharoni, R., Szpektor, I., Bansal, M., & Dagan, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.598 · Cited by 3 · see citing works
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 2 · see citing works
Rocha, A., de Oliveira, P. C., Ferreira, J. P., Alves, M., & Sousa, A. J. (2026) · Applied Sciences · doi:10.3390/app16157745 · Cited by 1 · see citing works
ConSens: Assessing Context Grounding in Open-Book Question Answering
Vankov, I., Ivanov, M., Correia, A. D., & Botev, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_13 · Cited by 1 · see citing works
Does It Still Make Sense? Organizing and Summarizing Resources in Cross-Session Aggregated Search
Momeni, M., & Hoeber, O. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787873 · Cited by 1 · see citing works
Benchmarking international construction contract knowledge of large language models
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131754 · Cited by 1 · see citing works
Trust Me on This: A User Study of Trustworthiness for RAG Responses
Łajewska, W., & Balog, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_44 · Cited by 1 · see citing works
Zhang, H., & Wu, W. (2026) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2026.086343 · Cited by 1 · see citing works
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1 · see citing works
LingyuKong, & Seong, D. (2026) · npj Heritage Science · doi:10.1038/s40494-026-02926-5 · Cited by 1 · see citing works
Saifullah, Wahid, S. H., & Fazrian, M. I. (2026) · Jurnal Studi Ilmu-ilmu Al-Qur an dan Hadis · doi:10.14421/qh.v27i1.7499 · Cited by 1 · see citing works
PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems
Wilie, B., Xu, Y., Chung, W., Cahyawijaya, S., Lovenia, H., & Fung, P. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.63 · Cited by 1 · see citing works
Benchmarking International Construction Contract Knowledge of Large Language ModelsPreprint
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5247570 · Cited by 1 · see citing works
Learning to Say "I Don't Know": A Vision for Abstention in Large Language ModelsPreprint
Adhikari, T. (2025) · doi:10.36227/techrxiv.175682660.02761872/v1 · Cited by 1 · see citing works
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation
Daheim, N., Dziri, N., Sachan, M., Gurevych, I., & Ponti, E. M. (2023) · arXiv · doi:10.48550/arxiv.2303.17574
Estimating the Confidence of Language Model Generation using Training Data
Yoshikawa, H., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.91
Stylometric Features Embedding for the Task of Author Attribution
Al-Shboul, B. A., Abudayeh, R., & Qudah, D. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_33
DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling
Sun, H., Qiao, Z., Wang, B., Chen, G., Hou, Y., Jiang, Y., Xie, P., Huang, F., & Zhang, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.222
DocRouter: Path-Controllable Generation for Discharge Summaries via Document-Level Routing
Chen, X., Ge, C., & Shao, Q. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356182
CAN LARGE LANGUAGE MODELS ACT AS “CO-AUDITORS”?
Emekci, H. (2026) · Denetişim · doi:10.58348/denetisim.1782835
Shukla, S., & Joshi, H. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00050
吉川, 和., Yoshikawa, H., 岡崎, 直., & Okazaki, N. (2022) · Institutional Repositories DataBase (IRDB)
Yang, K., Liu, D., & Wang, Z. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_10
Query-Aware Adaptive Chunking for RAG in Mental Health and Well-Being Policy Question Answering
Kondapalli, P., Saimon, M. Z. R., Polok, S. D., Chakraborty, G. S., Abrar, M. A., & Batra, S. (2026) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm69057.2026.11465692
AgentCite: Trustworthy AI Agents for Information Verification Across Referenced Documents
Yigit, E., & Aygun, R. (2025) · Conference on AI x Software Engineering (AIxSE) · doi:10.1109/aixse64906.2025.00010
LLM‐Based Scientific Assistants for Knowledge Extraction: Which Design Choices Matter?
Exler, D., Raimann, M., Münker, M. F., Rosin, M., Gómez, J. E. U., Niemeyer, C. M., Reischl, M., & Rettenberger, L. (2026) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500197
Nguyen, T. N., Bui, D. B., Le, T., & Nguyen, H. T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1527-0_10
AgentCite: Task-Isolated Multi-Agent Collaboration for Cost-Effective Long-Context Citations
Cao, Z., Wang, H., Jin, L., Zhu, T., & Ren, L. (2026) · International Symposium on Autonomous Systems (ISAS) · doi:10.1109/icaisisas68969.2026.11567713
Ferreira, D. M., Santos, D. R., Afonso, L. C., Pacheco, O. M. D. R., & Almeida, J. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7051341
Faithful LLM-Assisted Question Answering for Technical and Academic Search
Filgueiras, J. L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808357
Humans, LLMs, and Measures Do Not Align in Attributed Information Retrieval
Gienapp, L., Lang, J., Potthast, M., & Scells, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809867
Deletion Isn't Enough: Auditing RAG for Selective Forgetting
Tavakoli, L., & Sanderson, M. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808545
- Authors
- Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.12870
- OpenAlex
- W4226157795
- Cited by
- 34
- References
- 0
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 25 · 2022: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation, attribution, source attribution, human evaluation, annotation pipeline, conversational question answering, summarization, table-to-text generation, factual grounding, external knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
