The DOI matches its DataCite record.
Measuring Attribution in Natural Language Generation Models
Open accessAbstract
With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world. In this work, we present a new evaluation framework entitled Attributable to Identified Sources (AIS) for assessing the output of natural language generation models, when such output pertains to the external world. We first define AIS and introduce a two-stage annotation pipeline for allowing annotators to appropriately evaluate model output according to AIS guidelines. We empirically validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset) via human evaluation studies that suggest that AIS could serve as a common framework for measuring whether model-generated statements are supported by underlying sources. We release guidelines for the human evaluation studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work in model-based evaluation mainly reports aggregated scores since the goal is a comparison between different systems being evaluated (Zhang et al., 2020; Rashkin et al., 2021; Gao et al., 2022)”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Rashkin et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the Attributable to Identified Sources (AIS) benchmark (Rashkin et al., 2021) assesses whether the source documents support the output of text generation models.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Rashkin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Prior work in model-based evaluation mainly reports aggregated scores since the goal is a comparison between different systems being evaluated (Zhang et al., 2020; Rashkin et al., 2021; Gao et al., 2022) while we report both to see the relationship between two types of metrics.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Rashkin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Previously, Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution.”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Rashkin et al. (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Research works printed but not linked: 37
- Adiwardana, Daniel, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 . · arXiv
- Anantha, Raviteja, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of NAACL .
- Belz, Anja, Simon Mille, and David M Howcroft. 2020. Disentangling the properties of human evaluation methods: Aclassification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation , pages 183-194.
- Carston, Robyn. 1988. Implicature, explicature, and truth-theoretic semantics. In Ruth Kempson, editor, Mental Representations: The Interface Between Language and Reality . Cambridge University Press, pages 155-181.
- Choi, Eunsol, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of EMNLP .
- Choi, Eunsol, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making Sentences Stand-Alone. Transactions of the Association for Computational Linguistics , 9:447-461.
- Dalton, Jeffrey, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. Cast-19: A dataset for conversational information seeking. Proceedings of SIGIR . Dinan, Emily, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of ICLR .
- Durmus, Esin, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of ACL .
- Dziri, Nouha, Hannah Rashkin, Tal Linzen, and David Reitter. 2021. Evaluating groundedness in dialogue systems: The BEGIN benchmark. ArXiv , abs/2105.00071. · arXiv
- Gehrmann, Sebastian, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondˇ rej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics .
- Gopnik, Alison and Henry M. Wellman. 1992. Why the child's theory of mind really is a theory. Mind & Language , 7(1-2):145-171.
- Grice, Herbert P. 1975. Logic and conversation. In Speech acts . Brill, pages 41-58.
- ArXiv , abs/2110.08222. Harrington, Leo A, Michael D Morley, AŠcedrov, and Stephen G Simpson. 1985. Harvey Friedman's research on the foundations of mathematics . Elsevier. · arXiv
- Honovich, Or, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of EMNLP .
- Howcroft, David M, Anja Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation , pages 169-182.
- Kwiatkowski, Tom, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:452-466.
- Ladhak, Faisal, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1410-1421, Association for Computational Linguistics, Dublin, Ireland.
- Maynez, Joshua, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of ACL .
- Mehri, Shikib and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of ACL , Online.
- Nallapati, Ramesh, Bowen Zhou, Cicero dos Santos, Ça˘ glar G ˙ ulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of CoNLL .
- McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In Proceedings of ACL .
- Nie, Yixin, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of AAAI .
- Parikh, Ankur, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of EMNLP .
- Pavlick, Ellie and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics , 7:677-694.
- Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:140:1-140:67.
- Rashkin, Hannah, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of ACL .
- Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 . · arXiv
- Santhanam, Sashank, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation. arXiv preprint arXiv:2110.05456 . · arXiv
- See, Abigail, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1073-1083, Association for Computational Linguistics, Vancouver, Canada.
- Thorne, James, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. CoRR , abs/2104.00640. · arXiv
- Thorne, James and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of COLING .
- Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of NAACL .
- Wang, Alex, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of ACL .
- Welleck, Sean, Jason Weston, Arthur D. Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of ACL .
- Wiseman, Sam, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of EMNLP .
- Zaheer, Manzil, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems , volume 33, pages 17283-17297, Curran Associates, Inc.
- Zhong, Ming, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6197-6208, Association for Computational Linguistics, Online.
Cited by
Sorted by citations
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216
Evaluating Open-Domain Question Answering in the Era of Large Language Models
Kamalloo, E., Dziri, N., Clarke, C. L. A., & Rafiei, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.307 · Cited by 232
Lyu, Y., li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., & Chen, E. (2024) · ACM Transactions on Information Systems · doi:10.1145/3701228 · Cited by 199
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 113
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74
Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Koopman, B., & Zuccon, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.928 · Cited by 73
“In addition, we did not ask to attribute such claims to sources (Bohnet et al., 2022; Menick et al., 2022; Rashkin et al., 2021): correct attribution appears to be a critical requirement to improve the quality, interpretability and trustworthiness of these methods.”
BackgroundAn automated framework for assessing how well LLMs cite relevant medical references
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025) · Nature Communications · doi:10.1038/s41467-025-58551-6 · Cited by 72
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 86
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00506 · Cited by 79
Self-citationRetrieving Supporting Evidence for Generative Question Answering
Huo, S., Arabzadeh, N., & Clarke, C. L. A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625336 · Cited by 75
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 112
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
Large language models present new questions for decision support
Handler, A., Larsen, K. R. T., & Hackathorn, R. D. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102811 · Cited by 75
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591687 · Cited by 51
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., & Abend, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.559 · Cited by 66
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Roit, P., Ferret, J., Shani, L., Aharoni, R., Cideron, G., Dadashi, R., Geist, M., Girgin, S., Hussenot, L., Keller, O., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.344 · Cited by 63
Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution
Li, X., Cao, Y., Pan, L., Ma, Y., & Sun, A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.28 · Cited by 39
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36
Multilingual Summarization with Factual Consistency Evaluation
Aharoni, R., Narayan, S., Maynez, J., Herzig, J., Clark, E., & Lapata, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.220 · Cited by 47
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data
Weller, O., Marone, M., Weir, N., Lawrie, D., Khashabi, D., & Durme, B. V. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.140 · Cited by 52
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R. R., Parvez, M. R., & Feng, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.59 · Cited by 55
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28
Diving Deep into Modes of Fact Hallucinations in Dialogue Systems
Das, S., Saha, S., & Srihari, R. K. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.48 · Cited by 34
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 33
Correctness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21
Chiesurin, S., Dimakopoulos, D., Cabezudo, M. A. S., Eshghi, A., Papaioannou, I. V., Rieser, V., & Konstas, I. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.60 · Cited by 34
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Malaviya, C., Shaw, P., Chang, M.-W., Lee, K., & Toutanova, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.784 · Cited by 36
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 32
Query Refinement Prompts for Closed-Book Long-Form QA
Amplayo, R. K., Webster, K., Collins, M., Das, D., & Narayan, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.444 · Cited by 23
Self-citationFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 14
Attributable and Scalable Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.473 · Cited by 16
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
Lu, X., Brahman, F., West, P., Jung, J., Chandu, K. R., Ravichander, A., Ammanabrolu, P., Jiang, L., Ramnath, S., Dziri, N., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.424 · Cited by 30
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Clark, E., Rijhwani, S., Gehrmann, S., Maynez, J., Aharoni, R., Nikolaev, V. A., Sellam, T., Siddhant, A., Das, D., & Parikh, A. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.584 · Cited by 25
Self-citationAttributionBench: How Hard is Automatic Attribution Evaluation?
Li, Y., Yue, X., Liao, Z., & Sun, H. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.886 · Cited by 11
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 11
Evaluating Synthetic Data Generation from User Generated Text
Chim, J., Ive, J., & Liakata, M. (2024) · Computational Linguistics · doi:10.1162/coli_a_00540 · Cited by 12
Chen, Z., Qiu, Y., Yang, L., Liao, B., & Cao, D. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.104795 · Cited by 10
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 17
Guo, S., Zhong, Y., Feng, Z., Kang, S., & Chen, J. (2025) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2025.12.005 · Cited by 9
GINGER: Grounded Information Nugget-Based Generation of Responses
Łajewska, W., & Balog, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730166 · Cited by 18
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 13
Evaluating and Modeling Attribution for Cross-Lingual Question Answering
Müller, B., Wieting, J., Clark, J. H., Kwiatkowski, T., Ruder, S., Soares, L. B., Aharoni, R., Herzig, J., & Wang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.10 · Cited by 9
TaTA: A Multilingual Table-to-Text Dataset for African Languages
Gehrmann, S., Ruder, S., Nikolaev, V. A., Botha, J., Chavinda, M., Parikh, A. P., & Rivera, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.118 · Cited by 10
Self-citationEnhancing Retrieval-Augmented Generation via Evidence Tree Search
Sun, H., Cai, H., Li, Y., Fan, X., Wei, X., Wang, S., Zhang, Y., & Yin, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1175 · Cited by 12
Hierarchical Indexing for Retrieval-Augmented Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00703 · Cited by 7
ir_explain: A Python Library of Explainable IR Methods
Saha, S., Agarwal, H., Venktesh, V., Anand, A., Mohanty, S., Majumdar, D., & Mitra, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730343 · Cited by 7
Lu, X., Mahesh, A., Shen, Z., Dudley, M., Sano, L., & Wang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_28 · Cited by 7
1-PAGER: One Pass Answer Generation and Evidence Retrieval
Jain, P., Soares, L. B., & Kwiatkowski, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.967 · Cited by 5
Nandwani, Y., Kumar, V., Raghu, D., Joshi, S., & Lastras, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.639 · Cited by 6
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 5
Quantifying Factual Divergence in Generative Models: SHAP-LIME Based Hallucination Score for LLMs
Haq, I., Saqib, M., Zhang, Y., & Khan, I. A. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02150-4 · Cited by 4
Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models
Foley, M., Rawat, A., Lee, T., Hou, Y., Picco, G., & Zizzo, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.410 · Cited by 8
From RAG to Riches: Retrieval Interlaced with Sequence Generation
Jain, P., Soares, L. B., & Kwiatkowski, T. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.502 · Cited by 6
Enhancing LLM’s Reliability by Iterative Verification Attributions with Keyword Fronting
Sui, Y., Ren, J., Tan, H., Chen, H., Li, Z., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70365-2_15 · Cited by 3
On the Capacity of Citation Generation by Large Language Models
Qian, H., Fan, Y., Zhang, R., & Guo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_9 · Cited by 6
LXR: Learning to eXplain Recommendations
Gurevitch, L., Bogina, V., Barkan, O., Schein, Y., Elisha, Y., & Koenigstein, N. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3732292 · Cited by 3
Localizing Factual Inconsistencies in Attributable Text Generation
Cattan, A., Roit, P., Zhang, S., Wan, D., Aharoni, R., Szpektor, I., Bansal, M., & Dagan, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.598 · Cited by 11
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 14
Rocha, A., de Oliveira, P. C., Ferreira, J. P., Alves, M., & Sousa, A. J. (2026) · Applied Sciences · doi:10.3390/app16157745 · Cited by 1
ConSens: Assessing Context Grounding in Open-Book Question Answering
Vankov, I., Ivanov, M., Correia, A. D., & Botev, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_13 · Cited by 1
Does It Still Make Sense? Organizing and Summarizing Resources in Cross-Session Aggregated Search
Momeni, M., & Hoeber, O. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787873 · Cited by 1
Benchmarking international construction contract knowledge of large language models
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131754 · Cited by 1
Trust Me on This: A User Study of Trustworthiness for RAG Responses
Łajewska, W., & Balog, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_44 · Cited by 1
Zhang, H., & Wu, W. (2026) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2026.086343 · Cited by 3
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1
LingyuKong, & Seong, D. (2026) · npj Heritage Science · doi:10.1038/s40494-026-02926-5 · Cited by 1
Saifullah, Wahid, S. H., & Fazrian, M. I. (2026) · Jurnal Studi Ilmu-ilmu Al-Qur an dan Hadis · doi:10.14421/qh.v27i1.7499 · Cited by 2
PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems
Wilie, B., Xu, Y., Chung, W., Cahyawijaya, S., Lovenia, H., & Fung, P. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.63 · Cited by 1
Benchmarking International Construction Contract Knowledge of Large Language ModelsPreprint
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5247570 · Cited by 1
Learning to Say "I Don't Know": A Vision for Abstention in Large Language ModelsPreprint
Adhikari, T. (2025) · doi:10.36227/techrxiv.175682660.02761872/v1 · Cited by 1
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation
Daheim, N., Dziri, N., Sachan, M., Gurevych, I., & Ponti, E. M. (2023) · arXiv · doi:10.48550/arxiv.2303.17574
Estimating the Confidence of Language Model Generation using Training Data
Yoshikawa, H., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.91
Stylometric Features Embedding for the Task of Author Attribution
Al-Shboul, B. A., Abudayeh, R., & Qudah, D. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_33
DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling
Sun, H., Qiao, Z., Wang, B., Chen, G., Hou, Y., Jiang, Y., Xie, P., Huang, F., & Zhang, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.222
DocRouter: Path-Controllable Generation for Discharge Summaries via Document-Level Routing
Chen, X., Ge, C., & Shao, Q. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356182
CAN LARGE LANGUAGE MODELS ACT AS “CO-AUDITORS”?
Emekci, H. (2026) · Denetişim · doi:10.58348/denetisim.1782835
Shukla, S., & Joshi, H. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00050
吉川, 和., Yoshikawa, H., 岡崎, 直., & Okazaki, N. (2022) · Institutional Repositories DataBase (IRDB)
Yang, K., Liu, D., & Wang, Z. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_10
Query-Aware Adaptive Chunking for RAG in Mental Health and Well-Being Policy Question Answering
Kondapalli, P., Saimon, M. Z. R., Polok, S. D., Chakraborty, G. S., Abrar, M. A., & Batra, S. (2026) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm69057.2026.11465692
AgentCite: Trustworthy AI Agents for Information Verification Across Referenced Documents
Yigit, E., & Aygun, R. (2025) · Conference on AI x Software Engineering (AIxSE) · doi:10.1109/aixse64906.2025.00010
LLM‐Based Scientific Assistants for Knowledge Extraction: Which Design Choices Matter?
Exler, D., Raimann, M., Münker, M. F., Rosin, M., Gómez, J. E. U., Niemeyer, C. M., Reischl, M., & Rettenberger, L. (2026) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500197
Nguyen, T. N., Bui, D. B., Le, T., & Nguyen, H. T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1527-0_10
AgentCite: Task-Isolated Multi-Agent Collaboration for Cost-Effective Long-Context Citations
Cao, Z., Wang, H., Jin, L., Zhu, T., & Ren, L. (2026) · International Symposium on Autonomous Systems (ISAS) · doi:10.1109/icaisisas68969.2026.11567713
Ferreira, D. M., Santos, D. R., Afonso, L. C., Pacheco, O. M. D. R., & Almeida, J. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7051341
Faithful LLM-Assisted Question Answering for Technical and Academic Search
Filgueiras, J. L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808357
Humans, LLMs, and Measures Do Not Align in Attributed Information Retrieval
Gienapp, L., Lang, J., Potthast, M., & Scells, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809867
Deletion Isn't Enough: Auditing RAG for Selective Forgetting
Tavakoli, L., & Sanderson, M. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808545
Metadata
- Authors
- Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.12870
- OpenAlex
- W4226157795
- Cited by
- 34
- References
- 0
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 25 · 2022: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation, attribution, source attribution, human evaluation, annotation pipeline, conversational question answering, summarization, table-to-text generation, factual grounding, external knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
