Measuring Attribution in Natural Language Generation Models
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Rashkin et al. propose23 and formalize a framework called Attributable to Identifiable Sources, which defines the AIS score of a given language model’s response”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Citation-grade attribution requires locatable evidence: a reader must be able to retrieve and check the cited source (Rashkin et al., 2023;Bohnet et al., 2022).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Rashkin et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The AIS framework evaluates whether statements are supported by identified sources (Rashkin et al., 2023).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Rashkin et al., 2023
In the citing paper
“Compliance with stringent legal standards within applicable jurisdictions, such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA), is essential, necessitating purpose limitation and data minimization [230], [231], [232].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [230]
In the citing paper
“Rashkin et al. (2023) define the”
From Enabling Large Language Models to Generate Text with Citations (2023) · cites this paper as Rashkin et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“…that paper, we also compared with three models that are seq2seq-style conversation models: the original answer generation system from Dinan et al. (2019), the Dodecadialogue multitask system from Shuster et al. (2020) and a T5-base model (Raffel et al., 2020) finetuned on Wizard of Wikipedia data.”
BackgroundGet To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · cited 1×
“Lastly, we used Pointer-generator Networks from See et al. (2017) — a hybrid approach that is uses an abstractive seq2seq model but with an explicit copy mechanism that can extract information from the source document.”
MethodAbstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · cited 7×
“We observed that the reference summaries in CNN/DM may sometimes refer to external pieces of information that may have accompanied the article (a picture, a headline, etc.) or sometimes make assumptions about what the intended audience of the article might already know that can affect either the interpretability or AI…”
BackgroundMethodInfluentialNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Self-citationBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · cited 1×
“We also used an abstractive summarization system, BigBird (Zaheer et al., 2020).”
MethodFEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 1×
“A related field of study has dealt with the topic of fact or claim verification (Thorne et al., 2018; Thorne and Vlachos, 2018; Thorne et al., 2021, inter alia).”
BackgroundOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 4×
“Maynez et al. (2020) presented an extensive characterization of hallucinations, discussed behavior of models that generate content that are present in larger corpora beyond a given source and conducted a significant human study.”
BackgroundResultInfluential
Research works printed but not linked: 1
- Wilson, Deirdre and Dan Sperber. 2004. Relevance theory. In L.R. Horn and G. Ward, editors, The Handbook of Pragmatics. Blackwell, chapter 27, pages 607-632.
Cited by
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
“…generated by large language models are coherent or factually correct, as underscored by the recent work evaluating the factuality of language model generations and explanations (Maynez et al., 2020; Rashkin et al., 2021; Ye and Durrett, 2022; Marasovi ´ c et al., 2022; Wiegreffe et al., 2022).”
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2305.11747 · Cited by 340
“…the utterances generated by dialogue systems into three categories, i.e., fully attributable, not fully attributable, and generic; and the Attributable to Identified Sources (AIS) benchmark (Rashkin et al., 2021) assesses whether the source documents support the output of text generation models.”
BackgroundChain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Lyu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., & Chen, E. (2025) · ACM Transactions on Information Systems · doi:10.1145/3701228 · Cited by 99
“To address the shortcomings of previous benchmarks, we introduce CRUD-RAG, a comprehensive Chinese benchmark for RAG.”
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“• Natural Language Generation (NLG) : NLG emphasizes the training of models to generate coherent, contextually-relevant, and creative text responses, a critical component in chatbots, virtual assistants, and automated content creation tools [199], [36], [200], [201].”
BackgroundLM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · arXiv · doi:10.48550/arxiv.2305.13281 · Cited by 85
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · arXiv · doi:10.48550/arxiv.2204.04991 · Cited by 84
“Finally, we encourage data curators to use the binary annotation scheme, which is inline with the recommendations of Rashkin et al. (2021a).”
BackgroundMethodResultInfluentialEvaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74
“Specifically, we investigate the performance of current instruction-following models in comparison to models that have been specifically fine-tuned for those tasks.”
MethodEvaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2023) · arXiv · doi:10.48550/arxiv.2307.16877 · Cited by 74
An automated framework for assessing how well LLMs cite relevant medical references
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025) · Nature Communications · doi:10.1038/s41467-025-58551-6 · Cited by 72
“Rashkin et al. propose 23 and formalize a framework called Attri-butable to Identi fi able Sources , which de fi nes the AIS score of a given language model ’ s response, y , supported by evidence A as 1 if a human reviewer would agree that “ y is true, given A ” or 0 if not.”
Retrieving Supporting Evidence for Generative Question Answering
Huo, S., Arabzadeh, N., & Clarke, C. L. A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625336 · Cited by 106
“Attribution [5, 29, 40] focuses on connecting generated texts to supporting evidence to make them more trustworthy.”
BackgroundLM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57
“Other approaches include comparing the generated and knowledge texts using BLUERT [78], BERTScore [100], and other text similarity metrics.”
BackgroundMethodTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
“This is also in line with recent work on evaluating attribution in text generation (Rashkin et al., 2021a), where humans are required to judge whether a generated text is attributable to a grounding text.”
BackgroundMethodResultInfluentialEvaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2105.00071 · Cited by 47
Self-citationChain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42
“Other approaches include using tools for fact-checking (Chern et al., 2023a; Galitsky, 2023; Peng et al., 2023), or linking to external documents with attribution (Menick et al., 2022; Rashkin et al., 2023; Gao et al., 2023).”
BackgroundMethodLarge language models present new questions for decision support
Handler, A., Larsen, K. R. T., & Hackathorn, R. D. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102811 · Cited by 107
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R. R., Parvez, M. R., & Feng, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.59 · Cited by 98
“Rashkin et al. (2023) presents Attributable to Identified Sources or AIS as a framework for evaluating if the text generated by a model is supported by external sources (i.e., not hallucinated).”
BackgroundCorrectness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 17
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14
“Prior work in model-based evaluation mainly reports aggregated scores since the goal is a comparison between different systems being evaluated (Zhang et al., 2020; Rashkin et al., 2021; Gao et al., 2022) while we report both to see the relationship between two types of metrics.”
BackgroundInfluentialEvaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143
“To collect these binary judgments, we use the attributable to identified sources (AIS) evaluation framework of (Rashkin et al., 2022).”
BackgroundMethodInfluentialWang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
“Then, automated algorithms or human raters can examine whether the reference source really supports the generated answers using different criteria (Gao et al. 2023; Yue et al. 2023; Rashkin et al. 2023; Bohnet et al. 2023; Menick et al. 2022).”
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 45
Evaluating Synthetic Data Generation from User Generated Text
Chim, J., Ive, J., & Liakata, M. (2025) · Computational Linguistics · doi:10.1162/coli_a_00540 · Cited by 39
“…relative to grounding documents (Gabriel et al. 2021; Devaraj et al. 2022; Dziri et al. 2022) and external knowledge sources (Gupta et al. 2022; Rashkin et al. 2023), we consider original data to be grounding documents and assess synthetic data on how well they preserve their semantic and…”
Chen, Z., Qiu, Y., Yang, L., Liao, B., & Cao, D. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.104795 · Cited by 10
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 15
“ChatGPT 2 often generates replies that seem reasonable but are incorrect [63] and this phenomenon is also very common in other generation models [64, 65].”
GINGER: Grounded Information Nugget-Based Generation of Responses
Łajewska, W., & Balog, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730166 · Cited by 27
Guo, S., Zhong, Y., Feng, Z., Kang, S., & Chen, J. (2025) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2025.12.005 · Cited by 15
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 34
Hierarchical Indexing for Retrieval-Augmented Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00703 · Cited by 13
Lu, X., Mahesh, A., Shen, Z., Dudley, M., Sano, L., & Wang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_28 · Cited by 11
Enhancing LLM’s Reliability by Iterative Verification Attributions with Keyword Fronting
Sui, Y., Ren, J., Tan, H., Chen, H., Li, Z., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70365-2_15 · Cited by 5
Quantifying Factual Divergence in Generative Models: SHAP-LIME Based Hallucination Score for LLMs
Haq, I., Saqib, M., Zhang, Y., & Khan, I. A. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02150-4 · Cited by 4
“Rashkin et al. [21] extended attribution methods to evaluate factuality in text generation, while Yang et al. [22] proposed token-level preference-grounded guidance for fine-tuning language models to improve fac - tual consistency.”
ir_explain: A Python Library of Explainable IR Methods
Saha, S., Agarwal, H., Venktesh, V., Anand, A., Mohanty, S., Majumdar, D., & Mitra, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730343 · Cited by 8
“While several answer attribution approaches have been proposed [6, 32], they still rely on an external validator, such as NLI (Natural language Inference) models. hence, such approaches cannot guarantee the faithfulness of the attribution process.”
Attribute or Abstain: Large Language Models as Long Document Assistants
Buchmann, J., Liu, X., & Gurevych, I. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.463 · Cited by 3
On the Capacity of Citation Generation by Large Language Models
Qian, H., Fan, Y., Zhang, R., & Guo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_9 · Cited by 11
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 7
“Another research avenue focuses on assessing hallucinations across different NLP tasks (Dziri et al., 2022a, 2022b; Gupta et al., 2021 ; Li et al., 2023; Rashkin et al., 2023 ).”
LXR: Learning to eXplain Recommendations
Gurevitch, L., Bogina, V., Barkan, O., Schein, Y., Elisha, Y., & Koenigstein, N. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3732292 · Cited by 9
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 25
“Two common approaches to implementing v TE are human reasoning [7, 18] and automated systems capable of natural language inference (NLI) [21, 22].”
BackgroundMethodInfluentialSimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1
“…externally provided information; this includes grounding evaluations, which test factuality with respect to a given context (Jacovi et al., 2025; Rashkin et al., 2022), and retrieval-augmented benchmarks which evaluate the use of search tools to access fresh or real-time information beyond the…”
Self-citationFrom Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Ahn, J., & Kim, M. (2026) · arXiv · doi:10.48550/arxiv.2607.08028 · Cited by 1
ConSens: Assessing Context Grounding in Open-Book Question Answering
Vankov, I., Ivanov, M., Correia, A. D., & Botev, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_13 · Cited by 1
Benchmarking international construction contract knowledge of large language models
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131754 · Cited by 3
Trust Me on This: A User Study of Trustworthiness for RAG Responses
Łajewska, W., & Balog, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_44 · Cited by 1
Rocha, A., de Oliveira, P. C., Ferreira, J. P., Alves, M., & Sousa, A. J. (2026) · Applied Sciences · doi:10.3390/app16157745 · Cited by 1
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1
Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences
Russinovich, M., Kumar, R. S. S., & Salem, A. (2026) · arXiv · doi:10.48550/arxiv.2607.00738 · Cited by 4
“…line of works makes generations checkable through attribution and grounded citation, measuring whether a statement is supported by its source [25], evaluating verifiability in generative search engines [26], and training models to emit inline citations grounded in retrieved evidence [27],…”
Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Goo, T., Kim, J., Han, K., Jo, G., Kim, J.-S., & Kim, T.-H. (2026) · arXiv · doi:10.48550/arxiv.2607.20527
Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization
Guan, S. (2026) · arXiv · doi:10.48550/arxiv.2606.23989
“A growing line of work asks models to emit text together with supporting evidence (Rashkin et al., 2023; Bohnet et al., 2022).”
Estimating the Confidence of Language Model Generation using Training Data
Yoshikawa, H., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.91
Stylometric Features Embedding for the Task of Author Attribution
Al-Shboul, B. A., Abudayeh, R., & Qudah, D. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_33
Benchmarking International Construction Contract Knowledge of Large Language ModelsPreprint
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5775312
Localizing Factual Inconsistencies in Attributable Text Generation
Cattan, A., Roit, P., Zhang, S., Wan, D., Aharoni, R., Szpektor, I., Bansal, M., & Dagan, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.598 · Cited by 16
DocRouter: Path-Controllable Generation for Discharge Summaries via Document-Level Routing
Chen, X., Ge, C., & Shao, Q. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356182
CAN LARGE LANGUAGE MODELS ACT AS “CO-AUDITORS”?
Emekci, H. (2026) · Denetişim · doi:10.58348/denetisim.1782835
Does It Still Make Sense? Organizing and Summarizing Resources in Cross-Session Aggregated Search
Momeni, M., & Hoeber, O. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787873 · Cited by 2
Shukla, S., & Joshi, H. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00050 · Cited by 2
Query-Aware Adaptive Chunking for RAG in Mental Health and Well-Being Policy Question Answering
Kondapalli, P., Saimon, M. Z. R., Polok, S. D., Chakraborty, G. S., Abrar, M. A., & Batra, S. (2026) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm69057.2026.11465692 · Cited by 1
“They have located that numerous generation models generate statements with weak foundations on retrieved sources, even within a retrieval-based context, it is necessary to use attribution-aware evaluation of retrieval-enhanced generation systems [12].”
AgentCite: Trustworthy AI Agents for Information Verification Across Referenced Documents
Yigit, E., & Aygun, R. (2025) · Conference on AI x Software Engineering (AIxSE) · doi:10.1109/aixse64906.2025.00010
“Rashkin et al. [9] measured the attribution to identifiable sources.”
LLM‐Based Scientific Assistants for Knowledge Extraction: Which Design Choices Matter?
Exler, D., Raimann, M., Münker, M. F., Rosin, M., Gómez, J. E. U., Niemeyer, C. M., Reischl, M., & Rettenberger, L. (2026) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500197 · Cited by 1
“LLMs struggle with inaccuracy resulting from hallucinations [9, 10], inconsistent use of source material [56], or other unidentified issues [5, 6].”
Nguyen, T. N., Bui, D. B., Le, T., & Nguyen, H. T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1527-0_10
AgentCite: Task-Isolated Multi-Agent Collaboration for Cost-Effective Long-Context Citations
Cao, Z., Wang, H., Jin, L., Zhu, T., & Ren, L. (2026) · International Symposium on Autonomous Systems (ISAS) · doi:10.1109/icaisisas68969.2026.11567713
“To mitigate hallucinations and enhance verifiability, “At-tributed Text Generation” has emerged as a prominent research trajectory [12], [13].”
Ferreira, D. M., Santos, D. R., Afonso, L. C., Pacheco, O. M. D. R., & Almeida, J. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7051341
Faithful LLM-Assisted Question Answering for Technical and Academic Search
Filgueiras, J. L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808357
Humans, LLMs, and Measures Do Not Align in Attributed Information Retrieval
Gienapp, L., Lang, J., Potthast, M., & Scells, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809867
“Evaluating AIR systems requires assessing two complementary dimensions: informativeness , i.e., whether the response correctly addresses the query; and attributability , i.e., whether claims are supported by the cited documents [6].”
Deletion Isn't Enough: Auditing RAG for Selective Forgetting
Tavakoli, L., & Sanderson, M. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808545
Lin, Z., & Cai, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_48
IG-Guided Synonym Substitution Attacks on IndoBERT AI Detection Across Domains
Amelia, A. N. Z., Muliono, Y., & Isa, S. M. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620875
“Nonetheless, existing attacks primarily focus on English models and lack precise attribution guidance, leaving Indonesian-specific models like IndoBERT untested in realistic cross-domain scenarios [13].”
El‐Shafai, W., Azar, A. T., & Khalifa, M. A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133857
Pandilova, E., Petrov, M., Chorbev, I., Gjorgjevikj, D., & Trajanov, D. (2026) · Information · doi:10.3390/info17080790
“The four strands of that literature play distinct roles: Rashkin et al. [2] define the criterion itself, formalising attribution as AIS; Bohnet et al. [3] operationalise it into an evaluation protocol for attributed question answering; Gao et al. [4] use it as a revision target, editing generated…”
SwedTI: Evidence-Grounded Multi-source Web Threat Triage with Context-Aware Script Classification
Süren, E., Hacks, S., & Sigurðarson, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35586-7_7
Framework for Grounding Healthcare LLMs in a Causal Knowledge Graph: A Cardiovascular ExamplePreprint
Mumtaz, U., Noor, A., & Ahmed, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10720346/v1
Karimi, A., Westin, J., & Almevik, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36042-7_6
A Domain-Agnostic Agentic Architecture for Structured Extraction of Engineering Knowledge
Ikechukwu, O. C., Tarkian, M., Nambiar, S., Jonsson, M., & Brax, C. (2026) · Applied Artificial Intelligence · doi:10.1080/08839514.2026.2712892
Vantaku, C., & Avakian, A. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02326-y
From capability to assertability: epistemic regulation of LLM-mediated claim presentation
Wu, Q. (2026) · Ethics and Information Technology · doi:10.1007/s10676-026-09922-0
“Attribution research evaluates whether generated claims are supported by identified source material (Rashkin et al., 2023 ); document-grounded systems connect answering to source support or abstention (Buchmann et al., 2024); and evidence-sufficiency work shows that a system may answer correctly…”
Sethi, S., & Singh, S. (2026) · ACM Transactions on Social Computing · doi:10.1145/3849483
“Transparency mechanismsÐincluding uncertainty quantiication [22], provenance attribution [37], and explanation interfaces [25]Ðhave been proposed as mitigations, but ield evidence of their uptake by non-expert users remains limited.”
Reliable extraction of clinical-trial evidence with provenance, verification, and targeted reviewPreprint
Najpande, G., Titiya, P., Khan, M. A., Khakwani, K. Z. R., Sonbol, M. B., Riaz, I. B., & Gupta, V. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.10.01.756091
An Evidence-Grounded Multimodal AI Framework for Historical Digital Personae in Cultural Heritage
Lipianina-Honcharenko, K., & Yurkiv, K. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10100351
From Retrieval to Evidence: A Task-Adaptive Two-Layer Knowledge Architecture for LLM Agents
Volkov, A., Pustovoy, T., Istomin, D., & Otbetkina, T. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100322
Precision Evidence Bench: Assessing PICO alignment and faithful reporting in clinical AIPreprint
Mukerji, A., Sanghavi, N., Lauritzen, T., Muralidharan, J., & Gombar, S. (2026) · medRxiv · doi:10.64898/2026.10.07.26364976
Compact Wisdom at Small Scale: Small Language Models as Cultural Assistants
Manimegalai, R., M, S. S., Aravinda Krishnan, L., & Subramanian, V. (2026) · International Conference on Smart Systems for Electrical, Electronics, Communication and Computer Engineering (ICSSEECC) · doi:10.1109/icsseecc69062.2026.11722292
“During evaluation, citing a Kural not present in the evidence block is counted as an attribution error [29].”
Source-Linked Research Directions in ETDs: Preserving Evidence and Context
Aboelnaga, A. A. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3847125
Cho, H., Kim, H., Lee, C. H., Woon, P. S., Lee, C., & Lee, S. (2026) · Journal of Marine Science and Engineering · doi:10.3390/jmse14201904
Evidence-grounded multimodal knowledge graph for Dunhuang mural description generation
Xu, Z., & Zhao, C. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134815
Kanishkavardhan, P., Gowtham, S., & Maheswari, N. (2026) · Frontiers in Big Data · doi:10.3389/fdata.2026.1933830
Towards a Provenance-Aware Communication EcosystemPreprint
Protopopescu, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7654421
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
Godbole, A., & Jia, R. (2025) · arXiv · doi:10.48550/arxiv.2501.14883
“1 Details of our code and data are available at github.com/ameyagodbole/verify-with-caution In this work, we investigate automated metrics for evaluating “Attribution to Identified Sources” (AutoAIS; Rashkin et al., 2023), i.e., judging whether a source document fully supports a claim.”
CiteEval: Principle-Driven Citation Evaluation for Source Attribution
Xu, Y., Qi, P., Chen, J., Liu, K., Han, R., Liu, L., Min, B., Castelli, V., Gupta, A., & Wang, Z. (2025) · arXiv · doi:10.48550/arxiv.2506.01829
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3
“Our guidelines are mostly consistent with a recently proposed set of guidelines for checking attribution in text generation (Rashkin et al., 2021).”
Metadata
- Authors
- Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Google (United States) (US)
- Storia AI. iulia@iuliaturc.com
- ORCID
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 49
- Issue
- 4
- Pages
- 777-840
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4381432831
- Cited by
- 71
- References
- 46
- Citations by year
- 2026: 37 · 2025: 20 · 2024: 8 · 2023: 6
- FWCI
- 6.69
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attribution, natural language generation, evaluation framework, annotation guidelines, annotation pipeline, conversational question answering, summarization, table-to-text generation, model safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.