Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering

Phukan, Anirudh, Somasundaram, Shwetha, Saxena, Apoorv, Goswami, Koustava, Srinivasan, Balaji Vasan·2024·arXiv
DOI 10.48550/arxiv.2405.17980arXiv 2405.17980
Open this paper in CoChat — free account

Abstract

With the enhancement in the field of generative artificial intelligence (AI), contextual question answering has become extremely relevant. Attributing model generations to the input source document is essential to ensure trustworthiness and reliability. We observe that when large language models (LLMs) are used for contextual question answering, the output answer often consists of text copied verbatim from the input prompt which is linked together with "glue text" generated by the LLM. Motivated by this, we propose that LLMs have an inherent awareness from where the text was copied, likely captured in the hidden states of the LLM. We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of LLMs. Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. Our experimental results demonstrate that our method performs on par or better than GPT-4 at identifying verbatim copied segments in LLM generations and in attributing these segments to their source. Importantly, our method shows robust performance across various LLM architectures, highlighting its broad applicability. Additionally, we present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.17980

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overall picture of our proposed methodology; our method utilizes hidden layer representations from both the document and answer to determine the attribution of answer tokens. Initially, we identify extractive answer tokens (§ 4.2.1) through a cosine similarity matrix between document and answer tokens. Subsequently, we map these tokens to document token sequences by identifying anchor tokens and generating candidates, later ranked based on their cosine similarity to achieve attribution. (§ 4.2.2)
Figure 1 Overall picture of our proposed methodology; our method utilizes hidden layer representations from both the document and answer to determine the attribution of answer tokens. Initially, we identify extractive answer tokens (§ 4.2.1) through a cosine similarity matrix between document and answer tokens. Subsequently, we map these tokens to document token sequences by identifying anchor tokens and generating candidates, later ranked based on their cosine similarity to achieve attribution. (§ 4.2.2)CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 4 Comparison of model token F1 performance across layers of different models, for identifying output tokens extracted from the document on QuoteSum train set.
Figure 4 Comparison of model token F1 performance across layers of different models, for identifying output tokens extracted from the document on QuoteSum train set.CC-BY · From the PDF
Figure 5 Comparison of model accuracy across layers for attributing extractive spans on QuoteSum train set.
Figure 5 Comparison of model accuracy across layers for attributing extractive spans on QuoteSum train set.CC-BY · From the PDF
Figure 6 Limitations of GPT models in identifying directly copied spans from a source document
Figure 6 Limitations of GPT models in identifying directly copied spans from a source documentCC-BY · From the PDF
Figure 8 Performance of different models over span position on Sub-task 2.
Figure 8 Performance of different models over span position on Sub-task 2.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Figure 10 Sample datapoint from the paraphrased QuoteSum dataset
Figure 10 Sample datapoint from the paraphrased QuoteSum datasetCC-BY · From the PDF
Figure 12 Given a document and a question, the user selects "Generate response" to get the answer. Once the answer is generated, to identify extractive answer tokens, the user selects the "Generate Attributions" button. Answer tokens copied from the document are highlighted and are clickable. When the user selects "auto super.." in the generated response, the corresponding attribution is shown on the left panel.
Figure 12 Given a document and a question, the user selects "Generate response" to get the answer. Once the answer is generated, to identify extractive answer tokens, the user selects the "Generate Attributions" button. Answer tokens copied from the document are highlighted and are clickable. When the user selects "auto super.." in the generated response, the corresponding attribution is shown on the left panel.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Evaluating Verifiability in Generative Search Engines

    Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143 · cited 1×

    “While LLMs excel in generating informative responses, they often fall short in providing explicit references or attributions to the specific sections or sources within the context from which their answers derive (Liu et al., 2023b).”
    Background

Research works printed but not linked: 35

  1. Marco Ancona, Enea Ceolini, A. Cengiz Öztireli, and Markus H. Gross. 2017. A unified view of gradient- based attribution methods for deep neural networks. CoRR, abs/1711.06104. · arXiv
  2. Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037. · arXiv
  3. Brooke Borel. 2023. The Chicago guide to fact- checking. University of Chicago Press.
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. arXiv preprint arXiv:1704.00051. · arXiv
  5. Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue sys- tems: The BEGIN benchmark. Transactions of the Association for Computational Linguistics, 10:1066- 1083. · DOI
  6. Owain Evans, Owen Cotton-Barratt, Lukas Finnve- den, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: developing and governing AI that does not lie. CoRR, abs/2110.06674. · arXiv
  7. William Ferreira and Andreas Vlachos. 2016. Emer- gent: a novel data-set for stance classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1163-1168. The Association for Computational Lin- guistics. · DOI · Paper page
  8. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023a. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 16477-16508.
  9. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627. · arXiv
  10. Andreas Holzinger, Bernd Malle, Anna Saranti, and Bastian Pfeifer. 2021. Towards multi-modal causabil- ity with graph neural networks enabling information fusion for explainable AI. Inf. Fusion, 71:28-37. · DOI · Paper page
  11. Jie Huang and Kevin Chen-Chuan Chang. 2023. Ci- tation: A key to building responsible and ac- countable large language models. arXiv preprint arXiv:2307.02185. · arXiv
  12. Siqing Huo, Negar Arabzadeh, and Charles Clarke. 2023. Retrieving supporting evidence for generative question answering. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 11-20.
  13. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
  14. Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nan- dan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307.16883. · arXiv
  15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300. · arXiv
  16. Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. 2023. A survey of large language models attribution. ArXiv, abs/2311.03731. · arXiv
  17. Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2015. A survey on truth discovery. SIGKDD Explor., 17(2):1-16. · DOI · Paper page
  18. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. Lost in the middle: How lan- guage models use long contexts. arXiv preprint arXiv:2307.03172. · arXiv
  19. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2023. Ex- pertqa: Expert-curated questions and attributed an- swers. ArXiv, abs/2309.07852. · arXiv
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. · arXiv
  21. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv:2112.07899. · arXiv
  22. Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to- sequence model. arXiv preprint arXiv:2003.06713. · arXiv
  23. Fabio Petroni, Samuel Broscheit, Aleksandra Piktus, Patrick S. H. Lewis, Gautier Izacard, Lucas Hosseini, Jane Dwivedi-Yu, Maria Lomeli, Timo Schick, Pierre- Emmanuel Mazaré, Armand Joulin, Edouard Grave, and Sebastian Riedel. 2022. Improving wikipedia verifiability with AI. CoRR, abs/2207.06220. · DOI
  24. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural lan- guage generation models. Computational Linguistics, pages 1-64.
  25. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
  26. Gabriele Sarti, Grzegorz Chrupala, Malvina Nissim, and Arianna Bisazza. 2023a. Quantifying the plausibil- ity of context reliance in neural machine translation. CoRR, abs/2310.01188. · DOI
  27. Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Os- kar van der Wal. 2023b. Inseq: An interpretability toolkit for sequence generation models. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 421-435, Toronto, Canada. Association for Computational Linguistics. · DOI
  28. Tal Schuster, Adam D Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William W Cohen, and Donald Metzler. 2023. Semqa: Semi-extractive multi-source question answering. arXiv preprint arXiv:2311.04886. · arXiv
  29. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and verification (FEVER) shared task. CoRR, abs/1811.10971. · arXiv
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  31. Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. " according to..." prompting language mod- els improves quoting from pre-training data. arXiv preprint arXiv:2305.13252. · arXiv
  32. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652. · arXiv
  33. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. · arXiv
  34. Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Do Xuan Long, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. 2023. Retrieving multimodal information for augmented generation: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 4736- 4756. Association for Computational Linguistics.
  35. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top- down approach to ai transparency. arXiv preprint arXiv:2310.01405. · arXiv

Cited by

See all 1

Metadata

Authors
Phukan, A., Somasundaram, S., Saxena, A., Goswami, K., & Srinivasan, B. V.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2405.17980
arXiv
2405.17980
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.