CoChat

ContextCite: Attributing Model Generation to Context

Cohen-Wang, Benjamin, Shah, Harshay, Georgiev, Kristian, Madry, Aleksander · 2024 · Advances in Neural Information Processing Systems 37
Cited by 27

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Abstract

How do language models use information provided as context when generating a response? Can we infer whether a particular generated statement is actually grounded in the context, a misinterpretation, or fabricated? To help answer these questions, we introduce the problem of context attribution: pinpointing the parts of the context (if any) that led a model to generate a particular statement. We then present ContextCite, a simple and scalable method for context attribution that can be applied on top of any existing language model. Finally, we showcase the utility of ContextCite through three applications: (1) helping verify generated statements (2) improving response quality by pruning the context and (3) detecting poisoning attacks. We provide code for ContextCite at https://github.com/MadryLab/context-cite.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/079017-3035

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 30 references

Research works printed but not linked: 39

  1. Samira Abnar and Willem Zuidema. "Quantifying attention flow in transformers". In: arXiv preprint arXiv:2005.00928 (2020). · arXiv
  2. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc.", 2009.
  3. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. "Tydi qa: A benchmark for information-seeking ques- tion answering in ty pologically di verse languages". In: Transactions of the Association for Computational Linguistics 8 (2020), pp. 454-470.
  4. Ting-Yun Chang and Robin Jia. "Data curation alone can stabilize in-context learning". In: arXiv preprint arXiv:2212.10378 (2022). · arXiv
  5. Ian Covert, Scott Lundberg, and Su-In Lee. "Explaining by removing: A unified framework for model explanation". In: Journal of Machine Learning Research 22.209 (2021), pp. 1-90.
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. "The llama 3 herd of models". In: arXiv preprint arXiv:2407.21783 (2024). · arXiv
  7. Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. "Visualizing and understanding neural machine translation". In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017, pp. 1150-1159.
  8. Joseph Enguehard. "Sequential Integrated Gradients: a simple but effective method for ex- plaining language models". In: arXiv preprint arXiv:2305.15853 (2023). · arXiv
  9. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. "Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection". In: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. 2023, pp. 79-90.
  10. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. "Enabling large language models to generate text with citations". In: arXiv preprint arXiv:2305.14627 (2023). · arXiv
  11. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. "A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions". In: arXiv preprint arXiv:2311.05232 (2023). · arXiv
  12. Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. "Datamodels: Predicting Predictions from Training Data". In: International Conference on Machine Learning (ICML). 2022.
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. "Natural questions: a benchmark for question answering research". In: Transactions of the Association for Computational Linguistics 7 (2019), pp. 453-466.
  14. Enja Kokalj, Blaž Škrlj, Nada Lavrač, Senja Pollak, and Marko Robnik-Šikonja. "BERT meets shapley: Extending SHAP explanations to transformer-based classifiers". In: Proceedings of the EACL hackashop on news media content analysis and automated report generation. 2021, pp. 16-21.
  15. Adam Tauman Kalai and Santosh S Vempala. "Calibrated language models must hallucinate". In: arXiv preprint arXiv:2311.14648 (2023). · arXiv
  16. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. "Visualizing and understanding neural models in NLP". In: arXiv preprint arXiv:1506.01066 (2015). · arXiv
  17. Scott Lundberg and Su-In Lee. "A unified approach to interpreting model predictions". In: Neural Information Processing Systems (NeurIPS). 2017.
  18. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. "Lost in the middle: How language models use long contexts". In: Transactions of the Association for Computational Linguistics 12 (2024), pp. 157-173.
  19. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. "Interactive visualization and manipulation of attention-based neural machine translation". In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2017, pp. 121-126.
  20. Nelson F Liu, Tianyi Zhang, and Percy Liang. "Evaluating verifiability in generative search engines". In: arXiv preprint arXiv:2304.09848 (2023). · arXiv
  21. Behnam Mohammadi. "Wait, It's All Token Noise? Always Has Been: Interpreting LLM Behavior Using Shapley Value". In: arXiv preprint arXiv:2404.01332 (2024). · arXiv
  22. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016. URL: https://openreview.net/forum?id=Hk1iOLcle.
  23. Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. "Neural Exec: Learning (and Learn- ing from) Execution Triggers for Prompt Injection Attacks". In: arXiv preprint arXiv:2403.03792 (2024). · arXiv
  24. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. "Scikit-learn: Machine Learning in Python". In: vol. 12. 2011, pp. 2825-2830.
  25. Jirui Qi, Gabriele Sarti, Raquel Fern'andez, and Arianna Bisazza. "Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation". In: 2024. URL: https: //api.semanticscholar.org/CorpusID:270619780.
  26. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. "Measuring attribution in natural language generation models". In: Computational Linguistics 49.4 (2023), pp. 777-840.
  27. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ""Why Should I Trust You?": Explain- ing the Predictions of Any Classifier". In: International Conference on Knowledge Discovery and Data Mining (KDD). 2016.
  28. Kacper Sokol, Alexander Hepburn, Raul Santos-Rodriguez, and Peter Flach. "bLIMEy: Surro- gate Prediction Explanations Beyond LIME". In: Arxiv preprint arXiv:1910.13016. 2019. · arXiv
  29. Charles Spearman. "The Proof and Measurement of Association between Two Things". In: The American Journal of Psychology. 1904.
  30. Sofia Serrano and Noah A Smith. "Is attention interpretable?" In: arXiv preprint arXiv:1906.03731 (2019). · arXiv
  31. Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn. "Design and Analysis of Computer Experiments". In: Statistical Science. Vol. 4. Institute of Mathematical Statistics, 1989, pp. 409-423. URL: http://www.jstor.org/stable/2245858.
  32. Robert Tibshirani. "Regression Shrinkage and Selection Via the Lasso". In: Journal of the Royal Statistical Society, Series B. 1994.
  33. Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge university press, 2019.
  34. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. "Transformers: State-of- the-art natural language processing". In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 2020, pp. 38-45.
  35. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. "Universal adversar- ial triggers for attacking and analyzing NLP". In: arXiv preprint arXiv:1908.07125 (2019). · arXiv
  36. Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. "Unifying corroborative and contributive attributions in large language models". In: arXiv preprint arXiv:2311.12233 (2023). · arXiv
  37. Rose E Wang, Pawan Wirawarn, Omar Khattab, Noah Goodman, and Dorottya Demszky. "Backtracing: Retrieving the Cause of the Query". In: arXiv preprint arXiv:2403.03956 (2024). · arXiv
  38. Kayo Yin and Graham Neubig. "Interpreting language models with contrastive explanations". In: arXiv preprint arXiv:2202.10419 (2022). · arXiv
  39. Xi Ye, Ruoxi Sun, Sercan Arik, and Tomas Pfister. "Effective Large Language Model Adaptation for Improved Grounding and Citation Generation". In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024, pp. 6237-6251.

Cited by

Metadata

Authors
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A.
Venue
Advances in Neural Information Processing Systems 37
Published
2024
Pages
95764-95807
Cited by
27
References
30

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.