CoChat
Sign Up
PreprintCited by 25Explainable Artificial Intelligence (XAI)Topic ModelingAdversarial Robustness in Machine Learning

The DOI matches its DataCite record.

TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation

Open access
Goldshmidt, Roni, Horovicz, Miriam·2024·arXiv
DOI 10.48550/arxiv.2407.10114arXiv 2407.10114
Open this paper in CoChat — free accountOpen PDF

Abstract

As large language models (LLMs) become increasingly prevalent in critical applications, the need for interpretable AI has grown. We introduce TokenSHAP, a novel method for interpreting LLMs by attributing importance to individual tokens or substrings within input prompts. This approach adapts Shapley values from cooperative game theory to natural language processing, offering a rigorous framework for understanding how different parts of an input contribute to a model's response. TokenSHAP leverages Monte Carlo sampling for computational efficiency, providing interpretable, quantitative measures of token importance. We demonstrate its efficacy across diverse prompts and LLM architectures, showing consistent improvements over existing baselines in alignment with human judgments, faithfulness to model behavior, and consistency. Our method's ability to capture nuanced interactions between tokens provides valuable insights into LLM behavior, enhancing model transparency, improving prompt engineering, and aiding in the development of more reliable AI systems. TokenSHAP represents a significant step towards the necessary interpretability for responsible AI deployment, contributing to the broader goal of creating more transparent, accountable, and trustworthy AI systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.10114

Cites

Statistics

Figures

Figure 1 Flowchart of the TokenSHAP algorithm illustrating the process of Shapley value estimation for token importance in large language models by accepting parts of the text to the players and a cosine similarity measure to the base prompt as a gain.
Figure 1 Flowchart of the TokenSHAP algorithm illustrating the process of Shapley value estimation for token importance in large language models by accepting parts of the text to the players and a cosine similarity measure to the base prompt as a gain.CC-BY · From the PDF
Figure 2 A graph that shows the visualization of the prompt in blue-red colors.
Figure 2 A graph that shows the visualization of the prompt in blue-red colors.CC-BY · From the PDF
Figure 3 Box plot showing the distribution of importance values for the Random Baseline method.
Figure 3 Box plot showing the distribution of importance values for the Random Baseline method.CC-BY · From the PDF
Figure 4 Box plot showing the distribution of importance values for the Prompt Engineering method.
Figure 4 Box plot showing the distribution of importance values for the Prompt Engineering method.CC-BY · From the PDF
Figure 5 Box plot showing the distribution of importance values for TokenSHAP.
Figure 5 Box plot showing the distribution of importance values for TokenSHAP.CC-BY · From the PDF
Figure 6 Change in average similarity between true Shapley values and their approximations under different sampling ratios, with and without the condition of firstorder omission.
Figure 6 Change in average similarity between true Shapley values and their approximations under different sampling ratios, with and without the condition of firstorder omission.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 12

  1. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5), 1-42.
  2. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Why should I trust you? Explaining the predic- tions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowl- edge discovery and data mining (pp. 1135-1144).
  3. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Ad- vances in neural information processing systems (pp. 4765-4774).
  4. Simonyan, K., Vedaldi, A., & Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations.
  5. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.
  6. Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018). Explaining ex- planations: An overview of interpretability of ma- chine learning. In 2018 IEEE 5th International Con- ference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE.
  7. Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31, 841.
  8. Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). A survey of the state of explainable AI for natural language processing. arXiv preprint arXiv:2010.00711. · arXiv
  9. Vig, J. (2019). A multiscale visualization of atten- tion in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics: System Demonstrations (pp. 37-42).
  10. Sundararajan, M., Taly, A., & Yan, Q. (2017). Ax- iomatic attribution for deep networks. In Proceed- ings of the 34th International Conference on Ma- chine Learning (pp. 3319-3328).
  11. Tenney, I., Das, D., & Pavlick, E. (2019). BERT re- discovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4593-4601).
  12. Castro, J., Gómez, D., & Tejada, J. (2009). Polyno- mial calculation of the Shapley value based on sam- pling. Computers & Operations Research, 36(5), 1726-1730.

Cited by

Sorted by citations

See all 25

Related

Metadata

Authors
Goldshmidt, R., & Horovicz, M.
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.10114
arXiv
2407.10114
OpenAlex
W4400703199
Cited by
2
References
0
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, Shapley value, Monte Carlo estimation, interpretability, cooperative game theory, prompt engineering, model transparency, explainable AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review