CoChat
Sign Up
Cited by 681Explainable Artificial Intelligence (XAI)Topic ModelingAdversarial Robustness in Machine Learning

Explainability for Large Language Models: A Survey

Open access
Zhao, Haiyan, Chen, Hanjie, Yang, Fan, Liu, Ninghao, Deng, Huiqi, Cai, Hengyi, Wang, Shuaiqiang, Yin, Dawei +1 more·2024·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/3639372
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this article, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional deep learning models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3639372

Cites

118 of 118 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • As M-LLMs expand in size, the task of understanding and interpreting their decision-making processes becomes increasingly challenging.

    “First, as these models expand in size, the task of understanding and interpreting their decision-making processes becomes increasingly challenging [167].”

    From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [167]

    The Introduction states that as LLMs become larger, understanding and interpreting their decision-making becomes more difficult. The cited passage does not specifically address M-LLMs, so it supports the scaling point but not its application to that model category. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Most current interpretability research on M-LLMs neglects training dynamics, focusing mainly on post hoc explanations of fully trained models.

    “Most current interpretability research on M-LLMs neglects training dynamics, focusing mainly on post hoc explanations of fully trained models [167].”

    From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [167]

    Section 6.6, “Shifting from Snapshot Explainability to Temporal Analysis” (page not provided), says that current interpretability research neglects training dynamics and mainly offers post-hoc explanations of fully trained models. The survey makes this point about LLMs generally, not specifically about multimodal LLMs (M-LLMs), so it does not establish that the same pattern holds for M-LLM research. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Auditing why a model behaved a certain way is difficult because relevant knowledge is distributed across billions of parameters rather than encoded as inspectable modules.

    “Auditing why a model behaved a certain way is difficult because relevant knowledge is distributed across billions of parameters rather than encoded as inspectable modules [Zhao et al., 2024].”

    From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Zhao et al., 2024

    The Introduction (page unavailable) says that billions of parameters and increased internal complexity make LLM decisions difficult to interpret and debug. Section 3.2.4 (page unavailable) says features can be spread across many neurons, but it does not establish that the relevant knowledge is distributed across billions of parameters rather than encoded as inspectable modules. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Current LLMs primarily rely on parametric memory (knowledge encoded in model weights) and ephemeral activation memory (context-limited runtime states).

    “They primarily rely on parametric memory (knowledge encoded in model weights) and ephemeral activation memory (context-limited runtime states).”

    From MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models (2025) · cites this paper as [37]

    Section 3.2.4 (page unavailable) describes model components as a memory of training-text patterns and links model weights to factual knowledge. Although it mentions hidden-state activations, it does not describe them as ephemeral, context-limited memory or say that LLMs primarily rely on these two forms of memory. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

  • Gradient-based XAI approaches require model access unavailable through proprietary APIs.

    “gradient-based approaches require model access unavailable through proprietary APIs”

    From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [18]

    The “Gradient-Based Explanation” section (page not provided) says these methods analyze partial derivatives of model outputs, supporting the need for access to gradients. Our copy does not say that proprietary APIs withhold that access, so the API-specific part is unsupported. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 86

  1. AnthropicAI. 2023. Introducing Claude. Retrieved from https://www.anthropic.com/index/introducing-claude Omer Antverg and Yonatan Belinkov. 2022. On the Pitfalls of Analyzing Individual Neurons in Language Models. arXiv:2110.07483. · arXiv
  2. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Identifying and Con- trolling Important Neurons in Neural Machine Translation. arXiv:1811.01157. · arXiv
  3. Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, Taipei, Taiwan, 1-10. Retrieved from https://aclanthology.org/I17-1001
  4. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". arXiv:2309.12288. · arXiv
  5. Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2019. On iden- tifiability in transformers. arXiv:1908.04211. · arXiv
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712. · arXiv
  7. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. 2023. Are aligned neural networks adversarially aligned? arXiv:2306.15447. · arXiv
  8. Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. 2023c. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. arXiv:2309.07311. · arXiv
  9. Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. 2021. Probing BERT in Hyper- bolic Spaces. arXiv:2104.03869. · arXiv
  10. Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. 2023d. Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations. arXiv:2307.08678. · arXiv
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fe- dus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research 24, 240 (2023), 1-113.
  12. Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. 2022. Framework for evaluating faithfulness of local explanations. International Conference on Machine Learning (ICMR'22). PMLR, 4794-4815.
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies. Vol. 1, 4171-4186.
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL '19).
  15. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. stat 1050 (2017), 2.
  16. Timothy Dozat and Christopher D. Manning. 2016. Deep Biaffine Attention for Neural Dependency Parsing. arXiv: 1611.01734. · arXiv
  17. Mengnan Du, Ninghao Liu, Fan Yang, Shuiwang Ji, and Xia Hu. 2019b. On attribution of recurrent neural network predic- tions via additive decomposition. The World Wide Web Conference (WWW'19), 383-393.
  18. Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. 2021. Towards interpreting and mitigating shortcut learning behavior of NLU models. In Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 915-929.
  19. Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv:2307.01379. · arXiv
  20. Amirata Ghorbani and James Zou. 2019. Data shapley: Equitable valuation of data for machine learning. In Proceedings of the International Conference on Machine Learning. PMLR, 2242-2251.
  21. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyil- maz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv:2212.07919. · arXiv
  22. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying large language model generalization with influence functions. arXiv:2308.03296. Retrieved from https://arxiv.org/abs/2308.03296 · arXiv
  23. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary LLMs. arXiv:2305.15717. · arXiv
  24. Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. arXiv:2310.02207. · arXiv
  25. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentan- gled attention. In Proceedings of the International Conference on Learning Representations.
  26. Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. 2023. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv:2307.10236. · arXiv
  27. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 (2019), 3543-3556.
  28. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? natural language attack on text classification and entailment. In Proceedings of the AAAI Conference on Artificial Intelligence 34, 5 (2020), 8018-8025.
  29. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. In International Conference on Machine Learning. PMLR, 15696-15707.
  30. Cheongwoong Kang and Jaesik Choi. 2023. Impact of Co-occurrence on Factual Knowledge of Large Language Models. arXiv:2310.08256. Retrieved from https://arxiv.org/abs/2310.08256 · arXiv
  31. Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434 (2019). · arXiv
  32. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learning, PMLR, 2668-2677.
  33. Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. 2023. Are large language models Post Hoc Explainers? arXiv:2310.05797. · arXiv
  34. Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. 2023. When do pre-training biases propagate to downstream tasks? A case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.
  35. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702. Retrieved from https://arxiv.org/abs/2307.13702 · arXiv
  36. Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models. arXiv:2210.16978. Retrieved from https://arxiv.org/abs/2210.16978 · arXiv
  37. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Understanding Neural Networks through Representation Erasure. arXiv:1612.08220. · arXiv
  38. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023a. A survey on fairness in large language models. arXiv:2308.10149. · arXiv
  39. Zongxia Li, Paiheng Xu, Fuxiao Liu, and Hyemi Song. 2023b. Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps. arXiv:2307.05052. · arXiv
  40. Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit anal- ysis interpretability scale? Evidence from multiple choice capabilities in chinchilla. arXiv:2307.09458. · arXiv
  41. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241-253.
  42. Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Xiaotang Gai, Yang Feng, and Zuozhu Liu. 2023a. A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis. arXiv:2307.01981. · arXiv
  43. Ninghao Liu, Yunsong Meng, Xia Hu, Tie Wang, and Bo Long. 2020. Are interpretations fairly evaluated? A definition driven pipeline for post-hoc interpretability. arXiv:2009.07494. · arXiv
  44. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b. Lost in the middle: How language models use long contexts. arXiv:2307.03172. · arXiv
  45. Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. 2022. Rethinking attention-model explain- ability through faithfulness violation test. In International Conference on Machine Learning. PMLR, 2022, 13807-13824.
  46. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. · arXiv
  47. Scott M. Lundberg and Su-In Lee. 2017a. A unified approach to interpreting model predictions. Advances in Neural Infor- mation Processing Systems 30 (2017).
  48. Daniel D. Lundstrom, Tianjian Huang, and Meisam Razaviyayn. 2022. A rigorous study of integrated gradients method and extensions to internal neuron attributions. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 14485-14508. Retrieved from https://proceedings.mlr.press/v162/lundstrom22a.html ISSN: 2640-3498.
  49. Siwen Luo, Hamish Ivison, Caren Han, and Josiah Poon. 2022. Local Interpretations for Explainable Natural Language Processing: A Survey. arXiv:2103.11072. · arXiv
  50. Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv:2209.07686. Retrieved from https://arxiv.org/abs/2209.07686 · arXiv
  51. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model repre- sentations of true/false datasets. arXiv:2310.06824. · arXiv
  52. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023. The hydra effect: Emergent self-repair in language model computations. arXiv:2307.15771. · arXiv
  53. Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson. 2020. Investigating Satu- ration Effects in Integrated Gradients. arXiv:2010.12697. · arXiv
  54. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv:2306.02707. · arXiv
  55. Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, and Ana Lucic. 2021. Order in the court: Explainable ai methods prone to disagreement. arXiv:2105.03287. · arXiv
  56. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv:2112.00114. Retrieved from https://arxiv.org/abs/2112. 00114 · arXiv
  57. Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. 2020b. Naturally Occurring Equivariance in Neural Networks -distill.pub. Retrieved from https://distill.pub/2020/circuits/equivariance/. Accessed 27-11-2023.
  58. OpenAI. 2023a. GPT-4 Technical Report. arXiv:2303.08774. · arXiv
  59. Michael Petrov, Chelsea Voss, Ludwig Schubert, Nick Cammarata, Gabriel Goh, and Chris Olah. 2021. Weight Banding - distill.pub. https://distill.pub/2020/circuits/weight-banding/. Accessed 27-11-2023.
  60. Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yueyuan Zheng, Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. 2021. Resisting Out-of-Distribution Data Problem in Perturbation of XAI. arXiv:2107.14000. · arXiv
  61. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and Gretchen Krueger Ilya. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 8748-8763.
  62. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandrasekaran, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Question decomposition improves the faithfulness of model-generated reasoning. arXiv:2307.11768. Retrieved from https://arxiv.org/abs/2307.11768 · arXiv
  63. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Pe- ter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485-5551.
  64. Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation. arXiv:2307.11019. · arXiv
  65. Yaozong Shen, Lijie Wang, Ying Chen, Xinyan Xiao, Jing Liu, and Hua Wu. 2022. An Interpretability Evaluation Benchmark for Pre-trained Language Models. arXiv:2207.13948. · arXiv
  66. Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao. 2023. Ex- plaining black box text modules in natural language with language models. arXiv:2305.09863. · arXiv
  67. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. 3, 6 (2023), 7. https://crfm.stanford.edu/2023/03/13/alpaca.html
  68. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, et al. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316 (2019). · arXiv
  69. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https: //arxiv.org/abs/2302.13971 · arXiv
  70. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv:2305.04388. · arXiv
  71. Jesse Vig. 2019. BertViz: A tool for visualizing multi-head self-attention in the BERT model. In ICLR Workshop: Debugging Machine Learning Models Vol. 23, 353-355.
  72. Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. 2021. Branch Specialization -distill.pub. Retrieved from https://distill.pub/2020/circuits/branch-specialization/. [Accessed 26-11-2023].
  73. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023a. Simple Synthetic Data Reduces Sycophancy in Large Language Models. arXiv:2308.03958. · arXiv
  74. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023b. Larger language models do in-context learning differently. arXiv:2303.03846. · arXiv
  75. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Hendricks, Anne Isaac, Legassick William, Irving Sean, Geoffrey, and Iason Gabriel. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. · arXiv
  76. Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, and Himabindu Lakkaraju. 2023b. Analyzing chain-of-thought prompting in large language models via gradient-based feature attributions. arXiv preprint arXiv:2307.13339 · arXiv
  77. Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, and Dong Yu. 2023c. From language modeling to instruction following: Understanding the behavior shift in LLMs after instruction tuning. arXiv:2310.00492. · arXiv
  78. Zhengxuan Wu and Desmond C. Ong. 2021. On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification. arXiv:2101.00196. · arXiv
  79. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can LLMs express their uncer- tainty? An empirical evaluation of confidence elicitation in LLMs. arXiv:2306.13063. Retrieved from https://arxiv.org/ abs/2306.13063 · arXiv
  80. Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in Neural Information Processing Systems 35 (2022), 30378-30392. https://proceedings.neurips.cc/paper_files/paper/2022/ hash/c402501846f9fe03e2cac015b3f0e6b1-Abstract-Conference.html
  81. Catherine Yeh, Yida Chen, Aoyu Wu, Cynthia Chen, Fernanda Viégas, and Martin Wattenberg. 2023. AttentionViz: A Global View of Transformer Attention. arXiv:2305.03210. · arXiv
  82. Mert Yuksekgonul, Maggie Wang, and James Zou. 2023. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIR 2 Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data.
  83. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, and Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. · arXiv
  84. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219. · arXiv
  85. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. · arXiv
  86. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, and Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023. Representation engineer- ing: A top-down approach to AI transparency. arXiv:2310.01405. Retrieved from https://arxiv.org/abs/2310.01405 · arXiv

Cited by

Sorted by citations

Related

Authors
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M.
Affiliations
  • New Jersey Institute of Technology (US)
  • Johns Hopkins University (US)
  • Wake Forest University (US)
  • University of Georgia (US)
  • Shanghai Jiao Tong University (CN)
  • Chinese Academy of Sciences (CN)
  • Baidu (China) (CN)
ORCID
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2024-02-22
Volume
15
Issue
2
Pages
1-38
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/3639372
OpenAlex
W4390490761
Cited by
681
References
118
Citations by year
2026: 211 · 2025: 329 · 2024: 133 · 2023: 7 · 2019: 1
FWCI
99.11
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Intelligent Systems and Technology (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
explainability, large language models, transformer models, local explanations, global explanations, fine-tuning, prompting, explanation evaluation metrics, model debugging, model transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review