CoChat
Sign Up
Cited by 63Explainable Artificial Intelligence (XAI)Topic ModelingText Readability and Simplification

Augmenting interpretable models with large language models during training

Open access
Singh, Chandan, Askari, Armin, Caruana, Rich, Gao, Jianfeng·2023·Nature Communications
DOI 10.1038/s41467-023-43713-1PMC PMC10689442
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent large language models (LLMs), such as ChatGPT, have demonstrated remarkable prediction performance for a growing array of tasks. However, their proliferation into high-stakes domains and compute-limited settings has created a burgeoning need for interpretability and efficiency. We address this need by proposing Aug-imodels, a framework for leveraging the knowledge learned by LLMs to build extremely efficient and interpretable prediction models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency and often a speed/memory improvement of greater than 1000x for inference compared to LLMs. We explore two instantiations of Aug-imodels in natural-language processing: Aug-Linear, which augments a linear model with decoupled embeddings from an LLM and Aug-Tree, which augments a decision tree with LLM feature expansions. Across a variety of text-classification datasets, both outperform their non-augmented, interpretable counterparts. Aug-Linear can even outperform much larger models, e.g. a 6-billion parameter GPT-J model, despite having 10,000x fewer parameters and being fully transparent. We further explore Aug-imodels in a natural-language fMRI study, where they generate interesting interpretations from scientific data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-023-43713-1

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Post-hoc methods offer capabilities that generative approaches structurally cannot: training-data attribution, mechanistic localization and intervention on internal representations, and formal faithfulness guarantees from transparent surrogates.

    “Yet these methods offer capabilities that generative approaches structurally cannot: training-data attribution [129], mechanistic localization and intervention on internal representations [114], and formal faithfulness guarantees from transparent surrogates [125].”

    From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [125]

    The Introduction (page unavailable) says the paper’s interpretable models allow complete inspection of their decisions, unlike post-hoc explanations, which are often unfaithful. The Methods’ “Background and related work” section (page unavailable) similarly contrasts fully transparent models with LLM-generated explanations. It does not establish formal faithfulness guarantees for transparent surrogates; the cited work concerns models designed to be transparent, not surrogates for a black-box model. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The few studies that move beyond purely local explanations point to promising but underexplored directions.

    “The few studies that move beyond purely local explanations [11, 74,80,81,104,116,125,129] point to promising but underexplored directions.”

    From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [125]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 85

  1. [1]Brown T Language models are few-shot learners Adv. Neural Inf. Process. Syst. 2020 33 1877 1901
  2. [2]Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. https://arxiv.org/abs/2303.12712 (2023). · arXiv
  3. [3]Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805 (2018). · arXiv
  4. [4]Angermueller C Pärnamaa T Parts L Stegle O Deep learning for computational biology Mol. Syst. Biol. 2016 12 878 10.15252/msb.20156651 27474269 PMC4965871 · DOI · Paper page
  5. [5]Kornblith, A. E. et al. Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma. PLOS Digit. Health10.1371/journal.pdig.0000076 (2022). PMC9931266 36812570
  6. [6]Brennan T Oliver WL The emergence of machine learning techniques in criminology Criminol. Public Policy 2013 12 551 562 10.1111/1745-9133.12055 · DOI
  7. [7]Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. 214–226 (ACM, 2012).
  8. [8]Goodman, B. & Flaxman, S. European union regulations on algorithmic decision-making and a “right to explanation”. https://arxiv.org/abs/1606.08813 (2016). · arXiv
  9. [11]Rudin, C. et al. Interpretable machine learning: Fundamental principles and 10 grand challenges. https://arxiv.org/abs/2103.11251 (2021). · arXiv
  10. [12]Ha, W., Singh, C., Lanusse, F., Upadhyayula, S., & Yu, B. Adaptive wavelet distillation from neural networks through interpretations. Adv. Neural Inf. Process. Syst. 34 https://arxiv.org/abs/2107.09145 (2021). · arXiv
  11. [13]Mignan A Broccardo M One neuron versus deep learning in aftershock prediction Nature 2019 574 1 3 10.1038/s41586-019-1582-8 31578475 · DOI
  12. [14]Tan, Y. S., Singh, C., Nasseri, K., Agarwal, A., & Yu, B. Fast interpretable greedy-tree sums (figs). https://arxiv.org/abs/2201.11931 (2022). PMC11848335 39951504 · arXiv
  13. [15]Adebayo, J. et al. Sanity checks for saliency maps. Adv. Neural Inf. Process. Syst. 9505–9515 https://arxiv.org/abs/1810.03292 (2018). · arXiv
  14. [16]Turpin, M., Michael, J., Perez, E., & Bowman, S. R. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. https://arxiv.org/abs/2305.04388 (2023). · arXiv
  15. [18]Saravia, E., Liu, H.-C.T., Huang, Y.-H., Wu, J. & Chen, Y.-S. Carer: Contextualized affect representations for emotion recognition. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3687–3697 (2018).
  16. [20]Pang, B. & Lee, L. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In: Proceedings of the ACL. https://arxiv.org/abs/cs/0506075 (2005). · arXiv
  17. [21]Socher, R. et al. Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1631–1642 (Association for Computational Linguistics, 2013).
  18. [22]LeBel, A. et al. A natural language fmri dataset for voxelwise encoding models. bioRxiv https://www.biorxiv.org/content/10.1101/2022.09.22.509104v1 (2022). PMC10447563 37612332 · DOI
  19. [23]Jones KS A statistical interpretation of term specificity and its application in retrieval J. Documentation 2021 60 493 502 10.1108/00220410410560573 · DOI
  20. [24]Pennington, J., Socher, R., & Manning, C.D. Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1532–1543 (Association for Computational Linguistics, 2014).
  21. [25]Quinlan JR Induction of decision trees Mach. Learn. 1986 1 81 106 10.1007/BF00116251 · DOI · Paper page
  22. [26]Breiman L Random forests Mach. Learn. 2001 45 5 32 10.1023/A:1010933404324 · DOI · Paper page
  23. [27]Bach, S.H. et al. Promptsource: an integrated development environment and repository for natural language prompts. https://arxiv.org/abs/2202.01279 (2022). · arXiv
  24. [28]Ribeiro, M.T., Singh, S., Guestrin, C. Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1135–1144 (ACM, 2016).
  25. [29]Lundberg, S. & Lee, S.-I. An unexpected unity among methods for interpreting model predictions. https://arxiv.org/abs/1611.07478 (2016). · arXiv
  26. [30]Huth AG De Heer WA Griffiths TL Theunissen FE Gallant JL Natural speech reveals the semantic maps that tile human cerebral cortex Nature 2016 532 453 458 10.1038/nature17637 27121839 PMC4852309 · DOI · Paper page
  27. [31]Schrimpf M The neural architecture of language: Integrative modeling converges on predictive processing Proc. Natl Acad. Sci. USA 2021 118 2105646118 10.1073/pnas.2105646118 PMC8694052 34737231 · DOI · Paper page
  28. [32]Antonello, R.J. & Huth, A. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiol. Lang. 3, 1–39 (2022). PMC11025645 38645616
  29. [33]Caucheteux C King J-R Brains and algorithms partially converge in natural language processing Commun. Biol. 2022 5 1 10 10.1038/s42003-022-03036-1 35173264 PMC8850612 · DOI · Paper page
  30. [35]McCullagh P Nelder JA Generalized linear models J. Am. Stat. Assoc. 1993 88 698 10.2307/2290358 · DOI
  31. [36]Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst.26 https://arxiv.org/abs/1310.4546 (2013). · arXiv
  32. [37]Mikolov, T., Chen, K., Corrado, G., & Dean, J. Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781 (2013). · arXiv
  33. [38]Tan, C.et al. A survey on deep transfer learning. In: Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4–7, 2018, Proceedings, Part III 27. 270–279 (Springer, 2018). Springer.
  34. [41]Hastie T Tibshirani R Generalized additive models Stat. Sci. 1986 1 297 318 8548102
  35. [42]Agarwal R Neural additive models: Interpretable machine learning with neural nets Adv. Neural Inf. Process. Syst. 2021 34 4699 4711
  36. [45]Lin, J., Zhong, C., Hu, D., Rudin, C., & Seltzer, M. Generalized and scalable optimal sparse decision trees. In: International Conference on Machine Learning. 6150–6160 (PMLR, 2020).
  37. [46]Hu, X., Rudin, C., & Seltzer, M. Optimal sparse decision trees. Adv. Neural Inf. Process. Syst. (NeurIPS) https://arxiv.org/abs/1904.12847 (2019). PMC10544768 37786624 · arXiv
  38. [47]Bertsimas D Dunn J Optimal classification trees Mach. Learn. 2017 106 1039 1082 10.1007/s10994-017-5633-9 · DOI
  39. [48]Agarwal, A., Tan, Y. S., Ronen, O., Singh, C. & Yu, B. Hierarchical shrinkage: improving the accuracy and interpretability of tree-based methods. https://arxiv.org/abs/2202.00858 (2022). · arXiv
  40. [49]Carreira-Perpinán, M. A. & Tavallali, P. Alternating optimization of decision trees, with application to learning sparse oblique trees. Advances in Neural Information Processing Systems. Vol. 31 (ACM, 2018).
  41. [50]Morris, J. X., Singh, C., Rush, A. M., Gao, J., & Deng, Y. Tree prompting: efficient task adaptation without fine-tuning. https://arxiv.org/abs/2310.14034 (2023). · arXiv
  42. [51]Yao, S. et al. Tree of thoughts: deliberate problem solving with large language models. https://arxiv.org/pdf/2305.10601.pdf (2023). · arXiv
  43. [52]Friedman JH Popescu BE Predictive learning via rule ensembles Ann. Appl. Stat. 2008 2 916 954 10.1214/07-AOAS148 · DOI · Paper page
  44. [53]Angelino, E., Larus-Stone, N., Alabi, D., Seltzer, M., & Rudin, C. Learning certifiably optimal rule lists for categorical data. https://arxiv.org/abs/1704.01701 (2017). · arXiv
  45. [54]Singh C Nasseri K Tan YS Tang T Yu B imodels: a python package for fitting interpretable models J. Open Source Softw. 2021 6 3192 10.21105/joss.03192 · DOI
  46. [55]Freund, Y. et al. Experiments with a new boosting algorithm. In: Icml, vol. 96. 148–156 (Citeseer, 1996).
  47. [56]Chen, T. & Guestrin, C. Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining. 785–794 (ACM, 2016).
  48. [57]Chipman HA George EI McCulloch RE Bart: Bayesian additive regression trees Ann. Appl. Stat. 2010 4 266 298 10.1214/09-AOAS285 · DOI
  49. [60]Devlin, S., Singh, C., Murdoch, W.J., & Yu, B. Disentangled attribution curves for interpreting random forests and boosted trees. https://arxiv.org/abs/1905.07631 (2019). · arXiv
  50. [62]Singh, C., Murdoch, W.J., & Yu, B. Hierarchical interpretations for neural network predictions. International Conference on Learning Representations, Vol. 26 https://arxiv.org/abs/1806.05337 (2019). · arXiv
  51. [63]Singh, C. et al. Transformation importance with applications to cosmology. https://arxiv.org/abs/2003.01926 (2020). · arXiv
  52. [64]Rudin, C. Please stop explaining black box models for high stakes decisions. https://arxiv.org/abs/1811.10154 (2018). PMC9122117 35603010 · arXiv
  53. [66]Zaidan, O. & Eisner, J. Modeling annotators: A generative approach to learning from annotator rationales. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 31–40 (ACM, 2008).
  54. [67]Sha, L., Camburu, O.-M. & Lukasiewicz, T. Learning from the best: Rationalizing predictions by adversarial information calibration. In: AAAI, 13771–13779. 10.1609/aaai.v35i15.17623 (2021). · DOI
  55. [68]Hendricks, L.A. et al. Generating visual explanations. In: European Conference on Computer Vision. 3–19 (Springer, 2016).
  56. [69]Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T. & Blunsom, P. e-snli: Natural language inference with natural language explanations. Adv. Neural Inf. Process. Syst.31 https://arxiv.org/abs/1812.01193 (2018). · arXiv
  57. [70]Wei J Chain-of-thought prompting elicits reasoning in large language models Adv. Neural Inf. Process. Syst. 2022 35 24824 24837
  58. [71]Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. https://arxiv.org/abs/1805.01070 (2018). · arXiv
  59. [72]Liu, F. & Avci, B. Incorporating priors with feature attribution on text classification. https://arxiv.org/abs/1906.08286 (2019). · arXiv
  60. [73]Wang X Xu X Tong W Roberts R Liu Z Inferbert: a transformer-based causal inference framework for enhancing pharmacovigilance Front. Artif. Intell. 2021 4 659622 10.3389/frai.2021.659622 34136800 PMC8202286 · DOI
  61. [74]Olah C The building blocks of interpretability Distill 2018 3 10 10.23915/distill.00010 · DOI · Paper page
  62. [75]Meng, K., Bau, D., Andonian, A. & Belinkov, Y. Locating and editing factual knowledge in GPT. https://arxiv.org/abs/2202.05262 (2022). · arXiv
  63. [76]Li, O., Liu, H., Chen, C., & Rudin, C. Deep learning for case-based reasoning through prototypes: a neural network that explains its predictions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (AAAI Pressm 2018).
  64. [77]Chen, C. et al. This looks like that: deep learning for interpretable image recognition. Adv. Neural Inf. Process. Syst. 32 https://arxiv.org/abs/1806.10574 (2019). · arXiv
  65. [78]Koh, P.W et al. Concept bottleneck models. In: International Conference on Machine Learning. 5338–5348 (PMLR, 2020).
  66. [79]Yang, Y. et al. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. https://arxiv.org/abs/2211.11158 (2022). · arXiv
  67. [80]Ghosh, S. et al. Dividing and conquering a blackbox to a mixture of interpretable models: route, interpret, repeat. https://arxiv.org/abs/2307.05350 (2023). PMC10500943 37711878 · arXiv
  68. [81]Yuksekgonul, M., Wang, M., & Zou, J. Post-hoc concept bottleneck models. https://arxiv.org/abs/2205.15480 (2022). · arXiv
  69. [82]McInerney, D.J., Young, G., Meent, J.-W. & Wallace, B.C. Chill: zero-shot custom interpretable feature extraction from clinical notes with large language models. https://arxiv.org/abs/2302.12343 (2023). PMC12747426 41473351 · arXiv
  70. [83]Frosst, N. & Hinton, G. Distilling a neural network into a soft decision tree. https://arxiv.org/abs/1711.09784 (2017). · arXiv
  71. [84]Zarlenga, M.E., Shams, Z. & Jamnik, M. Efficient decompositional rule extraction for deep neural networks. https://arxiv.org/abs/2111.12628 (2021). · arXiv
  72. [85]Tan, S., Caruana, R., Hooker, G., Koch, P. & Gordo, A. Learning global additive explanations for neural nets using model distillation. ICLR 2019 Conference Blind Submission (2018).
  73. [86]Hinton, G., Vinyals, O., & Dean, J. Distilling the knowledge in a neural network. https://arxiv.org/abs/1503.02531 (2015). · arXiv
  74. [87]Sanh, V., Debut, L., Chaumond, J. & Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. https://arxiv.org/abs/1910.01108 (2019). · arXiv
  75. [88]Wolf, T. et al. Huggingface’s transformers: state-of-the-art natural language processing. https://arxiv.org/abs/1910.03771 (2019). · arXiv
  76. [89]Hazourli, A. Financialbert-a pretrained language model for financial text mining. 10.13140/RG.2.2.34032.12803 (2022). · DOI
  77. [90]Morris, J. X. et al. Textattack: a framework for adversarial attacks, data augmentation, and adversarial training in nlp. https://arxiv.org/abs/2005.05909 (2020). · arXiv
  78. [91]Akl, H.A., Mariko, D., & De Mazancourt, H. Yseop at finsim-3 shared task 2021: Specializing financial domain learning with phrase representations. https://arxiv.org/abs/2108.09485 (2021). · arXiv
  79. [92]Liu, Y. et al. Roberta: a robustly optimized bert pretraining approach. https://arxiv.org/abs/1907.11692 (2019). · arXiv
  80. [93]Su, H. et al. One embedder, any task: Instruction-finetuned text embeddings. https://arxiv.org/abs/2212.09741 (2022). · arXiv
  81. [95]Touvron, H. et al. Llama: Open and efficient foundation language models. https://arxiv.org/abs/2302.13971 (2023). · arXiv
  82. [97]Zhang, X., Zhao, J., LeCun, Y. Character-level convolutional networks for text classification. Adv. Neural Inf, Process. Syst.28 https://arxiv.org/abs/1509.01626 (2015). PMC4831869 27087766 · arXiv
  83. [98]Lehmann J Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia Semant. Web. 2015 6 167 195 10.3233/SW-140134 · DOI
  84. [99]Li, X. & Roth, D. Learning question classifiers. In: COLING 2002: The 19th International Conference on Computational Linguistics. 10.3115/1072228.1072378 (2002). · DOI
  85. [100]Loper, E. & Bird, S. Nltk: The natural language toolkit. https://arxiv.org/abs/cs/0205028 (2002). · arXiv

Cited by

Sorted by citations

Authors
Singh, C., Askari, A., Caruana, R., & Gao, J.
Affiliations
  • Microsoft (United States) (US)
  • University of California, Berkeley (US)
Corresponding author
Singh, Chandan
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2023-11-30
Volume
14
Issue
1
Pages
7913-7913
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-023-43713-1
PMC
PMC10689442
OpenAlex
W4389165057
Cited by
63
References
8
Citations by year
2026: 15 · 2025: 31 · 2024: 16 · 2023: 1
FWCI
5.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
large language models, model interpretability, linear models, decision tree, text classification, model efficiency
MeSH terms
Language, Learning, Natural Language Processing, Linear Models, Knowledge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review