Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Open accessAbstract
The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, “inner” interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mechanistic interpretability is an active research area.
“Mechanistic interpretability [93] is another active area of research, which has the potential to be further accelerated by the availability of small language models like phi-2.”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [93]
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
(2014) · Cited by 33,419 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
(2017) · International Conference on Learning Representations · Cited by 3,248 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works
Superintelligence: paths, dangers, strategies
(2015) · Choice Reviews Online · doi:10.5860/choice.187536 · Cited by 1,977 · see citing works
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Luo, J.-H., Wu, J., & Lin, W. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.541 · Cited by 1,889 · see citing works
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · cited 3× · see citing works
“ours [3], [65], [67], [75], [108], [134], [152], [175], [197]– [199], [236], [243], [246], [247].”
BackgroundMethodNetwork Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Mallya, A., & Lazebnik, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00810 · Cited by 1,258 · see citing works
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995 · see citing works
Visual interpretability for deep learning: a survey
Zhang, Q., & Zhu, S. C. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700808 · Cited by 975 · see citing works
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.12 · Cited by 933 · see citing works
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903 · see citing works
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900 · see citing works
Challenging Common Assumptions in the Unsupervised Learning of Disentangled RepresentationsPreprint
Locatello, F., Bauer, S., Lučić, M., Rätsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12359 · Cited by 880 · see citing works
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880 · see citing works
Interpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833 · see citing works
Disentangling by FactorisingPreprint
Kim, H., & Mnih, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05983 · Cited by 833 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep ArchitecturesPreprint
Hu, H., Peng, R., Tai, Y., & Tang, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.03250 · Cited by 792 · see citing works
(2019) · Cited by 771 · see citing works
(2017) · Cited by 761 · see citing works
Similarity of Neural Network Representations Revisited
Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1905.00414 · Cited by 724 · see citing works
This Looks Like That: Deep Learning for Interpretable Image Recognition
Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., & Rudin, C. (2018) · arXiv · doi:10.48550/arxiv.1806.10574 · Cited by 702 · see citing works
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · see citing works
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works
Future Progress in Artificial Intelligence: A Survey of Expert Opinion
Müller, V. C., & Bostrom, N. (2016) · Synthese Library · doi:10.1007/978-3-319-26485-1_33 · Cited by 655 · see citing works
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_1 · Cited by 650 · see citing works
Ross, A. S., & Doshi‐Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11504 · Cited by 621 · see citing works
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · see citing works
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593 · see citing works
The Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591 · see citing works
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1198 · Cited by 572 · see citing works
Overcoming catastrophic forgetting with hard attention to the taskPreprint
Serr\`a, J., Surís, D., Miron, M., & Karatzoglou, A. (2018) · arXiv (Cornell University) · Cited by 564 · see citing works
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · see citing works
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Alvarez-Melis, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 545 · see citing works
Li, O., Liu, H., Chen, C., & Rudin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11771 · Cited by 541 · see citing works
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534 · see citing works
Data-free Parameter Pruning for Deep Neural Networks
Srinivas, S., & Babu, R. V. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.31 · Cited by 518 · see citing works
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504 · see citing works
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · see citing works
A Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.46 · Cited by 380 · see citing works
Understanding the role of individual units in a deep neural network
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., & Torralba, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907375117 · Cited by 375 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 374 · see citing works
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · cited 1× · see citing works
“by ten orders of magnitude from 2010 to 2022 [255].”
BackgroundSelf-citationTextual Explanations for Self-Driving Vehicles
Kim, J., Rohrbach, A., Darrell, T., Canny, J., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_35 · Cited by 355 · see citing works
Transformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.446 · Cited by 353 · see citing works
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2007.04612 · Cited by 338 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Aljundi, R., Kelchtermans, K., & Tuytelaars, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01151 · Cited by 335 · see citing works
Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05806 · Cited by 335 · see citing works
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317 · see citing works
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2021) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 311 · see citing works
The Alignment Problem: Machine Learning and Human Values
Christian, B. (2021) · Perspectives on Science and Christian Faith · doi:10.56315/pscf12-21christian · Cited by 311 · see citing works
Visualizing and Measuring the Geometry of BERTPreprint
Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viegas, F., & Wattenberg, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02715 · Cited by 298 · see citing works
Understanding disentangling in $\beta$-VAEPreprint
Burgess, C., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., & Lerchner, A. (2018) · arXiv (Cornell University) · Cited by 294 · see citing works
Mensah, P. K., Adekoya, A. F., Ayidzoe, M. A., & Baagyere, E. Y. (2019) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2019.09.014 · Cited by 291 · see citing works
Zoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288 · see citing works
Concept whitening for interpretable image recognition
Chen, Z., Bei, Y., & Rudin, C. D. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00265-z · Cited by 284 · see citing works
Interpretable Basis Decomposition for Visual Explanation
Zhou, B., Sun, Y., Bau, D., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_8 · Cited by 270 · see citing works
On the importance of single directions for generalizationPreprint
Morcos, A. S., Barrett, D. G. T., Rabinowitz, N. C., & Botvinick, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06959 · Cited by 269 · see citing works
Pruning by explaining: A novel criterion for deep neural network pruning
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S. M., Müller, K., & Samek, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107899 · Cited by 269 · see citing works
Gradient-Based Attribution Methods
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_9 · Cited by 263 · see citing works
Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration
He, Y., Ding, Y., Liu, P., Zhu, L., Zhang, H., & Yang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00208 · Cited by 262 · see citing works
Singh, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7475259 · Cited by 262 · see citing works
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865044 · Cited by 244 · see citing works
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 240 · see citing works
Compositional Explanations of NeuronsPreprint
Mu, J., & Andreas, J. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2006.14032 · Cited by 233 · see citing works
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230 · see citing works
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 1× · see citing works
“Shapley values, however, are limited in their ability to provide useful causal explanations [159].”
BackgroundAgainst Interpretability: a Critical Examination of the Interpretability Problem in Machine Learning
Krishnan, M. (2019) · Philosophy & Technology · doi:10.1007/s13347-019-00372-9 · Cited by 221 · see citing works
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 217 · see citing works
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213 · see citing works
Interpretable machine learning
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2022) · Communications of the ACM · doi:10.1145/3546036 · Cited by 213 · see citing works
Net2Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks
Fong, R., & Vedaldi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00910 · Cited by 209 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · cited 1× · see citing works
“However, the extent to which these explanations accurately explain the model’s decision making is very unclear [139].”
BackgroundKnowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187 · see citing works
Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
Gao, M., Zhang, Z., Yu, G., Arık, S. Ö., Davis, L. S., & Pfister, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58607-2_30 · Cited by 180 · see citing works
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171 · see citing works
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · see citing works
Unsupervised learning by competing hidden units
Krotov, D., & Hopfield, J. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1820458116 · Cited by 165 · see citing works
Interpretable convolutional neural networks via feedforward design
Kuo, C. ‐. J., Zhang, M., Li, S., Duan, J., & Chen, Y. (2019) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2019.03.010 · Cited by 163 · see citing works
Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent SpacePreprint
Nguyen, A. Q., Clune, J., Bengio, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00005 · Cited by 159 · see citing works
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
Geva, M., Caciularu, A., Wang, K. I., & Goldberg, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.3 · Cited by 159 · see citing works
Understanding Neural Networks via Feature Visualization: A Survey
Nguyen, A., Yosinski, J., & Clune, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_4 · Cited by 157 · see citing works
A Neural Dirichlet Process Mixture Model for Task-Free Continual LearningPreprint
Lee, S., Ha, J., Zhang, D., & Kim, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00689 · Cited by 156 · see citing works
Evaluation of sentence embeddings in downstream and linguistic probing tasksPreprint
Perone, C. S., Silveira, R. P., & Paula, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06259 · Cited by 154 · see citing works
TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI SystemsPreprint
Guo, W., Wang, L., Xing, X., Du, M., & Song, D. X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01763 · Cited by 154 · see citing works
Do Adversarially Robust ImageNet Models Transfer Better?Preprint
Salman, H., Ilyas, A., Engstrom, L., Kapoor, A., & Mądry, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08489 · Cited by 153 · see citing works
Uncertainty-based Continual Learning with Adaptive RegularizationPreprint
Ahn, H., Cha, S., Lee, D.-G., & Moon, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11614 · Cited by 149 · see citing works
(2020) · neural information processing systems · Cited by 142 · see citing works
Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression
Li, Y., Lin, S., Zhang, B., Liu, J., Doermann, D., Wu, Y., Huang, F., & Ji, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00291 · Cited by 140 · see citing works
Probing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · doi:10.18653/v1/w16-2524 · Cited by 139 · see citing works
Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image PerturbationsPreprint
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., & DiCarlo, J. J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.16.154542 · Cited by 137 · see citing works
Individual differences among deep neural network models
Mehrer, J., Spoerer, C. J., Kriegeskorte, N., & Kietzmann, T. C. (2020) · Nature Communications · doi:10.1038/s41467-020-19632-w · Cited by 133 · see citing works
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126 · see citing works
The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype ClassificationPreprint
Kim, B., Rudin, C. D., & Shah, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01161 · Cited by 116 · see citing works
NILE : Natural Language Inference with Faithful Natural Language Explanations
Kumar, S., & Talukdar, P. P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.771 · Cited by 114 · see citing works
Nguyen, T., Raghu, M., & Kornblith, S. (2020) · arXiv · doi:10.48550/arxiv.2010.15327 · Cited by 112 · see citing works
Similarity and Matching of Neural Network Representations
Csiszárik, A., Kőrösi-Szabó, P., Matszangosz, Á. K., et al. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2110.14633 · Cited by 107 · see citing works
In-context Learning and Induction Heads
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.11895 · Cited by 104 · see citing works
Revisiting the Importance of Individual Units in CNNs via AblationPreprint
Zhou, B., Sun, Y., Bau, D., & Torralba, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02891 · Cited by 103 · see citing works
Image Synthesis with a Single (Robust) ClassifierPreprint
Santurkar, S., Tsipras, D., Tran, B., Ilyas, A., Engstrom, L., & Mądry, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09453 · Cited by 100 · see citing works
(2020) · neural information processing systems · Cited by 100 · see citing works
Interpretable Image Classification with Differentiable Prototypes Assignment
Rymarczyk, D., Struski, Ł., Górszczak, M., Lewandowska, K., Tabor, J., & Zieliński, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_21 · Cited by 100 · see citing works
The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement
Peebles, W., Peebles, J., Zhu, J.-Y., Efros, A. A., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_35 · Cited by 99 · see citing works
Interpret Neural Networks by Identifying Critical Data Routing Paths
Wang, Y., Su, H., Zhang, B., & Hu, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00928 · Cited by 97 · see citing works
Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networks
Gao, Y., Kim, Y., Doan, B. G., Zhang, Z., Zhang, G., Nepal, ., Ranasinghe, D. C., & Kim, H. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3055844 · Cited by 96 · see citing works
Functional Regularisation for Continual Learning with Gaussian ProcessesPreprint
Titsias, M. K., Schwarz, J., Matthews, A., Pascanu, R., & Teh, Y. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11356 · Cited by 92 · see citing works
A review of modularization techniques in artificial neural networks
Amer, M., & Maul, T. (2019) · Artificial Intelligence Review · doi:10.1007/s10462-019-09706-7 · Cited by 88 · see citing works
Interactive Visualization and Manipulation of Attention-based Neural Machine Translation
Lee, J., Shin, J.-H., & Kim, J. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-2021 · Cited by 87 · see citing works
Implicit Representations of Meaning in Neural Language Models
Li, B. Z., Nye, M., & Andreas, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.143 · Cited by 84 · see citing works
SPINE: SParse Interpretable Neural Embeddings
Subramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T., & Hovy, E. H. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11935 · Cited by 83 · see citing works
Adversarial Training for High-Stakes ReliabilityPreprint
Ziegler, D. M., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., Haas, D., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.01663 · Cited by 79 · see citing works
Neuron Shapley: Discovering the Responsible NeuronsPreprint
Ghorbani, A., & Zou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09815 · Cited by 77 · see citing works
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
Ravichander, A., Belinkov, Y., & Hovy, E. H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.295 · Cited by 75 · see citing works
Distributional vectors encode referential attributes
Gupta, A., Boleda, G., Baroni, M., & Padó, S. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1002 · Cited by 70 · see citing works
Liu, Y., Mondal, A., Chakraborty, A., Zuzak, M., Jacobsen, N., Xing, D., & Srivastava, A. (2020) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed48828.2020.9137011 · Cited by 70 · see citing works
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Zhao, M., Lin, T., Mi, F., Jaggi, M., & Schütze, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.174 · Cited by 67 · see citing works
On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors
Lucieri, A., Bajwa, M. N., Braun, S. A., Malik, M. I., Dengel, A. R., & Ahmed, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206946 · Cited by 67 · see citing works
Analyzing Individual Neurons in Pre-trained Language Models
Durrani, N., Sajjad, H., Dalvi, F., & Belinkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.395 · Cited by 60 · see citing works
A Closer Look at Rehearsal-Free Continual Learning *
Smith, J. S., Tian, J., Halbe, S., Hsu, Y.-C., & Kira, Z. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00239 · Cited by 60 · see citing works
Scaleable input gradient regularization for adversarial robustness
Finlay, C., & Oberman, A. M. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100017 · Cited by 58 · see citing works
Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
Jiang, Y., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1455 · Cited by 57 · see citing works
Self-explaining AI as an Alternative to Interpretable AI
Elton, D. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52152-3_10 · Cited by 57 · see citing works
Adversarial Robustness on In- and Out-Distribution Improves Explainability
Augustin, M., Meinke, A., & Hein, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58574-7_14 · Cited by 56 · see citing works
Discovering Latent Knowledge in Language Models Without Supervision
Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2212.03827 · Cited by 55 · see citing works
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S. M., Linzen, T., & Belinkov, Y. (2021) · doi:10.18653/v1/2021.acl-long.144 · Cited by 52 · see citing works
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.10652 · Cited by 51 · see citing works
Model-Based Deep LearningPreprint
Shlezinger, N., Whang, J., Eldar, Y. C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08405 · Cited by 50 · see citing works
QED: A Framework and Dataset for Explanations in Question Answering
Lamm, M., Palomaki, J., Alberti, C., Andor, D., Choi, E., Soares, L. B., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00398 · Cited by 47 · see citing works
Modular representation of layered neural networks
Watanabe, C., Hiramatsu, K., & Kashino, K. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.09.017 · Cited by 46 · see citing works
Investigating Transferability in Pretrained Language Models
Tamkin, A., Singh, T., Giovanardi, D., & Goodman, N. D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.125 · Cited by 46 · see citing works
Ortiz-Jiménez, G., Modas, A., Moosavi-Dezfooli, S.-M., & Pascal Frossard, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3050042 · Cited by 45 · see citing works
Causal abstractions of neural networks
Geiger, A., Lu, H., Icard, T., & Potts, C. (2021) · arXiv · doi:10.48550/arxiv.2106.02997 · Cited by 45 · see citing works
Neuron-level Interpretation of Deep NLP Models: A Survey
Sajjad, H., Durrani, N., & Dalvi, F. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00519 · Cited by 41 · see citing works
Dynamic Network Pruning with Interpretable Layerwise Channel Selection
Wang, Y., Zhang, X., Hu, X., Zhang, B., & Su, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6098 · Cited by 40 · see citing works
Extraction of an Explanatory Graph to Interpret a CNN
Zhang, Q., Wang, X., Cao, R., Wu, Y. N., Shi, F., & Zhu, S. C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992207 · Cited by 40 · see citing works
Proper Network Interpretability Helps Adversarial Robustness in ClassificationPreprint
Boopathy, A., Liu, S., Zhang, G., Liu, C., Chen, P., Chang, S., & Daniel, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14748 · Cited by 40 · see citing works
Deep neural network compression through interpretability-based filter pruning
Yao, K., Cao, F., Leung, Y., & Liang, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108056 · Cited by 40 · see citing works
Liu, S.-S., Li, T., Li, Z.-M., Srikumar, V., Pascucci, V., & Bremer, P. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2007 · Cited by 39 · see citing works
Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?Preprint
Kaur, S., Cohen, J., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08640 · Cited by 39 · see citing works
Towards Interpretable Deep Neural Networks by Leveraging Adversarial ExamplesPreprint
Dong, Y., Su, H., Zhu, J. J., & Bao, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05493 · Cited by 39 · see citing works
Post-hoc Concept Bottleneck ModelsPreprint
Yüksekgönül, M., Wang, M. H., & Zou, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15480 · Cited by 35 · see citing works
Interpreting Adversarial Examples by Activation Promotion and SuppressionPreprint
Xu, K., Liu, S., Zhang, G., Sun, M., Zhao, P., Fan, Q., Gan, C., & Lin, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02057 · Cited by 33 · see citing works
Learning Perceptually-Aligned Representations via Adversarial Robustness.Preprint
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Tran, B., & Mądry, A. (2019) · arXiv (Cornell University) · Cited by 33 · see citing works
Pitfalls to Avoid when Interpreting Machine Learning ModelsPreprint
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse‐Wentrup, M., & Bischl, B. (2020) · arXiv (Cornell University) · Cited by 33 · see citing works
Explainability and Adversarial Robustness for RNNs
Hartl, A., Bachl, M., Fabini, J., & Zseby, T. (2020) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice49289.2020.00030 · Cited by 32 · see citing works
An Overview of Early Vision in InceptionV1
Olah, C., Cammarata, N., Schubert, L., et al. (2020) · Distill · doi:10.23915/distill.00024.002 · Cited by 32 · see citing works
An Interpretability Illusion for BERTPreprint
Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F. B., & WATTENBERG, M. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07143 · Cited by 32 · see citing works
Why the Failure? How Adversarial Examples Can Provide Insights for Interpretable Machine Learning
Tomsett, R., Widdicombe, A., Xing, T., Chakraborty, S., Julier, S., Gurram, P. K., Rao, R., & Srivastava, M. B. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455710 · Cited by 31 · see citing works
Interpretability Beyond Classification Output: Semantic Bottleneck NetworksPreprint
Losch, M., Fritz, M., & Schiele, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.10882 · Cited by 31 · see citing works
An Empirical Study on the Relation Between Network Interpretability and Adversarial Robustness
Noack, A., Ahern, I., Dou, D., & Li, B. (2021) · SN Computer Science · doi:10.1007/s42979-020-00390-x · Cited by 31 · see citing works
Lateral Inhibition-Inspired Convolutional Neural Network for Visual Attention and Saliency Detection
Cao, C., Huang, Y., Wang, Z., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12238 · Cited by 28 · see citing works
Analysing Neural Network Topologies: a Game Theoretic Approach
Stier, J., Gianini, G., Granitzer, M., & Ziegler, K. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.07.257 · Cited by 28 · see citing works
An overview of 11 proposals for building safe advanced AIPreprint
Hubinger, E. (2020) · arXiv.org · doi:10.48550/arxiv.2012.07532 · Cited by 28 · see citing works
Is Sparse Attention more Interpretable?
Meister, C., Lazov, S., Augenstein, I., & Cotterell, R. (2021) · doi:10.18653/v1/2021.acl-short.17 · Cited by 28 · see citing works
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight MasksPreprint
Csordás, R., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02066 · Cited by 27 · see citing works
Bridging Adversarial Robustness and Gradient InterpretabilityPreprint
Kim, B., Seo, J., & Jeon, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.11626 · Cited by 26 · see citing works
Revisiting Model Stitching to Compare Neural Representations
Bansal, Y., Nakkiran, P., & Barak, B. (2021) · arXiv · doi:10.48550/arxiv.2106.07682 · Cited by 26 · see citing works
Linear Adversarial Concept ErasurePreprint
Ravfogel, S., Twiton, M., Goldberg, Y., & Cotterell, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.12091 · Cited by 26 · see citing works
Explaining AI-Based Decision Support Systems Using Concept Localization Maps
Lucieri, A., Bajwa, M. N., Dengel, A. R., et al. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-63820-7_21 · Cited by 24 · see citing works
Understanding community structure in layered neural networks
Watanabe, C., Hiramatsu, K., & Kashino, K. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.020 · Cited by 20 · see citing works
Knowledge Consistency between Neural Networks and BeyondPreprint
Liang, R., Li, T., Li, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01581 · Cited by 20 · see citing works
Naturally Occurring Equivariance in Neural Networks
Olah, C., Cammarata, N., Voss, C., et al. (2020) · Distill · doi:10.23915/distill.00024.004 · Cited by 20 · see citing works
A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron AttributionsPreprint
Lundström, D., Huang, T., & Razaviyayn, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.11912 · Cited by 20 · see citing works
Cammarata, N., Goh, G., Carter, S., Schubert, L., Petrov, M., & Olah, C. (2020) · Distill · doi:10.23915/distill.00024.003 · Cited by 20 · see citing works
Promises and Pitfalls of Black-Box Concept Learning ModelsPreprint
Mahinpei, A., Clark, J., Lage, I., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13314 · Cited by 19 · see citing works
Natural Language Descriptions of Deep Visual FeaturesPreprint
Hernandez, E. J., Schwettmann, S., Bau, D. A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11114 · Cited by 17 · see citing works
Explaining Deep Neural Networks using Unsupervised ClusteringPreprint
Liu, Y.-H., & Arik, S. O. (2020) · arXiv.org · doi:10.48550/arxiv.2007.07477 · Cited by 16 · see citing works
Schubert, L., Voss, C., & Olah, C. (2021) · Distill · doi:10.23915/distill.00024.005 · Cited by 16 · see citing works
3DB: A Framework for Debugging Computer Vision ModelsPreprint
Leclerc, G., Salman, H., Ilyas, A., Vemprala, S. H., Engstrom, L., Vineet, V., Xiao, K., Zhang, P., Santurkar, S., Yang, G., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03805 · Cited by 16 · see citing works
On Saliency Maps and Adversarial Robustness
Mangla, P., Singh, V., & Balasubramanian, V. N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67661-2_17 · Cited by 15 · see citing works
Optimizing Mode Connectivity via Neuron AlignmentPreprint
Tatro, N. J., Chen, P., Das, P., Melnyk, I., Sattigeri, P., & Lai, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02439 · Cited by 15 · see citing works
Is a Modular Architecture Enough?Preprint
Mittal, S., Bengio, Y., & Lajoie, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.02713 · Cited by 15 · see citing works
Similarity of Neural Networks with GradientsPreprint
Tang, S., Maddox, W. J., Dickens, C., Diethe, T., & Damianou, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11498 · Cited by 14 · see citing works
Lepori, M. A., & McCoy, R. T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.325 · Cited by 14 · see citing works
A Survey of Neural Trojan Attacks and Defenses in Deep LearningPreprint
Wang, J., Hassan, G. M., & Akhtar, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.07183 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Red-teaming can also be complemented by interpretability techniques [Rastogi et al., 2023, Räuker et al., 2023], especially for purposes of identifying adversarial inputs [Ziegler et al., 2022, Casper et al., 2023c,a] or anomalous inputs [Pang et al., 2021].”
BackgroundSelf-citationIdentifying indicators of consciousness in AI systems
Butlin, P., Long, R., Bayne, T., Bengio, Y., Birch, J., Chalmers, D., Constant, A., Deane, G., Elmoznino, E., Fleming, S. M., et al. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.10.011 · Cited by 56 · see citing works
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36 · see citing works
Exploring the Robustness of In-Context Learning with Noisy Labels
Cheng, C., Yu, X., Wen, H., Sun, J., Yue, G., Zhang, Y., & Wei, Z. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889281 · Cited by 11 · see citing works
Colas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
Algorithmic Collusion by Large Language Models
Fish, S., Gonczarowski, Y. A., & Shorrer, R. I. (2024) · arXiv · doi:10.48550/arxiv.2404.00806 · Cited by 10 · see citing works
- Authors
- Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D.
- Affiliations
- Epoch
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)
- Publisher
- IEEE
- Published
- 2023-02
- Pages
- 464-483
- OpenAlex
- W4378976798
- Semantic Scholar
- 2c709ef6186bd607494a3344c903552ea500e449
- Cited by
- 122
- References
- 248
- Citations by year
- 2026: 19 · 2025: 52 · 2024: 37 · 2023: 13 · 2022: 1
- FWCI
- 21.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- deep neural network interpretability, mechanistic interpretability, post-hoc explanation, neuronal activation, subnetwork analysis, latent representations, adversarial robustness, continual learning, network modularity, network compression, human visual system, trustworthy AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
