The DOI matches its DataCite record; 7 of the 7 references it cites were found in a registry or an index.
An Interpretability Illusion for BERT
Open accessAbstract
We describe an "interpretability illusion" that arises when analyzing the BERT model. Activations of individual neurons in the network may spuriously appear to encode a single, simple concept, when in fact they are encoding something far more complex. The same effect holds for linear combinations of activations. We trace the source of this illusion to geometric properties of BERT's embedding space as well as the fact that common text corpora represent only narrow slices of possible English sentences. We provide a taxonomy of model-learned concepts and discuss methodological implications for interpretability research, especially the importance of testing hypotheses on multiple data sets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bias in data sets can create misleading explanations even when top activating examples are selected from real data sets, as opposed to synthetically created data.
“Bolukbasi et al. (2021) show that bias in data sets can create misleading explanations even when top activating examples are selected from real data sets, as opposed to synthetically created data.”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Bolukbasi et al., 2021
The highly activating data set examples approach has the potential for 'interpretability illusions'.
“Another issue with this approach is the potential for 'interpretability illusions'.”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Bolukbasi et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Self-citationRaghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05806 · Cited by 335
Zoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288
Unsupervised Domain Clusters in Pretrained Language Models
Aharoni, R., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.692 · Cited by 181
Flagged first, then by citations
Research works printed but not linked: 19
- Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
- Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neu- ral probabilistic language model. The journal of machine learning research, 3:1137-1155, 2003.
- Dalvi, F., Durrani, N., and Sajjad, H. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pp. 6309-6317, 2019.
- Durrani, N., Dalvi, F., Sajjad, H., and Belinkov, Y. Ana- lyzing individual neurons in pre-trained language mod- els. 2020. URL https://arxiv.org/pdf/2010. 02695.pdf.
- Kaneko, M. and Bollegala, D. Gender-preserving de- biasing for pre-trained word embeddings. CoRR, abs/1906.00742, 2019. URL http://arxiv.org/ abs/1906.00742. · arXiv
- Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677, 2018.
- Li, J., Chen, X., Hovy, E. H., and Jurafsky, D. Visualiz- ing and understanding neural models in NLP. CoRR, abs/1506.01066, 2015. URL http://arxiv.org/ abs/1506.01066. · arXiv
- Manning, C., Hewitt, J., Clark, K., Khandelwal, U., and Levy, O. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences of the United States of America, 2020.
- Manzini, T., Lim, Y. C., Tsvetkov, Y., and Black, A. W. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. CoRR, abs/1904.04047, 2019. URL http://arxiv.org/ abs/1904.04047. · arXiv
- Na, S., Choe, Y. J., Lee, D.-H., and Kim, G. Discovery of natural language concepts in individual units of cnns. 2019. In the Proceedings of ICLR.
- Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Ad- vances in Neural Information Processing Systems, pp. 3387-3395, 2016.
- Olah, C., Mordvintsev, A., and Schubert, L. Feature vi- sualization. Distill, 2017. doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization. · DOI · Paper page
- Poerner, N., Roth, B., and Schütze, H. Interpretable textual neuron representations for NLP. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NLP, pp. 325-327, Brussels, Belgium, November 2018. Associ- ation for Computational Linguistics. doi: 10.18653/ v1/W18-5437. URL https://www.aclweb.org/ anthology/W18-5437. · DOI
- Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 3982--3992, 2019.
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Er- han, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. In International Confer- ence on Learning Representations, 2014. URL http: //arxiv.org/abs/1312.6199.
- Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S., Das, D., and Pavlick, E. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019.
- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and anal- ysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
- Zeiler, M. D. and Fergus, R. Visualizing and understand- ing convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
- Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Tor- ralba, A. Object detectors emerge in deep scene cnn. In International Conference on Learning Representations, 2015.
Cited by
Sorted by citations
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Self-citationEight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
“Often, ad-hoc techniques that at first seem to provide insight into the behavior of an LLM are later found to be severely misleading (Feng et al., 2018; Jain & Wallace, 2019; Bolukbasi et al., 2021; Wang et al., 2022).”
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
Improving Dictionary Learning with Gated Sparse Autoencoders
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., & Nanda, N. (2024) · arXiv · doi:10.48550/arxiv.2404.16014 · Cited by 3
“Classic approaches, like interpreting neurons (i.e. directions in the standard basis) are insufficient, as many are polysemantic and tend to activate for a range of different seemingly unrelated concepts (Bolukbasi et al., 2021; Elhage et al., 2022a,b, Empirical Phenomena).”
Background
Metadata
- Authors
- Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F., & Wattenberg, M.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.07143
- OpenAlex
- W3152624744
- Cited by
- 113
- References
- 7
- Citations by year
- 2026: 30 · 2025: 30 · 2024: 22 · 2023: 21 · 2022: 8 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- BERT, neuronal activation, embedding space geometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
