Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Open accessThe DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Abstract
A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To improve the causal relevance of probe vectors, one approach is to use counterfactual data, which involves intervening on the concept of interest (for instance, observing the resulting output if the dog in an image was changed to a cat) (Elazar et al., 2021;Mueller, 2024;Geiger et al., 2024b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Early studies showed that it is possible to use linear [51,52] or nonlinear [53] updates to a model’s hidden state in order to manipulate the model’s representation of grammatical role”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“In the past few years, multiple studies have suggested that these apparently symbolic representations do indeed play a causal role in models’ subsequent predictions, each study using a different method to arrive at this conclusion.”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“However, training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al., 2021;Geiger et al., 2024b;Elazar et al., 2021;Belinkov, 2022b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“The results up to this section treat all of BERT’s ‘Transformer blocks’ (Vaswani et al., 2017) as the encoding function and the embedding matrix”
BackgroundScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“(Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b)) spiked interest in understanding how these models work and what is”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · cited 1×
“One commonly used method for information removal relies on adversarial training through the gradient reversal layer technique (Ganin et al., 2016).”
MethodModeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · cited 1×
“Voita and Titov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts.”
The Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,781
Research works printed but not linked: 5
- Giulianelli, 2018, Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information, 240, 10.18653/v1/W18-5426 · DOI
- Hewitt, 2019, A structural probe for finding syntax in word representations, 4129
- Hupkes, 2018, Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure, Journal of Artificial Intelligence Research, 61, 907, 10.1613/jair.1.11196 · DOI · Paper page
- Reif, 2019, Visualizing and measuring the geometry of bert, 8592
- Weischedel, 2013, Ontonotes release 5.0 ldc2013t19, Linguistic Data Consortium, Philadelphia, PA, 23
Cited by
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2022) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 249
“Figure 1 Components constituting the probing classifiers framework. stable form by several groups who studied sentence embeddings (Ettinger, Elgohary, and Resnik 2016; Adi et al. 2017; Conneau et al. 2018) and recurrent/recursive neural networks (Belinkov et al. 2017a; Hupkes, Veldhoen, and Zuidema 2018).3 The same id…”
BackgroundInfluentialPretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“…in analyzing how well PLMs (Rogers et al., 2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Man-ning, 2019; Tenney et al., 2019; Elazar et al., 2021), commonsense (Forbes et al., 2019; Da and Kasai, 2019; Zhang et al., 2020) and reasoning (Talmor et al., 2020; Kassner et al.,…”
BackgroundResultSelf-citationCausal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
“To overcome the counterfactual generation problem, another class of approaches proposes to manipulate the representation of the text and not the text itself (Feder et al., 2021; Elazar et al., 2021; Ravfogel et al., 2021).”
BackgroundBlack-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
“Methods for this are imperfect and still an active area of research [14, 84, 257], but interpretations like these offer a potentially powerful way to identify whether a model represents specific concepts.”
BackgroundTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 120
“Two pieces of concurrent work, Amnesic Probing (Elazar et al. 2021) and CausalLM (Feder et al. 2021), are representative examples.”
BackgroundMethodInfluentialEditing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180
“certain behaviour has been studied in the context of research in interpreting neural networks (Lakretz et al., 2019; Vig et al., 2020; Elazar et al., 2021; Csordás et al., 2021).”
BackgroundA Primer on the Inner Workings of Transformer-based Language ModelsPreprint
Ferrando, J., Sarti, G., Bisazza, A., & Costa‐jussà, M. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.00208 · Cited by 218
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
“In a subsequent paper, [83] aimed to address this problem by pairing probing with experiments that manipulated the data in order to analyze the causal influence of perturbations on performance.”
BackgroundWarstadt, A., Zhang, Y., Li, X., Liu, H., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 98
Investigating Gender Bias in BERT
Bhardwaj, R., Majumder, N., & Poria, S. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09881-2 · Cited by 112
“Amnesic probing [19] observes the change in the prediction of the classifier caused by the removal of property Z from the representation of the input.”
BackgroundWhat Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
LEACE: Perfect linear concept erasure in closed form
Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., & Biderman, S. (2023) · neural information processing systems · doi:10.52202/075280-2884 · Cited by 121
Self-citationSymbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
Explainable Information Retrieval: A SurveyPreprint
Anand, A., Lyu, L., Idahl, M., Wang, Y., Wallat, J., & Zhang, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.02405 · Cited by 105
“To resolve this, recent studies borrow ideas from causality research to understand whether a specific concept is utilized during the inference using counterfactual representations, where the concept is voided [34, 66].”
BackgroundWhen Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85
Hanna, M., Liu, O., & Variengien, A. (2023) · neural information processing systems · doi:10.52202/075280-3322 · Cited by 420
Hanna, M., Liu, O., & Variengien, A. (2023) · arXiv · doi:10.48550/arxiv.2305.00586 · Cited by 84
“However, it has significant pitfalls: most crucially, probes can sometimes extract information from model representations that is irrelevant to model behavior [13, 40, 22].”
BackgroundContrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83
“…approach to causality in our work follows several recent works in NLP (Giu-lianelli et al., 2018; Meyes et al., 2020; Vig et al., 2020; Elazar et al., 2021; Feder et al., 2021), and is justified by accumulating empirical evidence for the inability to draw causal interpretation from…”
BackgroundMethodInfluentialSelf-citationCan Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation
Dankers, V., Lucas, C., & Titov, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.252 · Cited by 139
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 103
“However, across models (even those with the same architecture but just trained on different data), results can differ quite drastically [50], which shows differences between pre-trained and fine-tuned BERT models.”
BackgroundInfluentialFirst Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
Müller, B., Elazar, Y., Sagot, B., & Seddah, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.189 · Cited by 113
Self-citationWu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
(2021) · doi:10.18653/v1/2021.conll-1.15 · Cited by 51
Self-citationNeuron-level Interpretation of Deep NLP Models: A Survey
Sajjad, H., Durrani, N., & Dalvi, F. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00519 · Cited by 114
“• Although considerable work has been done on analyzing how knowledge is encoded within the learned representations, the question whether it is used by the model during prediction is a less explored area (Feder et al., 2021; Elazar et al., 2021).”
BackgroundProbing for the Usage of Grammatical Number
Lasri, K., Pimentel, T., Lenci, A., Poibeau, T., & Cotterell, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.603 · Cited by 89
“We thus run a control experiment proposed by Elazar et al. (2021), removing random directions at each layer (as opposed to the specific directions found by our amnesic probes).”
BackgroundMethodInfluentialSlobodkin, A., Goldman, O., Caciularu, A., Dagan, I., & Ravfogel, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.220 · Cited by 143
Self-citationCausal abstractions of neural networks
Geiger, A., Lu, H., Icard, T., & Potts, C. (2021) · arXiv · doi:10.48550/arxiv.2106.02997 · Cited by 45
“Probes are fundamentally unable to directly measure causal properties of neural representations, and Ravichander et al. [21], Elazar et al. [9], and Geiger et al. [11] have argued that probes are limited in their ability to provide even indirect evidence of causal properties.”
BackgroundLarge language models for whole-learner support: opportunities and challenges
Mannekote, A., Davies, A., Pinto, J. D. G., Zhang, S., Olds, D., Schroeder, N. L., Lehman, B. A., Zapata‐Rivera, D., & Zhai, C. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1460364 · Cited by 53
“For instance, current interpretability methods can detect what latent representations are used by models in producing a particular output (Elazar et al., 2021; Belinkov, 2022; Davies et al., 2023) and characterize how these representations are leveraged in producing particular behaviors (Elhage et…”
MethodHow Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
Li, S., Li, X., Shang, L., Dong, Z., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.136 · Cited by 30
Language Models as Models of Language
Millière, R. (2026) · Oxford University Press eBooks · doi:10.1093/oso/9780198879640.003.0025 · Cited by 44
Operationalising Representation in Natural Language Processing
Harding, J. (2023) · The British Journal for the Philosophy of Science · doi:10.1086/728685 · Cited by 53
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Hou, Y., Li, J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.299 · Cited by 90
Liu, K., Casper, S., Hadfield-Menell, D., & Andreas, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.291 · Cited by 78
“…being probed for, and an unsuccessful probe does not indicate that the LM does not use the feature being probed for (Ravichander et al., 2020; Elazar et al., 2021; Belinkov, 2022 regression objective) to classify (question, answer) pairs as correct or incorrect using the training split of…”
MethodOn the Pitfalls of Analyzing Individual Neurons in Language ModelsPreprint
Antverg, O., & Belinkov, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07483 · Cited by 48
“This approach is more of a causal one, inspired by similar prior work (Giulianelli et al., 2018; Elazar et al., 2021; Feder et al., 2020; Lovering et al., 2021; Ravfogel et al., 2021).”
BackgroundResultThe Low-Dimensional Linear Geometry of Contextualized Word Representations
Hernandez, E. J., & Andreas, J. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.7 · Cited by 61
Eliciting Latent Predictions from Transformers with the Tuned Lens
Belrose, N., Ostrovsky, I., McKinney, L., Furman, Z., Smith, L., Halawi, D., Biderman, S., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2303.08112 · Cited by 27
“Amnesic probing (Elazar et al., 2021) provides one way to do this—it seeks a direction whose erasure maximally degrades a model’s accuracy.”
BackgroundFact Checking with Insufficient Evidence
Atanasova, P., Simonsen, J. G., Lioma, C., & Augenstein, I. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00486 · Cited by 64
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26
Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios
Li, J., Yu, L., & Ettinger, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.70 · Cited by 49
Alajrami, A., & Αλέτρας, Ν. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.16 · Cited by 32
BERTologiCoMix: How does Code-Mixing interact with Multilingual BERT?
Santy, S., Srinivasan, A., & Choudhury, M. (2021) · Cited by 57
Sparse Interventions in Language Models with Differentiable Masking
Cao, N. D., Schmid, L., Hupkes, D., & Titov, I. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.2 · Cited by 41
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Perlitz, Y., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 56
“Other methodological work addressed the validity of obtained results with control tasks (Hewitt and Liang, 2019) or from an information theory perspective (Voita and Titov, 2020; Pimentel et al., 2020), or studied causal effects (Elazar et al., 2021).”
Adversarial Concept Erasure in Kernel Space
Ravfogel, S., Vargas, F., Goldberg, Y., & Cotterell, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.405 · Cited by 76
Self-citationLanguage Models Use Monotonicity to Assess NPI Licensing
Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., & Steinert‐Threlkeld, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.439 · Cited by 48
Data Efficient Masked Language Modeling for Vision and Language
Bitton, Y., Elhadad, M., Stanovsky, G., & Schwartz, R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.259 · Cited by 20
Causal Distillation for Language Models
Wu, Z., Geiger, A., Rozner, J., Kreiss, E., Lu, H., Icard, T., Potts, C., & Goodman, N. D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.318 · Cited by 31
Diagnosing AI Explanation Methods with Folk Concepts of Behavior
Jacovi, A., Bastings, J., Gehrmann, S., Goldberg, Y., & Filippova, K. (2023) · Journal of Artificial Intelligence Research · doi:10.1145/3593013.3593993 · Cited by 30
Self-citationPredicting Fine-Tuning Performance with Probing
Zhu, Z., Shahtalebi, S., & Rudzicz, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.793 · Cited by 22
Enhancing deep neural networks with morphological information
Klemen, M., Krsnik, L., & Robnik‐Šikonja, M. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000080 · Cited by 27
Schrödinger's tree—On syntax and neural language models
Kulmizev, A., & Nivre, J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796788 · Cited by 34
“and Liang, 2019), or causal intervention (Elazar et al., 2021).”
Log-linear Guardedness and its Implications
Ravfogel, S., Goldberg, Y., & Cotterell, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.523 · Cited by 32
“It was previously implied (Rav-fogel et al., 2020; Elazar et al., 2021) that linear concept erasure prevents information leakage about Z through the labeling of a log-linear classifier (cid:98) Y , i.e., it was assumed that Theorem 3.2 in § 3.2 can be generalized to the multiclass case.”
BackgroundInfluentialSelf-citationLanguage model and its interpretability in biomedicine: A scoping review
Lyu, D., Wang, X., Chen, Y., & Wang, F. (2024) · iScience · doi:10.1016/j.isci.2024.109334 · Cited by 26
“Counterfactual intervention can also happen in the feature representations in the model, such as the work of Amnesic Probing 148 and CausalLM.”
MethodProbing BERT for Ranking Abilities
Wallat, J., Beringer, F., Anand, A., & Anand, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_17 · Cited by 27
Decoding Prompt Syntax: Analysing its Impact on Knowledge Retrieval in Large Language Models
Linzbach, S., Tressel, T., Kallmeyer, L., Dietze, S., & Jabeen, H. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587655 · Cited by 25
Probing artificial neural networks: insights from neurosciencePreprint
Ivanova, A. A., Hewitt, J., & Zaslavsky, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08197 · Cited by 22
“Researchers can leverage the ability of probes to identify ANN activity patterns that are linked to a given feature set, and then eliminate (Elazar et al., 2021) or alter (Giulianelli et al., 2018) those patterns to determine whether they affect the system’s behavior in expected ways.”
BackgroundMethodWhat if This Modified That? Syntactic Interventions with Counterfactual Embeddings
Tucker, M., Qian, P., & Levy, R. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.76 · Cited by 16
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
Timkey, W., & Linzen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.582 · Cited by 43
Finding Neurons in a Haystack: Case Studies with Sparse Probing
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., & Bertsimas, D. (2023) · arXiv · doi:10.48550/arxiv.2305.01610 · Cited by 15
“While probing in general has a number of limitations [14, 28, 15, 29, 30], our work seeks to address, at least partially, the shortcomings most associated with probing individual neurons [13].”
BackgroundInfluentialNaturalistic Causal Probing for Morpho-Syntax
Amini, A., Pimentel, T., Meister, C., & Cotterell, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00554 · Cited by 36
“1 For instance, probing a pre-trained model for grammatical gender can only tell us whether information about gender is present in the representations,2 it cannot, however, tell us how or if the model actually uses information about gender in its predictions (Ravichander et al., 2021; Elazar et al., 2021; Ravfogel et…”
BackgroundA Reply to Makelov et al. (2023)'s "Interpretability Illusion" ArgumentsPreprint
Wu, Z., Geiger, A., Huang, J., Arora, A., Icard, T., Potts, C., & Goodman, N. D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.12631 · Cited by 20
“…presuppose that neurons will play unique causal roles (Vig et al., 2020; Csord ´ as et al., 2021; Feder et al., 2021; Ravfogel et al., 2020; Elazar et al., 2020; De Cao et al., 2022; Abraham et al., 2023; Chan et al., 2022; Wang et al., 2023) Thus, these methods will inevitably fail to…”
MethodInfluentialMediators in Determining what Processing BERT Performs First
Slobodkin, A., Choshen, L., & Abend, O. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.8 · Cited by 13
“Causal considerations in interpreting probing results were also emphasized by several recent works (e.g., Kaushik et al., 2020; Vig et al., 2020; Elazar et al., 2021).”
ResultAdversarial Scrubbing of Demographic Information for Text Classification
Chowdhury, S. B. R., Ghosh, S., Li, Y., Oliva, J. B., Srivastava, S., & Chaturvedi, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.43 · Cited by 21
“Prior works have shown that intermediate representations in a deep learning model encode demographic associations of the author or person being spoken about (Blod-gett et al., 2016; Elazar and Goldberg, 2018; Elazar et al., 2021).”
BackgroundUnit Testing for Concepts in Neural Networks
Lovering, C., & Pavlick, E. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00514 · Cited by 12
Probing LLMs for Joint Encoding of Linguistic Categories
Starace, G., Papakostas, K., Choenni, R., Panagiotopoulos, A., Rosati, M., Leidinger, A., & Shutova, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.476 · Cited by 30
Arithmetic with language models: From memorization to computation
Maltoni, D., & Ferrara, M. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106550 · Cited by 21
“So to further strengthen our hypothesis we applied an amnesic probing technique (Elazar et al., 2021) and proved that, upon removal of value information from the embeddings, the LM is no longer capable of performing the right computation.”
BackgroundMethodLi, B., Wisniewski, G., & Crabbé, B. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00531 · Cited by 20
“lated to Elazar et al. (2021), who proposed to erase the “part-of-speech property” encoded in a model representation to measure how important this information is for word prediction.”
BackgroundUnsupervised Distillation of Syntactic Information from Contextualized Word Representations
Ravfogel, S., Elazar, Y., Goldberger, J., & Goldberg, Y. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.9 · Cited by 16
Self-citationEger, S., Daxenberger, J., & Gurevych, I. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.8 · Cited by 15
Mueller, A. (2024) · arXiv · doi:10.48550/arxiv.2407.04690 · Cited by 12
“…problems affect methods that derive projections to erase or guard particular concepts in latent space (Belrose et al., 2023; Ravfogel et al., 2020; Elazar et al., 2021), or methods that leverage gradients from parametric classifiers to update the model at specific token positions (e.g.,…”
BackgroundLearning Fair Representations via Rate-Distortion Maximization
Chowdhury, S. B. R., & Chaturvedi, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00512 · Cited by 20
“Fairness can be achieved by ensuring demographic information does not get encoded in representations used by these predictive models (Blodgett et al., 2016; Elazar and Goldberg, 2018; Elazar et al., 2021).”
BackgroundProbing for Incremental Parse States in Autoregressive Language Models
Eisape, T., Gangireddy, V., Lévy, R., & Kim, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.203 · Cited by 31
Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords
Karidi, T., Zhou, Y., Schneider, N., Abend, O., & Srikumar, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.806 · Cited by 18
“Elazar et al. (2021) argue that while probing methods might show that certain linguistic properties exist in a representation, they do not reveal how and if this information is being used by the probing model.”
BackgroundData-Efficient Pretraining via Contrastive Self-SupervisionPreprint
Rethmeier, N., & Augenstein, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01061 · Cited by 11
Competence-Based Analysis of Language ModelsPreprint
Davies, A., Jiang, J., & Zhai, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.00333 · Cited by 12
“Hypernym Prompting The performance of BERT-like models on lexical inference tasks such as hypernym prediction is known to be highly variable under small changes to prompts (Hanna and Mareˇcek, 2021; Ravichander et al., 2020; Ettinger, 2020; Elazar et al., 2021a).”
InfluentialChintam, A., Beloch, R., Zuidema, W. H., Hanna, M., & van der Wal, O. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.29 · Cited by 29
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes
Tucker, M., Eisape, T., Qian, P., Lévy, R., & Shah, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.394 · Cited by 15
On the Transformation of Latent Space in Fine-Tuned NLP Models
Durrani, N., Sajjad, H., Dalvi, F., & Alam, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.97 · Cited by 20
Nejadgholi, I., Balkır, E., Fraser, K., & Kiritchenko, S. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.18 · Cited by 13
Investigating Negation in Pre-trained Vision-and-language Models
Dobreva, R., & Keller, F. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.27 · Cited by 10
Anand, S., Lepori, M. A., Merullo, J., & Pavlick, E. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.00053 · Cited by 11
Gumbel Counterfactual Generation From Language Models
Ravfogel, S., Svete, A., Snæbjarnarson, V., & Cotterell, R. (2024) · International Conference on Learning Representations · doi:10.3929/ethz-c-000790212 · Cited by 16
Self-citationWhat’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models
Geva, M., Katz, U., Ben-Arie, A., & Berant, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.646 · Cited by 13
“Moreover, we perform interventions (Woodward, 2005; Elazar et al., 2021), where we modify the representation based on the output of the extractive head, and show this leads to predictable changes in the behaviour of the generative head.”
MethodOn the evolution of syntactic information encoded by BERT’s contextualized representations
Pérez-Mayos, L., Carlini, R., Ballesteros, M., & Wanner, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.191 · Cited by 14
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Mikhailov, V., Serikov, O., & Artemova, E. (2021) · doi:10.18653/v1/2021.sigtyp-1.10 · Cited by 13
Haghighatkhah, P., Fokkens, A. S., Sommerauer, P., Speckmann, B., & Verbeek, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.575 · Cited by 20
Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering
Chaturvedi, A., Bhar, S., Saha, S., Garain, U., & Asher, N. M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00493 · Cited by 16
“Elazar et al. (2021b) explores an intervention-based model analysis, called amnesic probing .”
BackgroundInfluentialInterpretable modular knowledge reasoning for machine reading comprehension
Ren, M., Huang, H., & Gao, Y. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06975-2 · Cited by 8
“First, for structural analysis, it is a common practice to design probe classifiers to analyze model characteristics, such as syntactic structural features in machine translation task [16, 28] and semantic features in embedding learning task [1, 67].”
MethodOn Linear Representations and Pretraining Data Frequency in Language ModelsPreprint
Merullo, J., Smith, N. A., Wiegreffe, S., & Elazar, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.12459 · Cited by 17
Self-citationKunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 8
Analyzing Gender Representation in Multilingual Models
Gonen, H., Ravfogel, S., & Goldberg, Y. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.8 · Cited by 16
Self-citationCausal interventions expose implicit situation models for commonsense language understanding
Yamakoshi, T., McClelland, J. L., Goldberg, A. E., & Hawkins, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.839 · Cited by 14
Lexical Generalization Improves with Larger Models and Longer Training
Bandel, E., Goldberg, Y., & Elazar, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.323 · Cited by 12
Self-citationUncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning
Davis, F., & van Schijndel, M. (2021) · doi:10.18653/v1/2021.acl-long.93 · Cited by 12
“The goal of such modification in our work is quite similar both to work which attempts to remove targeted linguistic knowledge in model representations (e.g., Ravfogel et al., 2020; Elazar et al., 2021) and to work which”
MethodResultThe Architectural Bottleneck Principle
Pimentel, T., Valvoda, J., Stoehr, N., & Cotterell, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.788 · Cited by 10
What if This Modified That? Syntactic Interventions via Counterfactual EmbeddingsPreprint
Tucker, M., Qian, P., & Lévy, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14002 · Cited by 7
“Our work is most closely related to that of Elazar et al. (2020), who, as in this work, used probes to generate counterfactual embeddings within a network.”
BackgroundResultNeural Networks as Cognitive Models of the Processing of Syntactic Constraints
Arehalli, S., & Linzen, T. (2024) · Open Mind · doi:10.1162/opmi_a_00137 · Cited by 11
Metadata
- Authors
- Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Corresponding author
- Elazar, Yanai
- Ravfogel, Shauli
- Jacovi, Alon
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021-03-11
- Volume
- 9
- Pages
- 160-175
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3159900299
- Cited by
- 126
- References
- 48
- Citations by year
- 2026: 13 · 2025: 4 · 2024: 11 · 2023: 35 · 2022: 33 · 2021: 28
- FWCI
- 9.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- probing, causal intervention, neural representations, BERT, word prediction, parts of speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.