Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Open accessThe DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Abstract
A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To improve the causal relevance of probe vectors, one approach is to use counterfactual data, which involves intervening on the concept of interest (for instance, observing the resulting output if the dog in an image was changed to a cat) (Elazar et al., 2021;Mueller, 2024;Geiger et al., 2024b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Early studies showed that it is possible to use linear [51,52] or nonlinear [53] updates to a model’s hidden state in order to manipulate the model’s representation of grammatical role”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“In the past few years, multiple studies have suggested that these apparently symbolic representations do indeed play a causal role in models’ subsequent predictions, each study using a different method to arrive at this conclusion.”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“However, training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al., 2021;Geiger et al., 2024b;Elazar et al., 2021;Belinkov, 2022b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“The results up to this section treat all of BERT’s ‘Transformer blocks’ (Vaswani et al., 2017) as the encoding function and the embedding matrix”
BackgroundScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“(Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b)) spiked interest in understanding how these models work and what is”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · cited 1×
“One commonly used method for information removal relies on adversarial training through the gradient reversal layer technique (Ganin et al., 2016).”
MethodModeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · cited 1×
“Voita and Titov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts.”
The Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,781
Research works printed but not linked: 5
- Giulianelli, 2018, Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information, 240, 10.18653/v1/W18-5426 · DOI
- Hewitt, 2019, A structural probe for finding syntax in word representations, 4129
- Hupkes, 2018, Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure, Journal of Artificial Intelligence Research, 61, 907, 10.1613/jair.1.11196 · DOI · Paper page
- Reif, 2019, Visualizing and measuring the geometry of bert, 8592
- Weischedel, 2013, Ontonotes release 5.0 ldc2013t19, Linguistic Data Consortium, Philadelphia, PA, 23
Cited by
Items 201–214 of 214
zadeh, A. E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.32670
“Elazar et al. [7] move from decoding to intervention, Belrose et al. [2] give a closed form for that removal, and Geiger et al. [8] arrive independently at causal abstraction.”
PhysioTRACE: Provenance-Aware Stress Tests for Physiological Foundation ModelsPreprint
Mussabayeva, A., Aimoldin, A., Oullier, O., Liu, X., & Zhang, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34466
“The operation builds on linear concept-erasure methods such as iterative null-space projection (INLP) (Ravfogel et al., 2020), Amnesic Probing (Elazar et al., 2021), and least-squares concept erasure (LEACE) (Belrose et al., 2023); PhysioTRACE contributes acquisition-specific paired or conditional localization and beh…”
What Does an Embedding Mean? A Semiotic Account of Learned Representation
Le, J. (n.d.) · PhilPapers (PhilPapers Foundation)
Where Does Jagged Competence Come From?Preprint
Tsiokos, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.03831
“Probe accuracy alone does not show that a network uses what the probe finds: probes need controls for their own capacity [13, 2], high probing accuracy can coexist with task irrelevance [22], and removing information tests use more directly [6].”
Khetan, S., & Balasubramanian, S. (2026) · ACM International Conference on AI in Finance (ICAIF) · doi:10.48550/arxiv.2610.07798
“…what information a representation carries (Alain and Bengio, 2016; Conneau et al., 2018; Tenney et al., 2019), with well known caveats: a probe may learn the task itself (Hewitt and Liang, 2019), and decodability does not imply use (Ravichander et al., 2021; Belinkov, 2022; Elazar et al., 2021).”
Does the Model Use the Feature? Separating Steering from Mechanism in LLMsPreprint
Che, T., & Li, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07270 · Cited by 1
“Amnesic probing tests behavioral dependence by removing decodable information and measuring the resulting behavioral change (Elazar et al., 2021).”
CALR: Continuous Anchored Latent Reasoning via Render-of-Thought CompressionPreprint
Wei, Z., Jiang, B., Hao, Y., Ding, W., Wei, Z., Wu, S., Han, Z., & Jiao, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07175
“Attention interventions (Jain & Wallace, 2019) assess prediction sensitivity to attention distributions, while Amnesic Probing (Elazar et al., 2021) removes decodable properties to test their contribution to task behavior.”
Probing isn’t Causality in Molecular ModelsPreprint
Srivastava, S. P., Chundru, S. K., Singh, H., & Singh, R. K. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.10.05.756781
“Amnesic probing tests behavioral changes after removing information (Elazar et al., 2021).”
DANGERLENS: Probing Dangerous Representations Hidden by Safety AlignmentPreprint
Yang, T., Yan, S., Lu, L., Chen, J., Wang, K., Meng, F., Liu, M., & Yang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19365
“Mediation, erasure, activation patching, tracing, and causal scrubbing offer additional tools for testing how such representations contribute to computation (Vig et al., 2020; Elazar et al., 2021; Wang et al., 2022; Meng et al., 2023; Chan et al., 2022).”
InfluentialLearning Persistent Referential Identity for Two-Hop Retrieval in Event-Stream Models
Kadyrbek, N., Мансурова, М., Talasbayeva, Z. T., & Zhumasheva, Z. T. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100334
Verma, A., & Greenberg, C. (2026) · Journal of High School Science · doi:10.64336/001c.172667
Computation, Not Memory: Chain-of-Thought and State Tracking in LLM AgentsPreprint
Mouseli, P., & Moayedi, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.25760
“High probe accuracy does not show that a model uses the decoded information (Hewitt and Liang, 2019; Belinkov, 2022; Elazar et al., 2021).”
EMOTE: An Explainable and Reliability-Aware Framework for Conversational Emotion RecognitionPreprint
Nadiger, S., Karthik, T. S., Ghanshyam, P., & Saumya, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7643923
The Geometry of Emotion Representations Across Scale, Depth, and Instruction TuningPreprint
Hollowell, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.35411
“No result here shows that or how Gemma 2 models use these representations when they generate text, and neither the presence of emotion geometry nor the readability of emotion words licenses that inference (Elazar et al., 2021; Ravichander, Belinkov, and Hovy, 2021).”
Metadata
- Authors
- Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Corresponding author
- Elazar, Yanai
- Ravfogel, Shauli
- Jacovi, Alon
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021-03-11
- Volume
- 9
- Pages
- 160-175
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3159900299
- Cited by
- 126
- References
- 48
- Citations by year
- 2026: 13 · 2025: 4 · 2024: 11 · 2023: 35 · 2022: 33 · 2021: 28
- FWCI
- 9.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- probing, causal intervention, neural representations, BERT, word prediction, parts of speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.