Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Open accessThe DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Abstract
A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To improve the causal relevance of probe vectors, one approach is to use counterfactual data, which involves intervening on the concept of interest (for instance, observing the resulting output if the dog in an image was changed to a cat) (Elazar et al., 2021;Mueller, 2024;Geiger et al., 2024b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Early studies showed that it is possible to use linear [51,52] or nonlinear [53] updates to a model’s hidden state in order to manipulate the model’s representation of grammatical role”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“In the past few years, multiple studies have suggested that these apparently symbolic representations do indeed play a causal role in models’ subsequent predictions, each study using a different method to arrive at this conclusion.”
From Symbols and grounding in large language models (2023) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“However, training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al., 2021;Geiger et al., 2024b;Elazar et al., 2021;Belinkov, 2022b).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Elazar et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“The results up to this section treat all of BERT’s ‘Transformer blocks’ (Vaswani et al., 2017) as the encoding function and the embedding matrix”
BackgroundScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2×
“The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest”
BackgroundMethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“(Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b)) spiked interest in understanding how these models work and what is”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · cited 1×
“One commonly used method for information removal relies on adversarial training through the gradient reversal layer technique (Ganin et al., 2016).”
MethodModeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · cited 1×
“Voita and Titov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts.”
The Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,781
Research works printed but not linked: 5
- Giulianelli, 2018, Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information, 240, 10.18653/v1/W18-5426 · DOI
- Hewitt, 2019, A structural probe for finding syntax in word representations, 4129
- Hupkes, 2018, Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure, Journal of Artificial Intelligence Research, 61, 907, 10.1613/jair.1.11196 · DOI · Paper page
- Reif, 2019, Visualizing and measuring the geometry of bert, 8592
- Weischedel, 2013, Ontonotes release 5.0 ldc2013t19, Linguistic Data Consortium, Philadelphia, PA, 23
Cited by
Items 101–200 of 214
Parthasarathi, P., Pineau, J., & Chandar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10427 · Cited by 6
(2022) · doi:10.18653/v1/2022.conll-1.8 · Cited by 5
Seeing Through VisualBERT: A Causal Adventure on Memetic Landscapes
Bandyopadhyay, D., Hasanuzzaman, M., & Ekbal, A. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.629 · Cited by 6
Sustaining Fairness via Incremental Learning
Chowdhury, S. B. R., & Chaturvedi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25833 · Cited by 10
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Pérez-Mayos, L., García, A. T., Mille, S., & Wanner, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.333 · Cited by 14
Understanding Game-Playing Agents with Natural Language Annotations
Tomlin, N., He, A., & Klein, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.90 · Cited by 9
On the data requirements of probing
Zhu, Z., Wang, J., Li, B., & Rudzicz, F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.326 · Cited by 6
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences
Schuster, C. M., & Hegelich, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.89 · Cited by 5
“Amnesic probing measures what information gets used in the probing tasks by removing selected properties (e.g. part-of-speech) from the activations (Elazar et al., 2021).”
BackgroundQuantifying the Task-Specific Information in Text-Based ClassificationsPreprint
Zhu, Z., Balagopalan, A., Ghassemi, M., & Rudzicz, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08931 · Cited by 5
“Following072 the motivations of recent causal analysis papers073 (e.g., Elazar et al. (2021); Pryzant et al. (2021)),074 we want to factor out the impact of the shortcuts075 while still quantifying the amount of information a076 neural network model needs to learn for a task.077 This paper presents…”
BackgroundHow much pretraining data do language models need to learn syntax?Preprint
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03160 · Cited by 8
Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces
Levy, T. J., Goldman, O., & Tsarfaty, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.348 · Cited by 6
Causal Probing for Dual Encoders
Wallat, J., Hinrichs, H., & Anand, A. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679556 · Cited by 4
“Since the standard probing setup is limited to identifying whether information can be decoded from embeddings, causal probing meth-ods have been developed to investigate the abilities’ relevance for downstream tasks ([4, 14, 46] inter alia).”
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
Nikoulina, V., Tezekbayev, M., Kozhakhmet, N., Babazhanova, M., Gallé, M., & Assylbekov, Z. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12788 · Cited by 9
“Following Elazar et al. (2021) we keep iterating the INLP procedure or increasing λ in GRL until the performance of a linear probe that predicts T from the filtered embeddings ˜ x drops to the majority-class accuracy.”
BackgroundMethodResultInfluentialAsk to Understand: Question Generation for Multi-hop Question Answering
Li, J., Ren, M., Gao, Y., & Yang, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6207-5_2 · Cited by 7
Darkness can not drive out darkness: Investigating Bias in Hate SpeechDetection Models
Elsafoury, F. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.4 · Cited by 8
“Some of these studies have focused on understanding the causal inference of concepts (e.g. social bias in the datasets) on the task of text classification using counterfactual causal inference (Feder et al., 2021b; Qian et al., 2021; Elazar et al., 2021).”
BackgroundAligned Probing: Relating Toxic Behavior and Model Internals
Waldis, A., Gautam, V., Lauscher, A., Klakow, D., & Gurevych, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.613 · Cited by 5
Influence Patterns for Explaining Information Flow in BERTPreprint
Lu, K., Wang, Z., Mardziel, P., & Datta, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00740 · Cited by 4
Adversarial Scrubbing of Demographic Information for Text ClassificationPreprint
Chowdhury, S. B. R., Ghosh, S., Li, Y., Oliva, J. B., Srivastava, S., & Chaturvedi, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08613 · Cited by 5
“Prior works have shown that intermediate representations in a deep learning model encode demographic associations of the author or person being spoken about (Blod-gett et al., 2016; Elazar and Goldberg, 2018; Elazar et al., 2021).”
BackgroundLatent Causal Probing: A Formal Perspective on Probing with Causal Models of DataPreprint
Jin, C., & Rinard, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.13765 · Cited by 5
“These works typically intervene on either the model’s representations (Elazar et al., 2021; Geiger et al., 2021; Meng et al., 2022; Abraham et al., 2022; Li et al., 2022) or the model’s inputs (Kaushik et al., 2020; Vig et al., 2020; Gangal & Hovy, 2020; Amini et al., 2023), and analyze the causal…”
BackgroundExplainable AI for NLP: Enhancing Transparency in Sentiment Analysis and Named Entity Recognition
Elkharrat, F., Ghoniem, M., Sherif, A., & Sabty, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_11 · Cited by 3
Islam, M. S., Xiangdong, L., & Ahmed, J. (2026) · Journal of Electrical Systems and Information Technology · doi:10.1186/s43067-026-00341-1 · Cited by 3
“…question-answering system uses BERT to take two parameters, the input question and the passage, as a single-packed sequence model maintain awareness that one sequence is distinct from the other, a necessity for determining which parts of the sequence are essential to answering a question[74, 75].”
Interventional Probing in High Dimensions: An NLI Case Study
Rozanova, J., Valentino, M., Cordeiro, L. C., & Freitas, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.188 · Cited by 10
Knowledge Acquired by Foundation Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_4 · Cited by 3
Probing with Noise: Unpicking the Warp and Weft of Embeddings
Klubička, F., & Kelleher, J. D. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.34 · Cited by 6
Yi, D., Bruno, J. V., Han, J., Zukerman, P., & Steinert‐Threlkeld, S. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.12 · Cited by 13
Geometric Probing of Word Vectors
Babazhanova, M., Tezekbayev, M., & Assylbekov, Z. (2021) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2021.es2021-105 · Cited by 3
On the Usefulness of Embeddings, Clusters and Strings for Text Generator EvaluationPreprint
Pimentel, T., Meister, C., & Cotterell, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.16001 · Cited by 10
“Our probing analyses are thus exempt from most recent criticism against probing methodologies (Hewitt and Liang, 2019; Pimentel et al., 2020a,b; Ravichander et al., 2021; Elazar et al., 2021).”
MethodUnlearning as Ablation: Toward a Falsifiable Benchmark for Generative Scientific DiscoveryPreprint
Yang, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.17681 · Cited by 4
“• Representation-level interventions: e.g., Amnesic Probing [4] removes specific linguistic features from hidden states.”
Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and ActionsPreprint
Sobotka, J., Karabag, M. O., & Topcu, U. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.00226 · Cited by 3
“First, the goal is to quantify the inferential capability already present in the model rather than to add expressive power to it [21, 39], and more expressive probes risk learning the target function themselves instead of reading out what the model has computed [73, 38].”
Counterfactually Probing Language Identity in Multilingual Models
Srinivasan, A., Govindarajan, V. S., & Mahowald, K. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.3 · Cited by 2
CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation
Bandyopadhyay, D., Bhattacharjee, S., Hasanuzzaman, M., & Ekbal, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.686 · Cited by 2
Analyzing the Domain Robustness of Pretrained Language Models, Layer by Layer
Kashyap, A. R., Mehnaz, L., Malik, B., Waheed, A., Hazarika, D., Kan, M., & Shah, R. R. (2021) · National University of Singapore · Cited by 6
Causal Inference and Natural Language Processing
Chen, W., & Chu, Z. (2023) · Springer eBooks · doi:10.1007/978-3-031-35051-1_9 · Cited by 7
The 7th Workshop on Online Abuse and Harms (WOAH)
(2023) · doi:10.18653/v1/2023.woah-1 · Cited by 2
Orlov, A. V., Makus, Y. V., Ashniev, G. A., Orlova, N. N., & Nikitin, P. I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.04.709491 · Cited by 3
Trait-space Monitoring for Emergent Misalignment During Supervised FinetuningPreprint
Nghiem, H. T., Ho, S.-T., Wiegreffe, S., & Daumé, H. (2026) · arXiv (Cornell University) · doi:10.13140/rg.2.2.34447.14243 · Cited by 2
Quirke, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.08292 · Cited by 3
“The combination is widely employed but has no community-agreed validation protocol, and probing alone has long been known to overstate causal claims [Elazar et al., 2021, Hewitt and Liang, 2019].”
Tiwari, D., Davis, C., Sinha, S., Weaver, T., Shah, A., & Chaudhary, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.18080 · Cited by 4
“Amnesic probing made this distinction explicit by removing probe-identified information and measuring downstream behavior, finding that conventional probing performance need not correlate with task importance (Elazar et al., 2021).”
Verbalizable Representations Form a Global Workspace in Language Models
Gurnee, W., Sofroniew, N., Pearce, A., Piotrowski, M., Kauvar, I., Chen, R., Soligo, A., Bogdan, P., Ong, E., Wang, R., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.15495 · Cited by 1
Stengel-Eskin, E., & Durme, B. V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.760 · Cited by 5
Nejadgholi, I., Kiritchenko, S., Fraser, K., & Balkır, E. (2023) · doi:10.18653/v1/2023.woah-1.14 · Cited by 4
When and Why Does Bias Mitigation Work?
Ravichander, A., Stacey, J., & Rei, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.619 · Cited by 11
“In this work, we study how debiasing procedures affect model behavior , as probe performance is not necessarily indicative of the information which a model actually uses to make predictive decisions (Ravichander et al., 2021; Elazar et al., 2021).”
BackgroundCausal Post-hoc XAI: Categorisation and Systematic Literature Review
Bjøru, A. R., Langseth, H., Strümke, I., & Bach, K. (2026) · Machine Learning · doi:10.1007/s10994-026-07103-0 · Cited by 1
Kong, L., & Shin, J. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100350 · Cited by 1
Lan, Z., & Yang, J. (2026) · IEEE Access · doi:10.1109/access.2026.3740087 · Cited by 1
“Amnesic probing, causal tracing, model editing, and causal abstraction address this question [40], [41], [42], [43], [44].”
What does BERT Learn from Arabic Machine Reading Comprehension Datasets
Albilali, E., Al-Twairesh, N., & Hosny, M. (2021) · Cited by 8
Parthasarathi, P., Pineau, J., & Chandar, S. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.50 · Cited by 1
Which Visual Features Impact the Performance of Target Task in Self-supervised Learning?
Oleszkiewicz, W., Basaj, D., Trzcinski, T. P., & Zieliński, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08751-6_24 · Cited by 1
“However, some recent work extends the usability of probing tasks by introducing the concept of amnesic probing [10] to measure the influence of the phenomenons on the target task performance.”
BackgroundMethodExploring Category Structure with Contextual Language Models and Lexical Semantic Networks
Renner, J., Denis, P., Gilleron, R., & Brunellière, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.167 · Cited by 1
Tracing and Manipulating intermediate values in Neural Math Problem Solvers
Matsumoto, Y., Heinzerling, B., Yoshikawa, M., & Inui, K. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.1 · Cited by 3
NxPlain: A Web-based Tool for Discovery of Latent Concepts
Dalvi, F., Durrani, N., Sajjad, H., Jaban, T., Husaini, M., & Abbas, U. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.10 · Cited by 1
Training dynamics of neural language models
Saphra, N. (2021) · ERA · doi:10.7488/era/1421 · Cited by 9
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language ModelsPreprint
Pérez-Mayos, L., García, A. T., Mille, S., & Wanner, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04688 · Cited by 2
Emergent Causal-Geometric Dynamics Across Depth in Large Language ModelsPreprint
Haim, S., & McNamee, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.04931 · Cited by 1
“As a result, interventions that target easily decodable features may fail to generalize or may exert only indirect control [20–22].”
Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a TransformerPreprint
Darade, I., & Thorat, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.22488 · Cited by 1
“Thus, high decodability need not mark a learned causal intermediate [11, 2, 6].”
MANCE: Manifold Aware Concept ErasurePreprint
Avitan, M., Goldberg, Y., & Elazar, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.03973 · Cited by 1
“…encode many attributes of their inputs: semantic content, style, demographic information, task labels, and other latent factors [Conneau et al., 2018, Tenney et al., 2019, Goldberg, 2019, Hewitt and Manning, 2019, Adi et al., 2016, Rogers et al., 2020, Bolukbasi et al., 2016, Elazar et al., 2021].”
Self-citationCichoń, M., & Dmitruk, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.32890 · Cited by 1
“Whether the model uses that direction was not tested; that would need an intervention along it, as has been done for truthfulness directions [Li et al., 2023, Marks and Tegmark, 2024], and probe accuracy without an intervention is descriptive [Elazar et al., 2021].”
LoopVL: Recurrent Visual IntelligencePreprint
Qian, Z., Gong, Z., Xu, Z., Li, H., Wang, Z., Liu, H., Qian, L., Pei, X., Xu, P., Luo, F., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38426 · Cited by 2
“(Blayney et al., 2026; Lu et al., 2025; Hewitt and Liang, 2019; Elazar et al., 2021) The same distinction is important for attention.”
Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM AgentsPreprint
Wang, H., Zhao, W., Zhang, Y., POSKITT, C. M., & Sun, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.00400 · Cited by 1
When Do Biological Reasoning Models Use Their Biological Inputs?Preprint
Fang, A., Thoduguli, N., Fesser, L., Zhang, H., Kakade, S. M., & Žitnik, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.00898 · Cited by 2
“Linear probes provide a simple test of predictability from learned representations (Alain & Bengio, 2017; Hewitt & Liang, 2019), while subsequent work has shown that probe performance and downstream model behavior should be evaluated separately (Ravichander et al., 2021; Elazar et al., 2021).”
Copied, Mixed or Computed? Testing Latent Intermediate Values with Minimal PairsPreprint
Merin, A. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.28498 · Cited by 1
“Probing critiques (Belinkov, 2022) stress that a readable property need not be used (Ravichander et al., 2021; Elazar et al., 2021); at P1, Ouro’s a + b is a case of it.”
An, J., Yang, L., Wang, M., Lu, L., An, W., & Yang, E. (2026) · arXiv · doi:10.48550/arxiv.2603.26323
“By comparison, most studies of spatial reasoning in large neural models do not directly examine internal representations, leaving the basis of their spatial behavior largely opaque(Yamada et al., 2023; Li et al., 2024a; Hewitt and Manning, 2019; Elazar et al., 2020).”
ABSTRACTING INFLUENCE PATHS FOR EXPLAINING (CONTEXTUALIZATION OF) BERT MODELSPreprint
Lu, K., Wang, Z., Mardziel, P., & Datta, A. (2021) · arXiv (Cornell University)
Probe-Based Interventions for Modifying Agent BehaviorPreprint
Tucker, M., Kuhl, W., Shahid, K., Karten, S., Sycara, K. P., & Shah, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.12938
“Prior art has hinted at this possibility: TCAV, for example, instantiates many linear classifiers to see if, on average, they produce an effect, and Elazar et al. [2020] use an iterative approach to create multiple classifiers.”
MethodInfluentialVisualizing the Relationship Between Encoded Linguistic Information and Task Performance
Xiang, J., Li, H., Lian, D., Huang, G., Watanabe, T., & Liu, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.35 · Cited by 3
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_5
Disentangling the Linguistic Competence of Privacy-Preserving BERT
Arnold, S., Kemmerzell, N., & Schreiner, A. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.5
Tracing and Manipulating Intermediate Results in Neural Math Problem Solvers
Matsumoto, Y., Heinzerling, B., Yoshikawa, M., & Inui, K. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.1151
Invariant neural representation of parts of speech in the human brain
Misra, P., Shih, Y., Yu, H., Weisholtz, D. S., Madsen, J. R., Sceillig, S., & Kreiman, G. (2026) · Current Biology · doi:10.1016/j.cub.2026.07.029 · Cited by 2
Less than Necessary or More than Sufficient: Validating Probing Dataset Size
Orlov, E., & Serikov, O. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54534-4_8
Fact Checking with Insufficient Evidence
Atanasova, P. (2022) · Springer eBooks · doi:10.1007/978-3-031-51518-7_2
(2024) · Frontiers research topics · doi:10.3389/978-2-8325-5822-5
Explainability of Language Models (XAI and LLM)
Cecco, A. D., & Gianfagna, L. (2025) · Springer eBooks · doi:10.1007/978-3-031-92229-9_8 · Cited by 1
Shortcut Learning Forms Universal Transfer Patterns Across Language Understanding Tasks
Stow, M. S. M. (2025) · Journal of Software Engineering and Simulation · doi:10.35629/3795-11080827
EXPLORING THE INFLUENCE OF RELEVANT KNOWLEDGE FOR NATURAL LANGUAGE GENERATION INTERPRETABILITY
Martínez-Murillo, I., Moreda, P., & Lloret, E. (2025) · International Conference on Machine Learning Techniques and NLP (MLNLP) · doi:10.5121/csit.2025.152002
“Am-nesic Probing [13] addresses this gap by using counterfactual examples to analyze the causal influence of injected knowledge on model predictions.”
Towards AI Safety via Interpretability and Oversight
Kantamneni, S. (2025) · DSpace@MIT (Massachusetts Institute of Technology)
Diagnosing Capability Preservation and Task Sensitivity in Memory Augmented Document ClassifiersPreprint
Nti, I. K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.06582
When Less Is More? Diagnosing ASR Predictions in Sardinian via Layer-Wise DecodingPreprint
Cristofaro, D. D., Vietti, A., Pouplier, M., & Block, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10350
“Amnesic probing [5] further shows that linguistic properties can be selectively removed from representations, suggesting that such information is not uniformly distributed across layers.”
The Efficacy of Intervention Time on Probing Results
Hussein, A. A., Alnaqshabandi, N. M. M., & Abdulrahman, N. H. (2026) · Advanced medical journal · doi:10.56056/amj.2026.433
Panchal, M. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.ijcnlp-srw.12
“However, they often struggle with systematic ontological inference and can be inconsistent in their application of hierarchical knowledge (Elazar et al., 2020; Kassner et al., 2021).”
Müller, B. (2022) · theses.fr (ABES) · doi:10.70675/6984dc63z4731z4eabzb3fazaf5d012fe143
Facing Data Scarcity in Dialogues for Discourse Structure Discovery and Prediction
Li, C. (2023) · theses.fr (ABES) · doi:10.70675/c379d1f5z5324z4865z9c13z84c9508836f1 · Cited by 7
Study of the abstraction capabilities of neural language models
Li, B. (2023) · theses.fr (ABES) · doi:10.70675/0236e084zee1ez4013z8f5bz6eebee3f0e4e
Jackson, P., Li, R., & Edelstein, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.10151
Xanthopoulos, E. (2026) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.32762
Validating Causal Abstraction Metrics on Simulated Complex SystemsPreprint
Méloux, M., Pimentel, T., Portet, F., & Peyrard, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.00267
53A survey on latent linguistic patterns in bidirectional transformer-based language models
Kugler, K., & Rettinger, A. (2026) · De Gruyter eBooks · doi:10.1515/9783111199740-003
Denoising Models Develop Human-Like Perceptual Illusion Representations Across ArchitecturesPreprint
Ranka, G., & Chopra, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.17138
“[Elazar et al., 2021] used iterative null-space projection to remove probe-detected linguistic properties from language model representations and found several were not behaviorally used.”
雷, 俊. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7226340
Singla, P., Singh, J., & Chhabra, G. S. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116785
Testing Whether Internal Backdoor Precursors Precede Behavior: A Causal, Power-Matched ProtocolPreprint
Yuchen, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-10594749/v1
He, R., Altay, N., & Hinzen, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.25999
“More generally, discourse-level entity information can be encoded in contextualized representations without necessarily producing a corresponding e ff ect in model behavior (Davis and van Schijndel, 2020; Elazar et al., 2021).”
Sun, F., Tong, Z., Wei, H., Racharak, T., & Nguyen, L.-M. (2026) · IEEE Access · doi:10.1109/access.2026.3731685
“Diagnostic probing typically employs controlled perturbations to analyze how models encode linguistic and semantic properties [28], [29].”
Zhang, Y., Li, J., Yu, G., Guo, Y., Luo, X., Li, W., Xu, J., Liu, X., Xu, Z., Liang, M., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7417400
Zhou, C., Jiang, Q., Wu, S., & Zhou, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13012
“Methodologically, our utilization readout is in the ablate-and-measure lineage of amnesic probing (Elazar et al., 2021) — intervene on a representation and measure the behavioral change — applied at the granularity of a single visual KV unit.”
Three Lenses on Linguistic Knowledge in Pretrained Transformers: A Phenomenon-Centered ReviewPreprint
Wang, Y., & Ling, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-10776566/v1
Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMsPreprint
Matsutani, K., Minegishi, G., Park, C. F., Kojima, T., Iwasawa, Y., & Matsuo, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.24352
“Following (Elazar et al., 2021), we repeat the probing and removal procedure.”
InfluentialTRACE: Tractable Routing Autoencoder for Clinical ECGPreprint
Jia, S., Ma, R., Lyu, H., Zhang, H., Yang, Q., & Liao, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.23460
“Probing methods supply evidence about what a representation contains, from observational correlation to causal erasure [32, 33].”
Metadata
- Authors
- Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Corresponding author
- Elazar, Yanai
- Ravfogel, Shauli
- Jacovi, Alon
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021-03-11
- Volume
- 9
- Pages
- 160-175
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3159900299
- Cited by
- 126
- References
- 48
- Citations by year
- 2026: 13 · 2025: 4 · 2024: 11 · 2023: 35 · 2022: 33 · 2021: 28
- FWCI
- 9.46
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- probing, causal intervention, neural representations, BERT, word prediction, parts of speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.