CoChat
Sign Up
Cited by 132Topic ModelingNatural Language Processing TechniquesExplainable Artificial Intelligence (XAI)

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?

Open access
Ravichander, Abhilasha, Belinkov, Yonatan, Hovy, Eduard·2021·Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
DOI 10.18653/v1/2021.eacl-main.295
Open this paper in CoChat — free accountOpen PDF

Abstract

Although neural models have achieved impressive results on several NLP benchmarks, little is understood about the mechanisms they use to perform language tasks.Thus, much recent attention has been devoted to analyzing the sentence representations learned by neural encoders, through the lens of 'probing' tasks.However, to what extent was the information encoded in sentence representations, as discovered through a probe, actually used by the model to perform its task?In this work, we examine this probing paradigm through a case study in Natural Language Inference, showing that models can learn to encode linguistic properties even if they are not needed for the task on which the model was trained.We further identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments.Finally, through a set of controlled synthetic tasks, we demonstrate models can encode these properties considerably above chance-level even when distributed in the data as random noise, calling into question the interpretation of absolute claims on probing tasks. 1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.eacl-main.295

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.

    “Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Ravichander et al., 2021

    Section 4.1 reports high probing accuracy for linguistic properties that the main task does not require. Section 6 says probing can reveal a correlated property in a representation but cannot determine whether the model relies on it. The paper does not establish that the hidden activations have no causal connection to the output or that probes localize particular activation vectors. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all.

    “However, training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al., 2021;Geiger et al., 2024b;Elazar et al., 2021;Belinkov, 2022b).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Ravichander et al., 2021

  • Probing has two main challenges: the need for carefully chosen data for well-defined concepts, and probes detect correlations instead of causal variables in hidden activations.

    “Although relatively simple to implement, probing has two main challenges (Ravichander et al., 2021;Belinkov, 2022b): (1) The need for carefully chosen data for well-defined concepts, and (2) probes detect correlations instead of causal variables in hidden activations.”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Ravichander et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained anal- ysis of sentence embeddings using auxiliary predic- tion tasks. In International Conference on Learning Representations.
  2. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. Probing linguistic features of sentence-level representations in neural relation ex- traction. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 1534-1545, Online. Association for Computa- tional Linguistics. · DOI
  3. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and An- tonio Torralba. 2018. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Represen- tations.
  4. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural ma- chine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872.
  5. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
  6. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Ben- jamin Van Durme, and Alexander Rush. 2019a. Don't take the premise for granted: Mitigating ar- tifacts in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891, Flo- rence, Italy. Association for Computational Linguis- tics. · DOI · Paper page
  7. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Ben- jamin Van Durme, and Alexander Rush. 2019b. On adversarial removal of hypothesis-only bias in natu- ral language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Se- mantics (*SEM 2019), pages 256-262, Minneapolis, Minnesota. Association for Computational Linguis- tics. · DOI
  8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics. · DOI
  9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pages 670-680, Copen- hagen, Denmark. Association for Computational Linguistics. · DOI · Paper page
  10. Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics. · DOI · Paper page
  11. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. The fourth pascal recognizing textual entailment challenge. Journal of Natural Language Engineering.
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evalu- ating predictive uncertainty, visual object classifica- tion, and recognising tectual entailment, pages 177- 190. Springer.
  13. Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11- 21.
  14. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceed- ings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 134-139.
  15. Yaroslav Ganin and Victor Lempitsky. 2015. Unsu- pervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pages 1180-1189. JMLR. org.
  16. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9. Association for Computa- tional Linguistics.
  17. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Un- der the hood: Using diagnostic classifiers to in- vestigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium. Association for Computational Linguistics. · DOI
  18. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative ad- versarial nets. In Advances in neural information processing systems, pages 2672-2680.
  19. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Associa- tion for Computational Linguistics. · DOI · Paper page
  20. Aria Haghighi, Andrew Ng, and Christopher Manning. 2005. Robust textual inference via graph match- ing. In Proceedings of Human Language Technol- ogy Conference and Conference on Empirical Meth- ods in Natural Language Processing.
  21. Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain ques- tion answering. In Proceedings of the 21st Interna- tional Conference on Computational Linguistics and the 44th annual meeting of the Association for Com- putational Linguistics, pages 905-912. Association for Computational Linguistics.
  22. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' re- veal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
  23. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, pages 2486-2496, Copenhagen, Denmark. As- sociation for Computational Linguistics. · DOI
  24. Pratik Joshi, Somak Aditya, Aalok Sathe, and Mono- jit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41-55, Online. Association for Computational Lin- guistics. · DOI
  25. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, Brussels, Belgium. Association for Computational Linguistics. · DOI
  26. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bow- man, and Ellie Pavlick. 2019. Probing what dif- ferent NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Se- mantics (*SEM 2019), pages 235-249, Minneapolis, Minnesota. Association for Computational Linguis- tics. · DOI
  27. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Lin- guistics. · DOI · Paper page
  28. Nelson F Liu, Roy Schwartz, and Noah A Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 2171-2179.
  29. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In Association for Compu- tational Linguistics (ACL) System Demonstrations, pages 55-60.
  30. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A SICK cure for the evaluation of com- positional distributional semantic models. In Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 216-223, Reykjavik, Iceland. European Lan- guages Resources Association (ELRA).
  31. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Lin- guistics. · DOI · Paper page
  32. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 130-136, Berlin, Germany. As- sociation for Computational Linguistics. · DOI
  33. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Com- putational Linguistics.
  34. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopou- los, Miguel Ballesteros, David Chiang, Daniel Cloth- iaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980. · arXiv
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language process- ing (EMNLP), pages 1532-1543.
  36. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  37. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language in- ference. In Proceedings of the Seventh Joint Con- ference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Associa- tion for Computational Linguistics. · DOI
  38. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A bench- mark evaluation framework for quantitative reason- ing in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349-361, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  39. Abdelrhman Saleh, Tovly Deutsch, Stephen Casper, Yonatan Belinkov, and Stuart Shieber. 2020. Prob- ing neural dialog models for conversational under- standing. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 132-143, Online. Association for Computa- tional Linguistics. · DOI
  40. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings. OpenReview.net.
  41. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1526- 1534.
  42. Naftali Tishby and Noga Zaslavsky. 2015. Deep learn- ing and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE.
  43. Eva Vanmassenhove, Jinhua Du, and Andy Way. 2017. Investigating 'aspect'in nmt and smt: Translating the english simple past and present perfect. Computa- tional Linguistics in the Netherlands Journal, 7:109- 128.
  44. Sara Veldhoen, Dieuwke Hupkes, and Willem H Zuidema. 2016. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@ NIPS.
  45. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for inter- preting neural nlp: The case of gender bias. arXiv preprint arXiv:2004.12265. · arXiv
  46. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the trans- former: A study with machine translation and lan- guage modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4387-4397.
  47. Elena Voita and Ivan Titov. 2020. Information- theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  48. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha- gen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. Investi- gating BERT's knowledge of language: Five anal- ysis methods with NPIs. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2870-2880, Hong Kong, China. Association for Computational Linguistics. · DOI
  49. John Wieting and Douwe Kiela. 2018. No train- ing required: Exploring random encoders for sen- tence classification. In International Conference on Learning Representations.
  50. Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Work- shop BlackboxNLP: Analyzing and Interpreting Neu- ral Networks for NLP, pages 359-361, Brussels, Bel- gium. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Metadata

Authors
Ravichander, A., Belinkov, Y., & Hovy, E.
Affiliations
  • Carnegie Mellon University (US)
  • Technion – Israel Institute of Technology (IL)
ORCID
Venue
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Publisher
Association for Computational Linguistics
Published
2021
Pages
3363-3377
DOI
10.18653/v1/2021.eacl-main.295
OpenAlex
W3022518881
Cited by
132
References
7
Citations by year
2026: 33 · 2025: 15 · 2024: 12 · 2023: 30 · 2022: 18 · 2021: 18
FWCI
7.33
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1801316, 1914486, 1330596)
  • Brandeis University
  • Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
probing tasks, task relevance, natural language inference, sentence representation, pre-trained word embeddings, linguistic features, interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review