CoChat
Sign Up
PreprintCited by 110Natural Language Processing TechniquesTopic ModelingExplainable Artificial Intelligence (XAI)

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

Designing and Interpreting Probes with Control Tasks

Open access
Hewitt, John, Liang, Percy·2019·arXiv
DOI 10.48550/arxiv.1909.03368arXiv 1909.03368
Open this paper in CoChat — free accountOpen PDF

Abstract

Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just that the probe has learned the linguistic task? In this paper, we propose control tasks, which associate word types with random outputs, to complement linguistic tasks. By construction, these tasks can only be learned by the probe itself. So a good probe, (one that reflects the representation), should be selective, achieving high linguistic task accuracy and low control task accuracy. The selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types. We construct control tasks for English part-of-speech tagging and dependency edge prediction, and show that popular probes on ELMo representations are not selective. We also find that dropout, commonly used to control probe complexity, is ineffective for improving selectivity of MLPs, but that other forms of regularization are effective. Finally, we find that while probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective, which raises the question of which layer better represents parts-of-speech.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.03368

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • From an information-theoretic point of view, an arbitrarily powerful probe measures the mutual information between a hidden representation and a concept.

    “Indeed, from an information-theoretic point of view, an arbitrarily powerful probe measures the mutual information between a hidden representation and a concept (Hewitt & Liang, 2019;Pimentel et al., 2020).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Hewitt & Liang, 2019

    The Introduction says that a sufficiently expressive probe with enough training data can learn any task from a lossless representation. That supports the point about probe power, but our copy does not say that a probe measures mutual information between a representation and a concept. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained anal- ysis of sentence embeddings using auxiliary predic- tion tasks. In International Conference on Learning Representations.
  2. Guillaume Alain and Yoshua Bengio. 2016. Under- standing intermediate layers using linear classifier probes. In International Conference on Learning Representations.
  3. Peter L Bartlett and Shahar Mendelson. 2001. Rademacher and gaussian complexities: Risk bounds and structural results. In International Con- ference on Computational Learning Theory, pages 224-240. Springer.
  4. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Has- san Sajjad, and James Glass. 2017. What do neu- ral machine translation models learn about morphol- ogy? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 861-872. Association for Computational Linguistics. · DOI · Paper page
  5. Yonatan Belinkov and James Glass. 2017. Analyz- ing hidden representations in end-to-end automatic speech recognition systems. In Advances in Neural Information Processing Systems, pages 2441-2451.
  6. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
  7. Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural ma- chine translation on part-of-speech and semantic tag- ging tasks. arXiv preprint arXiv:1801.07772. · arXiv
  8. Arianna Bisazza and Clara Tump. 2018. The lazy en- coder: A fine-grained analysis of the role of mor- phology in neural machine translation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2871-2876.
  9. Alexis Conneau, Germán Kruszewski, Guillaume Lam- ple, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single \$&!#* vector: Prob- ing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136. Association for Computational Linguistics.
  10. Zied Elloumi, Laurent Besacier, Olivier Galibert, and Benjamin Lecouteux. 2018. Analyzing learned rep- resentations of a deep ASR performance prediction model. arXiv preprint arXiv:1808.08573. · arXiv
  11. Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sen- tence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790-1801.
  12. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceed- ings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 134-139.
  13. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Un- der the hood: Using diagnostic classifiers to in- vestigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium. Association for Computational Linguistics. · DOI
  14. Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 12-21.
  15. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies (NAACL). Association for Com- putational Linguistics.
  16. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' re- veal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
  17. Najoung Kim, Roma Patel, Adam Poliak, Alex Wang, Patrick Xia, R Thomas McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, et al. 2019. Probing what different nlp tasks teach machines about function word comprehension. arXiv preprint arXiv:1904.11544. · arXiv
  18. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. · arXiv
  19. Arne Köhn. 2015. Whatâ Ȃ Źs in an embedding? ana- lyzing word embeddings through multilingual eval- uation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067-2073.
  20. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies.
  21. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC.
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their composition- ality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Ad- vances in Neural Information Processing Systems 26, pages 3111-3119.
  23. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word rep- resentation. In Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1532-1543.
  24. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
  25. Peng Qi and Christopher D. Manning. 2017. Arc-swift: A novel transition system for dependency parsing. In Association for Computational Linguistics (ACL).
  26. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016. Investigating language universal and specific prop- erties in word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), vol- ume 1, pages 1478-1488.
  27. Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov, and Iryna Gurevych. 2019. Linspector: Multilingual probing tasks for word representations. arXiv preprint arXiv:1903.09442. · arXiv
  28. Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with svcca. In North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (NAACL). Association for Computational Linguistics.
  29. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1526- 1534.
  30. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958.
  31. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben- jamin Recht, and Oriol Vinyals. 2017. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Represen- tations.
  32. Kelly W Zhang and Samuel R Bowman. 2018. Lan- guage modeling teaches you more syntax than trans- lation does: Lessons learned through auxiliary task analysis. arXiv preprint arXiv:1809.10040. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Hewitt, J., & Liang, P.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1909.03368
arXiv
1909.03368
OpenAlex
W2972296365
Cited by
110
References
4
Citations by year
2026: 31 · 2025: 20 · 2024: 12 · 2023: 9 · 2022: 9 · 2021: 16 · 2020: 12 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
probe selectivity, control tasks, ELMo, word representation, part-of-speech tagging, regularization, dropout

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review