The DOI matches its Crossref record; 36 of the 41 references it cites were found in a registry or an index, and 1 are not checked yet.
Designing and Interpreting Probes with Control Tasks
Open accessAbstract
John Hewitt, Percy Liang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our CCC designs are heuristic: as with control tasks in the probing literature (Hewitt and Liang, 2019), we rely on intuition to craft a g w that is “simpler” than f w .”
From Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024) · cites this paper as Hewitt and Liang, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Generating Typed Dependency Parses from Phrase Structure Parses
de Marneffe, M.-C., MacCartney, B., & Manning, C. D. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3wtkaidk6ea8 · Cited by 2,289
Research works printed but not linked: 5
- John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies (NAACL). Association for Com- putational Linguistics.
- Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' re- veal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
- Arne Köhn. 2015. Whatâ Ȃ Źs in an embedding? ana- lyzing word embeddings through multilingual eval- uation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067-2073.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their composition- ality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Ad- vances in Neural Information Processing Systems 26, pages 3111-3119.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationAutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
FNet: Mixing Tokens with Fourier Transforms
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontañón, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.319 · Cited by 614
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 288
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2021) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 491
Revisiting Few-sample BERT Fine-tuningPreprint
Zhang, T., Wu, F., Katiyar, A., Weinberger, K. Q., & Artzi, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05987 · Cited by 361
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 283
Hydrological concept formation inside long short-term memory (LSTM) networks
Lees, T., Reece, S., Kratzert, F., Klotz, D., Gauch, M., de Bruijn, J., Sahu, R. K., Greve, P., Slater, L. J., & Dadson, S. (2022) · Hydrology and earth system sciences · doi:10.5194/hess-26-3079-2022 · Cited by 267
Post-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2022) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 283
Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
Dai, J., Yan, H. R., Sun, T., Liu, P., & Qiu, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.146 · Cited by 190
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 251
Information-Theoretic Probing for Linguistic Structure
Pimentel, T., Valvoda, J., Hall Maudslay, R., Zmigrod, R., Williams, A., & Cotterell, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.420 · Cited by 41
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
Wu, Z., Chen, Y., Kao, B., & Liu, Q. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.383 · Cited by 187
Interpretability research of deep learning: A literature survey
Xu, B., & Yang, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102721 · Cited by 176
What Happens To BERT Embeddings During Fine-tuning?
Merchant, A., Rahimtoroghi, E., Pavlick, E., & Tenney, I. F. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.4 · Cited by 208
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.431 · Cited by 163
(2020) · neural information processing systems · Cited by 142
Towards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 196
Finding Universal Grammatical Relations in Multilingual BERT
Chi, E. A., Hewitt, J., & Manning, C. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.493 · Cited by 138
Self-citationAmnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders
He, H. W., & Choi, J. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.451 · Cited by 133
A Survey on Contextual EmbeddingsPreprint
Liu, Q., Kusner, M. J., & Blunsom, P. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.12 · Cited by 243
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPTPreprint
Chen, Y. T., & Zou, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.562533 · Cited by 106
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Han, X., Wallace, B., & Tsvetkov, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.492 · Cited by 164
Warstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 115
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 152
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification
Xiao, Z., Wu, J., Chen, Q., & Deng, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.724 · Cited by 96
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85
Li, B., Li, R., Sun, T., Gong, A., Tian, F., Khan, M. Y. A., & Ni, G. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.129401 · Cited by 90
Contrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83
Implicit Representations of Meaning in Neural Language Models
Li, B. Z., Nye, M., & Andreas, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.143 · Cited by 84
Iteratively Prompt Pre-trained Language Models for Chain of Thought
Wang, B., Deng, X. B., & Sun, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.174 · Cited by 123
Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
Shi, P., Ng, P., Wang, Z., Zhu, H., Li, A. H., Wang, J., dos Santos, C. N., & Xiang, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17627 · Cited by 75
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation
Miaschi, A., & Dell’Orletta⋄, F. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.15 · Cited by 83
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
Ravichander, A., Belinkov, Y., & Hovy, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.295 · Cited by 132
Semantic Structure in Deep Learning
Pavlick, E. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 82
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.9 · Cited by 64
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender BiasPreprint
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., & Shieber, S. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12265 · Cited by 212
Analyzing Redundancy in Pretrained Transformer Models
Dalvi, F., Sajjad, H., Durrani, N., & Belinkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.398 · Cited by 97
Language Model Evaluation Beyond Perplexity
Meister, C., & Cotterell, R. (2021) · doi:10.18653/v1/2021.acl-long.414 · Cited by 102
BERT syntactic transfer: A computational experiment on Italian, French and English languages
Guarasci, R., Silvestri, S., Pietro, G. D., Fujita, H., & Esposito, M. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101261 · Cited by 56
Analyzing Individual Neurons in Pre-trained Language Models
Durrani, N., Sajjad, H., Dalvi, F., & Belinkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.395 · Cited by 92
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
(2021) · doi:10.18653/v1/2021.conll-1.15 · Cited by 51
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 73
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
Müller, B., Elazar, Y., Sagot, B., & Seddah, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.189 · Cited by 62
A Tale of a Probe and a Parser
Maudslay, R. H., Valvoda, J., Pimentel, T., Williams, A., & Cotterell, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.659 · Cited by 60
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 57
Ecco: An Open Source Library for the Explainability of Transformer Language Models
Alammar, J. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.30 · Cited by 53
Causal abstractions of neural networks
Geiger, A., Lu, H., Icard, T., & Potts, C. (2021) · arXiv · doi:10.48550/arxiv.2106.02997 · Cited by 45
Merrill, W., Goldberg, Y., Schwartz, R., & Smith, N. A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00412 · Cited by 44
“An alternate approach is probing (Adi et al., 2017; Conneau et al., 2018; Hupkes and Zuidema, 2018; Hewitt and Liang, 2019; Belinkov and Glass, 2019), which investigates how directly a model’s representations encode semantic relations by measuring if they can be easily decoded from them.”
BackgroundCausal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S. M., Linzen, T., & Belinkov, Y. (2021) · doi:10.18653/v1/2021.acl-long.144 · Cited by 99
A Closer Look at How Fine-tuning Changes BERT
Zhou, Y., & Srikumar, V. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.75 · Cited by 69
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
Geiger, A., Richardson, K., & Potts, C. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.16 · Cited by 97
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Papadimitriou, I., & Jurafsky, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.554 · Cited by 58
Pareto Probing: Trading Off Accuracy for Complexity
Pimentel, T., Saphra, N., Williams, A., & Cotterell, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.254 · Cited by 57
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 41
Let’s Play Mono - Poly : BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses
Soler, A. G., & Apidianaki, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00400 · Cited by 63
Computational Language Modeling and the Promise of In Silico Experimentation
Jain, S., Vo, V. A., Wehbe, L., & Huth, A. G. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00101 · Cited by 43
Linguistic Profiling of a Neural Language Model
Miaschi, A., Brunato⋄, D., Dell’Orletta⋄, F., & Venturi⋄, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.65 · Cited by 50
Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models
Balkır, E., Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2022) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2022.trustnlp-1.8 · Cited by 49
Human-Model Divergence in the Handling of Vagueness
Stengel-Eskin, E., Guallar-Blasco, J., & Durme, B. V. (2021) · Workshop on Understanding Implicit and Underspecified Language (UnImplicit) · doi:10.18653/v1/2021.unimplicit-1.6 · Cited by 39
Neuron-level Interpretation of Deep NLP Models: A Survey
Sajjad, H., Durrani, N., & Dalvi, F. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00519 · Cited by 52
Intrinsic Probing through Dimension Selection
Hennigen, L. T., Williams, A., & Cotterell, R. (2020) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000462314 · Cited by 38
Probing Across Time: What Does RoBERTa Know and When?
Liu, Z., Wang, Y., Kasai, J., Hajishirzi, H., & Smith, N. A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.71 · Cited by 59
What Do Self-Supervised Speech Models Know About Words?
Pasad, A., Chien, C.-M., Settle, S., & Livescu, K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00656 · Cited by 54
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.439 · Cited by 39
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 40
Do Neural Language Models Show Preferences for Syntactic Formalisms?
Kulmizev, A., Ravishankar, V., Abdou, M., & Nivre, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.375 · Cited by 41
Syntactic Structure Distillation Pretraining for Bidirectional Encoders
Kuncoro, A., Kong, L., Fried, D., Yogatama, D., Rimell, L., Dyer, C., & Blunsom, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00345 · Cited by 37
Zhao, Y., & Bethard, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.429 · Cited by 36
Dodrio: Exploring Transformer Models with Interactive Visualization
Wang, Z. J., Turko, R., & Chau, D. H. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.16 · Cited by 45
Assessing BERT’s ability to learn Italian syntax: a study on null-subject and agreement phenomena
Guarasci, R., Silvestri, S., Pietro, G. D., Fujita, H., & Esposito, M. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03297-4 · Cited by 34
Vilares, D., Strzyz, M., Søgaard, A., & Gómez‐Rodríguez, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6446 · Cited by 37
Asking without Telling: Exploring Latent Ontologies in Contextual Representations
Michael, J., Botha, J. A., & Tenney, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.552 · Cited by 38
Pretrained Language Model Embryology: The Birth of ALBERT
Chiang, C.-H., Huang, S.-F., & Lee, H.-Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.553 · Cited by 42
Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing
Wang, L., Qin, B., Hui, B., Li, B., Yang, M., Wang, B., Li, B., Sun, J., Huang, F., Si, L., et al. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539305 · Cited by 33
Probing for Referential Information in Language Models
Sorodoc, I.-T., Gulordava, K., & Boleda, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.384 · Cited by 37
Word Order Does Matter and Shuffled Language Models Know It
Abdou, M., Ravishankar, V., Kulmizev, A., & Søgaard, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.476 · Cited by 32
Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT
Papadimitriou, I., Chi, E. A., Futrell, R., & Mahowald, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.215 · Cited by 34
Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing
Liu, Q., Yang, D., Zhang, J., Guo, J., Zhou, B., & Lou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.100 · Cited by 32
Probing for the Usage of Grammatical Number
Lasri, K., Pimentel, T., Lenci, A., Poibeau, T., & Cotterell, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.603 · Cited by 51
Putting representations to use
Cao, R. (2022) · Synthese · doi:10.1007/s11229-022-03522-3 · Cited by 35
Eliciting Latent Predictions from Transformers with the Tuned Lens
Belrose, N., Ostrovsky, I., McKinney, L., Furman, Z., Smith, L., Halawi, D., Biderman, S., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2303.08112 · Cited by 27
“This is especially important for probing techniques, since it’s known that probes can learn to rely on spurious features unrelated to the model’s performance (Hewitt and Liang, 2019; Be-linkov, 2022).”
BackgroundInterpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 64
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26
Conditional probing: measuring usable information beyond a baseline
Hewitt, J., Ethayarajh, K., Liang, P., & Manning, C. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.122 · Cited by 40
Self-citationAutomatic Text Evaluation through the Lens of Wasserstein Barycenters
Colombo, P., Staerman, G., Clavel, C., & Piantanida, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.817 · Cited by 35
Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
Zhong, R., Ghosh, D., Klein, D., & Steinhardt, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.334 · Cited by 23
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.27 · Cited by 22
When Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM PredictionsPreprint
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2020) · arXiv (Cornell University) · Cited by 22
A matter of framing: The impact of linguistic formalism on probing results
Kuznetsov, I., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.13 · Cited by 22
Emergence of Separable Manifolds in Deep Language RepresentationsPreprint
Mamou, J., Le, H., Rio, M. D., Stephenson, C., Tang, H., Kim, Y., & Chung, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01095 · Cited by 25
Interpretable Entity Representations through Large-Scale Typing
Onoe, Y., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.54 · Cited by 20
DirectProbe: Studying Representations without Classifiers
Zhou, Y., & Srikumar, V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.401 · Cited by 24
Metadata
- Authors
- Hewitt, J., & Liang, P.
- Affiliations
- Stanford University (US)
- Corresponding author
- Liang, Percy
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2733-2743
- OpenAlex
- W2970862333
- Cited by
- 393
- References
- 41
- Citations by year
- 2026: 20 · 2025: 14 · 2024: 20 · 2023: 63 · 2022: 81 · 2021: 119 · 2020: 76
- FWCI
- 26.40
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- probing, control tasks, neural network interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
