The DOI matches its OpenAlex record; 49 of the 49 references it cites were found in a registry or an index.
What you can cram into a single vector: Probing sentence embeddings for linguistic properties
Open accessAbstract
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts
Pang, B., & Lee, L. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1218990 · Cited by 3,301
(2005) · Cited by 3,125
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · cited 6×
“Also refer to the SentEval page and to Conneau et al. (2017) for the specifics of training and figures of merit for each task.”
BackgroundMethodInfluentialSelf-citationSukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293
(2017) · International Conference on Learning Representations · Cited by 1,075
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
Socher, R., Huang, E., Pennin, J., Manning, C. D., & Ng, A. Y. (2011) · neural information processing systems · Cited by 782
(2017) · International Conference on Machine Learning · Cited by 758
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Dupoux, E., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 737
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01303-4 · Cited by 719
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 625
Self-citationVisualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · arXiv · doi:10.48550/arxiv.1803.05449 · Cited by 477 · cited 1×
“…downstream tasks Figure 2 reports correlation between performance on our probing tasks and the downstream tasks available in the SentEval 5 suite (Conneau and Kiela, 2018), which consists of classification (MR, CR, SUBJ, MPQA, SST2, SST5, TREC), natural language inference (SICK-E), semantic…”
BackgroundSelf-citationVinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., & Hinton, G. (2014) · arXiv · doi:10.48550/arxiv.1412.7449 · Cited by 464
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2gscircifffd · Cited by 449
Self-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C. E., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.05326 · Cited by 381
Advances in Pre-Training Distributed Word Representations
Mikolov, T., Grave, É., Bojanowski, P., Puhrsch, C., & Joulin, A. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4b3prw5a5tze · Cited by 354
Neurophysiological dynamics of phrase-structure building during sentence processing
Nelson, M. J., El Karoui, I., Giber, K., Yang, X., Cohen, L., Koopman, H., Cash, S. S., Naccache, L., Hale, J. T., Pallier, C., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1701590114 · Cited by 319 · cited 1×
“…an un-trained human subject to be explicitly aware of tree depth or top constituency, similar information must be implicitly computed to correctly parse sentences, and there is suggestive evidence that the brain tracks something akin to tree depth during sentence processing (Nelson et al., 2017).”
BackgroundWhat do Neural Machine Translation Models Learn about Morphology?
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., & Glass, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1080 · Cited by 307
Does String-Based Neural MT Learn Source Syntax?
Shi, X., Padhi, I., & Knight, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1159 · Cited by 300
Revisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_44 · Cited by 218
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
Zhou, J., Cao, Y., Wang, X., Li, P., & Xu, W. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00105 · Cited by 216 · cited 1×
“Seq2seq systems have shown strong results in machine translation (Zhou et al., 2016).”
BackgroundFine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
Adi, Y. (2024) · arXiv (Cornell University) · doi:10.57702/tcr06vve · Cited by 212
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Subramanian, S., Trischler, A., Bengio, Y., & Pal, C. J. (2018) · arXiv · doi:10.48550/arxiv.1804.00079 · Cited by 207
Belinkov, Y., Màrquez, L., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07772 · Cited by 200
Sennrich, R. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2060 · Cited by 199
Representation of Linguistic Form and Function in Recurrent Neural Networks
Kádár, Á., Chrupała, G., & Alishahi, A. (2017) · Computational Linguistics · doi:10.1162/coli_a_00300 · Cited by 148
Illinois-LH: A Denotational and Distributional Approach to Semantics
Lai, A., & Hockenmaier, J. (2014) · International Workshop on Semantic Evaluation (SemEval) · doi:10.3115/v1/s14-2055 · Cited by 137
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87
Self-citationExploring how deep neural networks form phonemic categories
Nagamine, T., Seltzer, M. L., & Mesgarani, N. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-422 · Cited by 83
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model
Pham, N. T., Kruszewski, G., Lazaridou, A., & Baroni, M. (2015) · doi:10.3115/v1/p15-1094 · Cited by 54
Self-citationUnderstanding and Improving Morphological Learning in the Neural Machine Translation Decoder
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., & Vogel, S. (2017) · International Joint Conference on Natural Language Processing · Cited by 52
Revisiting Visual Question Answering BaselinesPreprint
Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.08390 · Cited by 47
Deep Recurrent Models with Fast-Forward Connections for Neural Machine TranslationPreprint
Zhou, J., Cao, Y., Wang, X., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04199 · Cited by 17
Hupkes, D., Veldhoen, S., & Zuidema, W. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11196 · Cited by 14
Trimming and Improving Skip-thought VectorsPreprint
Tang, S., Jin, H., Chen, F., Wang, Z., & de Sa, V. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03148 · Cited by 9
Hupkes, D., Veldhoen, S., & Zuidema, W. (2017) · arXiv · doi:10.48550/arxiv.1711.10203
Research works printed but not linked: 3
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. Advances in neural information processing systems (NIPS) .
- Jiwei Li, Monroe Will, and Dan Jurafsky. 2017. Ef- ficient estimation of word representations in vec- tor space. https://arxiv.org/abs/1612. 08220.
- Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Ad- vances in pre-training distributed word representa- tions. In Proceedings of LREC. Miyazaki, Japan.
Cited by
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Metadata
- Authors
- Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1805.01070
- OpenAlex
- W4292353911
- Cited by
- 263
- References
- 46
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.