The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Enriching Word Vectors with Subword Information
Open accessAbstract
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrary to string similarity-based approaches, continuous word representations capture not only lexical but also semantic properties of tokens.”
From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Bojanowski et al. (2017)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We screened 4 different word embeddings following a similar procedure reported in our previous study [46] and found that the Common Crawl embeddings—released by Facebook and trained using the fastText on the Common Crawl data set [47]—achieved better performance compared to other embeddings on a validation data set.”
From Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models (2020) · cites this paper as [47]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“I use Fasttext (Bojanowski et al. 2017) as a word representation model because of its capacity to collect sub-word information, which can be helpful in the case of compound words.”
From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Bojanowski et al. 2017
The cited source says what the claim attributes to it.
In the citing paper
“This technique involves learning large-scale word embeddings (Bojanowski et al. 2017; Joulin et al. 2016), representing an advancement from Word2Vec.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Bojanowski et al. 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Self-citationDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Self-citationThe proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
From Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,949
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Producing high-dimensional semantic spaces from lexical co-occurrence
Lund, K., & Burgess, C. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204766 · Cited by 1,875
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,590
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts
dos Santos, C., & de Bayser, M. G. (2014) · International Conference on Computational Linguistics · Cited by 1,162
Mapping authors in intellectual space: A technical overview
McCain, K. W. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6 · Cited by 987
Neurocomputing: Foundations of Research
Anderson, J. A., & Rosenfeld, E. (1988) · Cited by 851
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2001) · International World Wide Web Conference (WWW) · doi:10.1145/371920.372094 · Cited by 841
Better Word Representations with Recursive Neural Networks for Morphology
Luong, M.-T., Socher, R., & Manning, C. D. (2013) · Conference on Computational Natural Language Learning · Cited by 789
Distributional Memory: A General Framework for Corpus-Based Semantics
Baroni, M., & Lenci, A. (2010) · Computational Linguistics · doi:10.1162/coli_a_00016 · Cited by 660
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
Wang, L., Dyer, C., Black, A. W., Trancoso, I. M., Fermandez, R., Amir, S., Marujo, L., & Luis, T. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1176 · Cited by 579
Learning Character-level Representations for Part-of-Speech Tagging
dos Santos, C., & Zadrozny, B. (2014) · Cited by 519
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
Luong, M.-T., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1100 · Cited by 375
Schütze, H. (1992) · Cited by 300
Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs
Ballesteros, M., Dyer, C., & Smith, N. A. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1041 · Cited by 293
Joint learning of character and word embeddings
Chen, X., Xu, L., Liu, Z., Sun, M., & Luan, H. (2015) · International Conference on Artificial Intelligence · Cited by 262
Schütze, H. (1992) · neural information processing systems · Cited by 212
Compositional Morphology for Word Representations and Language ModellingPreprint
Botha, J. A., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4273 · Cited by 209
Charagram: Embedding Words and Sentences via Character n-grams
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1157 · Cited by 189
Unsupervised Morphology Induction Using Word Embeddings
Soricut, R., & Och, F. J. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1186 · Cited by 135
Cross-lingual semantic relatedness using encyclopedic knowledge
Hassan, S., & Mihalcea, R. F. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699648.1699665 · Cited by 124
Using the Structure of a Conceptual Network in Computing Semantic Relatedness
Gurevych, I. (2005) · Lecture notes in computer science · doi:10.1007/11562214_67 · Cited by 117
Lazaridou, A., Marelli, M., Zamparelli, R., & Baroni, M. (2013) · Cited by 112
Cotterell, R., & Schütze, H. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1140 · Cited by 100
Factored neural language models
Alexandrescu, A., & Kirchhoff, K. (2006) · doi:10.3115/1614049.1614050 · Cited by 97
Co-learning of Word Representations and Morpheme Representations
Qiu, S., Cui, Q., Bian, J., Gao, B., & Liu, T. (2014) · Cited by 84
Normalizing tweets with edit scripts and recurrent neural embeddings
Chrupała, G. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2111 · Cited by 81
Automatically creating datasets for measures of semantic relatedness
Zesch, T., & Gurevych, I. (2006) · doi:10.3115/1641976.1641980 · Cited by 73
Identifying Opportunities for Valuable Encounters
Mayer, J. M., Jones, Q., & Hiltz, S. R. (2015) · ACM Transactions on Information Systems · doi:10.1145/2751557 · Cited by 66
Joubarne, C., & Inkpen, D. Z. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21043-3_26 · Cited by 52
Word Embeddings Go to Italy: A Comparison of Models and Training Datasets.
Berardi, G., Esuli, A., & Marcheggiani, D. (2015) · IIR eBooks · Cited by 45
Multilingual Reliability and "Semantic" Structure of Continuous Word Spaces
Köper, M., Scheible, C., & Walde, S. S. I. (2015) · Cited by 43
Morphology-based and sub-word language modeling for Turkish speech recognition
Sak, H., Saraçlar, M., & Güngör, T. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494927 · Cited by 40
Human and Machine Judgements for Russian Semantic Relatedness
Panchenko, A., Ustalov, D., Arefyev, N., Paperno, D., Konstantinova, N., Loukachevitch, N., & Biemann, C. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-52920-2_21 · Cited by 40
Alternative structures for character-level RNNsPreprint
Bojanowski, P., Joulin, A., & Mikolov, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06303 · Cited by 34
Self-citationLetter N-Gram-based Input Encoding for Continuous Space Language Models
Sperr, H., Niehues, J., & Waibel, A. (2013) · doi:10.5445/ir/1000037718 · Cited by 22
Cui, Q., Gao, B., Bian, J., Qiu, S., Dai, H., & Liu, T. (2015) · ACM Transactions on Information Systems · doi:10.1145/2797137 · Cited by 15
New Word Analogy Corpus for Exploring Embeddings of Czech Words
Svoboda, L., & Brychcín, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75477-2_6 · Cited by 13
Flagged first, then by citations
Research works printed but not linked: 1
- Cui Qing, 2015, ACM Transactions on Information Systems, 34, 1
Cited by
Sorted by citations
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
Self-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 2,008
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736
Metadata
- Authors
- Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T.
- Affiliations
- Meta (United States) (US)
- Facebook AI Research,
- Corresponding author
- Bojanowski, Piotr
- Joulin, Armand
- ORCID
- Bojanowski, Piotr
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2017-12
- Volume
- 5
- Pages
- 135-146
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2493916176
- Cited by
- 9,645
- References
- 47
- Citations by year
- 2026: 389 · 2025: 677 · 2024: 790 · 2023: 1188 · 2022: 1253 · 2021: 1691 · 2020: 1561 · 2019: 1288 · 2018: 648 · 2017: 142
- FWCI
- 593.75
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- word embeddings, character n-grams, skip-gram model, out-of-vocabulary words, word similarity, word analogy, multilingual word representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
