CoChat
Sign Up
Cited by 9,645Topic ModelingNatural Language Processing TechniquesAdvanced Graph Neural Networks

Enriching Word Vectors with Subword Information

Open access
Bojanowski, Piotr, Grave, Edouard, Joulin, Armand, Mikolov, Tomas·2017·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00051
Open this paper in CoChat — free accountOpen PDF

Abstract

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00051

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Continuous word representations capture not only lexical but also semantic properties of tokens, contrary to string similarity-based approaches.

    “Contrary to string similarity-based approaches, continuous word representations capture not only lexical but also semantic properties of tokens.”

    From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Bojanowski et al. (2017)

    The Introduction describes continuous word representations learned from co-occurrence statistics, and §5.1 evaluates their correspondence with human word-similarity judgments. The paper also models word-form information through character n-grams. It does not establish the claimed contrast with string-similarity-based approaches or say that those approaches cannot capture semantic properties. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • We screened 4 different word embeddings and found that the Common Crawl embeddings achieved better performance compared to other embeddings on a validation data set.

    “We screened 4 different word embeddings following a similar procedure reported in our previous study [46] and found that the Common Crawl embeddings—released by Facebook and trained using the fastText on the Common Crawl data set [47]—achieved better performance compared to other embeddings on a validation data set.”

    From Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models (2020) · cites this paper as [47]

    The Implementation details section says the paper’s subword word-vector model is publicly available, and the copy links to facebookresearch/fastText. But the Datasets section says its models were trained chiefly on Wikipedia, not Common Crawl; the copy does not identify a Common Crawl embedding release. The validation-set comparison is presented as the citing authors’ own result, not a result attributed to [47]. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Fasttext has the capacity to collect sub-word information, which can be helpful in the case of compound words.

    “I use Fasttext (Bojanowski et al. 2017) as a word representation model because of its capacity to collect sub-word information, which can be helpful in the case of compound words.”

    From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Bojanowski et al. 2017

  • FastText involves learning large-scale word embeddings.

    “This technique involves learning large-scale word embeddings (Bojanowski et  al. 2017; Joulin et al. 2016), representing an advancement from Word2Vec.”

    From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Bojanowski et al. 2017

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Andrei Alexandrescu and Katrin Kirchhoff. 2006. Fac- tored neural language models. In Proc. NAACL.
  2. Ryan Cotterell and Hinrich Schütze. 2015. Morphologi- cal word-embeddings. In Proc. NAACL.
  3. Qing Cui, Bin Gao, Jiang Bian, Siyu Qiu, Hanjun Dai, and Tie-Yan Liu. 2015. KNET: A general frame- work for learning word embedding using morpholog- ical knowledge. ACM Transactions on Information Systems, 34(1):4:1-4:25.
  4. Hinrich Schütze. 1993. Word space. In Adv. NIPS.
  5. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL.

Cited by

Sorted by citations

Related

Authors
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T.
Affiliations
  • Meta (United States) (US)
  • Facebook AI Research,
Corresponding author
  • Bojanowski, Piotr
  • Joulin, Armand
ORCID
Bojanowski, Piotr
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2017-12
Volume
5
Pages
135-146
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl_a_00051
OpenAlex
W2493916176
Cited by
9,645
References
47
Citations by year
2026: 389 · 2025: 677 · 2024: 790 · 2023: 1188 · 2022: 1253 · 2021: 1691 · 2020: 1561 · 2019: 1288 · 2018: 648 · 2017: 142 · 2016: 15 · 2013: 1 · 2012: 2
FWCI
593.75
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
Keywords
word embeddings, character n-grams, skip-gram model, out-of-vocabulary words, word similarity, word analogy, multilingual word representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review