CoChat

Neural Machine Translation of Rare Words with Subword Units

Open access
Sennrich, Rico, Haddow, Barry, Birch, Alexandra · 2016 · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 11 of the 25 sources its claims cite were checked for support.

Abstract

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem.Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary.In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units.This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations).We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English→German and English→Russian by up to 1.1 and 1.3 BLEU, respectively.

Overview

CoChat Summary

TL;DR

This paper introduces subword-unit representations, especially byte pair encoding (BPE), for open-vocabulary neural machine translation, avoiding back-off dictionaries for rare and unseen words. On WMT 2015 English→German and English→Russian, subword models improve over a word-level back-off dictionary baseline by up to 1.1 and 1.3 BLEU, respectively.

Key findings

  • Subword models improve rare-word translation: unigram F1 for rare words rises from 36.8% to 41.8% for English→German and from 26.5% to 29.7% for English→Russian.
  • For OOVs, the back-off dictionary baseline works relatively well for English→German because many OOVs are names, but for English→Russian it performs poorly (9.2% precision, 5.2% recall), while BPE-J90k improves both (21.9% precision, 15.6% recall).
  • Joint BPE (BPE-J90k) is more effective than independently learned BPE (BPE-60k) for unigram F1; for English→German, C2-50k achieves the best BLEU of 25.3, while BPE-J90k achieves the best CHRF3 of 54.1.
  • In an analysis of 100 rare German tokens, the majority are potentially translatable via smaller units: 56 compounds, 21 names, 6 loanwords with common origin, 5 transparent affixation cases, 1 number, and 1 computer language identifier.

Methods

  • The NMT system follows Bahdanau et al. (2015): a bidirectional GRU encoder and an attention-based recurrent decoder, implemented with Groundhog; hidden layer size 1000, embedding size 620, shortlist of 30,000 words, Adadelta, minibatch size 80, beam size 12, and training for approximately 7 days followed by continued training of the last 4 saved models.
  • Data are from WMT 2015: English→German uses 4.2 million sentence pairs (~100 million tokens); English→Russian uses 2.6 million sentence pairs (~50 million tokens); newstest2013 is the development set, and newstest2014/newstest2015 are test sets.
  • The paper compares word-level baselines (WUnk, WDict), character bigrams, BPE, and joint BPE, evaluating with BLEU, CHRF3, and unigram F1 for all words, rare words, and OOVs.

Limitations

  • The paper states that performance variability remains an open problem in NMT, with differences of up to 1 BLEU between models on the development set, and that the choice of vocabulary size is somewhat arbitrary and mainly motivated by comparison to prior work.
  • The paper notes that not every segmentation is transparent, that BPE-60k suffers from transliteration/copy errors due to inconsistent segmentations, and that joint BPE can produce unknown segments, though these are rare (0.05% of test tokens).
  • Evident from the text: experiments cover only two language pairs and WMT 2015 data, and the authors note that comparisons to some other neural systems are not fully controlled because those systems use dropout or larger ensembles.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/p16-1162

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

10 Valid · 3 Not valid · 7 Not checked · 98 Own results and interpretations

56 more claims

Create a free account to see all 118 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 38
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 38 references

Research works printed but not linked: 1

  1. [23]23. Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2012. Machine Translation without Words through Substring Alignment. In The 50th Annual Meeting of the Association for Compu- tational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea -Volume 1: Long Papers, pages 165-174.

Cited by

  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404

    “In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [31] and by…”
    Background
  • Learning Transferable Visual Models From Natural Language Supervision

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293

    “The transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015).”
    Background
  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929

    “We use SentencePiece [Kudo and Richardson, 2018] to encode text as WordPiece tokens [Sennrich et al., 2015; Kudo, 2018].”
    Method
  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550

    “We train language models on WebText2, an extended version of the WebText [RWC + 19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n vocab = 50257 .”
    Method
  • LLaMA: Open and Efficient Foundation Language Models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602

    “We tokenize the data with the byte-pair encoding (BPE) algorithm (Sennrich et al., 2015), using the implementation from Sentence-Piece (Kudo and Richardson, 2018).”
    Method

See all 3,005

Metadata

Authors
Sennrich, R., Haddow, B., & Birch, A.
Affiliations
University of Edinburgh (GB)
Venue
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Published
2016
Pages
1715-1725
ISSN
0736-587X (ISSN-L)
OpenAlex
W1816313093
Cited by
974
References
31
Citations by year
2026: 74 · 2025: 85 · 2024: 104 · 2023: 135 · 2022: 123 · 2021: 139 · 2020: 148 · 2019: 83 · 2018: 48 · 2017: 27 · 2016: 7
FWCI
29.08
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • European Commission (HORIZON2020, 645452)
  • Child Health Research Foundation
  • Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
neural machine translation, subword units, rare words, out-of-vocabulary words, word segmentation, byte pair encoding, transliteration, compositional translation, loanwords, cognates, English-German translation, English-Russian translation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Neural Machine Translation of Rare Words with Subword Units. https://app.cochat.ai/papers/10.18653/v1/p16-1162
The paper
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/p16-1162

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.