Neural Machine Translation of Rare Words with Subword Units
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 11 of the 25 sources its claims cite were checked for support.
Abstract
Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem.Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary.In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units.This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations).We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English→German and English→Russian by up to 1.1 and 1.3 BLEU, respectively.
Overview
CoChat Summary
TL;DR
This paper introduces subword-unit representations, especially byte pair encoding (BPE), for open-vocabulary neural machine translation, avoiding back-off dictionaries for rare and unseen words. On WMT 2015 English→German and English→Russian, subword models improve over a word-level back-off dictionary baseline by up to 1.1 and 1.3 BLEU, respectively.
Key findings
- Subword models improve rare-word translation: unigram F1 for rare words rises from 36.8% to 41.8% for English→German and from 26.5% to 29.7% for English→Russian.
- For OOVs, the back-off dictionary baseline works relatively well for English→German because many OOVs are names, but for English→Russian it performs poorly (9.2% precision, 5.2% recall), while BPE-J90k improves both (21.9% precision, 15.6% recall).
- Joint BPE (BPE-J90k) is more effective than independently learned BPE (BPE-60k) for unigram F1; for English→German, C2-50k achieves the best BLEU of 25.3, while BPE-J90k achieves the best CHRF3 of 54.1.
- In an analysis of 100 rare German tokens, the majority are potentially translatable via smaller units: 56 compounds, 21 names, 6 loanwords with common origin, 5 transparent affixation cases, 1 number, and 1 computer language identifier.
Methods
- The NMT system follows Bahdanau et al. (2015): a bidirectional GRU encoder and an attention-based recurrent decoder, implemented with Groundhog; hidden layer size 1000, embedding size 620, shortlist of 30,000 words, Adadelta, minibatch size 80, beam size 12, and training for approximately 7 days followed by continued training of the last 4 saved models.
- Data are from WMT 2015: English→German uses 4.2 million sentence pairs (~100 million tokens); English→Russian uses 2.6 million sentence pairs (~50 million tokens); newstest2013 is the development set, and newstest2014/newstest2015 are test sets.
- The paper compares word-level baselines (WUnk, WDict), character bigrams, BPE, and joint BPE, evaluating with BLEU, CHRF3, and unigram F1 for all words, rare words, and OOVs.
Limitations
- The paper states that performance variability remains an open problem in NMT, with differences of up to 1 BLEU between models on the development set, and that the choice of vocabulary size is somewhat arbitrary and mainly motivated by comparison to prior work.
- The paper notes that not every segmentation is transparent, that BPE-60k suffers from transliteration/copy errors due to inconsistent segmentations, and that joint BPE can produce unknown segments, though these are rare (0.05% of test tokens).
- Evident from the text: experiments cover only two language pairs and WMT 2015 data, and the authors note that comparisons to some other neural systems are not fully controlled because those systems use dropout or larger ensembles.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 7 Not checked · 98 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“It outperforms our WDict baseline by 1.5 BLEU.”
§ Neural Machine Translation of Rare Words with Subword Units > 4.2 Translation experiments
What the sources say
- Supportedcites Haddow et al. (2015) source
Section 4.4 describes Haddow et al.’s English→Russian system, and Table 4 reports its BLEU score. The 1.5-BLEU comparison with the citing authors’ own WDict baseline is their result, not a comparison attributed to Haddow et al. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For word-level NMT models, the translation of out-of-vocabulary words has been addressed through a back-off to a dictionary look-up (Jean et al., 2015;Luong et al., 2015b).”
§ Neural Machine Translation of Rare Words with Subword Units > 1 Introduction
What the sources say
- Supportedcites Jean et al., 2015 source
Section 3.3 says the model replaces generated [UNK] tokens with the aligned source word or its most likely translation from a word-alignment model. Section 4.1 confirms that the experiments use a bilingual dictionary to replace unknown words in translations. This supports the described dictionary-lookup back-off for word-level NMT. (LLM, conf=0.92)
- Not matchedcites Luong et al., 2015b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For English→Russian, the state of the art is the phrase-based system by Haddow et al. (2015).”
§ Neural Machine Translation of Rare Words with Subword Units > 4.2 Translation experiments
What the sources say
- Supportedcites Haddow et al. (2015) source
Section 4.4 describes the authors’ English→Russian system, and the Conclusion reports that their submission ranked first in cased BLEU on the WMT 2015 newstest2015 evaluation set for English→Russian. This supports the state-of-the-art characterization in that evaluation, though it does not establish superiority on every measure or dataset. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the unigram representation performed poorly in preliminary experiments”
§ Neural Machine Translation of Rare Words with Subword Units > 4.1 Subword statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For OOVs, the baseline strategy of copying unknown words works well for English→German.”
§ Neural Machine Translation of Rare Words with Subword Units > 4.2 Translation experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
56 more claims
Create a free account to see all 118 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 38
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The most widely adopted method is byte-pair encoding (BPE) [42], which iteratively merges frequent byte sequences into single tokens.”
From A Survey of Large Language Models (2026) · cites this paper as [42]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d , which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [38] and byte-pair [31] representations.”
From Attention Is All You Need (2017) · cites this paper as [31]
Citations
Cites
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 2×
“The encoder is a bidirectional neural network with gated recurrent units (Cho et al., 2014) that reads an input sequence x = ( x 1 , ..., x m ) and calculates a forward sequence of hidden states ( −→ h 1 , ..., −→ h m ) , and a backward sequence ←− −→ ←− The hidden states h j and h j are…”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 4×
“Regarding other neural systems, Luong et al. (2015a) report a B LEU score of 25.9 on newstest2015, but we note that they use an ensemble of 8 independently trained models, and also report strong improvements from applying dropout, which we did not use.”
BackgroundMethodInfluentialADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · cited 2×
“We perform two independent training runs for each models, once with cut-off for gradient clipping (Pascanu et al., 2013) of 5.0, once with a cut-off of 1.0 – the latter produced better single models for most settings.”
MethodInfluentialCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · cited 2×
“Various techniques have been proposed to produce fixed-length continuous word vectors based on characters or morphemes (Luong et al., 2013; Botha and Blunsom, 2014; Ling et al., 2015a; Kim et al., 2015).”
BackgroundCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,491 · cited 2×
“Various techniques have been proposed to produce fixed-length continuous word vectors based on characters or morphemes (Luong et al., 2013; Botha and Blunsom, 2014; Ling et al., 2015a; Kim et al., 2015).”
BackgroundRecurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1176 · Cited by 1,232
Research works printed but not linked: 1
- [23]23. Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2012. Machine Translation without Words through Substring Alignment. In The 50th Annual Meeting of the Association for Compu- tational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea -Volume 1: Long Papers, pages 165-174.
Cited by
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
“In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [31] and by…”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
“The transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
“We use SentencePiece [Kudo and Richardson, 2018] to encode text as WordPiece tokens [Sennrich et al., 2015; Kudo, 2018].”
MethodScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“We train language models on WebText2, an extended version of the WebText [RWC + 19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n vocab = 50257 .”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
“We tokenize the data with the byte-pair encoding (BPE) algorithm (Sennrich et al., 2015), using the implementation from Sentence-Piece (Kudo and Richardson, 2018).”
Method
Metadata
- Authors
- Sennrich, R., Haddow, B., & Birch, A.
- Affiliations
- University of Edinburgh (GB)
- Venue
- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2016
- Pages
- 1715-1725
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W1816313093
- Cited by
- 974
- References
- 31
- Citations by year
- 2026: 74 · 2025: 85 · 2024: 104 · 2023: 135 · 2022: 123 · 2021: 139 · 2020: 148 · 2019: 83 · 2018: 48 · 2017: 27 · 2016: 7
- FWCI
- 29.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission (HORIZON2020, 645452)
- Child Health Research Foundation
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- neural machine translation, subword units, rare words, out-of-vocabulary words, word segmentation, byte pair encoding, transliteration, compositional translation, loanwords, cognates, English-German translation, English-Russian translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Neural Machine Translation of Rare Words with Subword Units. https://app.cochat.ai/papers/10.18653/v1/p16-1162
- The paper
- Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/p16-1162
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.