Neural Machine Translation of Rare Words with Subword Units
Open accessAbstract
Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A good choice of tokenization can be critical to the performance of sequence models.
“It is widely known that a good choice of tokenization can be critical to the performance of sequence models [55, 57].”
From FAST: Efficient Action Tokenization for Vision-Language-Action Models (2025) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 35
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Represen- tations (ICLR).
- Issam Bazzi and James R. Glass. 2000. Modeling out- of-vocabulary words for robust speech recognition. In Sixth International Conference on Spoken Lan- guage Processing, ICSLP 2000 / INTERSPEECH 2000, pages 401-404, Beijing, China.
- Jan A. Botha and Phil Blunsom. 2014. Compositional Morphology for Word Representations and Lan- guage Modelling. In Proceedings of the 31st Inter- national Conference on Machine Learning (ICML), Beijing, China.
- Rohan Chitnis and John DeNero. 2015. Variable- Length Word Encodings for Neural Translation Models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learn- ing Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Pro- ceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
- Mathias Creutz and Krista Lagus. 2002. Unsupervised Discovery of Morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonolog- ical Learning, pages 21-30. Association for Compu- tational Linguistics.
- Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn. 2014. Integrating an Unsupervised Translit- eration Model into Statistical Machine Translation. In Proceedings of the 14th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, EACL 2014, pages 148-153, Gothen- burg, Sweden.
- Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparame- terization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 644-648, At- lanta, Georgia. Association for Computational Lin- guistics.
- Philip Gage. 1994. A New Algorithm for Data Com- pression. C Users J., 12(2):23-38, February.
- Barry Haddow, Matthias Huck, Alexandra Birch, Niko- lay Bogoychev, and Philipp Koehn. 2015. The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 126-133, Lisbon, Portugal. Association for Computational Linguistics.
- Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1-10, Beijing, China. Association for Computa- tional Linguistics.
- Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Nat- ural Language Processing, Seattle. Association for Computational Linguistics.
- Yoon Kim, Yacine Jernite, David Sontag, and Alexan- der M. Rush. 2015. Character-Aware Neural Lan- guage Models. CoRR, abs/1508.06615. · arXiv
- Philipp Koehn and Kevin Knight. 2003. Empirical Methods for Compound Splitting. In EACL '03: Proceedings of the Tenth Conference on European Chapter of the Association for Computational Lin- guistics, pages 187-193, Budapest, Hungary. Asso- ciation for Computational Linguistics.
- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007 Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
- Franklin M. Liang. 1983. Word hy-phen-a-tion by com-put-er. Ph.D. thesis, Stanford University, De- partment of Linguistics, Stanford, CA.
- Wang Ling, Chris Dyer, Alan W. Black, Isabel Tran- coso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015a. Finding Function in Form: Compositional Character Models for Open Vocab- ulary Word Representation. In Proceedings of the 2015 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pages 1520- 1530, Lisbon, Portugal. Association for Computa- tional Linguistics.
- Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015b. Character-based Neural Machine Translation. ArXiv e-prints, November.
- Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better Word Representations with Recursive Neural Networks for Morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pages 104- 113.
- Thang Luong, Hieu Pham, and Christopher D. Man- ning. 2015a. Effective Approaches to Attention- based Neural Machine Translation. In Proceed- ings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing, pages 1412- 1421, Lisbon, Portugal. Association for Computa- tional Linguistics.
- Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the Rare Word Problem in Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 11-19, Beijing, China. Association for Computa- tional Linguistics.
- Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2012. Machine Translation without Words through Substring Alignment. In The 50th Annual Meeting of the Association for Compu- tational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea -Volume 1: Long Papers, pages 165-174.
- Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In 18th Int. Conf. on Computational Linguistics, pages 1081-1085.
- Razvan Pascanu, Tomas Mikolov, and Yoshua Ben- gio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th Inter- national Conference on Machine Learning, ICML 2013, pages 1310-1318, Atlanta, USA.
- Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
- Rico Sennrich and Barry Haddow. 2015. A Joint Dependency Model of Morphological and Syntac- tic Structure for Statistical Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2081-2087, Lisbon, Portugal. Association for Com- putational Linguistics.
- Benjamin Snyder and Regina Barzilay. 2008. Unsu- pervised Multilingual Learning for Morphological Segmentation. In Proceedings of ACL-08: HLT, pages 737-745, Columbus, Ohio. Association for Computational Linguistics.
- David Stallard, Jacob Devlin, Michael Kayser, Yoong Keok Lee, and Regina Barzilay. 2012. Unsu- pervised Morphology Rivals Supervised Morphol- ogy for Arabic MT. In The 50th Annual Meeting of the Association for Computational Linguistics, Pro- ceedings of the Conference, July 8-14, 2012, Jeju Island, Korea -Volume 2: Short Papers, pages 322- 327.
- Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Met- rics Shared Task. In Proceedings of the Tenth Work- shop on Statistical Machine Translation, pages 256- 273, Lisbon, Portugal. Association for Computa- tional Linguistics.
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Net- works. In Advances in Neural Information Process- ing Systems 27: Annual Conference on Neural Infor- mation Processing Systems 2014, pages 3104-3112, Montreal, Quebec, Canada.
- Jörg Tiedemann. 2009. Character-based PSMT for Closely Related Languages. In Proceedings of 13th Annual Conference of the European Association for Machine Translation (EAMT'09), pages 12-19.
- Jörg Tiedemann. 2012. Character-Based Pivot Trans- lation for Under-Resourced Languages and Do- mains. In Proceedings of the 13th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 141-151, Avignon, France. Association for Computational Linguistics.
- David Vilar, Jan-Thorsten Peter, and Hermann Ney. 2007. Can We Translate Letters? In Second Work- shop on Statistical Machine Translation, pages 33- 39, Prague, Czech Republic. Association for Com- putational Linguistics.
- Sami Virpioja, Jaakko J. Väyrynen, Mathias Creutz, and Markus Sadeniemi. 2007. Morphology-Aware Statistical Machine Translation Based on Morphs Induced in an Unsupervised Manner. In Proceed- ings of the Machine Translation Summit XI, pages 491-498, Copenhagen, Denmark.
- Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701. · arXiv
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
Full Line Code Completion: Bringing AI to Desktop
Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.08704 · Cited by 8 · see citing works
WorldVLA: Towards Autoregressive Action World Model
Cen, J., Yu, C., Yuan, H., Jiang, Y., Huang, S., Guo, J., Li, X., Song, Y., Luo, H., Wang, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21539 · Cited by 1 · see citing works
FAST: Efficient Action Tokenization for Vision-Language-Action Models
Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., & Levine, S. (2025) · arXiv · doi:10.48550/arxiv.2501.09747 · Cited by 73 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
- Authors
- Sennrich, R., Haddow, B., & Birch, A.
- Affiliations
- University of Edinburgh (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- Pages
- 1715-1725
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 1508.07909
- OpenAlex
- W1816313093
- Cited by
- 950
- References
- 0
- Citations by year
- 2026: 67 · 2025: 85 · 2024: 104 · 2023: 135 · 2022: 123 · 2021: 135 · 2020: 143 · 2019: 81 · 2018: 46 · 2017: 23 · 2016: 7 · 2015: 1
- FWCI
- 28.20
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission (HORIZON2020, 645452)
- Child Health Research Foundation
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- neural machine translation, subword units, rare words, out-of-vocabulary words, word segmentation, byte pair encoding, transliteration, compositional translation, loanwords, cognates, English-German translation, English-Russian translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
