CoChat
Sign Up
PreprintCited by 165Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

Nearest Neighbor Machine Translation

Open access
Khandelwal, Urvashi, Fan, Angela, Jurafsky, Dan, Zettlemoyer, Luke, Lewis, Mike·2020·arXiv
DOI 10.48550/arxiv.2010.00710arXiv 2010.00710
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct access to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest neighbor search improves a state-of-the-art German-English translation model by 1.5 BLEU. $k$NN-MT allows a single model to be adapted to diverse domains by using a domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results -- without training on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of 3 BLEU for translating from English into German and Chinese. Qualitatively, $k$NN-MT is easily interpretable; it combines source and target context to retrieve highly relevant examples.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.00710

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The kNN-LM outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, domain adaptation settings, and conditional generation tasks such as machine translation.

    “This model is particularly notable for its large improvements in performance it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation (Khandelwal et al., 2020).”

    From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2020

    The Introduction (p. 1) reports improvements in machine translation and domain adaptation, but attributes the earlier unconditional language-model work to another paper. Section 2 (p. 2) identifies kNN-MT as a generalization of kNN-LM. This source does not establish the claimed margin over parametric models on standard language-modeling benchmarks, and its translation results concern kNN-MT rather than kNN-LM. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Random pruning has been shown to work well with a billion-scale datastore in machine translation tasks.

    “Random pruning has been shown to work well with a billion-scale datastore in machine translation tasks (Khandelwal et al., 2020).”

    From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2020

    Section 5 (p. 7) reports that smaller datastores retain much of the translation benefit, while Section 3 (p. 4) describes retrieval from datastores containing billions of items. Our copy does not say those smaller datastores were produced by random pruning, so it does not establish that random pruning works well. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Similar approximations were adopted to apply kNN-LM to machine translation tasks.

    “Similar approximations were adopted to apply k NN-LM to machine translation tasks (Khandelwal et al., 2020).”

    From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 42 · Cited by 165

Research works printed but not linked: 39

  1. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019 , 2019. · arXiv
  2. Ankur Bapna and Orhan Firat. Non-parametric adaptation for neural machine translation. In North American Association of Computational Linguistics (NAACL) , 2019.
  3. Lo¨ ıc Barrault, Ondˇ rej Bojar, Marta R. Costa-juss` a, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M¨ uller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5301. URL https://www.aclweb.org/anthology/W19-5301 . · DOI
  4. Ondˇ rej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers , pp. 272-303, Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-6401. URL https://www.aclweb.org/anthology/W18-6401 . · DOI
  5. Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics , 19(2): 263-311, 1993.
  6. Bram Bulte and Arda Tezcan. Neural fuzzy repair: Integrating fuzzy matches into neural machine translation. In Association of Computational Linguistics (ACL) , 2019.
  7. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Empirical Methods in Natural Language Processing (EMNLP) , 2020.
  8. Trevor Cohn and Mirella Lapata. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics , pp. 728-735, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/P07-1092 .
  9. Takao Doi, Hirofumi Yamamoto, and Eiichiro Sumita. Example-based machine translation using efficient sentence retrieval based on edit-distance. ACM Transactions on Asian Language Information Processing , 4(4):377-399, 2005.
  10. Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. Augmenting transformers with knn-based composite memory for dialogue. arXiv preprint arXiv:2004.12744 , 2020. · arXiv
  11. M Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation , pp. 127-137, 2017.
  12. Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. In NIPS , pp. 6042-6052, 2017a.
  13. Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In International Conference on Learning Representations (ICLR) , 2017b.
  14. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided neural machine translation. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.
  15. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning (ICML) , 2020.
  16. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909 , 2020. · arXiv
  17. Rejwanul Haque, Sudip Naskar, Yanjun Ma, and Andy Way. Using supertags as source language context in smt. 13th Annual Conference of the European Association for Machine Translation, (EAMT) , 01 2009.
  18. Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.
  19. Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime G Carbonell. Domain adaptation of neural machine translation by lexicon induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2989-3001, 2019.
  20. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations (ICLR) , 2020.
  21. Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In First Workshop on Neural Machine Translation (WNMT) , 2017.
  22. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) , 2018.
  23. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨ aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401 , 2020. · arXiv
  24. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. In Fourth Conference on Machine Translation (WMT) , 2019.
  25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In North American Association of Computational Linguistics: System Demonstrations (NAACL) , 2019.
  26. Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pp. 186-191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ W18-6319 .
  27. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? In North American Association of Computational Linguistics (NAACL) , 2018.
  28. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. Ccmatrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944 , 2019. · arXiv
  29. Nicolas Stroppa, Antal van den Bosch, and Andy Way. Exploiting source similarity for smt using context-informed features. Proceedings of The 11th Conference on Theoretical and Methodological Issues in Machine Translation , 01 2007.
  30. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with a continuous cache. In Transactions of the Association of Computational Linguistics (TACL) , 2018.
  31. Masao Utiyama and Hitoshi Isahara. A comparison of pivot methods for phrase-based statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference , pp. 484-491, Rochester, New York, April 2007. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/N07-1061 .
  32. Antal van den Bosch, Nicolas Stroppa, and Andy Way. A memory-based classification approach to marker-based ebmt. METIS-II Workshop on New Approaches to Machine Translation , 04 2007.
  33. Maarten van Gompel, Antal van den Bosch, and Peter Berck. Extending memory-based machine translation to phrases. In Computational Linguistics in the Netherlands (CLIN) , pp. 79-86, 01 2010.
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS) , 2017.
  35. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm´ an, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 , 2019. · arXiv
  36. Jason Weston, Emily Dinan, and Alexander H Miller. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.04776 , 2018. · arXiv
  37. Jitao Xu, Josep Crego, and Jean Senellart. Boosting neural machine translation with similar translations. In Association of Computational Linguistics (ACL) , 2020.
  38. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867 , 2020. · arXiv
  39. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. Guiding neural machine translation with retrieved translation pieces. In North American Association of Computational Linguistics (NAACL) , 2018.

Authors
Khandelwal, U., Fan, A., Jurafsky, D., Zettlemoyer, L., & Lewis, M.
Affiliations
  • Meta (United States) (US)
  • Meta (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.00710
arXiv
2010.00710
OpenAlex
W3090145300
Cited by
165
References
42
Citations by year
2025: 9 · 2024: 22 · 2023: 67 · 2022: 43 · 2021: 23 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Learning Representations
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
nearest neighbor search, example-based translation, zero-shot transfer, domain adaptation, multilingual machine translation, token prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review