The DOI matches its DataCite record; 25 of the 28 references it cites were found in a registry or an index.
Generalization through Memorization: Nearest Neighbor Language Models
Open accessAbstract
We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“d ( · , · ) is a distance function between the two vectors, and L 2 was shown to be more effective than other alternatives (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This model is particularly notable for its large improvements in performance it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation (Khandelwal et al., 2020).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The k NN-LMcomputes the probability of the next token by interpolating a parametric LM with a distribution calculated from the k nearest context-token pairs in the datastore, as demonstrated in Figure 2.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Note that the dataset to contruct the datastore may not necessarily be the training data that trains the parametric NLM in Eq. 1 - a separate dataset may be used for the datastore construction which would lead to potential applications such as training-free domain adaptation or a gradient-free way to utilize extra training data (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The perplexity continues improving as k grows as shown in (Khandelwal et al., 2019) and confirmed by us.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Note that the application of k NN-LM requires no additional training; the parameters of the NLM remain as-is, and Eq. 1 is only applied at test time.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 1,476
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 729
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep LearningPreprint
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 592
Learning to Remember Rare EventsPreprint
Kaiser, Ł., Nachum, O., Roy, A., & Samy Bengio, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03129 · Cited by 322
MBT: A Memory-Based Part of Speech Tagger-Generator
Daelemans, W. M. P., Zavrel, J., Berck, P., & Gillis, S. A. (1996) · Research portal (Tilburg University) · doi:10.48550/arxiv.cmp-lg/9607012 · Cited by 305
Improving Neural Language Models with a Continuous CachePreprint
Grave, É., Joulin, A., & Usunier, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04426 · Cited by 281
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255
Search Engine Guided Neural Machine Translation
Gu, J., Wang, Y., Cho, K., & Li, V. O. K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12013 · Cited by 185
Retrieve and Refine: Improved Sequence Generation Models For Dialogue
Weston, J., Dinan, E., & Miller, A. (2018) · Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI · doi:10.18653/v1/w18-5713 · Cited by 165
Efficient softmax approximation for GPUs
Grave, É., Joulin, A., Cissé, M., & Jeǵou, H. (2017) · International Conference on Machine Learning · Cited by 101
Dynamic Evaluation of Transformer Language Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2019) · arXiv · doi:10.48550/arxiv.1904.08378 · Cited by 30
Lightweight Adaptive Mixture of Neural and N-gram Language ModelsPreprint
Bakhtin, A., Szlam, A. D., Ranzato, M., & Grave, É. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07705 · Cited by 15
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
Luo, H., Jiang, L., Belinkov, Y., & Glass, J. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1144 · Cited by 10
Zhao, J., & Cho, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09502 · Cited by 9
A Simple Cache Model for Image Recognition
Orhan, E. (2018) · neural information processing systems · Cited by 5
Research works printed but not linked: 4
- Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research , 3(Feb):1137-1155, 2003.
- Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. In NIPS , pp. 6042-6052, 2017a.
- Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics , 6:437-450, 2018.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS , 2019.
Cited by
Sorted by citations
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
“For “what to retrieve” research has progressed from simple token [ Khandelwal et al. , 2019 ] and entity retrieval [ Nishikawa et al. , 2022 ] to more complex structures like chunks [ Ram et al. , 2023 ] and knowledge graph [ Kang et al. , 2023 ] , with studies focusing on the granularity of…”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“Technical approaches to reducing risks of environmental harm include segmenting LMs into less large LMs that search and retrieve information from a distinct data corpus [25, 88, 97, 102, 115].”
BackgroundRetrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497
Metadata
- Authors
- Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M.
- Affiliations
- Meta (United States) (US)
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1911.00172
- OpenAlex
- W2988841832
- Cited by
- 285
- References
- 28
- Citations by year
- 2026: 1 · 2025: 34 · 2024: 34 · 2023: 111 · 2022: 52 · 2021: 39 · 2020: 13 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Styrelsen för Internationellt Utvecklingssamarbete
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- linear interpolation, pre-trained language models, embedding space, perplexity reduction, WikiText-103, domain adaptation, factual knowledge, next-word prediction, sequence similarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
