The DOI matches its DataCite record; 25 of the 28 references it cites were found in a registry or an index.
Generalization through Memorization: Nearest Neighbor Language Models
Open accessAbstract
We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“d ( · , · ) is a distance function between the two vectors, and L 2 was shown to be more effective than other alternatives (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This model is particularly notable for its large improvements in performance it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation (Khandelwal et al., 2020).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The k NN-LMcomputes the probability of the next token by interpolating a parametric LM with a distribution calculated from the k nearest context-token pairs in the datastore, as demonstrated in Figure 2.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Note that the dataset to contruct the datastore may not necessarily be the training data that trains the parametric NLM in Eq. 1 - a separate dataset may be used for the datastore construction which would lead to potential applications such as training-free domain adaptation or a gradient-free way to utilize extra training data (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The perplexity continues improving as k grows as shown in (Khandelwal et al., 2019) and confirmed by us.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Note that the application of k NN-LM requires no additional training; the parameters of the NLM remain as-is, and Eq. 1 is only applied at test time.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 1,476
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Research works printed but not linked: 4
- Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research , 3(Feb):1137-1155, 2003.
- Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. In NIPS , pp. 6042-6052, 2017a.
- Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics , 6:437-450, 2018.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS , 2019.
Cited by
Sorted by citations
Items 201–300 of 305
Haji, S., Suekane, K., Sano, H., & Takagi, T. (2023) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc56153.2023.00036 · Cited by 4
Decouple knowledge from paramters for plug-and-play language modeling
Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.901 · Cited by 9
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets
Laban, P., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.135 · Cited by 3
Memories for Virtual AI Characters
Landwehr, F., Doggett, E. V., & Weber, R. M. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.17 · Cited by 4
HistAlign: Improving Context Dependency in Language Generation by Aligning with History
Wan, D., Zhang, S., & Bansal, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.179 · Cited by 6
Attending to Long-Distance Document Context for Sequence Labeling
Jörke, M., Gillick, J., Sims, M., & Bamman, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.330 · Cited by 4
A Neighborhood Framework for Resource-Lean Content Flagging
Sarwar, S. M., Zlatkova, D., Hardalov, M., Dinkov, Y., Augenstein, I., & Nakov, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00472 · Cited by 9
Taking Notes on the Fly Helps BERT Pre-trainingPreprint
Wu, Q., Chen, X., Li, Y., Ke, G., He, D., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01466 · Cited by 6
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models
Doostmohammadi, E., Norlund, T., Kuhlmann, M., & Johansson, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.45 · Cited by 8
RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling
Deng, J., Pang, L., Shen, H., & Cheng, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.164 · Cited by 9
N-Gram Nearest Neighbor Machine Translation
Lv, R., Guo, J., Wang, R., Tan, X., Qin, T., & Liu, Q. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3407537 · Cited by 4
Santra, P., Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70021 · Cited by 5
$k-\text{NN}$ embeded space conditioning for enhanced few-shot object detection
Matcovici, S., Voinea, D., & Popa, A.-I. (2023) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw58289.2023.00044 · Cited by 4
On-the-Fly Text Retrieval for end-to-end ASR Adaptation
Yusuf, B., Gourav, A., Gandhe, A., & Bulyko, I. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095857 · Cited by 3
Suffix Retrieval-Augmented Language Modeling
Wang, Z., & Tam, Y.-C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096450 · Cited by 4
GRACE: Gradient-guided Controllable Retrieval for Augmenting Attribute-based Text Generation
Wen, Z., Tian, Z., Huang, Z., Yang, Y., Jian, Z., Wang, C., & Li, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.530 · Cited by 4
Wen, J., Zhou, H., Guan, J., Zhou, J., & Huang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.612 · Cited by 3
Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization
Hou, G., Shen, Y., Zhang, W., Xue, W., & Lü, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.842 · Cited by 3
Applied Federated Model Personalization in the Industrial Domain: A Comparative Study
Siniosoglou, I., Argyriou, V., Fragulis, G. F., Fouliras, P., Papadopoulos, G. T., Lytos, A., & Sarigiannidis, P. G. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3457803 · Cited by 3
Graph Data Management and Graph Machine Learning: Synergies and Opportunities
Khan, A., Ke, X., & Wu, Y. (2025) · ACM SIGMOD Record · doi:10.1145/3749116.3749122 · Cited by 5
Improving Entity Recall in Automatic Speech Recognition with Neural Embeddings
Li, C., Rondon, P., Caseiro, D., Velikovich, L., Velez, X., & Aleksic, P. S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413756 · Cited by 3
Meta-Learning Fast Weight Language Models
Clark, K., Guu, K., Chang, M.-W., Pasupat, P., Hinton, G. E., & Norouzi, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.661 · Cited by 15
On the Generalization Ability of Retrieval-Enhanced Transformers
Norlund, T., Doostmohammadi, E., Johansson, R., & Kuhlmann, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.109 · Cited by 8
Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer
Gao, R., Zhang, Z., Du, Y., Liu, L., & Wang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.964 · Cited by 3
An Instance Level Approach for Shallow Semantic Parsing in Scientific Procedural Text
Swarup, D., Bajaj, A., Mysore, S., O’Gorman, T., Das, R., & McCallum, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.270 · Cited by 3
Wu, Z., Kong, L., Bi, W., Li, X., & Kao, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14462 · Cited by 3
Larimar: Large Language Models with Episodic Memory Control
Das, P., Chaudhury, S., Nelson, E., Melnyk, I., Swaminathan, S., Dai, S., Lozano, A., Kollias, G., Chenthamarakshan, V., Jiří, et al. (2024) · arXiv · doi:10.48550/arxiv.2403.11901 · Cited by 3
MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models
Li, Z., Song, S., Wang, H., Niu, S., Chen, D., Yang, J., Xi, C., Lai, H., Zhao, J., Wang, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.22101 · Cited by 2
“Explicit long-term memory leverages external retrieval mechanisms, increasingly adopting structured formats—such as graphs and trees—to improve semantic integration and retrieval efficiency [8, 15, 31, 35].”
Memory-Based Invariance Learning for Out-of-Domain Text Classification
Chen, J., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.101 · Cited by 2
WebSurfer: Enhancing LLM Agents with Web-Wise Feedback for Web Navigation
Hu, D., Ge, J., Tang, W., Li, G., Li, L., & Wu, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889961 · Cited by 3
Shin, C. (2025) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10346-7 · Cited by 2
(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets43072 · Cited by 2
Corpus Complexity Matters in Pretraining Language Models
Agrawal, A., & Singh, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.20 · Cited by 3
ERATE: Efficient Retrieval Augmented Text Embeddings
Raina, V., Kassner, N., Popat, K., Lewis, P., Cancedda, N., & Martin, L. (2023) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2023.insights-1.2 · Cited by 2
Rajpoot, P. K., & Parikh, A. P. (2023) · doi:10.18653/v1/2023.pandl-1.1 · Cited by 2
Leveraging Biases in Large Language Models: "bias-kNN" for Effective Few-Shot Learning
Zhang, Y., Li, H., Li, Z., Cheng, N., Li, M., Xiao, J. Z., & Wang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447730 · Cited by 2
Buddiga, P. (2024) · Global Conference on Communications and Information Technologies (GCCIT) · doi:10.1109/gccit63234.2024.10862867 · Cited by 2
LLM+KG@VLDB 24 Workshop Summary
Khan, A., Wu, T., & Chen, X. (2025) · ACM SIGMOD Record · doi:10.1145/3749116.3749132 · Cited by 2
Knowledge Base Index Compression via Dimensionality and Precision Reduction
Zouhar, V., Mosbach, M., Zhang, M., & Klakow, D. (2022) · doi:10.18653/v1/2022.spanlp-1.5 · Cited by 3
Pathway2Text: Dataset and Method for Biomedical Pathway Description Generation
Yang, J., Liu, Z., Zhang, M., & Wang, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.108 · Cited by 2
Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries
Mittal, A., Sarawagi, S., Jyothi, P., Saon, G., & Kurata, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.916 · Cited by 2
Adaptive Chinese Pinyin IME for Most Similar Representation
Jiang, D., Cheng, X., & Han, T. (2022) · IEEE Access · doi:10.1109/access.2022.3218337 · Cited by 2
Improving Sequential Model Editing with Fact Retrieval
Han, X., Li, R., Tan, H., Wang, Y., Chai, Q., & Pan, J. Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.749 · Cited by 5
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition
Xu, G., Kordjamshidi, P., & Chai, J. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.818 · Cited by 2
Moratelli, N., Barraco, M., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3386099 · Cited by 2
Instance-Based Neural Dependency Parsing
Ouchi, H., Suzuki, J., Kobayashi, S., Yokoi, S., Kuribayashi, T., Yoshikawa, M., & Inui, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00439 · Cited by 2
Ember: No-Code Context Enrichment via Similarity-Based Keyless JoinsPreprint
Suri, S., Ilyas, I. F., Ré, C., & Rekatsinas, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01501 · Cited by 2
Zhang, C., Raghu, M., Kleinberg, J. M., & Samy Bengio, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12580 · Cited by 21
Benchmarking nearest neighbor retrieval of zebra finch vocalizations across developmentPreprint
Tomka, T., Hao, X., Miao, A., Lee, K., Basha, M., Reimann, S., Zai, A. T., & Hahnloser, R. H. R. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.04.555475 · Cited by 2
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
“A tested way of ensuring more factuality is through retrieval augmented generation (Lewis et al., 2021; Karpukhin et al., 2020; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020).”
Self-citationDatastore Distillation for Nearest Neighbor Machine Translation
Dai, Y.-H., Zhang, Z., Du, Y., Liu, S.-C., Liu, L.-M., & Xu, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3337633 · Cited by 1
Now It Sounds Like You: Learning Personalized Vocabulary On Device
Shenoy, A., Wang, S., Chuang, P., & Nguyen, J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31224 · Cited by 5
Wang, L., Wloka, B., & Lepage, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129680 · Cited by 1
Verified Telcorag: A Verified Local-Rag for Telecommunication Uses
Salajeghe, Y., Kopaee, A. M., Najjari, S., Ahmadi, I., Dehmolaee, M. A., & Khalaj, B. H. (2025) · Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit) · doi:10.1109/eucnc/6gsummit63408.2025.11037160 · Cited by 1
Dynamic Fusion Nearest Neighbor Machine Translation via Dempster-Shafer Theory
Yang, Z., Hou, H., Sun, S., Wu, N., Wang, Y., Jian, W., & Wang, P. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-7960-6_9 · Cited by 1
PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
Chang, W.-C., Jiang, J., Zhang, J., Al-Darabsah, M., Teo, C. H., Hsieh, C., Yu, H., & Vishwanathan, S. V. N. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635791 · Cited by 1
TRAMS: Training-free Memory Selection for Long-range Language Modeling
Yu, H., Wang, C., Zhang, Y., & Bi, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.331 · Cited by 5
Luo, M., Gokhale, T., Varshney, N., Yang, Y., & Baral, C. R. (2024) · Synthesis lectures on computer vision · doi:10.1007/978-3-031-57816-8_5 · Cited by 1
Multi-Hop Arabic LLM Reasoning in Complex QA
Abdelazim, H., Begemy, T., Galal, A., Sedki, H., & Mohamed, A. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.10.179 · Cited by 1
Enhancing Automated Vulnerability Repair Through Dependency Embedding and Pattern Store
Dong, Q., Lin, Y., Sun, H., & Gao, X. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00026 · Cited by 1
Large Product Key Memory for Pretrained Language Models
Kim, G., & Jung, T. H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.362 · Cited by 5
Identifying Settlements Using SVM and U-Net
Mutreja, G., Kumar, S. S., Jha, D., Singh, A., & Singh, R. (2020) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss39084.2020.9324702 · Cited by 1
Cross-domain Generalization for AMR Parsing
Bai, X., Yang, S., Cui, L., Song, L., & Zhang, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.749 · Cited by 2
Generating (Formulaic) Text by Splicing Together Nearest NeighborsPreprint
Wiseman, S., Bačkurs, A., & Stratos, K. (2021) · arXiv (Cornell University) · Cited by 1
MemBERT: Injecting Unstructured Knowledge into BERTPreprint
Ruggeri, F., Lippi, M., & Torroni, P. (2021) · arXiv (Cornell University) · Cited by 1
Combining Transformers with Natural Language ExplanationsPreprint
Ruggeri, F., Lippi, M., & Torroni, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4531749 · Cited by 2
Studies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“…retrieval augmented generation (RAG) is an effective framework that retrieves relevant information from external knowledge sources to aid and ground language models’ generation (Lewis et al., 2020; Khandelwal et al., 2020; Borgeaud et al., 2021; Izacard and Grave, 2021; Izacard et al., 2022b).”
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · arXiv · Cited by 59
“A number of approaches have been proposed to infuse external knowledge in the input layer (Guu et al., 2020; Shi et al., 2023), intermediate layers (Borgeaud et al., 2022) or at the output layer (Khandelwal et al., 2020; Zhong et al., 2022).”
BackgroundSynapse: Empowering LLM Agents with Episodic-Semantic Memory via Spreading Activation
Jiang, H., Chen, J., Pan, Y., Chen, L., You, W., Zhou, Y., Zhang, R., Abate, Y., & Liu, T. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1108
Leveraging Reinforcement Learning for evaluating Robustness of KNN Search AlgorithmsPreprint
Vadiraja, P., & Balada, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06525
Regularized Training of Nearest Neighbor Language Models
Ton, J.-F., Talbott, W. A., Zhai, S., & Susskind, J. M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.4
A Data Selection Framework to Improve Unsupervised Sentence Simplification Using Fine-Tuned BertPreprint
Rashid, M. A., & Amirkhani, H. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4159706
Reducing classifier overconfidence against adversaries through graph algorithms
Teixeira, L. V., Jalaian, B. A., & Ribeiro, B. (2023) · Machine Learning · doi:10.1007/s10994-023-06307-y
Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation
Kambhatla, N., Born, L., & Sarkar, A. (2023) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2023.iwslt-1.27
Predicting Numerals in Text Using Nearest Neighbor Language Models
Sakamoto, T., & Aizawa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.295
Knowledge Corpus Error in Question Answering
Lee, Y., Oh, P., & Thorne, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.616
DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework
Kim, K., & Lee, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.150
Simple Temporal Adaptation to Changing Label Sets: Hashtag Prediction via Dense KNN
Mireshghallah, N., Vogler, N., He, J., Florez, O. U., El-Kishky, A., & Berg-Kirkpatrick, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.452 · Cited by 1
Multilingual k-Nearest-Neighbor Machine Translation-Nearest-Neighbor Machine Translation
Stap, D., & Monz, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.571
kNN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers
Bhardwaj, R., Li, Y., Majumder, N., Cheng, B. H., & Poria, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.903 · Cited by 1
Complex Skill Learning using Modified Transformers with Automatic Chunking and Forgetting
Singh, S., Katti, S., & Ghatnekar, V. (2023) · doi:10.1145/3637843.3637849
K-Nearest neighbor smart contract classification with semantic feature enhancement
Tian, G., Zhao, G., Wang, R., Wang, J., & He, C. (2024) · The Computer Journal · doi:10.1093/comjnl/bxae070
Zero-/Few-Shot Anomaly Classification for Transistor Using Multimodal CLIP Retrieval Augmented
Praneenatthavee, N., & Ploysuwan, T. (2024) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp64410.2024.10799271
An Explainable Scheme for Memorization of Noisy Instances by Downstream Evaluation
Tsai, C.-Y., Tsai, P.-H., & Chung, Y.-W. (2025) · Applied Sciences · doi:10.3390/app15052392
Precision Meets Innovation: Hybrid Ai Model in Brain Tumor ResearchPreprint
Sumona, R. B., Biswas, J. P., Shafkat, A., & Rahman, M. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5153385 · Cited by 2
Cui, J.-J., Huang, M., Xu, Y., & Guo, N.-W. (2025) · Other Conferences · doi:10.1117/12.3060306
kNN-CL: Enhancing Continual Learning with Nearest Neighbor Retrieval
Wang, E., Li, Q., Chen, H., Sun, R., & Zhou, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889711
Towards Personalized Federated Learning via Contrastive-Augmented Local Memorization Retrieval
Zhang, P., & Chen, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887972
Estimating the Confidence of Language Model Generation using Training Data
Yoshikawa, H., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.91
Retrieval-Native Language Models: Integrating Parametric and Vector Memory with Bayesian AttentionPreprint
Bhuiyan, M. M. (2025) · doi:10.36227/techrxiv.174494725.56070193/v1 · Cited by 1
Research on active retrieval enhancement for large language models based on multi-model integration
Yin, B., & Zhen-lai, X. (2025) · International Conference on Computer Application and Information Security (ICCAIS) · doi:10.1117/12.3062366
Benchmarking and Analyzing Generative Data for Visual Recognition
Li, B., Liu, H., Chen, L., Lee, Y. J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3572476 · Cited by 5
HyRACC: A Hybrid Retrieval-Augmented Framework for More Efficient Code Completion
Li, C., Shang, J., Feng, Y., & Luo, B. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00013
IMPACT: Identifying and Classifying Multiple Sourced and Categorized Self-Admitted Technical Debts
Li, Q., Yin, Z., Yang, Y., Li, C., Shen, Z., Ge, J., Zhong, W., Luo, B., & Ng, V. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747180
Unified Cognitive Consciousness Theory: Formalization, Validation, and Outlook
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749438
Flexible Prefrontal Control over Hippocampal Episodic Memory for Goal-Directed Generalization
Zheng, Y., Wolf, N., Ranganath, C., O'Reilly, R. C., & McKee, K. L. (2025) · Proceedings of Cognitive Computational Neuroscience 2025 · doi:10.32470/jnqbpro
Metadata
- Authors
- Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M.
- Affiliations
- Meta (United States) (US)
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1911.00172
- OpenAlex
- W2988841832
- Cited by
- 285
- References
- 28
- Citations by year
- 2026: 1 · 2025: 34 · 2024: 34 · 2023: 111 · 2022: 52 · 2021: 39 · 2020: 13 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Styrelsen för Internationellt Utvecklingssamarbete
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- linear interpolation, pre-trained language models, embedding space, perplexity reduction, WikiText-103, domain adaptation, factual knowledge, next-word prediction, sequence similarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
