Generalization through Memorization: Nearest Neighbor Language Models
Open accessAbstract
We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
L2 distance was shown to be more effective than other alternatives for kNN-LM.
“d ( · , · ) is a distance function between the two vectors, and L 2 was shown to be more effective than other alternatives (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The Implementation passage (Section 2, p. 2) says preliminary experiments found better kNN-LM performance with L2 distance than with inner product distance. That supports the comparison with inner product, but not the broader suggestion that L2 was tested against multiple other alternatives. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe kNN-LM outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, domain adaptation settings, and conditional generation tasks such as machine translation.
“This model is particularly notable for its large improvements in performance it outperforms the previous best parametric LMs by a large margin in standard language modeling benchmarks, in domain adaptation settings, and on other conditional generation tasks such as machine translation (Khandelwal et al., 2020).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Table 1 (p. 4) shows that kNN-LM substantially improves WIKITEXT-103 perplexity, and Section 4.3 (p. 6) reports a large improvement over an out-of-domain LM on BOOKS. The paper does not report kNN-LM outperforming parametric LMs on conditional generation or machine translation; Section 7 (p. 9) discusses translation as related work by others. In domain adaptation, the reported kNN-LM result also remains worse than the in-domain LM. (LLM, conf=0.89)
Experimental — accuracy not yet measuredNon-parametric LMs model text distributions by referencing both the parameters of the underlying model and examples from an external datastore.
“Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The kNN-LM computes the probability of the next token by interpolating a parametric LM with a distribution calculated from the k nearest context-token pairs in the datastore.
“The k NN-LMcomputes the probability of the next token by interpolating a parametric LM with a distribution calculated from the k nearest context-token pairs in the datastore, as demonstrated in Figure 2.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The dataset used to construct the datastore may differ from the training data that trains the parametric NLM, enabling applications such as training-free domain adaptation or gradient-free use of extra training data.
“Note that the dataset to contruct the datastore may not necessarily be the training data that trains the parametric NLM in Eq. 1 - a separate dataset may be used for the datastore construction which would lead to potential applications such as training-free domain adaptation or a gradient-free way to utilize extra training data (Khandelwal et al., 2019).”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Perplexity continues improving as k grows, as shown by Khandelwal et al. and confirmed by the authors.
“The perplexity continues improving as k grows as shown in (Khandelwal et al., 2019) and confirmed by us.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
The kNN-LM's application requires no additional training; the parameters of the NLM remain as-is, and Eq. 1 is only applied at test time.
“Note that the application of k NN-LM requires no additional training; the parameters of the NLM remain as-is, and Eq. 1 is only applied at test time.”
From Efficient Nearest Neighbor Language Models (2021) · cites this paper as Khandelwal et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Self-citationXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · see citing works
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 642 · see citing works
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep LearningPreprint
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 460 · see citing works
Improving Neural Language Models with a Continuous CachePreprint
Grave, E., Joulin, A., & Usunier, N. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1612.04426 · Cited by 316 · see citing works
MBT: A Memory-Based Part of Speech Tagger-Generator
Daelemans, W. M. P., Zavrel, J., Berck, P., & Gillis, S. A. (1996) · Research portal (Tilburg University) · doi:10.48550/arxiv.cmp-lg/9607012 · Cited by 305 · see citing works
Learning to Remember Rare EventsPreprint
Kaiser, Ł., Nachum, O., Roy, A., & Samy Bengio, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03129 · Cited by 296 · see citing works
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255 · see citing works
Retrieve and Refine: Improved Sequence Generation Models For Dialogue
Weston, J., Dinan, E., & Miller, A. (2018) · Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI · doi:10.18653/v1/w18-5713 · Cited by 165 · see citing works
Search Engine Guided Neural Machine Translation
Gu, J., Wang, Y., Cho, K., & Li, V. O. K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12013 · Cited by 147 · see citing works
Efficient softmax approximation for GPUs
Grave, É., Joulin, A., Cissé, M., & Jeǵou, H. (2017) · International Conference on Machine Learning · Cited by 96 · see citing works
Dynamic Evaluation of Transformer Language Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2019) · arXiv · doi:10.48550/arxiv.1904.08378 · Cited by 30 · see citing works
Zhao, J., & Cho, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09502 · Cited by 9 · see citing works
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
Luo, H., Jiang, L., Belinkov, Y., & Glass, J. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1144 · Cited by 9 · see citing works
Lightweight Adaptive Mixture of Neural and N-gram Language ModelsPreprint
Bakhtin, A., Szlam, A. D., Ranzato, M., & Grave, É. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07705 · Cited by 7 · see citing works
A Simple Cache Model for Image Recognition
Orhan, E. (2018) · neural information processing systems · Cited by 5 · see citing works
Flagged first, then by citations
4 of the 28 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research , 3(Feb):1137-1155, 2003.
- Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. In NIPS , pp. 6042-6052, 2017a.
- Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics , 6:437-450, 2018.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 , 2015. · arXiv
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS , 2019.
Cited by
Sorted by citations
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
“For “what to retrieve” research has progressed from simple token [ Khandelwal et al. , 2019 ] and entity retrieval [ Nishikawa et al. , 2022 ] to more complex structures like chunks [ Ram et al. , 2023 ] and knowledge graph [ Kang et al. , 2023 ] , with studies focusing on the granularity of…”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“Technical approaches to reducing risks of environmental harm include segmenting LMs into less large LMs that search and retrieve information from a distinct data corpus [25, 88, 97, 102, 115].”
BackgroundRetrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497 · see citing works
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., & Feichtenhofer, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.544 · Cited by 468 · see citing works
Self-citationIn-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449 · see citing works
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
Data augmentation approaches in natural language processing: A survey
Li, B., Hou, Y., & Che, W. (2022) · AI Open · doi:10.1016/j.aiopen.2022.03.001 · Cited by 357 · see citing works
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376 · see citing works
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340 · see citing works
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · see citing works
Self-citationMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303 · see citing works
Self-citationEnabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
A Survey of Knowledge Enhanced Pre-Trained Language Models
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310002 · Cited by 224 · see citing works
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204 · see citing works
Internet-Augmented Dialogue Generation
Komeili, M., Shuster, K., & Weston, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.579 · Cited by 192 · see citing works
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141 · see citing works
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
Chen, W., Hu, H., Chen, X., Verga, P., & Cohen, W. W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.375 · Cited by 134 · see citing works
Pre-training via ParaphrasingPreprint
Lewis, M., Ghazvininejad, M., Ghosh, G., Aghajanyan, A., Wang, S. I., & Zettlemoyer, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15020 · Cited by 120 · see citing works
Self-citationCase-based Reasoning for Natural Language Queries over Knowledge Bases
Das, R., Zaheer, M., Thai, D., Godbole, A., Perez, E., Lee, J. Y., Tan, L., Polymenakos, L., & McCallum, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.755 · Cited by 121 · see citing works
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Retrieval Augmented Classification for Long-Tail Visual Recognition
Long, A., Yin, W., Ajanthan, T., Nguyen, D.-V., Purkait, P., Garg, R., Blair, A., Shen, C., & van den Hengel, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00683 · Cited by 86 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81 · see citing works
Generative Knowledge Graph Construction: A Review
Ye, H., Zhang, N., Chen, H., & Chen, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.1 · Cited by 89 · see citing works
Adaptive Nearest Neighbor Machine Translation
Zheng, X. W., Zhang, Z., Guo, J., Huang, S., Chen, B., Luo, W., & Chen, J. (2021) · doi:10.18653/v1/2021.acl-short.47 · Cited by 74 · see citing works
Shuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A. D., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.27 · Cited by 78 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification
Su, X. H., Wang, R., & Dai, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.75 · Cited by 67 · see citing works
Wu, Z., Kong, L., Bi, W., Li, X., & Kao, B. (2021) · doi:10.18653/v1/2021.acl-long.480 · Cited by 65 · see citing works
Improving Transformer Models by Reordering their Sublayers
Press, O., Smith, N. A., & Levy, O. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.270 · Cited by 64 · see citing works
Self-citationNeural Machine Translation with Monolingual Translation Memory
Cai, D., Wang, Y., Li, H., Lam, W. P., & Liu, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.567 · Cited by 81 · see citing works
CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs
Ootomo, H., Naruse, A., Nolet, C., Wang, H., Fehér, T., & Wang, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00323 · Cited by 63 · see citing works
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 88 · see citing works
Training language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60 · see citing works
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60 · see citing works
Efficient Nearest Neighbor Language Models
He, J., Neubig, G., & Berg-Kirkpatrick, T. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.461 · Cited by 59 · see citing works
Epistemic Injustice in Generative AI
Kay, J., Kasirzadeh, A., & Mohamed, S. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31671 · Cited by 76 · see citing works
MeaCap: Memory-Augmented Zero-shot Image Captioning
Zeng, Z., Xie, Y., Zhang, H., Chen, C., Chen, B., & Wang, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01337 · Cited by 67 · see citing works
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“Studies show that we can augment generation with retrievers (Khandelwal et al., 2020; Guu et al., 2020) or search engines (Nakano et al.”
BackgroundAdaptive Semiparametric Language Models
Yogatama, D., d’Autume, C. D. M., & Kong, L. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00371 · Cited by 58 · see citing works
FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
Guo, H., Rajani, N. F., Hase, P., Bansal, M., & Xiong, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.808 · Cited by 63 · see citing works
When Large Language Models Meet Vector Databases: A Survey
Jing, Z., Su, Y., & Han, Y. (2025) · 2025 Conference on Artificial Intelligence x Multimedia (AIxMM) · doi:10.1109/aixmm62960.2025.00008 · Cited by 93 · see citing works
BiomedRAG: A retrieval augmented large language model for biomedicine
Li, M., Halil Kilicoglu, B., Xu, H., & Zhang, R. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104769 · Cited by 53 · see citing works
Few-shot Intent Classification and Slot Filling with Retrieved Examples
Yu, D., He, L., Zhang, Y., Du, X., Pasupat, P., & Li, Q. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.59 · Cited by 54 · see citing works
Agentic AI: The age of reasoning—A review
Nisa, U., Shirazi, M., Saip, M. A., & Pozi, M. S. M. (2025) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2025.08.003 · Cited by 71 · see citing works
Efficient Meta Lifelong-Learning with Limited Memory
Wang, Z., Mehta, S. V., Póczos, B., & Carbonell, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.39 · Cited by 51 · see citing works
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59 · see citing works
Retrieval-augmented Image Captioning
Ramos, R., Elliott, D., & Martins, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.266 · Cited by 47 · see citing works
Retrieve-and-Sample: Document-level Event Argument Extraction via Hybrid Retrieval Augmentation
Ren, Y., Cao, Y., Guo, P., Fang, F., Ma, W., & Lin, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.17 · Cited by 42 · see citing works
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 41 · see citing works
Nearest neighbor search on embeddings rapidly identifies distant protein relations
Schütze, K., Heinzinger, M., Steinegger, M., & Rost, B. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.1033775 · Cited by 39 · see citing works
Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future
Klie, J.-C., Webber, B., & Gurevych, I. (2022) · Computational Linguistics · doi:10.1162/coli_a_00464 · Cited by 44 · see citing works
AMGPT: A large language model for contextual querying in additive manufacturing
Chandrasekhar, A., Chan, J., Ogoke, F., Ajenifujah, O., & Farimani, A. B. (2024) · Additive Manufacturing Letters · doi:10.1016/j.addlet.2024.100232 · Cited by 40 · see citing works
Mitigating privacy risks in Retrieval-Augmented Generation via locally private entity perturbation
He, L., Tang, P., Zhang, Y., Zhou, P., & Su, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104150 · Cited by 45 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
Yang, Y., Hossain, M. Z., Stone, E. A., & Rahman, S. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00501 · Cited by 37 · see citing works
Hierarchical text classification with multi-label contrastive learning and KNN
Zhang, J., Li, Y., Shen, F., -, H. Y., Tan, H., & He, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127323 · Cited by 38 · see citing works
Privacy Preserving Prompt Engineering: A Survey
Edemacu, K., & Wu, X. (2025) · ACM Computing Surveys · doi:10.1145/3729219 · Cited by 48 · see citing works
Domain Adaptive Code Completion via Language Models and Decoupled Domain Databases
Tang, Z. J., Ge, J., Liu, S., Zhu, T., Xu, T., Huang, L., & Luo, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00076 · Cited by 40 · see citing works
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
Lei, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.602 · Cited by 37 · see citing works
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Mensink, T., Uijlings, J., Castrejón, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araújo, A. D. M., & Ferrari, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00289 · Cited by 35 · see citing works
Relation Extraction as Open-book Examination
Chen, X., Li, L., Zhang, N., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531746 · Cited by 33 · see citing works
Fast Nearest Neighbor Machine Translation
Meng, Y., Li, X., Zheng, X., Wu, F. Q., Sun, X., Zhang, T., & Li, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.47 · Cited by 33 · see citing works
Can Whisper Perform Speech-Based In-Context Learning?
Wang, S., Yang, C.-H. H., Wu, J. Y., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446502 · Cited by 32 · see citing works
Memory-Augmented Image Captioning
Fei, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16220 · Cited by 32 · see citing works
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation
Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021) · doi:10.18653/v1/2021.acl-long.259 · Cited by 32 · see citing works
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning
Barraco, M., Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00282 · Cited by 30 · see citing works
RetrievalFuse: Neural 3D Scene Reconstruction with a Database
Siddiqui, Y., Thies, J., Ma, F., Shan, Q., Nießner, M., & Dai, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01234 · Cited by 30 · see citing works
Limitations of Autoregressive Models and Their Alternatives
Lin, C., Jaech, A., Li, X., Gormley, M. R., & Eisner, J. M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.405 · Cited by 36 · see citing works
ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
Vulić, I., Su, P.-H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkšić, N., & Wen, T.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.88 · Cited by 31 · see citing works
RankGen: Improving Text Generation with Large Ranking Models
Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.15 · Cited by 30 · see citing works
Controllable Semantic Parsing via Retrieval Augmentation
Pasupat, P., Zhang, Y., & Guu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.607 · Cited by 30 · see citing works
Memorization and Generalization in Neural Code Intelligence Models
Rabin, M. R. I., Hussain, A., Alipour, M. A., & Hellendoorn, V. J. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107066 · Cited by 30 · see citing works
Addressing Some Limitations of Transformers with Feedback MemoryPreprint
Fan, A., Lavril, T., Grave, É., Joulin, A., & Sukhbaatar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09402 · Cited by 41 · see citing works
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Gur, S., Neverova, N., Stauffer, C., Lim, S.-N., Kiela, D., & Reiter, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.11 · Cited by 28 · see citing works
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 37 · see citing works
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
Reid, M., Marrese-Taylor, E., & Matsuo, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.344 · Cited by 33 · see citing works
BioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature
Frisoni, G., Mizutani, M., Moro, G., & Valgimigli, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.390 · Cited by 27 · see citing works
ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries
Zhang, X., Li, J. K., Chi, P.-W., Chandrasegaran, S. K., & Ma, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581260 · Cited by 25 · see citing works
Nonparametric Masked Language Modeling
Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-T., Hajishirzi, H., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.132 · Cited by 29 · see citing works
Self-citationOn Training Instance Selection for Few-Shot Neural Text Generation
Chang, E., Shen, X., Yeh, H.-S., & Demberg, V. (2021) · doi:10.18653/v1/2021.acl-short.2 · Cited by 23 · see citing works
Relational memory-augmented language models
Liu, Q., Yogatama, D., & Blunsom, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00476 · Cited by 23 · see citing works
ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval
Yu, Y., Zhuang, Y., Zhang, R., Yu, M., Shen, J., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.748 · Cited by 26 · see citing works
Aligning Language Models to User Opinions
Hwang, E., Majumder, B. P., & Tandon, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.393 · Cited by 22 · see citing works
Privacy Implications of Retrieval-Based Language Models
Huang, Y., Gupta, S., Zhong, Z., Li, K., & Chen, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.921 · Cited by 27 · see citing works
Transfer learning for language model adaptation
Saiful, B. M. (2023) · doi:10.32657/10356/169892 · Cited by 22 · see citing works
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Wang, B., Ping, W., Xu, P., McAfee, L., Liu, Z., Shoeybi, M., Dong, Y., Kuchaiev, O., Li, B., Xiao, C., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.482 · Cited by 21 · see citing works
Improving Image Recognition by Retrieving from Web-Scale Image-Text Data
İşcen, A., Fathi, A., & Schmid, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01849 · Cited by 20 · see citing works
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24 · see citing works
RetGen: A Joint Framework for Retrieval and Grounded Text Generation Modeling
Zhang, Y., Sun, S., Gao, X., Fang, Y., Brockett, C., Galley, M., Gao, J., & Dolan, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21429 · Cited by 27 · see citing works
Analogical Inference Enhanced Knowledge Graph Embedding
Yao, Z., Zhang, W., Chen, M., Huang, Y., Yang, Y., & Chen, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25605 · Cited by 18 · see citing works
CPR: Retrieval Augmented Generation for Copyright Protection
Golatkar, A., Achille, A., Zancato, L., Wang, Y., Swaminathan, A. D., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01176 · Cited by 20 · see citing works
GNN-LM: Language Modeling based on Global Contexts via GNNPreprint
Meng, Y., Zong, S., Li, X., Sun, X., Zhang, T., Wu, F. Q., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08743 · Cited by 22 · see citing works
Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis
Zhang, R., Wang, L., Guo, Z., & Shi, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00130 · Cited by 17 · see citing works
- Authors
- Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M.
- Affiliations
- Meta (United States) (US)
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1911.00172
- OpenAlex
- W2988841832
- Cited by
- 285
- References
- 28
- Citations by year
- 2026: 1 · 2025: 34 · 2024: 34 · 2023: 111 · 2022: 52 · 2021: 39 · 2020: 13 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Styrelsen för Internationellt Utvecklingssamarbete
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- linear interpolation, pre-trained language models, embedding space, perplexity reduction, WikiText-103, domain adaptation, factual knowledge, next-word prediction, sequence similarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
