When Large Language Models Meet Vector Databases: A Survey
Open accessAbstract
This survey explores the synergistic potential of Large Language Models (LLMs) and Vector Databases (VecDBs), a burgeoning but rapidly evolving research area. With the proliferation of LLMs comes a host of challenges, including hallucinations, outdated knowledge, prohibitive commercial application costs, and memory issues. VecDBs emerge as a compelling solution to these issues by offering an efficient means to store, retrieve, and manage the high-dimensional vector representations intrinsic to LLM operations. Through this nuanced review, we delineate the foundational principles of LLMs and VecDBs and critically analyze their integration’s impact on enhancing LLM functionalities. This discourse extends into a discussion on the speculative future developments in this domain, aiming to catalyze further research into optimizing the confluence of LLMs and VecDBs for advanced data handling and knowledge extraction capabilities.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For many real-world applications, large or heterogeneous tool outputs, such as web-search results and database-query results, can accumulate dramatically.
“Additionally, for many real-world applications, large or heterogeneous tool outputs, such as web-search results (Wei et al., 2025e) and database-query results (Jing et al., 2025), can also accumulate dramatically.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Jing et al., 2025
The RAG section (3.1; page unavailable) discusses large external databases, limited LLM context, and retrieval of selected data segments. The cost-effective storage section likewise concerns the size of stored data, not the size of database-query outputs. Searches for accumulating tool outputs and large query results found no passage addressing the claim attributed to Jing et al.; the web-search example has a separate citation. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“One major challenge is hallucinations in LLMs [20], which undermine the reliability of their outputs.”
BackgroundProduct Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · cited 2× · see citing works
“LLMs often generate plausible yet factually incorrect or nonsensical information [5].”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“In Oppenheimer, Cillian Murphy starred as J. Robert Oppenheimer.”
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 1× · see citing works
“Essentially, every dataset can be questioned with bias issues, including imitative falsehoods [9], duplicating biases [10], and social biases [11].”
BackgroundMeasuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497 · see citing works
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslpro.2025.3606231 · Cited by 136 · see citing works
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.08747 · Cited by 135 · see citing works
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81 · see citing works
Bang, F. (2023) · Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) · doi:10.18653/v1/2023.nlposs-1.24 · Cited by 72 · cited 1× · see citing works
“GPT-Cache [36], a VecDB that serves as a semantic cache, as shown in Figure 2, for instance, stores responses to previously asked queries and works as a cache before calling LLM APIs.”
RET-LLM: Towards a General Read-Write Memory for Large Language Models
Modarressi, A., Imani, A., Fayyaz, M., & Schütze, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14322 · Cited by 8 · see citing works
Long-Term Memory for Large Language Models Through Topic-Based Vector Database
Zhang, Y., Yu, Z., Jiang, W., Shen, Y., & Li, J. (2023) · 2023 International Conference on Asian Language Processing (IALP) · doi:10.1109/ialp61005.2023.10337079 · Cited by 6 · cited 2× · see citing works
“Our study also demonstrates both the research and engineering challenges in this fast-growing field and suggests directions for future work.”
Flagged first, then by citations
Research works printed but not linked: 45
- Josh Achiam, Steven Adler, et al. Gpt- 4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- Armen Aghajanyan, Bernie Huang, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022. · arXiv
- Alayrac et al., 2022] Jean Alayrac, Jeff Donahue, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. · arXiv
- Alexandr Andoni and Piotr Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Comm. ACM, 51(1):117- 122, 2008.
- Tom Brown, Benjamin Mann, et al. Language models are few-shot learners. In NeurIPS, pages 1877-1901, 2020.
- Qi Chen, Bing Zhao, et al. Spann: Highly-efficient billion-scale approximate nearest neigh- bor search. In NeurIPS 2021, 2021.
- Aakanksha Chowdhery, Sharan Narang, et al. Palm: Scaling language modeling with pathways. JMLR, 24(240):1-113, 2023.
- Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. · arXiv
- Ganguli et al., 2022] Deep Ganguli, Liane Lovitt, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. · arXiv
- Han et al., 2023] Yikun Han, Chunjiang Liu, et al. A comprehensive survey on vector database: Storage and retrieval technique, challenge. arXiv preprint arXiv:2310.11703, 2023. · arXiv
- Izacard et al., 2022] Gautier Izacard, Lewis Patrick, et al. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022. · arXiv
- Minki Kang and Jin M Kwak. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846, 2023. · arXiv
- Jing Yu Koh, Ruslan Salakhutdinov, et al. Grounding language models to images for multimodal inputs and outputs. In ICML, 2023.
- Li et al., 2023a] Haoran Li, Yulin Chen, et al. Privacy in large language models: Attacks, defenses and future directions. arXiv preprint arXiv:2310.10383, 2023. · arXiv
- Li et al., 2023b] Junnan Li Li, Dongxu Li, et al. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. · arXiv
- Yury A Malkov and Dmitry A Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. PAMI, 42(4):824-836, 2018.
- Muja and Lowe, 2009] Marius Muja and David G Lowe. Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration. In VISAPP, pages 331-340, 2009.
- Musser, 2023] Micah Musser. A cost analysis of generative language models and influence operations. arXiv preprint arXiv:2308.03740, 2023. · arXiv
- Pranav Narayanan Venkit, Sanjana Gautam, et al. Nationality bias in text generation. In EACL, pages 116-122, 2023.
- Nishikawa et al., 2022] Sosuke Nishikawa, Ryokan Ri, et al. EASE: Entity-aware contrastive learning of sentence embedding. In NAACL, pages 3870-3885, 2022.
- Yasumasa Onoe, Michael Zhang, et al. Entity cloze by date: What LMs know about unseen entities. In NAACL Findings, pages 693-702, 2022.
- James Jie Pan, Jianguo Wang, et al. Survey of vector database management systems. arXiv preprint arXiv:2310.14021, 2023. · arXiv
- Guilherme Penedo, Quentin Malartic, et al. The refinedweb dataset for falcon llm: Outperform- ing curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. · arXiv
- Alec Radford, Jeffrey Wu, et al. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.
- Ram et al., 2023] Ori Ram, Yoav Levine, et al. In-context retrieval-augmented language models. TACL, 11:1316- 1331, 2023.
- Sanca and Ailamaki, 2023] Viktor Sanca and Anastasia Ailamaki. E-scan: Consuming contextual data with model plugins. In VLDB Workshop, 2023.
- Roy Schwartz, Jesse Dodge, et al. Green ai. Commun. ACM, 63(12):54-63, 2020.
- Shi et al., 2015] Xingjian Shi, Zhourong Chen, et al. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In NeurIPS, 2015.
- Yu Shu, Siwei Dong, et al. Llasm: Large language and speech model. arXiv preprint arXiv:2308.15930, 2023. · arXiv
- Emma Strubell, Ananya Ganesh, et al. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019. · arXiv
- Yufei Tao, Ke Yi, et al. Quality and efficiency in high dimensional nearest neighbor search. In SIGMOD, page 563-576, 2009.
- Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS, 2017.
- Wang and et al., 2023] Mengzhao Wang and Xiangyu Ke et al. Must: An effective and scalable framework for multimodal search of target modality. arXiv preprint arXiv:2312.06397, 2023. · arXiv
- Jianguo Wang, Xiaomeng Yi, et al. Milvus: A purpose-built vector data management system. In SIGMOD, pages 2614-2627, 2021.
- Jason Wei, Yi Tay, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. · arXiv
- Carole J. Wu and Ramya oth- ers Raghavendra. Sustainable ai: Environmental implications, challenges and opportunities. In MLSys, pages 795-813, 2022.
- Felix Wu, Kwangyoun Kim, et al. Wav2seq: Pre-training speech-to-text encoder-decoder models using pseudo languages. arXiv preprint arXiv:2205.01086, 2022. · arXiv
- Cheng Xin, Luo Di, et al. Lift yourself up: Retrieval-augmented text generation with self memory. arXiv preprint arXiv:2305.02437, 2023. · arXiv
- Antoine Yang, Arsha Nagrani, et al. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. arXiv preprint arXiv:2302.14115, 2023. · arXiv
- Yao et al., 2023] Yunzhi Yao, Peng Wang, et al. Editing large language models: Problems, methods, and opportu- nities. arXiv preprint arXiv:2305.13172, 2023. · arXiv
- Jason Yosinski, Jeff Clune, et al. How transferable are features in deep neural networks? In NeurIPS, 2014.
- Wenhao Yu, Dan Iter, et al. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063, 2023. · arXiv
- Wojciech Zaremba, Ilya Sutskever, et al. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. · arXiv
- Zhang et al., 2023a] Chenshuang Zhang, Chaoning Zhang, et al. Text-to-image diffusion models in generative ai: A survey. arXiv preprint arXiv:2303.07909, 2023. · arXiv
Cited by
Sorted by citations
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works
Human-inspired Perspectives: A Survey on AI Long-term Memory
He, Z., Lin, W., Zheng, H., Zhang, F., Jones, M. W., Aitchison, L., Xu, X., Liu, M., Kristensson, P. O., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.00489 · Cited by 3 · see citing works
“Traditional relational databases require highly structured data and are not effective at data types like texts, images, and audios [254].”
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
- Authors
- Jing, Z., Su, Y., & Han, Y.
- Affiliations
- Carnegie Mellon University (US)
- Purdue University West Lafayette (US)
- University of Michigan (US)
- ORCID
- Venue
- 2025 Conference on Artificial Intelligence x Multimedia (AIxMM)
- Publisher
- IEEE
- Published
- 2025-02-03
- Pages
- 7-13
- OpenAlex
- W4410492435
- Cited by
- 93
- References
- 19
- Citations by year
- 2026: 37 · 2025: 43 · 2024: 13
- FWCI
- 56.20
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, vector databases, vector retrieval, hallucination mitigation, knowledge updating, LLM memory, knowledge extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
