MemVerse: Multimodal Memory for Lifelong Learning Agents
Open accessAbstract
Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember. Without reliable memory, agents catastrophically forget past experiences, struggle with long-horizon reasoning, and fail to operate coherently in multimodal or interactive environments. We introduce MemVerse, a model-agnostic, plug-and-play memory framework that bridges fast parametric recall with hierarchical retrieval-based memory, enabling scalable and adaptive multimodal intelligence. MemVerse maintains short-term memory for recent context while transforming raw multimodal experiences into structured long-term memories organized as hierarchical knowledge graphs. This design supports continual consolidation, adaptive forgetting, and bounded memory growth. To handle real-time demands, MemVerse introduces a periodic distillation mechanism that compresses essential knowledge from long-term memory into the parametric model, allowing fast, differentiable recall while preserving interpretability. Extensive experiments demonstrate that MemVerse significantly improves multimodal reasoning and continual learning efficiency, empowering agents to remember, adapt, and reason coherently across extended interactions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
InternVideo: General Video Foundation Models via Generative and Discriminative LearningPreprint
Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.03191 · Cited by 111 · see citing works
MemGen: Weaving Generative Latent Memory for Self-Evolving AgentsPreprint
Zhang, G.-M., Fu, M., & Yan, S.-C. (2025) · arXiv.org · doi:10.48550/arxiv.2509.24704 · Cited by 106 · see citing works
Disentangled Representation Learning for Text-Video RetrievalPreprint
Wang, Q., Zhang, Y., Zheng, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.07111 · Cited by 41 · see citing works
MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension TasksPreprint
Yang, X., Wu, W., Feng, S., Wang, M., Wang, D.-L., Li, Y., Sun, Q., Zhang, Y.-F., Fu, X.-M., & Poria, S. (2023) · arXiv.org · doi:10.48550/arxiv.2310.09036 · Cited by 29 · see citing works
Self-citationMem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025) · arXiv · doi:10.48550/arxiv.2504.19413 · Cited by 10 · see citing works
FireAct: Toward Language Agent Fine-tuning
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2310.05915 · Cited by 9 · see citing works
MEMORYLLM: Towards Self-Updatable Large Language Models
Wang, Y., Gao, Y., Chen, X., Jiang, H., Li, S., Yang, J., Yin, Q., Li, Z., Li, X., Yin, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.04624 · Cited by 4 · see citing works
Evaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.17753 · Cited by 3 · see citing works
MIRIX: Multi-Agent Memory System for LLM-Based Agents
Wang, Y., & Chen, X. (2025) · arXiv · doi:10.48550/arxiv.2507.07957 · Cited by 2 · see citing works
Yan, S., Yang, X., Huang, Z., Nie, E., Ding, Z., Li, Z., Ma, X., Bi, J., Kersting, K., Pan, J. Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.19828 · Cited by 2 · see citing works
Self-citationExpertized Caption Auto-Enhancement for Video-Text RetrievalPreprint
Yang, B., Chen, J., Li, W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.02885 · Cited by 1 · see citing works
Meng, S., Liu, J., Chen, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.09497
Self-citation
Flagged first, then by citations
Research works printed but not linked: 44
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems, pages 1877-1901.
- Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, and Stefan Leutenegger. Vidbot: Learning generalizable 3d actions from in-the-wild 2d human videos for zero-shot robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27661-27672, 2025.
- Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Sys- tems, 36, 2024. 6, 8
- Z. Chen et al. Knowledge graphs meet multi-modal learning, 2024. arXiv:2402.05391. 3 · arXiv
- Ian J. Deary, Lars Penke, and Wendy Johnson. The neuro- science of human intelligence differences. Nature Reviews Neuroscience, 11(3):201-211, 2010. 2
- Yihe Deng, Chenchen Ye, Zijie Huang, Mingyu Derek Ma, Yiwen Kou, and Wei Wang. Graphvis: Boosting llms with visual knowledge graph integration. In Advances in Neural Information Processing Systems, pages 67511-67534.
- Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 9062-9072, 2025. 2
- Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented mul- timodal agent for video understanding. In Computer Vision -ECCV 2024, pages 75-92, Cham, 2025. Springer Nature Switzerland. 2
- Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, et al. Clip2tv: An empirical study on transformer-based methods for video-text retrieval. arXiv preprint arXiv:2111.05610, 1 (2):6, 2021. 6, 8 · arXiv
- Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Otta- viano, and Linjun Yang. Embedding-based retrieval in face- book search. In Proceedings of the 26th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Min- ing, page 2553-2561, New York, NY, USA, 2020. Associa- tion for Computing Machinery. 2
- Chen Jiang, Hong Liu, Xuzheng Yu, et al. Dual-modal attention-enhanced text-video retrieval with triplet partial margin contrastive learning. In ACM International Confer- ence on Multimedia, pages 4626-4636, 2023. 6, 8
- Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen- tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pages 6769-6781. ACL, 2020. 3
- Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In IEEE/CVF In- ternational Conference on Computer Vision, pages 19948- 19960, 2023. 6, 8
- Yuqi Li, Chuanguang Yang, Hansheng Zeng, Zeyu Dong, Zhulin An, Yongjun Xu, Yingli Tian, and Hao Wu. Frequency-aligned knowledge distillation for lightweight spatiotemporal forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7262- 7272, 2025. 2
- Fang Liu, Clement Yu, Weiyi Meng, and Abdur Chowdhury. Effective keyword search in relational databases. In Proceed- ings of the 2006 ACM SIGMOD International Conference on Management of Data, page 563-574, New York, NY, USA, 2006. Association for Computing Machinery. 2
- Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 34892-34916.
- Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, and Botian Shi. Aligning vision to lan- guage: Annotation-free multimodal knowledge graph con- struction for enhanced llms reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 981-992, 2025. 2, 3, 6, 7
- Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, and Jun Ma. Hm-rag: Hierarchical multi-agent multimodal retrieval augmented generation. In Proceedings of the 33rd ACM International Conference on Multimedia, page 2781-2790, New York, NY, USA, 2025. Association for Computing Machinery. 2
- Yuqi Liu, Pengfei Xiong, Luhui Xu, et al. Ts2-net: Token shift and selection transformer for text-video retrieval. In European Conference on Computer Vision, pages 319-335, 2022. 6, 8
- Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reason- ing via thought chains for science question answering. In Advances in Neural Information Processing Systems, pages 2507-2521.
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. In Advances in Neural Information Processing Systems, pages 43447-43478.
- Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision- language instruction tuning for large language models. In Advances in Neural Information Processing Systems, pages 29615-29627.
- Yiwei Ma, Guohai Xu, Xiaoshuai Sun, et al. X-clip: End- to-end multi-grained contrastive learning for video-text re- trieval. In ACM International Conference on Multimedia, pages 638-647, 2022. 6, 8
- Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14420-14431, 2024. 6, 7
- Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, et al. Neural episodic control. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 2827-2836. PMLR, 2017. 2
- Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. Memorag: Boost- ing long context processing with global memory-enhanced retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, page 2366-2377, New York, NY, USA, 2025. Association for Computing Machinery. 3
- Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, pages 28492-28518. PMLR, 2023. 6
- Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Mem- net: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Com- puter Vision (ICCV), 2017. 2
- Kaibin Tian, Ruixiang Zhao, Zijie Xin, Bangxiang Lan, and Xirong Li. Holistic features are almost sufficient for text- to-video retrieval. In CVPR, pages 17138-17147, 2024. 6, 8
- Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13587-13597, 2022. 2
- Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Cap4video++: Enhancing video understanding with auxiliary captions. TPAMI, pages 1-15, 2024. 6, 8
- Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-aware prompt tuning. Proceedings of the 33rd ACM International Conference on Multimedia, 2025. 2
- Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, et al. mplug-2: A mod- ularized multi-modal foundation model across text, image and video. In International Conference on Machine Learn- ing, pages 38728-38748, 2023. 6, 8
- Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6
- Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language representation alignment. In International Conference on Learning Repre- sentation, 2023. 6, 8
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh In- ternational Conference on Learning Representations, 2023. 2
- Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12380-12403, Bangkok, Thailand, 2024. Association for Computational Linguistics. 3
- Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshap- ing long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259, 2025. 3 · arXiv
- Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene under- standing with multi-modal instruction tuning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14147-14157, 2025. 2
- Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model- based agents. ACM Trans. Inf. Syst., 43(6), 2025. 3
- Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model- based agents. ACM Trans. Inf. Syst., 43(6), 2025. 2
- Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. In Advances in Neural Information Processing Systems, pages 5168-5191.
- Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yan- lin Wang. Memorybank: Enhancing large language models with long-term memory. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 38(17):19724-19731, 2024.
- Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Pen- glei Sun, et al. Multi-modal knowledge graph construction and application. arXiv:2202.05786, 2022. 3 · arXiv
Cited by
Sorted by citations
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
Memory in the Age of AI Agents
Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., Guo, H., Dou, S., Xi, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.13564 · Cited by 1 · see citing works
Self-citationA Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
Self-citationHindsight: Structured Agent Memory that Retains, Recalls, and Reflects
Latimer, C., Boschi, N., Neeser, A., Bartholomew, C., Srivastava, G., Wang, X., & Ramakrishnan, N. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2026.acl-demo.27
- Authors
- Liu, J., Sun, Y., Cheng, W., Lei, H., Chen, Y., Wen, L., Yang, X., Fu, D., Cai, P., Deng, N., Yu, Y., Hu, S., Shi, B., & Wang, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2512.03627
- OpenAlex
- W4417031017
- Cited by
- 0
- References
- 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- multimodal memory, lifelong learning, AI agents, hierarchical knowledge graph, continual learning, catastrophic forgetting, knowledge distillation, long-term memory consolidation, multimodal reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
