Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
Open accessAbstract
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM agents begin to operate within hybrid environments that necessitate joint execution of linguistic reasoning and perception across multiple dimensions, such as embodied navigation and interactive web browsing.
“As LLM agents begin to operate within hybrid environments that necessitate the joint execution of linguistic reasoning and perception across multiple dimensions, such as embodied navigation and interactive web browsing”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Chen et al., 2026
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
“…multiple dimensions, such as embodied navigation and interactive web browsing, mechanisms for memory are required to facilitate the capture of the rich dependencies across modalities that arise within interactions situated in the real world (Wang et al., 2026; Chen et al., 2026; Yan et al., 2026).”
WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
Yeo, W., Kim, K., Yoon, J., & Hwang, S. J. (2025) · arXiv · doi:10.48550/arxiv.2512.02425
“The aforementioned strategies struggle in these settings due to the massive scale of frames and long-term temporal dependencies.”
- Authors
- Chen, G., Lu, L., Liu, Y., Dong, L., Zou, L., Lv, J., Li, Z., Mao, X., Pei, B., Wang, S., Li, Z., Sapra, K., Liu, F., Zheng, Y.-D., Huang, Y., Wang, L., Yu, Z., Tao, A., Liu, G., & Lu, T.
- Affiliations
- Nanjing University (CN)
- Zhejiang University (CN)
- Shanghai Jiao Tong University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2603.05484
- OpenAlex
- W7134049875
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video understanding, dynamic memory management, temporal bias, domain bias, out-of-distribution generalization, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
