LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Multimodal Large Language Models (MLLMs) are widely used for visual perception, understanding, and reasoning. However, long video processing and precise moment retrieval remain challenging due to LLMs’ limited context size and coarse frame extraction. We propose the Large Language-and-Vision Assistant for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame Selection (IFS) for capturing brief visual and motion patterns, and Dynamic Token Compression (DTC) to manage LLM context limitations. Evaluations on benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in R1@0.5 and 1.29% in mAP@0.5 on the QVHighlights dataset. Our implementation will be open-sourced upon acceptance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908
Self-citationTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469
Research works printed but not linked: 44
- Jin Y, Li J, Liu Y, Gu T, Wu K, Jiang Z, He M, Zhao B, Tan X, Gan Z, et al. Efficient multimodal large language models: A survey. _arXiv preprint arXiv:2405.10739_. 2024. · arXiv
- Li J, Lu W, Fei H, Luo M, Dai M, Xia M, Jin Y, Gan Z, Qi D, Fu C, Tai Y, Yang W, Wang Y, Wang C (2024). "A Survey on Benchmarks of Multimodal Large Language Models". _arXiv_. arXiv:2408.08632 [cs.CL] [https://arxiv.org/abs/2408.08632]. · arXiv
- Lei J, Berg TL, Bansal M. "QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries". _arXiv_ [cs.CV]. 2021. Available from: https://arxiv.org/abs/2107.09609. · arXiv
- Moon W, Hyun S, Lee S, Heo J (2024). "Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding". _arXiv_. arXiv:2311.08835 [cs.CV] [https://arxiv.org/abs/2311.08835]. · arXiv
- Shen Y, Xiong X, Nagrani A, Arnab A, Wang Z, Ge W, Ross D, Schmid C (2023). "UnLoc: A Unified Framework for Video Localization Tasks". _arXiv_. arXiv:2308.11062 [cs.CV] [https://arxiv.org/abs/2308.11062]. · arXiv
- Zeng Y, Zhong Y, Feng C, Ma L. _UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection_. 2024. Available from: https://arxiv.org/abs/2404.04933. · DOI
- Meinardus B, Batra A, Rohrbach A, Rohrbach M. "The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval". _arXiv_ [cs.CV]. 2024. Available from: https://arxiv.org/abs/2406.18113. · arXiv
- Gao J, Sun C, Yang Z, Nevatia R. "Tall: Temporal activity localization via language query". In: _ICCV_. 2017. p. 5267–5275. · DOI · Paper page
- Lin C, Li J, Wang Y, Tai Y, Luo D, Cui Z, Wang C, Li J, Huang F, Ji R. Fast learning of temporal action proposal via dense boundary generator. _Proceedings of the AAAI conference on artificial intelligence_. 2020; 34(07):11499-11506. · DOI · Paper page
- Anne Hendricks LA, Wang O, Shechtman E, Sivic J, Darrell T, Russell B. Localizing moments in video with natural language. In: _ICCV_. 2017. p. 5803–5812. · DOI · Paper page
- Yuan Y, Ma L, Wang J, Liu W, Zhu W (2019). "Semantic conditioned dynamic modulation for temporal sentence grounding in videos". _NeurIPS_. 32. · DOI · Paper page
- Zhang D, Dai X, Wang X, Wang YF, Davis LS. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In: _CVPR_. 2019. p. 1247–1257. · DOI · Paper page
- Zhang S, Peng H, Fu J, Luo J (2020). "Learning 2d temporal adjacent networks for moment localization with natural language". _Proceedings of the AAAI Conference on Artificial Intelligence_. 34 (07): 12870–12877. · DOI · Paper page
- Shou Z, Wang D, Chang SF. "Temporal action localization in untrimmed videos via multi-stage cnns." In: _CVPR_; 2016. p. 1049-1058. · DOI · Paper page
- Lin T, Zhao X, Su H, Wang C, Yang M. "Bsn: Boundary sensitive network for temporal action proposal generation." In: _ECCV_; 2018. p. 3-19. · DOI · Paper page
- Zeng R, Huang W, Tan M, Rong Y, Zhao P, Huang J, Gan C. Graph convolutional networks for temporal action localization. In: _ICCV_. 2019. p. 7094-7103. · DOI · Paper page
- Girdhar R, Carreira J, Doersch C, Zisserman A. Video action transformer network. In: _CVPR_; 2019. p. 244-253. · DOI
- Zhang CL, Wu J, Li Y. Actionformer: Localizing moments of actions with transformers. In: _ECCV_. Springer; 2022. p. 492-510. · DOI · Paper page
- Dai R, Das S, Bremond F (2021). "Ctrn: Class-temporal relational network for action detection". _arXiv preprint arXiv:2110.13473_. Available from: https://arxiv.org/abs/2110.13473. · arXiv
- Dai R, Das S, Kahatapitiya K, Ryoo MS, Brémond F. "Ms-tct: Multi-scale temporal convtransformer for action detection". In: _CVPR_. 2022. p. 20041–20051. · DOI
- Zhu Z, Tang W, Wang L, Zheng N, Hua G (2021). "Enriching local and global contexts for temporal action localization". In: _ICCV_. pp. 13516–13525. · DOI
- Sardari F, Mustafa A, Jackson PJ, Hilton A. Pat: Position-aware transformer for dense multi-label action detection. In: _CVPR_. 2023. p. 2988-2997. · DOI
- Liu Y, Li S, Wu Y, Chen CW, Shan Y, Qie X (2022). "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection". In: _CVPR_. pp. 3042–3051. · DOI · Paper page
- Mun J, Cho M, Han B. Local-global video-text interactions for temporal grounding. In: _CVPR_; 2020. p. 10810-10819. · DOI · Paper page
- Awadalla A, Gao I, Gardner J, Hessel J, Hanafy Y, Zhu W, Marathe K, Bitton Y, Gadre S, Sagawa S, Jitsev J, Kornblith S, Koh PW, Ilharco G, Wortsman M, Schmidt L. "OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models". _arXiv_ [cs.CV]. 2023. Available from: https://arxiv.org/abs/2308.01390. · arXiv
- Li B, Zhang Y, Chen L, Wang J, Pu F, Yang J, Li C, Liu Z (2023). "MIMIC-IT: Multi-Modal In-Context Instruction Tuning". _arXiv_. Available from: https://arxiv.org/abs/2306.05425. · arXiv
- Zhang R, Han J, Liu C, Gao P, Zhou A, Hu X, Yan S, Lu P, Li H, Qiao Y (2024). "LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention". _arXiv_. arXiv:2303.16199 [https://arxiv.org/abs/2303.16199] [cs.CV]. · arXiv
- Dai W, Li J, Li D, Tiong AMH, Zhao J, Wang W, Li B, Fung P, Hoi S (2023). "InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning". _arXiv_. arXiv:2305.06500 [https://arxiv.org/abs/2305.06500] [cs.CV]. · arXiv
- Li K, He Y, Wang Y, Li Y, Wang W, Luo P, Wang Y, Wang L, Qiao Y (2024). "VideoChat: Chat-Centric Video Understanding". _arXiv_. arXiv:2305.06355 [https://arxiv.org/abs/2305.06355]. · arXiv
- Zhang H, Li X, Bing L (2023). "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding". _arXiv_. Available from: https://arxiv.org/abs/2306.02858. · DOI · Paper page
- Maaz M, Rasheed H, Khan S, Khan FS. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models". _arXiv_ [cs.CV]. 2024. Available from: https://arxiv.org/abs/2306.05424. · DOI · Paper page
- Luo R, Zhao Z, Yang M, Dong J, Li D, Lu P, Wang T, Hu L, Qiu M, Wei Z (2023). "Valley: Video Assistant with Large Language Model Enhanced Ability". _arXiv_. arXiv:2306.07207 [cs.CV] [https://arxiv.org/abs/2306.07207]. · arXiv
- Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021). "Lora: Low-rank adaptation of large language models". _arXiv preprint arXiv:2106.09685_. 2021. · arXiv
- Loshchilov I, Hutter F (2019). "Decoupled Weight Decay Regularization". _arXiv_. arXiv:1711.05101 [cs.LG] [https://arxiv.org/abs/1711.05101]. · arXiv
- Jang J, Park J, Kim J, Kwon H, Sohn K (2023). "Knowing Where to Focus: Event-aware Transformer for Video Grounding". _arXiv_. arXiv:2308.06947 [cs.CV] [https://arxiv.org/abs/2308.06947]. · DOI
- Moon W, Hyun S, Park S, Park D, Heo JP (2023). "Query-Dependent Video Representation for Moment Retrieval and Highlight Detection". _arXiv_. arXiv:2303.13874 [cs.CV] [https://arxiv.org/abs/2303.13874]. · DOI · Paper page
- Shen Y, Xiong X, Nagrani A, Arnab A, Wang Z, Ge W, Ross D, Schmid C (2023). "UnLoc: A Unified Framework for Video Localization Tasks". _arXiv_. arXiv:2308.11062 [cs.CV] [https://arxiv.org/abs/2308.11062]. · arXiv
- Yu S, Cho J, Yadav P, Bansal M (2023). "Self-Chained Image-Language Model for Video Localization and Question Answering". _arXiv_. arXiv:2305.06988 [cs.CV] [https://arxiv.org/abs/2305.06988]. · arXiv
- Lin KQ, Zhang P, Chen J, Pramanick S, Gao D, Wang AJ, Yan R, Shou MZ (2023). "UniVTG: Towards Unified Video-Language Temporal Grounding". _arXiv_. arXiv:2307.16715 [cs.CV] [https://arxiv.org/abs/2307.16715]. · DOI
- Wang Y, Li K, Li X, Yu J, He Y, Wang C, Chen G, Pei B, Yan Z, Zheng R, Xu J, Wang Z, Shi Y, Jiang T, Li S, Zhang H, Huang Y, Qiao Y, Wang Y, Wang L (2024). "InternVideo2: Scaling Foundation Models for Multimodal Video Understanding". _arXiv_. arXiv:2403.15377 [cs.CV] [https://arxiv.org/abs/2403.15377]. · DOI · Paper page
- Krishna R, Hata K, Ren F, Fei-Fei L, Niebles JC (2017). "Dense-Captioning Events in Videos". _arXiv_. arXiv:1705.00754 [cs.CV] [https://arxiv.org/abs/1705.00754]. · DOI · Paper page
- Zeng R, Xu H, Huang W, Chen P, Tan M, Gan C (2020). "Dense Regression Network for Video Grounding". _arXiv_. arXiv:2004.03545 [cs.CV] [https://arxiv.org/abs/2004.03545]. · DOI · Paper page
- Soldan M, Xu M, Qu S, Tegner J, Ghanem B (2021). "VLG-Net: Video-Language Graph Matching Network for Video Grounding". _arXiv_. arXiv:2011.10132 [https://arxiv.org/abs/2011.10132] [cs.CV]. · DOI
- Wang T, Zhang J, Zheng F, Jiang W, Cheng R, Luo P (2023). "Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos". _arXiv_. arXiv:2303.06378 [https://arxiv.org/abs/2303.06378] [cs.CV]. · arXiv
Cited by
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Collaboration between clinicians and vision–language models in radiology report generation
Tanno, R., Barrett, D. G. T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03302-1 · Cited by 186
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
CXR-LLaVA: a multimodal large language model for interpreting chest X-ray images
Lee, S., Youn, J., Kim, H., Kim, M., & Yoon, S. H. (2025) · European Radiology · doi:10.1007/s00330-024-11339-6 · Cited by 76
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57426-0 · Cited by 75
Chiu, H.-H., Varghese, A., Shao, K., Lu, Y.-C., Nahar, R., Chen, H., Deng, Q., Bao, X., & Li, C. (2024) · doi:10.1101/2024.10.01.616063 · Cited by 6
Metadata
- Authors
- Li, J.
- Venue
- Qeios
- Publisher
- Qeios Ltd
- Published
- 2024-12-05
- ISSN
- 2632-3834 (ISSN-L)
- OpenAlex
- W4405081677
- Cited by
- 112
- References
- 6
- Type
- posted content
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.