CoChat

LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval

Open access
Li, Jian · 2024 · Qeios
PreprintCited by 112

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

Multimodal Large Language Models (MLLMs) are widely used for visual perception, understanding, and reasoning. However, long video processing and precise moment retrieval remain challenging due to LLMs’ limited context size and coarse frame extraction. We propose the Large Language-and-Vision Assistant for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame Selection (IFS) for capturing brief visual and motion patterns, and Dynamic Token Compression (DTC) to manage LLM context limitations. Evaluations on benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in R1@0.5 and 1.29% in mAP@0.5 on the QVHighlights dataset. Our implementation will be open-sourced upon acceptance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.32388/vlxb6m

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Research works printed but not linked: 44

  1. Jin Y, Li J, Liu Y, Gu T, Wu K, Jiang Z, He M, Zhao B, Tan X, Gan Z, et al. Efficient multimodal large language models: A survey. _arXiv preprint arXiv:2405.10739_. 2024. · arXiv
  2. Li J, Lu W, Fei H, Luo M, Dai M, Xia M, Jin Y, Gan Z, Qi D, Fu C, Tai Y, Yang W, Wang Y, Wang C (2024). "A Survey on Benchmarks of Multimodal Large Language Models". _arXiv_. arXiv:2408.08632 [cs.CL] [https://arxiv.org/abs/2408.08632]. · arXiv
  3. Lei J, Berg TL, Bansal M. "QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries". _arXiv_ [cs.CV]. 2021. Available from: https://arxiv.org/abs/2107.09609. · arXiv
  4. Moon W, Hyun S, Lee S, Heo J (2024). "Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding". _arXiv_. arXiv:2311.08835 [cs.CV] [https://arxiv.org/abs/2311.08835]. · arXiv
  5. Shen Y, Xiong X, Nagrani A, Arnab A, Wang Z, Ge W, Ross D, Schmid C (2023). "UnLoc: A Unified Framework for Video Localization Tasks". _arXiv_. arXiv:2308.11062 [cs.CV] [https://arxiv.org/abs/2308.11062]. · arXiv
  6. Zeng Y, Zhong Y, Feng C, Ma L. _UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection_. 2024. Available from: https://arxiv.org/abs/2404.04933. · DOI
  7. Meinardus B, Batra A, Rohrbach A, Rohrbach M. "The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval". _arXiv_ [cs.CV]. 2024. Available from: https://arxiv.org/abs/2406.18113. · arXiv
  8. Gao J, Sun C, Yang Z, Nevatia R. "Tall: Temporal activity localization via language query". In: _ICCV_. 2017. p. 5267–5275. · DOI · Paper page
  9. Lin C, Li J, Wang Y, Tai Y, Luo D, Cui Z, Wang C, Li J, Huang F, Ji R. Fast learning of temporal action proposal via dense boundary generator. _Proceedings of the AAAI conference on artificial intelligence_. 2020; 34(07):11499-11506. · DOI · Paper page
  10. Anne Hendricks LA, Wang O, Shechtman E, Sivic J, Darrell T, Russell B. Localizing moments in video with natural language. In: _ICCV_. 2017. p. 5803–5812. · DOI · Paper page
  11. Yuan Y, Ma L, Wang J, Liu W, Zhu W (2019). "Semantic conditioned dynamic modulation for temporal sentence grounding in videos". _NeurIPS_. 32. · DOI · Paper page
  12. Zhang D, Dai X, Wang X, Wang YF, Davis LS. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In: _CVPR_. 2019. p. 1247–1257. · DOI · Paper page
  13. Zhang S, Peng H, Fu J, Luo J (2020). "Learning 2d temporal adjacent networks for moment localization with natural language". _Proceedings of the AAAI Conference on Artificial Intelligence_. 34 (07): 12870–12877. · DOI · Paper page
  14. Shou Z, Wang D, Chang SF. "Temporal action localization in untrimmed videos via multi-stage cnns." In: _CVPR_; 2016. p. 1049-1058. · DOI · Paper page
  15. Lin T, Zhao X, Su H, Wang C, Yang M. "Bsn: Boundary sensitive network for temporal action proposal generation." In: _ECCV_; 2018. p. 3-19. · DOI · Paper page
  16. Zeng R, Huang W, Tan M, Rong Y, Zhao P, Huang J, Gan C. Graph convolutional networks for temporal action localization. In: _ICCV_. 2019. p. 7094-7103. · DOI · Paper page
  17. Girdhar R, Carreira J, Doersch C, Zisserman A. Video action transformer network. In: _CVPR_; 2019. p. 244-253. · DOI
  18. Zhang CL, Wu J, Li Y. Actionformer: Localizing moments of actions with transformers. In: _ECCV_. Springer; 2022. p. 492-510. · DOI · Paper page
  19. Dai R, Das S, Bremond F (2021). "Ctrn: Class-temporal relational network for action detection". _arXiv preprint arXiv:2110.13473_. Available from: https://arxiv.org/abs/2110.13473. · arXiv
  20. Dai R, Das S, Kahatapitiya K, Ryoo MS, Brémond F. "Ms-tct: Multi-scale temporal convtransformer for action detection". In: _CVPR_. 2022. p. 20041–20051. · DOI
  21. Zhu Z, Tang W, Wang L, Zheng N, Hua G (2021). "Enriching local and global contexts for temporal action localization". In: _ICCV_. pp. 13516–13525. · DOI
  22. Sardari F, Mustafa A, Jackson PJ, Hilton A. Pat: Position-aware transformer for dense multi-label action detection. In: _CVPR_. 2023. p. 2988-2997. · DOI
  23. Liu Y, Li S, Wu Y, Chen CW, Shan Y, Qie X (2022). "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection". In: _CVPR_. pp. 3042–3051. · DOI · Paper page
  24. Mun J, Cho M, Han B. Local-global video-text interactions for temporal grounding. In: _CVPR_; 2020. p. 10810-10819. · DOI · Paper page
  25. Awadalla A, Gao I, Gardner J, Hessel J, Hanafy Y, Zhu W, Marathe K, Bitton Y, Gadre S, Sagawa S, Jitsev J, Kornblith S, Koh PW, Ilharco G, Wortsman M, Schmidt L. "OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models". _arXiv_ [cs.CV]. 2023. Available from: https://arxiv.org/abs/2308.01390. · arXiv
  26. Li B, Zhang Y, Chen L, Wang J, Pu F, Yang J, Li C, Liu Z (2023). "MIMIC-IT: Multi-Modal In-Context Instruction Tuning". _arXiv_. Available from: https://arxiv.org/abs/2306.05425. · arXiv
  27. Zhang R, Han J, Liu C, Gao P, Zhou A, Hu X, Yan S, Lu P, Li H, Qiao Y (2024). "LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention". _arXiv_. arXiv:2303.16199 [https://arxiv.org/abs/2303.16199] [cs.CV]. · arXiv
  28. Dai W, Li J, Li D, Tiong AMH, Zhao J, Wang W, Li B, Fung P, Hoi S (2023). "InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning". _arXiv_. arXiv:2305.06500 [https://arxiv.org/abs/2305.06500] [cs.CV]. · arXiv
  29. Li K, He Y, Wang Y, Li Y, Wang W, Luo P, Wang Y, Wang L, Qiao Y (2024). "VideoChat: Chat-Centric Video Understanding". _arXiv_. arXiv:2305.06355 [https://arxiv.org/abs/2305.06355]. · arXiv
  30. Zhang H, Li X, Bing L (2023). "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding". _arXiv_. Available from: https://arxiv.org/abs/2306.02858. · DOI · Paper page
  31. Maaz M, Rasheed H, Khan S, Khan FS. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models". _arXiv_ [cs.CV]. 2024. Available from: https://arxiv.org/abs/2306.05424. · DOI · Paper page
  32. Luo R, Zhao Z, Yang M, Dong J, Li D, Lu P, Wang T, Hu L, Qiu M, Wei Z (2023). "Valley: Video Assistant with Large Language Model Enhanced Ability". _arXiv_. arXiv:2306.07207 [cs.CV] [https://arxiv.org/abs/2306.07207]. · arXiv
  33. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021). "Lora: Low-rank adaptation of large language models". _arXiv preprint arXiv:2106.09685_. 2021. · arXiv
  34. Loshchilov I, Hutter F (2019). "Decoupled Weight Decay Regularization". _arXiv_. arXiv:1711.05101 [cs.LG] [https://arxiv.org/abs/1711.05101]. · arXiv
  35. Jang J, Park J, Kim J, Kwon H, Sohn K (2023). "Knowing Where to Focus: Event-aware Transformer for Video Grounding". _arXiv_. arXiv:2308.06947 [cs.CV] [https://arxiv.org/abs/2308.06947]. · DOI
  36. Moon W, Hyun S, Park S, Park D, Heo JP (2023). "Query-Dependent Video Representation for Moment Retrieval and Highlight Detection". _arXiv_. arXiv:2303.13874 [cs.CV] [https://arxiv.org/abs/2303.13874]. · DOI · Paper page
  37. Shen Y, Xiong X, Nagrani A, Arnab A, Wang Z, Ge W, Ross D, Schmid C (2023). "UnLoc: A Unified Framework for Video Localization Tasks". _arXiv_. arXiv:2308.11062 [cs.CV] [https://arxiv.org/abs/2308.11062]. · arXiv
  38. Yu S, Cho J, Yadav P, Bansal M (2023). "Self-Chained Image-Language Model for Video Localization and Question Answering". _arXiv_. arXiv:2305.06988 [cs.CV] [https://arxiv.org/abs/2305.06988]. · arXiv
  39. Lin KQ, Zhang P, Chen J, Pramanick S, Gao D, Wang AJ, Yan R, Shou MZ (2023). "UniVTG: Towards Unified Video-Language Temporal Grounding". _arXiv_. arXiv:2307.16715 [cs.CV] [https://arxiv.org/abs/2307.16715]. · DOI
  40. Wang Y, Li K, Li X, Yu J, He Y, Wang C, Chen G, Pei B, Yan Z, Zheng R, Xu J, Wang Z, Shi Y, Jiang T, Li S, Zhang H, Huang Y, Qiao Y, Wang Y, Wang L (2024). "InternVideo2: Scaling Foundation Models for Multimodal Video Understanding". _arXiv_. arXiv:2403.15377 [cs.CV] [https://arxiv.org/abs/2403.15377]. · DOI · Paper page
  41. Krishna R, Hata K, Ren F, Fei-Fei L, Niebles JC (2017). "Dense-Captioning Events in Videos". _arXiv_. arXiv:1705.00754 [cs.CV] [https://arxiv.org/abs/1705.00754]. · DOI · Paper page
  42. Zeng R, Xu H, Huang W, Chen P, Tan M, Gan C (2020). "Dense Regression Network for Video Grounding". _arXiv_. arXiv:2004.03545 [cs.CV] [https://arxiv.org/abs/2004.03545]. · DOI · Paper page
  43. Soldan M, Xu M, Qu S, Tegner J, Ghanem B (2021). "VLG-Net: Video-Language Graph Matching Network for Video Grounding". _arXiv_. arXiv:2011.10132 [https://arxiv.org/abs/2011.10132] [cs.CV]. · DOI
  44. Wang T, Zhang J, Zheng F, Jiang W, Cheng R, Luo P (2023). "Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos". _arXiv_. arXiv:2303.06378 [https://arxiv.org/abs/2303.06378] [cs.CV]. · arXiv

Cited by

See all 112

Metadata

Authors
Li, J.
Venue
Qeios
Publisher
Qeios Ltd
Published
2024-12-05
ISSN
2632-3834 (ISSN-L)
OpenAlex
W4405081677
Cited by
112
References
6
Type
posted content
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval. https://app.cochat.ai/papers/10.32388/vlxb6m
The paper
Li, J. (2024). LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval. Qeios. https://doi.org/10.32388/vlxb6m

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.