CoChat
Sign Up
PreprintCited by 2Multimodal Machine Learning ApplicationsHuman Pose and Action RecognitionGenerative Adversarial Networks and Image Synthesis

The DOI matches its DataCite record; the references it cites are not checked yet.

VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models

Open access
Lan, Xiaohan, Yuan, Yitian, Jie, Zequn, Ma, Lin·2024·arXiv
DOI 10.48550/arxiv.2410.11417arXiv 2410.11417
Open this paper in CoChat — free accountOpen PDF

Abstract

Video-based multimodal large language models (Video-LLMs) possess significant potential for video understanding tasks. However, most Video-LLMs treat videos as a sequential set of individual frames, which results in insufficient temporal-spatial interaction that hinders fine-grained comprehension and difficulty in processing longer videos due to limited visual token capacity. To address these challenges, we propose VidCompress, a novel Video-LLM featuring memory-enhanced temporal compression. VidCompress employs a dual-compressor approach: a memory-enhanced compressor captures both short-term and long-term temporal relationships in videos and compresses the visual tokens using a multiscale transformer with a memory-cache mechanism, while a text-perceived compressor generates condensed visual tokens by utilizing Q-Former and integrating temporal contexts into query embeddings with cross attention. Experiments on several VideoQA datasets and comprehensive benchmarks demonstrate that VidCompress efficiently models complex temporal-spatial relations and significantly outperforms existing Video-LLMs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.11417

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional approaches to long video understanding do not scale effectively for infinitely long video streams.

    “Traditional approaches to long video understanding, such as extending the context window in multimodal models [4, 60] or compressing visual tokens to increase temporal coverage [21, 49, 49], do not scale effectively for infinitely long video streams.”

    From Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (2025) · cites this paper as [21]

    The Introduction says visual-token counts grow with video length while LLM input capacity is limited. Related Work says token reduction mitigates the resulting loss of efficiency, but the paper does not establish how those approaches perform on infinitely long video streams. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 2

Research works printed but not linked: 41

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Caba Heilbron, F.; Escorcia, V.; Ghanem, B.; and Car- los Niebles, J. 2015. Activitynet: A large-scale video bench- mark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recogni- tion, 961-970.
  3. Chen, D.; and Dolan, W. B. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies, 190-200.
  4. Chen, L.; Li, J.; Dong, X.; Zhang, P.; He, C.; Wang, J.; Zhao, F.; and Lin, D. 2023. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793. · arXiv
  5. Chen, L.; Wei, X.; Li, J.; Dong, X.; Zhang, P.; Zang, Y.; Chen, Z.; Duan, H.; Lin, B.; Tang, Z.; et al. 2024. Sharegpt4video: Improving video understanding and gener- ation with better captions. arXiv preprint arXiv:2406.04325. · arXiv
  6. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6.
  7. Christoph, R.; and Pinz, F. A. 2016. Spatiotemporal residual networks for video action recognition. Advances in neural information processing systems, 2: 3468-3476.
  8. Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500. · arXiv
  9. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186. Minneapolis, Min- nesota: Association for Computational Linguistics.
  10. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  11. Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 6824-6835.
  12. Fang, X.; Mao, K.; Duan, H.; Zhao, X.; Li, Y.; Lin, D.; and Chen, K. 2024. MMBench-Video: A Long-Form Multi- Shot Benchmark for Holistic Video Understanding. arXiv preprint arXiv:2406.14515. · arXiv
  13. Fang, Y.; Wang, W.; Xie, B.; Sun, Q.; Wu, L.; Wang, X.; Huang, T.; Wang, X.; and Cao, Y. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19358-19369.
  14. Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; Zhang, R.; Wang, Z.; Zhou, C.; Shen, Y.; Zhang, M.; et al. 2024. Video- MME: The First-Ever Comprehensive Evaluation Bench- mark of Multi-modal LLMs in Video Analysis. arXiv preprint arXiv:2405.21075. · arXiv
  15. Li, F.; Gan, C.; Liu, X.; Bian, Y.; Long, X.; Li, Y.; Li, Z.; Zhou, J.; and Wen, S. 2017. Temporal modeling ap- proaches for large-scale youtube-8m video understanding. arXiv preprint arXiv:1707.04555. · arXiv
  16. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. In International conference on machine learning, 19730-19742.
  17. PMLR.
  18. Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. Blip: Boot- strapping language-image pre-training for unified vision- language understanding and generation. In International conference on machine learning, 12888-12900.
  19. PMLR.
  20. Li, K.; He, Y.; Wang, Y.; Li, Y.; Wang, W.; Luo, P.; Wang, Y.; Wang, L.; and Qiao, Y. 2023b. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355. · arXiv
  21. Li, K.; Wang, Y.; He, Y.; Li, Y.; Wang, Y.; Liu, Y.; Wang, Z.; Xu, J.; Chen, G.; Luo, P.; et al. 2023c. Mvbench: A compre- hensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005. · arXiv
  22. Li, K.; Wang, Y.; Li, Y.; Wang, Y.; He, Y.; Wang, L.; and Qiao, Y. 2023d. Unmasked teacher: Towards training- efficient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 19948-19960.
  23. Li, Y.; Wang, C.; and Jia, J. 2023. LLaMA-VID: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043. · arXiv
  24. Lin, B.; Zhu, B.; Ye, Y.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-llava: Learning united visual representation by align- ment before projection. arXiv preprint arXiv:2311.10122. · arXiv
  25. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b. Visual Instruc- tion Tuning. Liu, R.; Li, C.; Ge, Y.; Shan, Y.; Li, T. H.; and Li, G. 2023c. One for all: Video conversation is feasible without video in- struction tuning. arXiv preprint arXiv:2309.15785. · arXiv
  26. Maaz, M.; Rasheed, H.; Khan, S.; and Khan, F. S. 2023. Video-chatgpt: Towards detailed video understand- ing via large vision and language models. arXiv preprint arXiv:2306.05424. · arXiv
  27. Miech, A.; Laptev, I.; and Sivic, J. 2017. Learnable pooling with context gating for video classification. arXiv preprint arXiv:1706.06905. · arXiv
  28. Reid, M.; Savinov, N.; Teplyashin, D.; Lepikhin, D.; Lilli- crap, T.; Alayrac, J.-b.; Soricut, R.; Lazaridou, A.; Firat, O.; Schrittwieser, J.; et al. 2024. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. · arXiv
  29. Ren, S.; Yao, L.; Li, S.; Sun, X.; and Hou, L. 2023. TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding. arXiv preprint arXiv:2312.02051. · arXiv
  30. Song, E.; Chai, W.; Wang, G.; Zhang, Y.; Zhou, H.; Wu, F.; Guo, X.; Ye, T.; Lu, Y.; Hwang, J.-N.; et al. 2023. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449. · arXiv
  31. Wang, L.; Xiong, Y.; Wang, Z.; Qiao, Y.; Lin, D.; Tang, X.; and Van Gool, L. 2016. Temporal segment networks: To- wards good practices for deep action recognition. In Euro- pean conference on computer vision, 20-36. Springer.
  32. Wang, W.; He, Z.; Hong, W.; Cheng, Y.; Zhang, X.; Qi, J.; Huang, S.; Xu, B.; Dong, Y.; Ding, M.; et al. 2024. LVBench: An Extreme Long Video Understanding Bench- mark. arXiv preprint arXiv:2406.08035. · arXiv
  33. Wu, C.-Y.; Feichtenhofer, C.; Fan, H.; He, K.; Krahenbuhl, P.; and Girshick, R. 2019. Long-term feature banks for de- tailed video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 284-293.
  34. Wu, C.-Y.; Li, Y.; Mangalam, K.; Fan, H.; Xiong, B.; Ma- lik, J.; and Feichtenhofer, C. 2022. Memvit: Memory- augmented multiscale vision transformer for efficient long- term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13587-13597.
  35. Xu, J.; Mei, T.; Yao, T.; and Rui, Y. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5288-5296.
  36. Xu, L.; Zhao, Y.; Zhou, D.; Lin, Z.; Ng, S. K.; and Feng, J. 2024. Pllava: Parameter-free llava extension from im- ages to videos for video dense captioning. arXiv preprint arXiv:2404.16994. · arXiv
  37. Ye, Q.; Xu, H.; Xu, G.; Ye, J.; Yan, M.; Zhou, Y.; Wang, J.; Hu, A.; Shi, P.; Shi, Y.; et al. 2023. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304.14178. · arXiv
  38. Zhang, H.; Li, X.; and Bing, L. 2023. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858. · arXiv
  39. Zhang, R.; Han, J.; Liu, C.; Gao, P.; Zhou, A.; Hu, X.; Yan, S.; Lu, P.; Li, H.; and Qiao, Y. 2023. Llama-adapter: Effi- cient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199. · arXiv
  40. Zhu, B.; Lin, B.; Ning, M.; Yan, Y.; Cui, J.; Wang, H.; Pang, Y.; Jiang, W.; Zhang, J.; Li, Z.; et al. 2023a. Language- bind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852. · arXiv
  41. Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023b. Minigpt-4: Enhancing vision-language understand- ing with advanced large language models. arXiv preprint arXiv:2304.10592. · arXiv

Metadata

Authors
Lan, X., Yuan, Y., Jie, Z., & Ma, L.
Affiliations
Meituan (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.11417
arXiv
2410.11417
OpenAlex
W4403574051
Cited by
2
References
0
Citations by year
2025: 1 · 2024: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
video understanding, video large language models, time compression, visual token compression, multi-scale transformer, cache memory, Q-Former, cross-attention, video question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review