MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Open accessAbstract
Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some video understanding systems draw upon the Atkinson-Shiffrin Model to construct hierarchical memory systems.
“For instance, some video understanding systems [11, 14] draw upon the Atkinson-Shiffrin Model [6] (a highly influential human memory model, detailed in Sec. 3) to construct hierarchical memory systems.”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [14]
The Atkinson-Shiffrin model is frequently referenced in AI memory research.
“Notably, this model and its derivative theories are frequently referenced in AI memory research [61, 11, 14], bridging the fields of human and AI memory studies.”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [14]
Some AI long-term memory research is directly inspired by human long-term memory, while other studies align with human long-term memory processing mechanisms without intentional imitation.
“some AI long-term memory research is directly inspired by human long-term memory [245, 11, 14, 190, 191], while other studies align with human long-term memory processing mechanisms without intentional imitation.”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Human Memory: A Proposed System and its Control Processes
Atkinson, R. C., & Shiffrin, R. M. (1968) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60422-3 · Cited by 7,216 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J.-Q., Wang, W., Li, B. A., Fung, P., & Hoi, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.06500 · Cited by 4,191 · see citing works
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294 · see citing works
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video UnderstandingPreprint
Zhang, H., Li, X., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2306.02858 · Cited by 2,019 · see citing works
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y.-H., Qin, Y.-L., Zhang, M.-D., Lin, X., Yang, J., Zheng, X.-W., Li, K., Sun, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13394 · Cited by 1,917
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsPreprint
Maaz, M., Rasheed, H., Khan, S. H., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2306.05424 · Cited by 1,339 · see citing works
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · see citing works
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
Mangalam, K., Akshulakov, R., & Malik, J. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2308.09126 · Cited by 857
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z.-Y., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C.-H., Yue, X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.15010 · Cited by 777 · see citing works
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric TasksPreprint
Wang, W., Chen, Z., Chen, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11175 · Cited by 728 · see citing works
Squire, L. R., Genzel, L., Wixted, J. T., & Morris, R. G. (2015) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a021766 · Cited by 700 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · see citing works
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · see citing works
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01855 · Cited by 540 · see citing works
A database for fine grained activity detection of cooking activities
Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247801 · Cited by 521 · see citing works
PandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · Tsinghua Interdisciplinary Workshop on Logic, Language and Meaning · doi:10.48550/arxiv.2305.16355 · Cited by 474 · see citing works
Grounding Action Descriptions in Videos
Regneri, M., Rohrbach, M., Wetzel, D., Thater, S., Schiele, B., & Pinkal, M. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00207 · Cited by 471 · see citing works
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
Learning Dynamic Memory Networks for Object Tracking
Yang, T., & Chan, A. B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_10 · Cited by 382 · see citing works
MeMOT: Multi-Object Tracking with Memory
Cai, J., Xu, M., Li, W., Xiong, Y., Xia, W., Tu, Z., & Soatto, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00792 · Cited by 265 · see citing works
Coherent Multi-sentence Video Description with Variable Level of Detail
Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., & Schiele, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11752-2_15 · Cited by 209 · see citing works
Kernelized Memory Network for Video Object Segmentation
Seong, H., Hyun, J., & Kim, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_38 · Cited by 204 · see citing works
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionPreprint
Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., & Qiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16199 · Cited by 202 · see citing works
Script Data for Attribute-Based Recognition of Composite Activities
Rohrbach, M., Regneri, M., Andriluka, M., Amin, S., Pinkal, M., & Schiele, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_11 · Cited by 186 · see citing works
Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data
Rohrbach, M., Rohrbach, A., Regneri, M., Amin, S., Andriluka, M., Pinkal, M., & Schiele, B. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0851-8 · Cited by 182 · see citing works
MovieNet: A Holistic Dataset for Movie Understanding
Huang, Q., Xiong, Y., Rao, A., Wang, J., & Lin, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_41 · Cited by 172 · see citing works
Temporal Aggregate Representations for Long-Range Video Understanding
Şener, F., Singhania, D., & Yao, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_10 · Cited by 139 · see citing works
MovieGraphs: Towards Understanding Human-Centric Situations from Videos
Vicol, P. A., Tapaswi, M., Castrejón, L., & Fidler, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00895 · Cited by 130 · see citing works
Learning Position and Target Consistency for Memory-based Video Object Segmentation
Hu, L., Zhang, P., Zhang, B., Pan, P., Xu, Y., & Jin, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00413 · Cited by 129 · see citing works
Hierarchical Memory Matching Network for Video Object Segmentation
Seong, H., Oh, S. W., Lee, J., Lee, S., Lee, S., & Kim, E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01265 · Cited by 129 · see citing works
MIST : Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
Gao, D., Zhou, L., Ji, L., Zhu, L., Yang, Y., & Shou, M. Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01419 · Cited by 96 · see citing works
MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions
Soldan, M., Pardo, A., Alcázar, J. L., Heilbron, F. C., Zhao, C., Giancola, S., & Ghanem, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00497 · Cited by 94 · see citing works
Multiple-Target Tracking: A Role for Working Memory?
Allen, R., McGeorge, P., Pearson, D., & Milne, A. B. (2005) · Quarterly Journal of Experimental Psychology · doi:10.1080/02724980543000097 · Cited by 89 · see citing works
VideoChat: Chat-Centric Video UnderstandingPreprint
Li, K., He, Y., Wang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06355 · Cited by 89 · see citing works
Otter: A Multi-Modal Model with In-Context Instruction TuningPreprint
Li, B., Zhang, Y., Chen, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03726 · Cited by 86 · see citing works
Self-citationTitle Generation for User Generated Videos
Zeng, K.-H., Chen, T.-H., Niebles, J. C., & Sun, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_38 · Cited by 78 · see citing works
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · see citing works
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70 · see citing works
Generic Event Boundary Detection: A Benchmark for Event Segmentation
Shou, M. Z., Lei, S. W., Wang, W., Ghadiyaram, D., & Feiszli, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00797 · Cited by 65 · see citing works
Generating Descriptions with Grounded and Co-referenced People
Rohrbach, A., Rohrbach, M., Tang, S., Oh, S. J., & Schiele, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.447 · Cited by 54 · see citing works
Hierarchical Video-Moment Retrieval and Step-Captioning
Zala, A., Cho, J., Kottur, S., Chen, X., Oğuz, B., Mehdad, Y., & Bansal, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02208 · Cited by 45 · see citing works
Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 43 · see citing works
Self-citationMacaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2306.09093 · Cited by 38 · see citing works
Memory Network With Pixel-Level Spatio-Temporal Learning for Visual Object Tracking
Zhou, Z., Zhou, X., Chen, Z., Guo, P., Liu, Q., & Zhang, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3272319 · Cited by 27 · see citing works
Awad, G., Butt, A. A., Curtis, K., Fiscus, J., Godil, A., Lee, Y., Delgado, A., Zhang, J., Godard, E., Chocot, B., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.13473 · Cited by 18 · see citing works
UMOTMA: Underwater multiple object tracking with memory aggregation
Hao, Z., Qiu, J., Zhang, H., Ren, G., & Liu, C. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.1071618 · Cited by 17 · see citing works
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding SystemPreprint
Wang, J., Chen, D., Luo, C., Dai, X., Yuan, L., Wu, Z., & Jiang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14407 · Cited by 9 · see citing works
MAVOT: Memory-Augmented Video Object TrackingPreprint
Liu, B., Wang, Y., Tai, Y., & Tang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09414 · Cited by 7 · see citing works
Multi-Object Tracking with Spatial-Temporal Correlation Memory Networks
Xin, M., Sun, W., Li, K., & Hui, G. (2022) · 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA) · doi:10.1109/cvidliccea56201.2022.9825193 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.00650
StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
Wang, Z., Zhao, L., & Xing, W. (2023) · arXiv · doi:10.48550/arxiv.2308.07863
Zero-Shot Video Question Answering Via Frozen Bidirectional Language Models
Yang, A., Miech, A., Sivic, J., Laptev, I., & Schmid, C. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0010
Flagged first, then by citations
Research works printed but not linked: 15
- George Awad, Asad A Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzal Godil, Andrew Delgado, Jesse Zhang, Eliot Godard, Lukas Diduch, et al. Trecvid 2019: An evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval. arXiv preprint arXiv:2009.09984, 2020. 2 · arXiv
- George Awad, Asad A Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzad Godil, David Joy, Andrew Delgado, Alan F Smeaton, Yvette Graham, et al. Trecvid 2018: Benchmarking video activity detection, video captioning and matching, video storytelling linking and video search. In Proceedings of TRECVID 2018, 2018. 2
- Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2022. 4
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877-1901, 2020. 1, 2
- David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies, pages 190-200, 2011.
- Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. In Computer Vision-ECCV 2022: 17th Eu- ropean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 640-658. Springer, 2022. 3
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019. 4
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888- 12900. PMLR, 2022. 1
- Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang. Adaptive correlation filters with long-term and short- term memory for object tracking. International Journal of Computer Vision, 126:771-796, 2018. 3
- Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631-4640, 2016. 2, 14, 15
- Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim- ing He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 284-293, 2019. 2
- Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1884-1894, 2021. 2, 5
- Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022. 2
- Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645-1653, 2017. 6
- Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 9127-9134, 2019. 6
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
Human-inspired Perspectives: A Survey on AI Long-term Memory
He, Z., Lin, W., Zheng, H., Zhang, F., Jones, M. W., Aitchison, L., Xu, X., Liu, M., Kristensson, P. O., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.00489 · Cited by 3 · see citing works
The AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
- Authors
- Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., Lu, Y., Hwang, J.-N., & Wang, G.
- Affiliations
- Zhejiang University (CN)
- University of Washington (US)
- Donghua University (CN)
- Microsoft Research Asia (China) (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 18221-18232
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 2307.16449
- OpenAlex
- W4402702917
- Cited by
- 176
- References
- 61
- Citations by year
- 2026: 82 · 2025: 60 · 2024: 32 · 2023: 2
- FWCI
- 35.92
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62106219,62206046)
- National Key Research and Development Program of China (2022ZD0162000)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Keywords
- long video understanding, video foundation models, large language models, temporal relations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
