CoChat
Sign Up
Cited by 24Multimodal Machine Learning ApplicationsSubtitles and Audiovisual MediaVideo Analysis and Summarization

MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning

Open access
Zhang, Chaoyi, Lin, Kevin, Yang, Zhengyuan, Wang, Jianfeng, Li, Linjie, Lin, Chung-Ching, Liu, Zicheng, Wang, Lijuan·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52733.2024.01295
Open this paper in CoChat — free accountOpen PDF

Abstract

We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the gener-ation of audio descriptions (AD). Unlike previous methods that primarily focused on downstream fine-tuning with short video clips, MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even be-yond hours, in an autoregressive manner. This capability is made possible by the proposed memory-augmented generation process, which effectively utilizes both the short-term textual context and long-term visual memory through an efficient register-and-recall mechanism. These contextual memories compile pertinent past information, including storylines and character identities, ensuring an accurate tracking and depicting of story-coherent and character-centric audio descriptions. Maintaining the training-free design of MM-Narrator, we further propose a complexity-based demonstration selection strategy to largely enhance its multi-step reasoning capability via few-shot multimodal in-context learning (MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics. Additionally, we introduce the first segment-based evaluator for recurrent text generation. Empowered by GPT-4, this evaluator comprehensively reasons and marks AD generation performance in various extendable dimensions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52733.2024.01295

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The memorization task introduces two key challenges beyond long video description: infinite information processing and world knowledge construction.

    “The memorization task relates to long video description [13, 18, 57] but goes beyond it, introducing two key challenges: (1) Infinite information processing .”

    From Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (2025) · cites this paper as [57]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [7][7] Nayyer Aafaq, Ajmal Mian, Wei Liu, Syed Zulqarnain Gilani, and Mubarak Shah. Video description: A survey of methods, datasets, and evaluation metrics. ACM Computing Surveys , 52(6):1-37, 2019. 2 -
  2. [8][8] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022. 3 -
  3. [9][9] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic propositional image caption evaluation. In ECCV , 2016. 6 -
  4. [10][10] Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. WhisperX: Time-accurate speech transcription of longform audio. INTERSPEECH 2023 , 2023. 4, 13 -
  5. [11][11] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop , 2005. 6 -
  6. [12][12] Aanisha Bhattacharya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, and Changyou Chen. A video is worth 4096 tokens: Verbalize story videos to understand them in zero shot. EMNLP , 2023. 3 -
  7. [13][13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 33:18771901, 2020. 3, 7, 8 -
  8. [16][16] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234 , 2022. 3, 4 - · arXiv
  9. [18][18] Jenny Rose Finkel, Trond Grenager, and Christopher D Manning. Incorporating non-local information into information extraction systems by gibbs sampling. In ACL , pages 363370, 2005. 4 -
  10. [19][19] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720 , 2022. 12 - · arXiv
  11. [21][21] Tengda Han, Max Bain, Arsha Nagrani, G¨ ul Varol, Weidi Xie, and Andrew Zisserman. AutoAD II: The Sequel - who, when, and what in movie audio description. In ICCV , 2023. 2, 4, 5, 6 -
  12. [22][22] Tengda Han, Max Bain, Arsha Nagrani, G¨ ul Varol, Weidi Xie, and Andrew Zisserman. AutoAD: Movie description in context. In CVPR , 2023. 2, 4, 5, 6, 13 -
  13. [23][23] Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. arXiv preprint arXiv:2306.15091 , 2023. 4 - · arXiv
  14. [24][24] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV , pages 706-715, 2017. 2 -
  15. [26][26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023. 6 -
  16. [27][27] Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-Centric video understanding. arXiv preprint arXiv:2305.06355 , 2023. 3 - · arXiv
  17. [28][28] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al. VALUE: A multi-task benchmark for video-and-language understanding evaluation. In NeurIPS Datasets and Benchmarks Track , 2021. 2 -
  18. [29][29] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. ACL , 2004. 6 -
  19. [30][30] Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. MM-Vid: Advancing video understanding with GPT-4V (ision). arXiv preprint arXiv:2310.19773 , 2023. 2, 3, 6, 8 - · arXiv
  20. [31][31] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. SwinBert: End-to-end transformers with sparse attention for video captioning. In CVPR , 2022. 2 -
  21. [33][33] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? arXiv preprint arXiv:2101.06804 , 2021. 3, 4, 7, 8 - · arXiv
  22. [35][35] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. GPTEval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634 , 2023. 5 - · arXiv
  23. [36][36] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS , 35:2507-2521, 2022. 3 -
  24. [37][37] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 , 2023. 3 - · arXiv
  25. [38][38] MuhammadMaaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 , 2023. 3 - · arXiv
  26. [41][41] Ron Mokady, Amir Hertz, and Amit H Bermano. ClipCap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021. 6, 8, 13 - · arXiv
  27. [42][42] David Nukrai, Ron Mokady, and Amir Globerson. Textonly training for image captioning using noise-injected clip. EMNLP Findings , 2022. 6 -
  28. [46][46] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL , 2002. 6 -
  29. [47][47] Elisa Perego. Gains and losses of watching audio described films for sighted viewers. Target , 28(3):424-444, 2016. 2 -
  30. [48][48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763. PMLR, 2021. 3, 4, 13 -
  31. [49][49] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML , 2023. 6 -
  32. [50][50] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In CVPR , pages 3202-3212, 2015. 2 -
  33. [53][53] Mattia Soldan, Alejandro Pardo, Juan Le´ on Alc´ azar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. MAD: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR , pages 5026-5035, 2022. 2 -
  34. [55][55] Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. An information-theoretic approach to prompt engineering without ground truth labels. arXiv preprint arXiv:2203.11364 , 2022. 3 - · arXiv
  35. [56][56] Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. Multilingual llms are better cross- lingual in-context learners with alignment. arXiv preprint arXiv:2305.05940 , 2023. 3 - · arXiv
  36. [57][57] Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070 , 2015. 2 - · arXiv
  37. [58][58] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 3 - · arXiv
  38. [59][59] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In CVPR , 2015. 6 -
  39. [60][60] Chien-Yao Wang, Alexey Bochkovskiy, and HongYuan Mark Liao. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In CVPR , pages 7464-7475, 2023. 3 -
  40. [61][61] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VaTeX: A large-scale, highquality multilingual dataset for video-and-language research. In ICCV , October 2019. 2 -
  41. [62][62] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS , 35:24824-24837, 2022. 2, 3 -
  42. [63][63] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. GRiT: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280 , 2022. 3, 6 - · arXiv
  43. [64][64] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR , pages 5288-5296, 2016. 2 -
  44. [65][65] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The Dawn of LMMs: Preliminary explorations with GPT-4V (ision). arXiv preprint arXiv:2309.17421 , 2023. 6 - · arXiv
  45. [66][66] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019. 6 - · arXiv

Cited by

Sorted by citations

Authors
Zhang, C., Lin, K., Yang, Z., Wang, J., Li, L., Lin, C.-C., Liu, Z., & Wang, L.
Affiliations
  • The University of Sydney (AU)
  • Microsoft (United States) (US)
  • Advanced Micro Devices (United States) (US)
ORCID
Venue
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2024-06-16
Pages
13647-13657
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52733.2024.01295
OpenAlex
W4402703105
Cited by
24
References
14
Citations by year
2026: 7 · 2025: 12 · 2024: 5
FWCI
4.95
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Keywords
GPT-4, demonstration selection, few-shot learning, visual memory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review