CoChat
Sign Up
PreprintCited by 2Multimodal Machine Learning ApplicationsVideo Analysis and SummarizationGenerative Adversarial Networks and Image Synthesis

The DOI matches its DataCite record; the references it cites are not checked yet.

StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification

Open access
He, Yichen, Lin, Yuan, Wu, Jianchao, Zhang, Hanchong, Zhang, Yuchen, Le, Ruicheng·2024·arXiv
DOI 10.48550/arxiv.2411.07076arXiv 2411.07076
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing large vision-language models (LVLMs) are largely limited to processing short, seconds-long videos and struggle with generating coherent descriptions for extended video spanning minutes or more. Long video description introduces new challenges, such as consistent character identification and plot-level descriptions incorporating both visual and audio information. To address these, we figure out audio-visual character identification, matching character names to each dialogue, as a key factor. We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information. StoryTeller uses a multimodal large language model that integrates visual, audio, and text modalities to perform audio-visual character identification on minute-long video clips. The results are then fed into a LVLM to enhance consistency of video description. We validate our approach on movie description tasks and introduce MovieStory101, a dataset with dense descriptions for three-minute movie clips. To evaluate long video descriptions, we create StoryQA, a large set of multiple-choice questions for MovieStory101 test set. We assess descriptions by inputting them into GPT-4 to answer these questions, using accuracy as an automatic evaluation metric. Experiments show that StoryTeller outperforms all open and closed-source baselines on StoryQA, achieving 9.5% higher accuracy than the strongest baseline, Gemini-1.5-pro, and demonstrating a +15.56% advantage in human side-by-side evaluations. Additionally, incorporating audio-visual character identification from StoryTeller improves the performance of all video description models, with Gemini-1.5-pro and GPT-4o showing relative improvement of 5.5% and 13.0%, respectively, in accuracy on StoryQA.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.07076

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Extracting cross-modal identity equivalences from video remains a challenging task, even for advanced models like Gemini-1.5-Pro, particularly in scenes with multiple faces and voices.

    “This remains a challenging task, even for advanced models like Gemini-1.5-Pro, particularly in scenes with multiple faces and voices [15].”

    From Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (2025) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 2

Research works printed but not linked: 52

  1. Chen, G.; Chai, S.; Wang, G.; Du, J.; Zhang, W.-Q.; Weng, C.; Su, D.; Povey, D.; Trmal, J.; Zhang, J.; et al. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv:2106.06909. · arXiv
  2. Chen, S.; Wu, Y.; Wang, C.; Liu, S.; Tompkins, D.; Chen, Z.; and Wei, F. 2022. Beats: Audio pre-training with acoustic tokenizers. arXiv:2212.09058. · arXiv
  3. Chen, Y.; Zheng, S.; Wang, H.; Cheng, L.; Chen, Q.; Zhang, S.; and Li, J. 2024a. ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Effi- ciency. arXiv:2406.02167. · arXiv
  4. Chen, Z.; Wang, W.; Tian, H.; Ye, S.; Gao, Z.; Cui, E.; Tong, W.; Hu, K.; Luo, J.; Ma, Z.; et al. 2024b. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites. arXiv:2404.16821. · arXiv
  5. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; Li, B.; Luo, P.; Lu, T.; Qiao, Y.; and Dai, J. 2023. InternVL: Scaling up Vi- sion Foundation Models and Aligning for Generic Visual- Linguistic Tasks. arXiv:2312.14238. · arXiv
  6. Chen, Z.; Yoshioka, T.; Lu, L.; Zhou, T.; Meng, Z.; Luo, Y.; Wu, J.; Xiao, X.; and Li, J. 2020. Continuous speech sep- aration: Dataset and analysis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7284-7288. IEEE.
  7. Chung, J. S.; Huh, J.; Nagrani, A.; Afouras, T.; and Zisser- man, A. 2020. Spot the conversation: speaker diarisation in the wild. arXiv:2007.01216. · arXiv
  8. Desplanques, B.; Thienpondt, J.; and Demuynck, K. 2020. Ecapa-tdnn: Emphasized channel attention, propa- gation and aggregation in tdnn based speaker verification. arXiv:2303.00332. · arXiv
  9. Drossos, K.; Lipping, S.; and Virtanen, T. 2020. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 736-740. IEEE.
  10. Fu, Y.; Cheng, L.; Lv, S.; Jv, Y.; Kong, Y.; Chen, Z.; Hu, Y.; Xie, L.; Wu, J.; Bu, H.; et al. 2021. Aishell-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario. arXiv:2104.03603. · arXiv
  11. Han, T.; Bain, M.; Nagrani, A.; Varol, G.; Xie, W.; and Zis- serman, A. 2023a. Autoad ii: The sequel-who, when, and what in movie audio description. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 13645-13655.
  12. Han, T.; Bain, M.; Nagrani, A.; Varol, G.; Xie, W.; and Zis- serman, A. 2023b. AutoAD: Movie description in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18930-18940.
  13. Han, T.; Bain, M.; Nagrani, A.; Varol, G.; Xie, W.; and Zis- serman, A. 2024. Autoad iii: The prequel-back to the pixels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18164-18174.
  14. He, M.-K.; Du, J.; and Lee, C.-H. 2022. End-to-end audio- visual neural speaker diarization. In Proc. Interspeech, 1461-1465.
  15. Huang, G.; Pang, B.; Zhu, Z.; Rivera, C.; and Soricut, R. 2020a. Multimodal Pretraining for Dense Video Captioning. In AACL-IJCNLP 2020.
  16. Huang, Q.; Xiong, Y.; Rao, A.; Wang, J.; and Lin, D. 2020b. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, 709-727. Springer.
  17. Islam, M. M.; Ho, N.; Yang, X.; Nagarajan, T.; Torresani, L.; and Bertasius, G. 2024. Video ReCap: Recursive Caption- ing of Hour-Long Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18198-18208.
  18. Ji, Y.; Zhang, S.; Wu, J.; Sun, P.; Chen, W.; Xiao, X.; Yang, S.; Yang, Y.; and Luo, P. 2024. IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model. arXiv:2407.07577. · arXiv
  19. Kim, C. D.; Kim, B.; Lee, H.; and Kim, G. 2019. Audio- caps: Generating captions for audios in the wild. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 119-132.
  20. Krishna, R.; Hata, K.; Ren, F.; Fei-Fei, L.; and Car- los Niebles, J. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on com- puter vision, 706-715.
  21. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Li, Y.; Liu, Z.; and Li, C. 2024. Llava-onevision: Easy visual task transfer. arXiv:2408.03326. · arXiv
  22. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. In International conference on machine learning, 19730-19742.
  23. PMLR.
  24. Li, K.; He, Y.; Wang, Y.; Li, Y.; Wang, W.; Luo, P.; Wang, Y.; Wang, L.; and Qiao, Y. 2023b. Videochat: Chat-centric video understanding. arXiv:2305.06355. · arXiv
  25. Lin, B.; Zhu, B.; Ye, Y.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-llava: Learning united visual representation by align- ment before projection. arXiv:2311.10122. · arXiv
  26. Lin, J.; Yin, H.; Ping, W.; Molchanov, P.; Shoeybi, M.; and Han, S. 2024. Vila: On pre-training for visual language mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 26689-26699.
  27. Carletta, J.; Kraaij, W.; Ashby, S.; Bour- ban, S.; Flynn, M.; Guillemot, M.; Hain, T.; Kadlec, J.; Karaiskos, V.; Kronenthal, M.; Lathoud, G.; Lincoln, M.; Lisowska Masson, A.; Post, W.; Reidsma, D.; and Wellner, P. 2005. The AMI meeting corpus. Int'l. Conf. on Methods and Techniques in Behavioral Research.
  28. Meng, C.; Liu, H.; Kong, Q.; Ko, T.; Zhao, C.; Plumbley, M. D.; Zou, Y.; and Wang, W. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing.
  29. Nagrani, A.; and Zisserman, A. 2018. From benedict cum- berbatch to sherlock holmes: Character identification in tv series without a script. arXiv:1801.10442. · arXiv
  30. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. · arXiv
  31. Panayotov, V.; Chen, G.; Povey, D.; and Khudanpur, S. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), 5206-5210. IEEE.
  32. Peng, W.; Shuai, B.; Sinan, T.; Shijie, W.; Zhihao, F.; Jinze, B.; Keqin, C.; Xuejing, L.; Jialin, W.; Wenbin, G.; Yang, F.; Kai, D.; Mengfei, D.; Xuancheng, R.; Rui, M.; Dayiheng, L.; Chang, Z.; Jingren, Z.; and Junyang, L. 2024. Qwen2- VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv:2409.12191. · arXiv
  33. Raajesh, H.; Desanur, N. R.; Khan, Z.; and Tapaswi, M. 2024. MICap: A Unified Model for Identity-aware Movie Descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14011-14021.
  34. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, 8748-8763.
  35. PMLR.
  36. Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, 28492-28518.
  37. PMLR.
  38. Reid, M.; Savinov, N.; Teplyashin, D.; Lepikhin, D.; Lilli- crap, T.; Alayrac, J.-b.; Soricut, R.; Lazaridou, A.; Firat, O.; Schrittwieser, J.; et al. 2024. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv:2403.05530. · arXiv
  39. Ryant, N.; Church, K.; Cieri, C.; Cristia, A.; Du, J.; Ganapathy, S.; and Liberman, M. 2019. The second di- hard diarization challenge: Dataset, task, and baselines. arXiv:1906.07839. · arXiv
  40. Tang, C.; Yu, W.; Sun, G.; Chen, X.; Tan, T.; Li, W.; Lu, L.; Ma, Z.; and Zhang, C. 2023. Salmonn: To- wards generic hearing abilities for large language models. arXiv:2310.13289. · arXiv
  41. Tapaswi, M.; Bäuml, M.; and Stiefelhagen, R. 2012. "Knock! Knock! Who is it?" probabilistic person identifi- cation in TV-series. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2658-2665. IEEE.
  42. Tapaswi, M.; Zhu, Y.; Stiefelhagen, R.; Torralba, A.; Urta- sun, R.; and Fidler, S. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, 4631-4640.
  43. Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4566-4575.
  44. Wang, J.; Yuan, L.; and Zhang, Y. 2024. Tarsier: Recipes for Training and Evaluating Large Video Description Models. arXiv:2407.00634. · arXiv
  45. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024. Qwen2-VL: En- hancing Vision-Language Model's Perception of the World at Any Resolution. arXiv:2409.12191. · arXiv
  46. Xu, E. Z.; Song, Z.; Tsutsui, S.; Feng, C.; Ye, M.; and Shou, M. Z. 2022. Ava-avd: Audio-visual speaker diarization in the wild. In Proceedings of the 30th ACM International Con- ference on Multimedia, 3838-3847.
  47. Yu, F.; Zhang, S.; Fu, Y.; Xie, L.; Zheng, S.; Du, Z.; Huang, W.; Guo, P.; Yan, Z.; Ma, B.; Xu, X.; and Bu, H. 2022a. M2MeT: The ICASSP 2022 Multi-Channel Multi- Party Meeting Transcription Challenge. In Proc. ICASSP. IEEE.
  48. Yu, F.; Zhang, S.; Guo, P.; Fu, Y.; Du, Z.; Zheng, S.; Huang, W.; Xie, L.; Tan, Z.-H.; Wang, D.; Qian, Y.; Lee, K. A.; Yan, Z.; Ma, B.; Xu, X.; and Bu, H. 2022b. Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Grand Challenge. In Proc. ICASSP. IEEE.
  49. Yue, Z.; Zhang, Q.; Hu, A.; Zhang, L.; Wang, Z.; and Jin, Q. 2023. Movie101: A New Movie Understanding Bench- mark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 4669-4684.
  50. Yue, Z.; Zhang, Y.; Wang, Z.; and Jin, Q. 2024. Movie101v2: Improved Movie Narration Benchmark. arXiv:2404.13370. · arXiv
  51. Zhang, C.; Lin, K.; Yang, Z.; Wang, J.; Li, L.; Lin, C.-C.; Liu, Z.; and Wang, L. 2024. Mm-narrator: Narrating long- form videos with multimodal in-context learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13647-13657.
  52. Zhou, L.; Xu, C.; and Corso, J. J. 2018. Towards Automatic Learning of Procedures From Web Instructional Videos. In AAAI Conference on Artificial Intelligence, 7590-7598.

Metadata

Authors
He, Y., Lin, Y., Wu, J., Zhang, H., Zhang, Y., & Le, R.
Affiliations
Peking University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2411.07076
arXiv
2411.07076
OpenAlex
W4404402436
Cited by
2
References
0
Citations by year
2026: 1 · 2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
multimodal large language models, vision-language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review