CoChat
Sign Up
PreprintCited by 20Multimodal Machine Learning ApplicationsHuman Pose and Action RecognitionVideo Analysis and Summarization

The DOI matches its DataCite record.

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

Open access
Ren, Shuhuai, Chen, Sishuo, Li, Shicheng, Sun, Xu, Hou, Lu·2023·arXiv
DOI 10.48550/arxiv.2310.19060arXiv 2310.19060
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.19060

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most current research on long video learning focuses on discriminative tasks such as video understanding.

    “Due to the inherent complexity and resource demands, most current research on long video learning focuses on discriminative tasks such as video understanding [75, 112, 24].”

    From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [112]

    The Introduction describes an efficiency bottleneck for long-form video understanding, and the paper evaluates retrieval and long-form video question answering. It does not establish that most long-video research focuses on discriminative tasks; the Introduction also lists video captioning among video-language tasks. No page number is available in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 58

  1. Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. 2021. Vivit: A video vision transformer. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816-6826.
  2. Max Bain, Arsha Nagrani, A. Brown, and Andrew Zisserman. 2020. Condensed movies: Story based retrieval with contextual embeddings. ArXiv, abs/2005.04208. · arXiv
  3. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1708-1718.
  4. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2022. A clip-hitchhiker's guide to long video retrieval. ArXiv, abs/2205.08508. · arXiv
  5. Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding? In International Conference on Ma- chine Learning.
  6. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. ArXiv, abs/2210.09461. · arXiv
  7. Qingqing Cao, Bhargavi Paranjape, and Hanna Ha- jishirzi. 2023. Pumer: Pruning and merging to- kens for efficient vision language models. ArXiv, abs/2305.17530. · arXiv
  8. Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long- tail visual concepts. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3557-3567.
  9. Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022. Litevl: Efficient video- language learning with enhanced spatial-temporal modeling. In Conference on Empirical Methods in Natural Language Processing.
  10. Feng Cheng, Xizi Wang, Jie Lei, David J. Crandall, Mohit Bansal, and Gedas Bertasius. 2022. Vindlu: A recipe for effective video-and-language pretraining. ArXiv, abs/2212.05051. · arXiv
  11. Ioana Croitoru, Simion-Vlad Bogolin, Yang Liu, Samuel Albanie, Marius Leordeanu, Hailin Jin, and Andrew Zisserman. 2021. Teachtext: Crossmodal generalized distillation for text-video retrieval. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11563-11573.
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs/1810.04805. · arXiv
  13. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929. · arXiv
  14. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2021. Violet : End-to-end video-language trans- formers with masked visual-token modeling. ArXiv, abs/2111.12681. · arXiv
  15. Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. 2022. Bridging video- text retrieval with multiple choice questions. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 16146-16155.
  16. Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Key- van Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022. X-pool: Cross-modal language-video attention for text-video retrieval. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4996-5005.
  17. Saurabh Goyal, Anamitra R. Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning.
  18. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll'ar, and Ross B. Girshick. 2021. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15979-15988.
  19. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738.
  20. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. 2017. Localizing moments in video with natural language. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804-5813.
  21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  22. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. 2017 IEEE International Confer- ence on Computer Vision (ICCV), pages 706-715.
  23. Jie Lei, Tamara L. Berg, and Mohit Bansal. 2022. Re- vealing single frame bias for video-and-language learning. ArXiv, abs/2206.03428. · arXiv
  24. Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7327-7337.
  25. Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C. H. Hoi. 2021a. Align and prompt: Video-and-language pre-training with entity prompts. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4943- 4953.
  26. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Ma- chine Learning.
  27. Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021b. Align before fuse: Vision and language representation learning with momentum distillation. ArXiv, abs/2107.07651. · arXiv
  28. Yanghao Li, Chaoxia Wu, Haoqi Fan, Karttikeya Man- galam, Bo Xiong, Jitendra Malik, and Christoph Fe- ichtenhofer. 2021c. Mvitv2: Improved multiscale vision transformers for classification and detection. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4794-4804.
  29. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021. Video swin trans- former. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192- 3201.
  30. Stuart P. Lloyd. 1982. Least squares quantization in pcm. IEEE Trans. Inf. Theory, 28:129-136.
  31. Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Chao Zhang, and Rongrong Ji. 2022. X-clip: End- to-end multi-grained contrastive learning for video- text retrieval. Proceedings of the 30th ACM Interna- tional Conference on Multimedia.
  32. Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish K. Prabhu, Mohammad Rastegari, and Oncel Tuzel. 2021. Token pooling in vision transformers. ArXiv, abs/2110.03860. · arXiv
  33. Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. ArXiv, abs/1804.02516. · arXiv
  34. Andreea-Maria Oncescu, João F. Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. 2020. Queryd: A video dataset with high-quality text and audio narrations. ICASSP 2021 -2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2265-2269.
  35. Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. 2021. Spatiotemporal contrastive video representa- tion learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6964-6974.
  36. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In International Conference on Machine Learning.
  37. Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Ef- ficient vision transformers with dynamic token sparsi- fication. In Neural Information Processing Systems.
  38. Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. 2017. Youtube- boundingboxes: A large high-precision human- annotated data set for object detection in video. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7464-7473.
  39. Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. 2023a. Delving into the openness of CLIP. In Findings of the Association for Com- putational Linguistics: ACL 2023. Association for Computational Linguistics.
  40. Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, Mu Li, Alexander J. Smola, and Xu Sun. 2023b. Prompt pre-training with twenty-thousand classes for open-vocabulary visual recognition. vol- ume abs/2304.04704. · arXiv
  41. Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. To- kenlearner: What can 8 learned tokens do for images and videos? ArXiv, abs/2106.11297. · arXiv
  42. Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. 2023. Crossget: Cross- guided ensemble of tokens for accelerating vision- language transformers. ArXiv, abs/2305.17455. · arXiv
  43. Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. 2022. Long-form video- language pre-training with multimodal temporal con- trastive learning. ArXiv, abs/2210.06031. · arXiv
  44. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training. ArXiv, abs/2203.12602. · arXiv
  45. Alex Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022a. All in one: Ex- ploring unified video-language pre-training. ArXiv, abs/2203.07303. · arXiv
  46. Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu- Gang Jiang, and Lu Yuan. 2022b. Omnivl: One foun- dation model for image-language and video-language tasks. ArXiv, abs/2209.07526. · arXiv
  47. Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xiansheng Hua. 2022c. Disentangled repre- sentation learning for text-video retrieval. ArXiv, abs/2203.07111. · arXiv
  48. Chaoxia Wu and Philipp Krähenbühl. 2021. Towards long-form video understanding. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1884-1894.
  49. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettle- moyer Christoph Feichtenhofer. 2021. Videoclip: Contrastive pre-training for zero-shot video-text un- derstanding. In Conference on Empirical Methods in Natural Language Processing.
  50. Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and X. Wang. 2022. Groupvit: Semantic segmentation emerges from text supervision. 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 18113-18123.
  51. Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Bain- ing Guo. 2021. Advancing high-resolution video- language representation with large-scale video tran- scriptions. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5026- 5035.
  52. Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Rui Song, Houqiang Li, and Jiebo Luo. 2022. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. ArXiv, abs/2209.06430. · arXiv
  53. Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. 2022. Video-text modeling with zero-shot transfer from contrastive captioners. ArXiv, abs/2212.04979. · arXiv
  54. Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2020. Just ask: Learning to answer questions from millions of narrated videos. 2021 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 1666-1677.
  55. Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Chao Zhang, and Fei Huang. 2022. Hitea: Hierarchical temporal-aware video-language pre-training. ArXiv, abs/2212.14546. · arXiv
  56. Zhou Yu, D. Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. ArXiv, abs/1906.02467. · arXiv
  57. Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. In Neural Information Processing Systems.
  58. Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gok- berk Cinbis, David F. Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. 2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 3532-3540.

Cited by

Sorted by citations

Metadata

Authors
Ren, S., Chen, S., Li, S., Sun, X., & Hou, L.
Affiliations
Peking University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
932-947
DOI
10.48550/arxiv.2310.19060
arXiv
2310.19060
OpenAlex
W4389519526
Cited by
20
References
0
Citations by year
2026: 6 · 2025: 9 · 2024: 5
FWCI
2.14
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (62176002)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Keywords
long-form video, frame aggregation, patch aggregation, video encoder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review