CoChat
Sign Up
Cited by 536Human Pose and Action RecognitionDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

Open access
Wang, Limin, Huang, Bingkun, Zhao, Zhiyu, Tong, Zhan, He, Yinan, Wang, Yi, Wang, Yali, Qiao, Yu·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52729.2023.01398
Open this paper in CoChat — free accountOpen PDF

Abstract

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01398

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multimodal large-language models and comparable video prediction methods making predictions in pixel-space perform around chance.

    “Surprisingly, we find that multimodal large-language models (Wang et al., 2024; Reid et al., 2024) and comparable video prediction methods making predictions in pixel-space (Wang et al., 2023) perform around chance.”

    From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Wang et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 98

  1. Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisser- man. Self-supervised multimodal versatile networks. In NeurIPS, 2020. 12
  2. Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020. 12
  3. Anonymous. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. In Submitted to The Eleventh International Conference on Learning Repre- sentations, 2023. under review. 9
  4. Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, pages 6816-6826, 2021. 8, 12
  5. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718, 2021. 5, 9
  6. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR, 2022. 1, 2, 3, 5, 7, 11, 13
  7. Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, 2020. 12
  8. Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Mar- keeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, pages 10925-10934, 2022. 11
  9. Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms-improving object detection with one line of code. In ICCV, pages 5561-5569, 2017. 10
  10. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020. 1, 2, 3, 8
  11. Guo Chen, Yin-Dong Zheng, Limin Wang, and Tong Lu. DCAN: improving temporal action detection via dual con- text aggregation. In AAAI, pages 248-257, 2022. 8
  12. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In ICML, pages 1691-1703, 2020. 3, 11, 13
  13. Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed perception for unsupervised video representation learning. In AAAI, 2021. 12
  14. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey E. Hinton. A simple framework for contrastive learn- ing of visual representations. In ICML, pages 1597-1607, 2020. 2
  15. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pages 15750-15758, 2021. 2
  16. Feng Cheng and Gedas Bertasius. Tallformer: Temporal action localization with a long-memory transformer. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Gio- vanni Maria Farinella, and Tal Hassner, editors, ECCV, pages 503-521, 2022. 8
  17. Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng. Masked spectrogram prediction for self-supervised audio pre-training. CoRR, abs/2204.12768, 2022. 1 · arXiv
  18. Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020. 11, 13
  19. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In CVPR, pages 248-255, 2009. 2
  20. Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool. Vi2clr: Video and image for visual contrastive learning of representation. In ICCV, 2021. 12
  21. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An im- age is worth 16x16 words: Transformers for image recog- nition at scale. In ICLR, 2021. 2, 3, 4
  22. Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichten- hofer. Multiscale vision transformers. In ICCV, pages 6804-6815, 2021. 12
  23. Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In NeurIPS, 2022. 1, 3, 7, 8, 12
  24. Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Gir- shick, and Kaiming He. A large-scale study on unsuper- vised spatiotemporal representation learning. In CVPR, 2021. 12
  25. Chunhui Gu, Chen Sun, David A. Ross, Carl Von- drick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- narasimhan, George Toderici, Susanna Ricco, Rahul Suk- thankar, Cordelia Schmid, and Jitendra Malik. AVA: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, pages 6047-6056, 2018. 2, 5, 9
  26. Tengda Han, Weidi Xie, and Andrew Zisserman. Memory- augmented dense predictive coding for video representation learning. In ECCV, 2020. 12
  27. Tengda Han, Weidi Xie, and Andrew Zisserman. Self- supervised co-training for video representation learning. In NeurIPS, 2020. 12
  28. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735, 2020. 2
  29. Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improv- ing generalization through instance repetition. In CVPR, pages 8129-8138, 2020. 11
  30. Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order representations for video self-supervised learning. In ICCV, 2021. 12
  31. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Fi- rat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Ji- quan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, pages 103-112, 2019. 3
  32. Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017. 2, 5, 9, 10 · arXiv
  33. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 10 · arXiv
  34. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, pages 491-507, 2020. 3
  35. Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature. CoRR, abs/2208.04164, 2022. 2, 4 · arXiv
  36. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2324, 1998. 2
  37. Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised representation learning by sort- ing sequence. In ICCV, pages 667-676, 2017. 12
  38. Ang Li, Meghana Thotakuri, David A. Ross, João Car- reira, Alexander Vostrikov, and Andrew Zisserman. The ava-kinetics localized human actions video dataset. CoRR, abs/2005.00214, 2020. 2, 5 · arXiv
  39. Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified trans- former for efficient spatial-temporal representation learn- ing. In ICLR, 2022. 3, 8
  40. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. CoRR, abs/2211.09552, 2022. 5, 9 · arXiv
  41. Tianhao Li and Limin Wang. Learning spatiotemporal fea- tures via video and text pair discrimination. arXiv preprint arXiv:2001.05691, 2020. 12 · arXiv
  42. Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. TEA: temporal excitation and aggregation for action recognition. In CVPR, pages 906-915, 2020. 8
  43. Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Man- galam, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, pages 4794-4804, 2022. 8, 12
  44. Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Learning salient boundary feature for anchor-free temporal action localization. In CVPR, pages 3320-3329, 2021. 8
  45. Ji Lin, Chuang Gan, and Song Han. TSM: temporal shift module for efficient video understanding. In ICCV, pages 7082-7092, 2019. 2, 8
  46. Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: boundary-matching network for temporal action pro- posal generation. In ICCV, pages 3888-3897, 2019. 8
  47. Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, and Xiang Bai. End-to-end temporal ac- tion detection with transformer. IEEE Trans. Image Pro- cess., 31:5427-5441, 2022. 8
  48. Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal ac- tion localization. IEEE Trans. Image Process., 31:6937- 6950, 2022. 2, 6, 10
  49. Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards an efficient architecture for video recognition. In AAAI, pages 11669-11676, 2020. 8
  50. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, pages 3192-3201, 2022. 2, 8, 12
  51. Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. TAM: temporal adaptive module for video recog- nition. In ICCV, pages 13688-13698, 2021. 2, 8
  52. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 11, 13 · arXiv
  53. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 11, 13 · arXiv
  54. Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pages 185-201, 2018. 3
  55. Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-End Learning of Visual Representations from Uncurated In- structional Videos. In CVPR, 2020. 12
  56. Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram mod- eling using masked autoencoders for learning general- purpose audio representation. CoRR, abs/2204.12260, 2022. 1 · arXiv
  57. Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor-context-actor relation net- work for spatio-temporal action localization. In CVPR, pages 464-474, 2021. 8
  58. Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In CVPR, 2021. 12
  59. Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2536-2544, 2016. 3
  60. Mandela Patrick, Yuki M. Asano, Polina Kuznetsova, Ruth Fong, João F. Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. In ICCV, 2021. 12
  61. Mandela Patrick, Dylan Campbell, Yuki M. Asano, Is- han Misra, Florian Metze, Christoph Feichtenhofer, An- drea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NeurIPS, pages 12493-12506, 2021. 8
  62. AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, 2020. 12
  63. Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, 2021. 12
  64. Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Xiang Wang, Yuehuan Wang, Yiliang Lv, Changxin Gao, and Nong Sang. MAR: masked autoencoders for efficient action recogni- tion. CoRR, abs/2207.11660, 2022. 4 · arXiv
  65. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021. 1, 2
  66. Karen Simonyan and Andrew Zisserman. Very deep con- volutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015. 3
  67. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 2, 5, 10, 11 · arXiv
  68. Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recognition. In CVPR, pages 1099-1108, 2020. 8
  69. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vin- cent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015. 3
  70. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2818-2826, 2016. 11
  71. Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and con- trastive learning. arXiv preprint arXiv:2106.11250, 2021. 3, 8 · arXiv
  72. Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Re- laxed transformer decoders for direct action proposal gen- eration. In ICCV, pages 13506-13515, 2021. 8
  73. Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, and Cewu Lu. Asynchronous interaction aggregation for action detection. In European Conference on Computer Vision, pages 71-87. Springer, 2020. 10
  74. Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotem- poral convolutions for action recognition. In CVPR, pages 6450-6459, 2018. 2
  75. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017. 1, 2
  76. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing ro- bust features with denoising autoencoders. In ICML, 2008. 3
  77. Chenhao Wang, Hongxiang Cai, Yuxin Zou, and Yichao Xiong. RGB stream is enough for temporal action detec- tion. CoRR, abs/2107.04362, 2021. 8 · arXiv
  78. Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation networks. In CVPR, pages 349-358, 2020. 8
  79. Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self- supervised video representation learning by pace predic- tion. In ECCV, 2020. 12
  80. Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classification. In CVPR, pages 1430-1439, 2018. 2
  81. Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. TDN: temporal difference networks for efficient action recogni- tion. In CVPR, pages 1895-1904, 2021. 2, 8, 12
  82. Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, pages 20-36, 2016. 2, 7
  83. Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of video transform- ers. In CVPR, pages 14713-14723, 2022. 3, 8
  84. Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In ECCV, pages 413-431, 2018. 8
  85. Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan L. Yuille, and Christoph Feichtenhofer. Masked feature pre- diction for self-supervised visual pre-training. In CVPR, pages 14648-14658, 2022. 1, 3, 8, 12
  86. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: a simple framework for masked image modeling. In CVPR, pages 9643-9653, 2022. 1, 2, 3
  87. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. CoRR, abs/2206.04664, 2022. 2, 6 · arXiv
  88. Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, pages 10334- 10343, 2019. 12
  89. Mengmeng Xu, Chen Zhao, David S. Rojas, Ali K. Thabet, and Bernard Ghanem. G-TAD: sub-graph localization for temporal action detection. In CVPR, pages 10153-10162, 2020. 8
  90. Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou. Video representation learning with visual tempo consis- tency. arXiv preprint arXiv:2006.15489, 2020. 12 · arXiv
  91. Min Yang, Guo Chen, Yin-Dong Zheng, Tong Lu, and Limin Wang. Basictad: an astounding rgb-only baseline for temporal action detection. CoRR, abs/2205.02717, 2022. 8 · arXiv
  92. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In CVPR, pages 6023-6032, 2019. 11, 13
  93. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, pages 1204-1213, 2022. 2, 3, 4, 5, 8
  94. Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, An- drew M. Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recog- nition. CoRR, abs/2112.07175, 2021. 8, 12 · arXiv
  95. Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In ECCV, pages 492-510, 2022. 8, 10
  96. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk mini- mization. arXiv preprint arXiv:1710.09412, 2017. 11, 13 · arXiv
  97. Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R. Scott, and Limin Wang. V4D: 4d convolutional neural networks for video-level representation learning. In ICLR, 2020. 8
  98. Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, Ivan Marsic, Cees G. M. Snoek, and Joseph Tighe. Tuber: Tubelet transformer for video action detection. In CVPR, pages 13588-13597, 2022. 7, 8

Cited by

Sorted by citations

Authors
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y.
Affiliations
  • Shanghai Artificial Intelligence Laboratory (CN)
  • Nanjing University,State Key Laboratory for Novel Software Technology,China
  • Chinese Academy of Sciences (CN)
ORCID
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
14549-14560
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52729.2023.01398
OpenAlex
W4386057769
Cited by
536
References
20
Citations by year
2026: 154 · 2025: 191 · 2024: 165 · 2023: 26
FWCI
54.58
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Key R&D Program of China (2022ZD0160900,2022ZD0160100)
  • National Natural Science Foundation of China (62076119,61921006)
  • Youth Innovation Promotion Association of Chinese Academy of Sciences (2020355)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
VideoMAE, video foundation models, self-supervised pre-training, progressive training, unlabeled video data, video representation learning, kinetics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review