ViViT: A Video Vision Transformer
Open accessAbstract
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Training video transformers from scratch yields unsatisfied results on video datasets.
“Similar to previous trials [3, 6], training video transformers from scratch yields unsatisfied results on video datasets.”
From VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 3× · see citing works
“This architecture corresponds to a “late fusion” [34, 56, 72, 46] of temporal information, and the initial spatial encoder is identical to the one used for image classi-fication.”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1× · see citing works
“ViT has inspired a large amount of follow-up work in the community, and we note that there are a number of concurrent approaches on extending it to other tasks in computer vision [71, 74, 84, 85] and improving its data-efficiency [64, 48].”
BackgroundSlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669 · see citing works
Self-citationIs Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663 · see citing works
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471 · see citing works
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“Many variants of the transformer have also been proposed to reduce the computational cost of self-attention when processing longer sequences [10, 11, 37, 62, 63, 73] and to improve parameter efficiency [40, 14].”
BackgroundCCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., & Huang, T. S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3007032 · Cited by 524 · see citing works
Axial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370 · see citing works
Flagged first, then by citations
Research works printed but not linked: 68
- Anurag Arnab, Chen Sun, and Cordelia Schmid. Unified graph structured models for video understanding. In ICCV, 2021. 1
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In arXiv preprint arXiv:1607.06450, 2016. 3 · arXiv
- Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional net- works. In ICCV, 2019. 1
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, et al. Language models are few-shot learners. In NeurIPS, 2020. 2
- Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Gcnet: Non-local networks meet squeeze-excitation net- works and beyond. In CVPR Workshops, 2019. 2
- Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 1, 2
- Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A2-nets: Double attention networks. In NeurIPS, 2018. 2
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. In arXiv preprint arXiv:1904.10509, 2019. 2 · arXiv
- Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In NeurIPS, 2020. 7, 13, 14
- Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. In arXiv preprint arXiv:2006.13256, 2020. 1, 6 · arXiv
- Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. In ICLR, 2019. 2
- Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A JAX library for computer vision research and beyond. arXiv preprint arXiv:2110.11403, 2021. 6 · arXiv
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 5, 6
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 1, 2, 3, 5, 7
- Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More is less: Learning efficient video repre- sentations by big-little network and depthwise temporal ag- gregation. In NeurIPS, 2019. 8
- Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020. 1, 2, 6, 8
- Christoph Feichtenhofer, Axel Pinz, and Richard Wildes. Spatiotemporal residual networks for video action recogni- tion. In NeurIPS, 2016. 2, 5, 6
- Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zis- serman. Video action transformer network. In CVPR, 2019. 1
- Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. In NeurIPS, 2017. 4
- Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). In arXiv preprint arXiv:1606.08415, 2016. 3 · arXiv
- Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. In CVPR, 2018. 2
- Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, 2016. 7, 13, 14
- Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In ICCV, 2019. 8
- Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The ki- netics human action video dataset. In arXiv preprint arXiv:1705.06950, 2017. 1, 2, 5, 6 · arXiv
- Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019. 8
- Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Re- former: The efficient transformer. In ICLR, 2020. 2
- Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open im- ages dataset v4: Unified image classification, object detec- tion, and visual relationship detection at scale. IJCV, 2020. 5
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. 2
- Ivan Laptev. On space-time interest points. IJCV, 64(2-3), 2005.
- Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020. 8
- Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019. 8, 9
- Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 8, 9
- Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ra- makrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. PAMI, 42(2):502-508, 2019. 1, 6
- Daniel Neimark, Omri Bar, Maya Zohar, and Dotan As- selmann. Video transformer network. In arXiv preprint arXiv:2102.00719, 2021. 2, 4 · arXiv
- Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vi- jayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015. 2
- Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jian- fei Cai. Scalable visual transformers with hierarchical pool- ing. In arXiv preprint arXiv:2103.10619, 2021. 2 · arXiv
- Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Im- age transformer. In ICML, 2018. 1, 2
- Will Price and Dima Damen. An evaluation of action recognition models on epic-kitchens. In arXiv preprint arXiv:1908.00867, 2019. 9 · arXiv
- Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. In CVPR, 2019. 8
- Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019. 1, 2
- Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. In ICLR, 2020. 8
- Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli, Xuhui Jia, and Ching-Hui Chen. Global self-attention networks for im- age recognition. In arXiv preprint arXiv:2010.03019, 2021. 2 · arXiv
- Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In NeurIPS, 2014. 2, 4
- Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, 2021. 2
- Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017. 5, 6, 9
- Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi. Human action recognition using factorized spatio-temporal convolu- tional networks. In ICCV, 2015. 2
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015. 1
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, 2016. 7, 13, 14
- Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebas- tian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In arXiv preprint arXiv:2011.04006, 2020. 2 · arXiv
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. Efficient transformers: A survey. In arXiv preprint arXiv:2009.06732, 2020. 2 · arXiv
- Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. In arXiv preprint arXiv:2012.12877, 2020. 1, 2, 7 · arXiv
- Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feis- zli. Video classification with channel-separated convolu- tional networks. In ICCV, 2019. 2, 8, 9
- Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. 2
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1, 2, 3, 4, 5, 7
- Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation networks. In CVPR, 2020. 8
- Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In arXiv preprint arXiv:2012.00759, 2020. 2 · arXiv
- Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, 2016. 4, 8
- Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Pier- giovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua. Attentionnas: Spatiotemporal attention cell search for video classification. In ECCV, 2020. 8
- Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In arXiv preprint arXiv:2011.14503, 2020. 2 · arXiv
- Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2020. 4
- Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim- ing He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019. 1
- Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Fe- ichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In CVPR, 2020. 8
- Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In ECCV, 2018. 2
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimiza- tion. In ICLR, 2018. 7, 13, 14
- Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dy- namic graph message passing networks. In CVPR, 2020. 2
- Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In arXiv preprint arXiv:2012.09164, 2020. 2 · arXiv
- Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers. In arXiv preprint arXiv:2012.15840, 2020. 2 · arXiv
- Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Tor- ralba. Temporal relational reasoning in videos. In ECCV, 2018. 4, 8
Cited by
Sorted by citations
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679 · see citing works
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294 · see citing works
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · see citing works
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · see citing works
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312 · see citing works
Self-citationSatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., & Ermon, S. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0015 · Cited by 303 · see citing works
ISTVT: Interpretable Spatial-Temporal Video Transformer for Deepfake Detection
Zhao, C., Wang, C., Hu, G., Chen, H., Liu, C., & Tang, J. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3239223 · Cited by 184 · see citing works
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
“The frame features can also be used directly, by appending a linear layer on top of the features [Radford et al., 2019], or by using more complex heads [Ni et al., 2022, Arnab et al., 2021].”
MethodMovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.16449 · Cited by 176 · see citing works
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01725 · Cited by 176 · see citing works
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155 · see citing works
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01826 · Cited by 143 · see citing works
M3T: three-dimensional Medical image classifier using Multi-plane and Multi-slice Transformer
Jang, J., & Hwang, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02006 · Cited by 133 · see citing works
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · see citing works
“Furthermore, ViTs have also been applied to more complexed vision applications, such as object detection [4, 49], semantic segmentation [5], and video recognition [1].”
BackgroundTALL: Thumbnail Layout for Deepfake Video Detection
Xu, Y., Liang, J., Jia, G., Yang, Z., Zhang, Y., & He, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02071 · Cited by 109 · see citing works
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92 · see citing works
“To do so, we introduce C-ViViT , a causal variation of ViViT [1] with additional architectural changes for video generation, which can compress the videos in temporal and spatial dimensions, while staying auto-regressive in time, This capability allows for generating videos of arbitrary length…”
MethodBenchmarking and scaling of deep learning models for land cover image classification
Papoutsis, I., Bountos, N. I., Zavras, A., Michail, D., & Tryfonopoulos, C. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.11.012 · Cited by 86 · see citing works
Spatio-temporal segments attention for skeleton-based action recognition
Qiu, H., Hou, B., Ren, B., & Zhang, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.10.084 · Cited by 86 · see citing works
Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation
Gan, Y., Yang, Z., Yue, X., Sun, L., & Yang, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02069 · Cited by 72 · see citing works
Prithvi-EO-2.0: A Versatile Multitemporal Foundation Model for Earth Observation Applications
Szwarcman, D., Roy, S., Fraccaro, P., Gislason, T. E., Blumenstiel, B., Ghosal, R., de Oliveira, P. H., de Sousa Almeida, J. L., Sedona, R., Kang, Y., et al. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3642610 · Cited by 100 · see citing works
“A full list can be found in Table 3 Table 3”
Transformer-Based Spatio-Temporal Unsupervised Traffic Anomaly Detection in Aerial Videos
Tran, T. M., Bui, D. C., Nguyen, T. V., & Nguyen, K. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376399 · Cited by 63 · see citing works
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2109.10686 · Cited by 63 · see citing works
Self-citationTowards Good Practices for Missing Modality Robust Action Recognition
Woo, S., Lee, S., Park, Y., Nugroho, M. A., & Kim, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25378 · Cited by 60 · see citing works
Ramadhani, K. N., Munir, R., & Utama, N. P. (2024) · IEEE Access · doi:10.1109/access.2024.3352890 · Cited by 57 · see citing works
CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging
Hamamci, I. E., Er, S., & Menze, B. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72390-2_45 · Cited by 47 · see citing works
Deepfake Video Detection with Spatiotemporal Dropout Transformer
Zhang, D., Lin, F., Hua, Y., Wang, P., Zeng, D., & Ge, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547913 · Cited by 50 · see citing works
PSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers
Li, Z., Chen, M., Xiao, J., & Gu, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301007 · Cited by 41 · see citing works
SwinAnomaly: Real-Time Video Anomaly Detection Using Video Swin Transformer and SORT
Bajgoti, A., Gupta, R., Balaji, P., Dwivedi, R., Siwach, M., & Gupta, D. (2023) · IEEE Access · doi:10.1109/access.2023.3321801 · Cited by 38 · see citing works
Paulraj, S., & Vairavasundaram, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109496 · Cited by 27 · see citing works
Development of an AI model for DILI-level prediction using liver organoid brightfield images
Tan, S., Ding, Y., Wang, W., Rao, J., Cheng, F., Zhang, Q., Xu, T., Hu, T., Hu, Q., Ye, Z., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08205-6 · Cited by 26 · see citing works
Lai, D. K.-H., Cheng, E. S.-W., So, B. P.-H., Mao, Y.-J., Cheung, S. M.-Y., Cheung, D. S. K., Wong, D. W.-C., & Cheung, J. C.-W. (2023) · Mathematics · doi:10.3390/math11143081 · Cited by 18 · see citing works
Vision Transformers (ViTs) for Feature Extraction and Classification of AI-Generated Visual Designs
Yun, Q. (2025) · IEEE Access · doi:10.1109/access.2025.3562130 · Cited by 13 · see citing works
Continual Egocentric Activity Recognition With Foreseeable-Generalized Visual–IMU Representations
He, C., Cheng, S., Qiu, Z., Xu, L., Meng, F., Wu, Q., & Li, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3371975 · Cited by 11 · see citing works
Wang, P., Zheng, W., Chen, T., & Wang, Z. (2022) · arXiv · doi:10.48550/arxiv.2203.05962 · Cited by 80 · see citing works
“…al., 2021; Zheng et al., 2021a; Sun et al., 2020), segmentation (Chen et al., 2021a; Wang et al., 2021), image generation (Parmar et al., 2018; Jiang et al., 2021a), video processing (Zhou et al., 2018; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021). relate to.”
BackgroundInfluentialSabale, S. B., & Khedkar, V. N. (2025) · International Journal of Safety and Security Engineering · doi:10.18280/ijsse.151017 · Cited by 5 · see citing works
Chen, N., Xu, T., Sun, M., Yao, C., & Yang, D. (2025) · Intelligent Computing · doi:10.34133/icomputing.0143 · Cited by 5 · see citing works
Clinical Annotations for Automatic Stuttering Severity Assessment
Valente, A., Marew, R., Toyin, H., Al-Ali, H., Bohnen, A., Becerra, I., Soares, E., Leal, G., & Aldarmaki, H. (2025) · Interspeech 2025 · doi:10.21437/interspeech.2025-1916
Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach
Françani, A. O., & Maximo, M. R. O. A. (2023) · arXiv · doi:10.48550/arxiv.2305.06121 · Cited by 38 · see citing works
“A competitive architecture is the ViViT [13] that explored different ways of extracting spatio-temporal features similar to TimeSformer.”
- Authors
- Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C.
- Affiliations
- Google (United States) (US)
- Google Research
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 6816-6826
- OpenAlex
- W3147387781
- Cited by
- 2,767
- References
- 18
- Citations by year
- 2026: 412 · 2025: 684 · 2024: 690 · 2023: 611 · 2022: 313 · 2021: 57
- FWCI
- 136.63
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video classification, vision transformer, transfer learning, regularization, limited training data, kinetics, EPIC-KITCHENS, Something-Something V2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
