The DOI matches its DataCite record; 54 of the 58 references it cites were found in a registry or an index.
Is Space-Time Attention All You Need for Video Understanding?
Open accessAbstract
We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vision Transformer (ViT) [17], and video-specific variants such as TimeSformer [ 18, 19 ] and Video Swin Transformer [20] capture longrange dependencies and global spatiotemporal context that CNNs often miss.”
From Cross-dataset video deepfake detection using Transformer and CNN architectures (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Similar to previous trials [3, 6], training video transformers from scratch yields unsatisfied results on video datasets.”
From VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“For this reason, due to real-time computation concerns in visual odometry applications, we selected the encoder architecture with the 'divided space-time' self-attention, which is more efficient in terms of processing complexity compared to the 'joint space-time' self-attention.”
From Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach (2023) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“We chose the TimeSformer [14] model for this work since it achieved state-of-the-art results in video action recognition tasks compared to previous works based on convolutional neural networks (CNN).”
From Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach (2023) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
(2008) · Journal of Machine Learning Research · Cited by 33,051
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Research works printed but not linked: 5
- Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, 2016.
- Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Par- mar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Chen, Z., Wu, Y., and Hughes, M. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics. Association for Computational Linguistics, 2018a.
- Chen, Y., Kalantidis, Y., Li, J., Yan, S., and Feng, J. Aˆ2- nets: Double attention networks. In Advances in Neural Information Processing Systems 31, 2018b.
- International Conference on Computer Vision (ICCV), October 2019.
- Cordonnier, J., Loukas, A., and Jaggi, M. On the relation- ship between self-attention and convolutional layers. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Cited by
Sorted by citations
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R. A., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,369
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407
“Concurrent vision-transformer based methods [78,6,1] require over 5× more computation and large-scale external pre-training on ImageNet-21K (IN-21K), to achieve equivalent MViT accuracy.”
BackgroundMethodInfluentialMViTv2: Improved Multiscale Vision Transformers for Classification and Detection
Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., & Feichtenhofer, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00476 · Cited by 909
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 719
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition
Li, K., Wang, Y., Zhang, J., Gao, P., Song, G., Liu, Y., Li, H., & Qiao, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3282631 · Cited by 682
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678
Masked Feature Prediction for Self-Supervised Visual Pre-Training
Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01426 · Cited by 587
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01398 · Cited by 536
GroupViT: Semantic Segmentation Emerges from Text Supervision
Xu, J., Mello, S. D., Liu, S., Byeon, W., Breuel, T. M., Kautz, J., & Wang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01760 · Cited by 518
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909
SimVP: Simpler yet Better Video Prediction
Gao, Z., Tan, C., Wu, L., & Li, S. Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00317 · Cited by 466
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and TextPreprint
Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S., Cui, Y., & Gong, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11178 · Cited by 469
MPViT: Multi-Path Vision Transformer for Dense Prediction
Lee, Y., Kim, J., Willette, J., & Hwang, S. J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00714 · Cited by 401
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale UpPreprint
Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07074 · Cited by 419
Escaping the Big Data Paradigm with Compact TransformersPreprint
Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., & Shi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05704 · Cited by 619
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01742 · Cited by 333
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Yu, Z., Shen, Y., Shi, J., Zhao, H., Torr, P. H. S., & Zhao, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00415 · Cited by 341
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312
End-to-End Temporal Action Detection With Transformer
Liu, X., Wang, Q., Hu, Y., Tang, X., Zhang, S., Bai, S., & Bai, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3195321 · Cited by 316
Attention Spiking Neural Networks
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241201 · Cited by 298
PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation
Zhao, Q., Zheng, C., Liu, M., Wang, P., & Chen, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00857 · Cited by 289
MoViNets: Mobile Video Networks for Efficient Video Recognition
Kondratyuk, D., Yuan, L., Li, Y., Zhang, L., Tan, M., Brown, M., & Gong, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01576 · Cited by 286
MeMOT: Multi-Object Tracking with Memory
Cai, J., Xu, M., Li, W., Xiong, Y., Xia, W., Tu, Z., & Soatto, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00792 · Cited by 265
XCiT: Cross-Covariance Image TransformersPreprint
El-Nouby, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J. J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09681 · Cited by 316
ObjectFormer for Image Manipulation Detection and Localization
Wang, J., Wu, Z., Chen, J., Han, X., Shrivastava, A., Lim, S.-N., & Jiang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00240 · Cited by 275
Transformer-Based Visual Segmentation: A Survey
Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3434373 · Cited by 305
Chen, X., Zhang, H., Zhao, F., Hu, Y., Tan, C., & Yang, J. J. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3170551 · Cited by 252
Recurrent Vision Transformers for Object Detection with Event Cameras
Gehrig, M., & Scaramuzza, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01334 · Cited by 247
STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition
Ahn, D., Kim, S., Hong, H., & Ko, B. C. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00333 · Cited by 215
ActionCLIP: A New Paradigm for Video Action RecognitionPreprint
Wang, M., Xing, J., & Liu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08472 · Cited by 215
Anticipative Video Transformer
Girdhar, R., & Grauman, K. L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01325 · Cited by 237
Nwoye, C. I., Yu, T., Gonzalez, C. A., Seeliger, B., Mascagni, P., Mutter, D., Marescaux, J., & Padoy, N. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102433 · Cited by 216
BEVT: BERT Pretraining of Video Transformers
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., & Yuan, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01432 · Cited by 234
Omnivore: A Single Model for Many Visual Modalities
Girdhar, R., Singh, M., Ravi, N., van der Maaten, L. J. P., Joulin, A., & Misra, I. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01563 · Cited by 235
MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection
Chen, Y., Liu, Z., Zhang, B., Fok, W., Qi, X., & Wu, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25112 · Cited by 230
AFTer-UNet: Axial Fusion Transformer UNet for Medical Image Segmentation
Yan, X., Tang, H., Sun, S., Ma, H., Kong, D., & Xie, X. S. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00333 · Cited by 208
UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection
Acsintoae, A., Florescu, A., Georgescu, M.-I., Mare, T., Sumedrea, P., Ionescu, R. T., Khan, F. S., & Shah, M. A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01951 · Cited by 194
Fine-tuned CLIP Models are Efficient Video Learners
Rasheed, H., Khattak, M. U., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00633 · Cited by 221
Benchmarking Micro-Action Recognition: Dataset, Methods, and Applications
Guo, D., Li, K., Hu, B., Zhang, Y., & Wang, M. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3358415 · Cited by 187
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
Li, D., Li, J., Li, H., Niebles, J. C., & Hoi, S. C. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00490 · Cited by 187
Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 192
Keeping Your Eye on the Ball: Trajectory Attention in Video TransformersPreprint
Patrick, M., Campbell, D., Asano, Y. M., Misra, I., Metze, F., Feichtenhofer, C., Vedaldi, A., & Henriques, J. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05392 · Cited by 187
MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition
Wu, C.-Y., Li, Y., Mangalam, K., Fan, H., Xiong, B., Malik, J., & Feichtenhofer, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01322 · Cited by 201
GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer
Li, S. C., Cao, Q., Liu, L., Yang, K., Liu, S., Hou, J. Y., & Yi, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01341 · Cited by 169
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01589 · Cited by 179
Learning Video Representations from Large Language Models
Zhao, Y., Misra, I., Krähenbühl, P., & Girdhar, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00637 · Cited by 190
FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
Kalluri, T., Pathak, D., Chandraker, M. K., & Tran, D. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00211 · Cited by 183
All in One: Exploring Unified Video-Language Pre-Training
Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K. Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., Shan, Y., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00638 · Cited by 199
Ghalkhani, M., & Habibi, S. R. (2022) · Energies · doi:10.3390/en16010185 · Cited by 171
Vision Mamba: A Comprehensive Survey and Taxonomy
Liu, X., Zhang, C., Huang, F., Xia, S., Wang, G., & Zhang, L. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3610435 · Cited by 188
TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer
Zhou, M., Xu, W., Kang, J., & Rosing, T. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00082 · Cited by 151
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01826 · Cited by 143
A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications
Wu, F. L., Wang, Q., Bian, J., Ding, N., Lu, F., Cheng, J. S., Dou, D., & Xiong, H. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3232034 · Cited by 167
MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
Chen, J., & Ho, C. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00086 · Cited by 144
Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00498 · Cited by 127
Video Frame Interpolation Transformer
Shi, Z., Xu, X., Liu, X., Chen, J., & Yang, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01696 · Cited by 135
Self-supervised Video Object Segmentation by Motion Grouping
Yang, C., Lamdouar, H., Lu, E., Zisserman, A., & Xie, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00709 · Cited by 153
Recurring the Transformer for Video Action Recognition
Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., & Yu, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01367 · Cited by 137
Wu, W., Wang, X., Luo, H., Wang, J., Yang, Y., & Ouyang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00640 · Cited by 136
Transformer-Based Imitative Reinforcement Learning for Multirobot Path Planning
Chen, L., Wang, Y., Miao, Z., Mo, Y., Feng, M., Zhou, Z., & Wang, H. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3240585 · Cited by 128
Self-citationWhen Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128
CycleMLP: A MLP-like Architecture for Dense PredictionPreprint
Chen, S., Xie, E., Ge, C., Chen, R., Liang, D., & Luo, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10224 · Cited by 128
HRFormer: High-Resolution Transformer for Dense Prediction
Yuan, Y., Fu, R., Huang, L., Lin, W., Zhang, C., Chen, X., & Wang, J. (2021) · arXiv · doi:10.48550/arxiv.2110.09408 · Cited by 178
Energy consumption prediction strategy for electric vehicle based on LSTM-transformer framework
Feng, Z., Zhang, J., Jiang, H., Yao, X., Yu, Q., & Zhang, H. (2024) · Energy · doi:10.1016/j.energy.2024.131780 · Cited by 127
SVFormer: Semi-supervised Video Transformer for Action Recognition
Xing, Z., Dai, Q., Han, H., Chen, J., Wu, Z., & Jiang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01804 · Cited by 130
CLIP2Video: Mastering Video-Text Retrieval via Image CLIPPreprint
Fang, H., Xiong, P., Xu, L., & Chen, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11097 · Cited by 289
CycleMLP: A MLP-Like Architecture for Dense Visual Predictions
Chen, S., Xie, E., Ge, C., Chen, R., Liang, D., & Luo, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3303397 · Cited by 311
Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting
Wasim, S. T., Naseer, M., Khan, S., Khan, F. S., & Shah, M. A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02206 · Cited by 140
Video Probabilistic Diffusion Models in Projected Latent Space
Yu, S., Sohn, K., Kim, S., & Shin, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01770 · Cited by 144
Selective Structured State-Spaces for Long-Form Video Understanding
Wang, J., Zhu, W., Wang, P., Yu, X., Liu, L. L., Omar, M., & Hamid, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00618 · Cited by 131
RegionViT: Regional-to-Local Attention for Vision TransformersPreprint
Chen, C.-F., Panda, R., & Fan, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02689 · Cited by 186
Self-supervised Video Transformer
Ranasinghe, K., Naseer, M., Khan, S. H., Khan, F. S., & Ryoo, M. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00289 · Cited by 128
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K. L., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V. B., Bansal, S., Boote, B., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01834 · Cited by 267
Self-citationVideo Understanding With Large Language Models: A Survey
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3566695 · Cited by 288
ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization
He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., & Shrivastava, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01355 · Cited by 120
Sub-word Level Lip Reading With Visual Attention
R, P. K., Afouras, T., & Zisserman, A. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00510 · Cited by 137
SSMTL++: Revisiting self-supervised multi-task learning for video anomaly detection
Bărbălau, A., Ionescu, R. T., Georgescu, M.-I., Dueholm, J. V., Ramachandra, B., Nasrollahi, K., Khan, F. S., Moeslund, T. B., & Shah, M. A. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103656 · Cited by 114
ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition
Wang, M., Xing, J., Mei, J., Liu, Y., & Jiang, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3331841 · Cited by 592
Zou, B., Han, J., Wang, Y., Liu, R., Zhao, S., Feng, L., Lyu, X., & Ma, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3181210 · Cited by 125
Unsupervised video anomaly detection via normalizing flows with implicit latent features
Cho, M., Kim, T., Kim, W. J., Cho, S., & Lee, S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108703 · Cited by 111
MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection
Dai, R., Das, S., Kahatapitiya, K., Ryoo, M. S., & Brémond, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01941 · Cited by 113
A Comprehensive Survey of Transformers for Computer Vision
Jamil, S., Piran, M. J., & Kwon, O. (2023) · Drones · doi:10.3390/drones7050287 · Cited by 128
Revisiting Classifier: Transferring Vision-Language Models for Video Recognition
Wu, W., Sun, Z., & Ouyang, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25386 · Cited by 120
Vision Transformers are Parameter-Efficient Audio-Visual Learners
Lin, Y.-B., Sung, Y.-L., Lei, J., Bansal, M., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00228 · Cited by 120
Self-citationCLIP4Clip: An Empirical Study of CLIP for End to End Video Clip RetrievalPreprint
Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., & Li, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08860 · Cited by 108
HiEve: A Large-Scale Benchmark for Human-Centric Video Analysis in Complex Events
Lin, W., Liu, H., Liu, S., Li, Y., Xiong, H., Qi, G.-J., & Sebe, N. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01842-6 · Cited by 141
Masked Vision Transformers for Hyperspectral Image Classification
Scheibenreif, L., Mommert, M., & Borth, D. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00210 · Cited by 121
VideoTrack: Learning to Track Objects via Video Transformer
Xie, F., Chu, L., Li, J., Lu, Y., & Ma, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02186 · Cited by 110
ViTGAN: Training GANs with Vision TransformersPreprint
Lee, K., Chang, H., Lu, J., Zhang, H., Tu, Z., & Liu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04589 · Cited by 163
TranSkeleton: Hierarchical Spatial–Temporal Transformer for Skeleton-Based Action Recognition
Liu, H., Liu, Y., Chen, Y.-X., Yuan, C.-F., Li, B., & Hu, W.-M. (2023) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2023.3240472 · Cited by 61
Zhai, Z., Zhao, J., Long, C., Xu, W., He, S., & Zhao, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02115 · Cited by 110
Chen, J., Hu, Y., Lai, Q., Wang, W. W., Chen, J., Liu, H., Srivastava, G., Bashir, A. K., & Hu, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102017 · Cited by 104
TrafficFormer: An Efficient Pre-trained Model for Traffic Data
Zhou, G., Guo, X., Liu, Z., Li, T., Li, Q., & Xu, K. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00102 · Cited by 104
Long Short-Term Transformer for Online Action DetectionPreprint
Xu, M., Xiong, Y., Chen, H., Li, X., Xia, W., Tu, Z., & Soatto, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03377 · Cited by 100
ASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170
Zhang, J., Yang, K., Constantinescu, A., Peng, K., Müller, K., & Stiefelhagen, R. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3161141 · Cited by 98
MIST : Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
Gao, D., Zhou, L., Ji, L., Zhu, L., Yang, Y., & Shou, M. Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01419 · Cited by 108
Metadata
- Authors
- Bertasius, G., Wang, H., & Torresani, L.
- Affiliations
- University of Pennsylvania (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 813-824
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05095
- OpenAlex
- W3126721948
- Cited by
- 1,663
- References
- 58
- Citations by year
- 2026: 161 · 2025: 390 · 2024: 418 · 2023: 387 · 2022: 199 · 2021: 108
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video classification, self-attention, spatio-temporal attention, TimeSformer, transformer models, divided attention, human action recognition, 3D convolutional networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
