CoChat
Sign Up
PreprintCited by 1,663Human Pose and Action RecognitionAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

Is Space-Time Attention All You Need for Video Understanding?

Open access
Bertasius, Gedas, Wang, Heng, Torresani, Lorenzo·2021·arXiv
DOI 10.48550/arxiv.2102.05095arXiv 2102.05095
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.05095

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Vision Transformer (ViT) and video-specific variants such as TimeSformer and Video Swin Transformer capture long-range dependencies and global spatiotemporal context that CNNs often miss.

    “Vision Transformer (ViT) [17], and video-specific variants such as TimeSformer [ 18, 19 ] and Video Swin Transformer [20] capture longrange dependencies and global spatiotemporal context that CNNs often miss.”

    From Cross-dataset video deepfake detection using Transformer and CNN architectures (2026) · cites this paper as [18]

    The TimeSformer results (Test, page not provided) report that the model exploits long-range temporal dependencies, and its space-time attention visualizations show attention to relevant video regions. The cited paper does not establish that CNNs often miss global spatiotemporal context; it also does not evaluate Video Swin Transformer. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Training video transformers from scratch yields unsatisfied results on video datasets.

    “Similar to previous trials [3, 6], training video transformers from scratch yields unsatisfied results on video datasets.”

    From VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022) · cites this paper as [6]

  • The divided space-time self-attention architecture is more efficient in terms of processing complexity compared to the joint space-time self-attention.

    “For this reason, due to real-time computation concerns in visual odometry applications, we selected the encoder architecture with the 'divided space-time' self-attention, which is more efficient in terms of processing complexity compared to the 'joint space-time' self-attention.”

    From Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach (2023) · cites this paper as [14]

  • The TimeSformer model achieved state-of-the-art results in video action recognition tasks compared to previous CNN-based works.

    “We chose the TimeSformer [14] model for this work since it achieved state-of-the-art results in video action recognition tasks compared to previous works based on convolutional neural networks (CNN).”

    From Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach (2023) · cites this paper as [14]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

5 of the 58 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, 2016.
  2. Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Par- mar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Chen, Z., Wu, Y., and Hughes, M. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics. Association for Computational Linguistics, 2018a.
  3. Chen, Y., Kalantidis, Y., Li, J., Yan, S., and Feng, J. Aˆ2- nets: Double attention networks. In Advances in Neural Information Processing Systems 31, 2018b.
  4. International Conference on Computer Vision (ICCV), October 2019.
  5. Cordonnier, J., Loukas, A., and Jaggi, M. On the relation- ship between self-attention and convolutional layers. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
  6. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv

Cited by

Sorted by citations

Related

Authors
Bertasius, G., Wang, H., & Torresani, L.
Affiliations
  • University of Pennsylvania (US)
  • Meta (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
813-824
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.05095
arXiv
2102.05095
OpenAlex
W3126721948
Cited by
1,663
References
58
Citations by year
2026: 161 · 2025: 390 · 2024: 418 · 2023: 387 · 2022: 199 · 2021: 108
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
video classification, self-attention, spatio-temporal attention, TimeSformer, transformer models, divided attention, human action recognition, 3D convolutional networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review