CoChat
Sign Up
PreprintCited by 3,664Human Pose and Action RecognitionHand Gesture Recognition SystemsAdvanced Neural Network Applications

A Closer Look at Spatiotemporal Convolutions for Action Recognition

Open access
Tran, Du, Wang, Heng, Torresani, Lorenzo, Ray, Jamie, LeCun, Yann, Paluri, Manohar·2017·arXiv
DOI 10.48550/arxiv.1711.11248arXiv 1711.11248
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly advantages in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which gives rise to CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101 and HMDB51.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1711.11248

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt. Sequential Deep Learning for Human Ac- tion Recognition, pages 29-39. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. 2
  2. N. Ballas, L. Yao, C. Pal, and A. Courville. Delving deeper into convolutional networks for learning video representa- tions. arXiv preprint arXiv:1511.06432, 2015. 2 · arXiv
  3. J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 1, 3, 5, 7, 8
  4. N. Dalal, B. Triggs, and C. Schmid. Human Detection Using Oriented Histograms of Flow and Appearance. In A. Leonardis, H. Bischof, and A. Pinz, editors, European Conference on Computer Vision (ECCV '06), volume 3952 of Lecture Notes in Computer Science (LNCS), pages 428- 441, Graz, Austria, May 2006. Springer-Verlag. 2
  5. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie. Behav- ior recognition via sparse spatio-temporal features. In Proc. ICCV VS-PETS, 2005. 2
  6. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Dar- rell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2625-2634, 2015. 2
  7. G. Farnebäck. Two-frame motion estimation based on poly- nomial expansion. In Image Analysis, 13th Scandinavian Conference, SCIA 2003, Halmstad, Sweden, June 29 -July 2, 2003, Proceedings, pages 363-370, 2003.
  8. C. Feichtenhofer, A. Pinz, and R. P. Wildes. Spatiotempo- ral residual networks for video action recognition. In NIPS, 2016. 2, 7, 8
  9. C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, 2016. 2, 8
  10. R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. C. Rus- sell. Actionvlad: Learning spatio-temporal aggregation for action classification. In CVPR, 2017. 2
  11. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. 5 · arXiv
  12. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 2, 3, 5, 7
  13. G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 1
  14. S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neu- ral networks for human action recognition. IEEE TPAMI, 35(1):221-231, 2013. 1, 2, 3
  15. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convo- lutional neural networks. In CVPR, 2014. 1, 2, 5, 7
  16. W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017. 1 · arXiv
  17. A. Kläser, M. Marszałek, and C. Schmid. A spatio-temporal descriptor based on 3d-gradients. In BMVC, 2008. 2
  18. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recogni- tion. In ICCV, 2011. 5, 8
  19. I. Laptev and T. Lindeberg. Space-time interest points. In ICCV, 2003. 2
  20. Q. V. Le, W. Y. Zou, S. Y. Yeung, and A. Y. Ng. Learn- ing hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR, 2011. 2
  21. P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In CVPR, 2016. 2
  22. Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, 2016. 2
  23. Z. Qiu, T. Yao, , and T. Mei. Learning spatio-temporal repre- sentation with pseudo-3d residual networks. In ICCV, 2017. 1, 2, 4, 7, 8
  24. S. Sadanand and J. Corso. Action bank: A high-level repre- sentation of activity in video. In CVPR, 2012. 2
  25. P. Scovanner, S. Ali, and M. Shah. A 3-dimensional sift de- scriptor and its application to action recognition. In ACM MM, 2007. 2
  26. Z. Shou, D. Wang, and S.-F. Chang. Temporal action local- ization in untrimmed videos via multi-stage cnns. In CVPR, 2016. 2
  27. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014. 2, 3, 7, 8
  28. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 1, 4
  29. N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsuper- vised learning of video representations using lstms. In Inter- national Conference on Machine Learning, pages 843-852, 2015. 2
  30. L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi. Human action recognition using factorized spatio-temporal convolutional networks. In ICCV, 2015. 2, 8
  31. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. 1
  32. G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolu- tional learning of spatio-temporal features. In ECCV, pages 140-153. Springer, 2010. 1, 2
  33. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional net- works. In ICCV, 2015. 1, 2, 3, 7
  34. G. Varol, I. Laptev, and C. Schmid. Long-term Temporal Convolutions for Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. 6, 7
  35. H. Wang and C. Schmid. Action recognition with improved trajectories. In ICCV, 2013. 1, 2
  36. L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool. Temporal segment networks: Towards good prac- tices for deep action recognition. In ECCV, 2016. 2, 8
  37. X. Wang, A. Farhadi, and A. Gupta. Actions ˜transforma- tions. In CVPR, 2016. 2, 8
  38. Z. Xu, Y. Yang, and A. G. Hauptmann. A discriminative cnn video representation for event detection. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1798-1807, 2015. 2
  39. J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snip- pets: Deep networks for video classification. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4694-4702, 2015. 2, 7, 8
  40. C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime tv-l 1 optical flow. Pattern Recognition, pages 214-223, 2007.

Cited by

Sorted by citations

Authors
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M.
Affiliations
  • Meta (United States) (US)
  • Dartmouth College (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
6450-6459
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.1711.11248
arXiv
1711.11248
OpenAlex
W2963155035
Cited by
3,664
References
1
Citations by year
2026: 252 · 2025: 475 · 2024: 571 · 2023: 610 · 2022: 557 · 2021: 597 · 2020: 369 · 2019: 202 · 2018: 29 · 2017: 1 · 2012: 1
FWCI
90.21
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
spatio-temporal convolution, human action recognition, 3D convolutional neural networks, residual learning, video analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review