The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Convolutional Two-Stream Network Fusion for Video Action Recognition
Abstract
Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters, (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy, finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194
Cited by
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Self-citationMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007
Self-citationMultiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
BackgroundSelf-citationA Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801
Self-citationLearning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
Hara, K., Kataoka, H., & Satoh, Y. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.373 · Cited by 726
“Many methods based on the two-stream CNNs are proposed to improve action recognition performance [5, 6, 20, 22].”
BackgroundMethodInfluentialBMN: Boundary-Matching Network for Temporal Action Proposal Generation
Lin, T., Liu, X., Li, X., Ding, E., & Wen, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00399 · Cited by 694
Detect to Track and Track to Detect
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.330 · Cited by 595
Self-citationWhat Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
A Comprehensive Survey of Vision-Based Human Action Recognition Methods
Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., Du, J.-X., & Chen, D.-S. (2019) · Sensors · doi:10.3390/s19051005 · Cited by 503
“[84] changed the process of spatial and temporal information fusion from the original final classification layer to the middle of the network, further improving the action recognition accuracy.”
BackgroundA review of Convolutional-Neural-Network-based action recognition
Yao, G., Lei, T., & Zhong, J. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.05.018 · Cited by 437
Self-supervised Co-training for Video Representation Learning
Han, T., Xie, W., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.09709 · Cited by 292
Self-citationTwo-Stream 3-D convNet Fusion for Action Recognition in Videos With Arbitrary Size and Length
Wang, X., Gao, L., Wang, P., Sun, X., & Liu, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2749159 · Cited by 263
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation
Pan, B., Cai, H., Huang, D.-A., Lee, K.-H., Gaidon, A., Adeli, E., & Niebles, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01088 · Cited by 281
Spatiotemporal Co-Attention Recurrent Neural Networks for Human-Skeleton Motion Prediction
Shu, X., Zhang, L., Qi, G.-J., Liu, W., & Tang, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3050918 · Cited by 276
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216
Joint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217
What Makes Multi-modal Learning Better than Single (Provably)
Huang, Y., Du, C., Xue, Z., Chen, X., Zhao, H., & Huang, L. (2021) · arXiv · doi:10.48550/arxiv.2106.04538 · Cited by 197
Convolutional Neural Networks for Human Activity Recognition Using Body-Worn Sensors
Moya Rueda, F., Grzeszick, R., Fink, G. A., Feldhorst, S., & Ten Hompel, M. (2018) · Informatics · doi:10.3390/informatics5020026 · Cited by 196
Furnari, A., & Farinella, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00635 · Cited by 182
“Action Recognition Our work is related to previous research on action recognition from third person vision [7, 13, 14, 30, 34, 35, 49, 57, 59, 60, 61, 67] and first person vision [11, 12, 36, 37, 38, 44, 48, 52, 54, 55, 56].”
BackgroundAlleviating Over-segmentation Errors by Detecting Action Boundaries
Ishikawa, Y., Kasai, S., Aoki, Y., & Kataoka, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00237 · Cited by 171
Recurring the Transformer for Video Action Recognition
Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., & Yu, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01367 · Cited by 137
Weakly supervised learning of actions from transcripts
Kuehne, H., Richard, A., & Gall, J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.004 · Cited by 119
Cross and learn: Cross-modal self-supervision
Sayed, N., Brattoli, B., & Ommer, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-12939-2_17 · Cited by 76
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
Liu, J., Li, Y., Song, S., Xing, J., Lan, C., & Zeng, W. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2799968 · Cited by 62
Stacked Memory Network for Video Summarization
Wang, J., Wang, W., Wang, Z., Wang, L., Feng, D., & Tan, T. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350992 · Cited by 60
Bilal, M., Maqsood, M., Yasmin, S., Hasan, N. U., & Rho, S. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-021-03957-4 · Cited by 51
An Improved Inter-Intra Contrastive Learning Framework on Self-Supervised Video Representation
Tao, L., Wang, X., & Yamasaki, T. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3141051 · Cited by 50
Ezzat, M. A., Abd El Ghany, M. A., Almotairi, S., & Salem, M. A.-M. (2021) · Sensors · doi:10.3390/s21093222 · Cited by 39
Huang, K., Jiao, Z., Cai, Y., & Zhong, Z. (2022) · Journal of Nursing Management · doi:10.1111/jonm.13787 · Cited by 38
N-HAR: A Neuromorphic Event-Based Human Activity Recognition System using Memory Surfaces
Pradhan, B. R., Bethi, Y., Narayanan, S., Chakraborty, A., & Thakur, C. S. (2019) · 2019 IEEE International Symposium on Circuits and Systems (ISCAS) · doi:10.1109/iscas.2019.8702581 · Cited by 31
Hear Me out: Fusional Approaches for Audio Augmented Temporal Action Localization
Bagchi, A., Mahmood, J., Fernandes, D., & Sarvadevabhatla, R. (2022) · Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications · doi:10.5220/0010832700003124 · Cited by 29
(2023) · doi:10.1109/cvpr52729.2023.00627 · Cited by 28
Paulraj, S., & Vairavasundaram, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109496 · Cited by 27
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17066 · Cited by 24
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450460 · Cited by 24
Memory-Augmented Temporal Dynamic Learning for Action Recognition
Yuan, Y., Wang, D., & Wang, Q. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019167 · Cited by 17
Deep Learning for Video Classification: A Review
Rehman, A., & Belhaouari, S. B. (2021) · doi:10.36227/techrxiv.15172920 · Cited by 17
Jiao, J., Sun, H., Huang, Y., Xia, M., Qiao, M., Ren, Y., Wang, Y., & Guo, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278259 · Cited by 15
Metadata
- Authors
- Feichtenhofer, C., Pinz, A., & Zisserman, A.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 1933-1941
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2342662179
- Cited by
- 2,684
- References
- 9
- FWCI
- 100.55
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.