The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Beyond short snippets: Deep networks for video classification
Abstract
Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Self-citationHmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 1×
“Optical flow is computed from two adjacent frames sampled at 15fps using the approach of [26].”
MethodGoing Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288 · cited 1×
“For this reason, LSTMs yield state-of-the-art results in handwriting recognition [8, 10], speech recognition [9, 7], phoneme detection [5], emotion detection [25], segmentation of meetings and events [18], and evaluating programs [27].”
Background
Research works printed but not linked: 20
- graves, 2008, Offline handwriting recognition with multidimensional recurrent neural networks, Proc NIPS, 545
- 10.1109/CVPR.2013.330 · DOI · Paper page
- 10.1109/TPAMI.2012.59 · DOI · Paper page
- 10.1109/CVPR.2008.4587756 · DOI
- 10.1109/ICASSP.2006.1660362 · DOI
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc NIPS, 568
- zeiler, 2014, Visualizing and understanding convolutional networks, Proc ECCV, 818
- boureau, 2010, A theoretical analysis of feature pooling in visual recognition, Proc ICML, 111
- zaremba, 2014, Learning to execute, CoRR abs/1410 4615
- 10.1109/72.279181 · DOI · Paper page
- gers, 2002, Learning precise timing with LSTM recurrent networks, JMLR, 3, 115
- fernandez, 2008, Phoneme recognition in TIMIT with BLSTM-CTC, CoRR abs/0804 3269
- 10.1109/TPAMI.2008.137 · DOI · Paper page
- graves, 2014, Towards end-to-end speech recognition with recurrent neural networks, Proc ICML, 1764
- baccouche, 2011, Sequential Deep Learning for Human Action Recognition, Human Behavior Unterstanding, 29, 10.1007/978-3-642-25446-8_4 · DOI · Paper page
- 10.1109/ICASSP.2013.6638947 · DOI · Paper page
- baccouche, 2010, Action classification in soccer videos with long short-term memory recurrent neural networks, Proc ICANN, 154
- soomro, 2012, UCF101: A dataset of 101 human actions classes from videos in the wild, CRCV-TR-12–01
- 10.1109/CVPR.2011.5995407 · DOI · Paper page
- 10.5244/C.23.124 · DOI · Paper page
Cited by
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
“A natural solution to video classification is to combine the success of CNNs for images and RNNs for sequences [55, 11].”
MethodQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761
“Results of a comparative analysis on the Sports-1Mand UCF-101 datasets showed that LSTM cells were necessary to take full advantage of the motion information contained in the video and yielded the highest reported performance [30].”
BackgroundMethodViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
“Large-scale Recognition: There has been substantial work on spatiotemporal models for large scale video classification and detection [31, 11, 12, 27, 33, 23, 19].”
BackgroundLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994
Temporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Flow-Guided Feature Aggregation for Video Object Detection
Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.52 · Cited by 729
“On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
BackgroundMethodPredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P. S., & Long, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3165153 · Cited by 677
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Chen, W., & McDuff, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_22 · Cited by 716
P-CNN: Pose-Based CNN Features for Action Recognition
Cheron, G., Laptev, I., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.368 · Cited by 633
Liu, M., Li, F., Yan, H., Wang, K., Ma, Y., Shen, L., & Xu, M. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116459 · Cited by 568
Video-based emotion recognition using CNN-RNN and C3D hybrid networks
Fan, Y., Lu, X., Li, D., & Liu, Y. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997632 · Cited by 556
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
A Comprehensive Survey of Vision-Based Human Action Recognition Methods
Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., Du, J.-X., & Chen, D.-S. (2019) · Sensors · doi:10.3390/s19051005 · Cited by 503
“[91] proposed a recurrent neural network to recognize human action that connects the LSTM cell with the output of the underlying CNN.”
BackgroundMethodInfluentialA Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461
A review of Convolutional-Neural-Network-based action recognition
Yao, G., Lei, T., & Zhong, J. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.05.018 · Cited by 437
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312
Factors of Transferability for a Generic ConvNet Representation
Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2500224 · Cited by 301
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290
Two-Stream 3-D convNet Fusion for Action Recognition in Videos With Arbitrary Size and Length
Wang, X., Gao, L., Wang, P., Sun, X., & Liu, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2749159 · Cited by 263
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257
BEVT: BERT Pretraining of Video Transformers
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., & Yuan, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01432 · Cited by 234
Temporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195
“used by [6, 35, 27, 33].”
Learning laparoscopic video shot classification for gynecological surgery
Petscharnig, S., & Schöffmann, K. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4699-5 · Cited by 182
Wang, X., Girshick, R., Gupta, A., & He, K. (2017) · arXiv · Cited by 11,689
Unsupervised Deep Video Hashing via Balanced Code for Large-Scale Video Retrieval
Wu, G., Han, J., Guo, Y., Liu, L., Ding, G., Ni, Q., & Shao, L. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2882155 · Cited by 148
ASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170
“Other approaches model the temporal action sequence with Conditional Random Fields [30, 34, 39], Markov model [20, 38] and RNN [9, 29, 35, 42, 49] to classify framewise actions.”
MethodDeep Fisher discriminant learning for mobile hand gesture recognition
Li, C., Xie, C., Zhang, B., Chen, C., & Han, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.12.023 · Cited by 72
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
Liu, J., Li, Y., Song, S., Xing, J., Lan, C., & Zeng, W. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2799968 · Cited by 62
Gao, Z., Guo, L., Guan, W., Liu, A.-A., Ren, T., & Chen, S. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3038372 · Cited by 63
Stacked Memory Network for Video Summarization
Wang, J., Wang, W., Wang, Z., Wang, L., Feng, D., & Tan, T. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350992 · Cited by 60
Taming unbalanced training workloads in deep learning with partial collective operations
Li, S., Ben-Nun, T., Girolamo, S. D., Alistarh, D., & Hoefler, T. (2020) · Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3332466.3374528 · Cited by 57
Cheng, D., & Liu, M. (2017) · 2017 IEEE International Conference on Imaging Systems and Techniques (IST) · doi:10.1109/ist.2017.8261461 · Cited by 49
Yurtsever, E., Liu, Y., Lambert, J., Miyajima, C., Takeuchi, E., Takeda, K., & Hansen, J. H. L. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917362 · Cited by 40
Hu, Y., Zhang, X.-Q., Xu, L., Xian He, F., Tian, Z., She, W., & Liu, W. (2020) · IEEE Access · doi:10.1109/access.2020.3003162 · Cited by 36
DTR-HAR: deep temporal residual representation for human activity recognition
Basly, H., Ouarda, W., Sayadi, F. E., Ouni, B., & Alimi, A. M. (2022) · The Visual Computer · doi:10.1007/s00371-021-02064-y · Cited by 36
A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., & Wang, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274783 · Cited by 38
A review of AI Technologies for Wearable Devices
Jin, C. Y. (2019) · IOP Conference Series: Materials Science and Engineering · doi:10.1088/1757-899x/688/4/044072 · Cited by 32
Spatio-Temporal FAST 3D Convolutions for Human Action Recognition
Stergiou, A., & Poppe, R. (2019) · 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) · doi:10.1109/icmla.2019.00036 · Cited by 21
Memory-Augmented Temporal Dynamic Learning for Action Recognition
Yuan, Y., Wang, D., & Wang, Q. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019167 · Cited by 17
Deep Learning for Video Classification: A Review
Rehman, A., & Belhaouari, S. B. (2021) · doi:10.36227/techrxiv.15172920 · Cited by 17
Motion-driven adaptive frame selection strategy for video action recognition
Ding, H., Guo, C., Sun, J., Jiang, X., Shi, H., & Li, J. (2025) · EURASIP Journal on Image and Video Processing · doi:10.1186/s13640-025-00675-2 · Cited by 9
Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations
Fan, L., Zhou, J., Xing, X., & Wu, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01551 · Cited by 6
“In contrast to passive visual recognition [10, 11, 20, 27, 45], where the input is obtained through human-operated or stationary devices, an active recognition agent is endowed with the ability to actively seek different observations, as illustrated in Figure 1a.”
BackgroundMultimodal social media video classification with deep neural networks
Trzcinski, T. (2018) · Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2018 · doi:10.1117/12.2501679 · Cited by 6
Deploying human activity recognition in embedded RISC-V processors
Nunes, W. A., Reusch, R. S., Luza, L., Bernardon, E., Dal Zotto, A. E., Juracy, L. R., & Moraes, F. G. (2024) · Design Automation for Embedded Systems · doi:10.1007/s10617-024-09288-w
Metadata
- Authors
- Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G.
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 4694-4702
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1923404803
- Cited by
- 2,062
- References
- 8
- FWCI
- 78.56
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.