The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences
Open accessAbstract
The interest in action and gesture recognition has grown considerably in the last years. In this paper, we present a survey on current deep learning methodologies for action and gesture recognition in image sequences. We introduce a taxonomy that summarizes important aspects of deep learning for approaching both tasks. We review the details of the proposed architectures, fusion strategies, main datasets, and competitions. We summarize and discuss the main works proposed so far with particular interest on how they treat the temporal dimension of data, discussing their main features and identify opportunities and challenges for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“However, these models typically faced some major mathematical difficulties identified by Hochreiter [39] and Bengio et al [9].”
BackgroundLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761
Phoneme recognition using time-delay neural networks
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.21701 · Cited by 2,643
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194 · cited 8×
“Advances in hybrid models combining handcrafted and new descriptors are expected [68, 112].”
BackgroundMethodInfluentialP-CNN: Pose-Based CNN Features for Action Recognition
Cheron, G., Laptev, I., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.368 · Cited by 633
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585 · cited 3×
“[130] introduces a fully end-to-end approach”
MethodA Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461
ModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290 · cited 13×
“[42] uses an LSTM to model each individual’s actions and a second-level LSTM aggregates the outputs of individual LSTMs.”
MethodInfluentialTemporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215 · cited 1×
“2014 THUMOS-14 A ECCV [44] [46] [95] [111] [88]”
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Jain, A., Tompson, J., LeCun, Y., & Bregler, C. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16808-1_21 · Cited by 142
Objects2action: Classifying and Localizing Actions without Any Video Example
Jain, M., Gemert, J. C. V., Mensink, T., & Snoek, C. G. M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.521 · Cited by 142
Progressively Parsing Interactional Objects for Fine Grained Action Detection
Ni, B., Yang, X., & Gao, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.116 · Cited by 69
Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network
Nishida, N., & Nakayama, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-29451-3_54 · Cited by 53
Research works printed but not linked: 121
- hochreiter, 1991, Untersuchungen zu dynamischen neuronalen netzen, Diplomarbeit Technischen Universität München, 91
- gers, 2002, Learning precise timing with 1stm recurrent networks, JMLR, 115
- 10.1109/CVPR.2016.213 · DOI · Paper page
- feichtenhofer, 2016, Spatiotemporal residual networks for video action recognition, NIPS, 3468
- 10.1007/978-3-319-46487-9_47 · DOI · Paper page
- 10.5244/C.29.50 · DOI
- han, 2016, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, Proc ICLR
- 10.1016/j.neucom.2016.08.032 · doi:10.1016/j.neucom.2016.08.032
- gkioxari, 2014, Finding action tubes, CoRR
- 10.1016/j.patrec.2016.01.013 · DOI
- escalante, 2015, Principal motion components for gesture recognition using a single example, PAA
- 10.1109/ICPR.2016.7899609 · DOI
- diba, 2016, Gool. Deepcamp: Deep convolutional action and attribute mid-level patterns, CVPR
- du, 2015, Hierarchical recurrent neural network for skeleton based action recognition, CVPR, 1110
- 10.1109/CVPR.2015.7298878 · DOI · Paper page
- 10.1007/978-3-319-51811-4_30 · DOI
- duan, 2016, Multi-modality fusion based on consensus-voting and 3d convolution for isolated gesture recognition, arXiv preprint arXiv 1611 06689
- sultani, 2016, Automatic action annotation in weakly labeled videos, CoRR abs/1605 08125
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1109/ICCV.2015.375 · DOI
- eleni, 2015, Gesture recognition with a convolutional long short term memory recurrent neural network, ESANN
- karaman, 2013, L1-regularized logistic regression stacking and transductive crf smoothing for action recognition in video, ICCV Workshops
- 10.1109/ICCV.2013.227 · DOI · Paper page
- 10.1016/j.imavis.2016.04.004 · DOI
- liu, 2016, Spatio-temporal with trust gates for 3d human action recognition, ECCV, 816
- 10.1109/TPAMI.2016.2537337 · DOI · Paper page
- 10.1109/CVPR.2016.215 · DOI
- 10.1109/ICCV.2015.326 · DOI · Paper page
- konecny, 2014, One-shot-learning gesture recognition using hog-hof features, JMLR, 15, 2513
- 10.1016/j.cviu.2016.10.004 · DOI
- aliakbarian, 2016, Deep action-and context-aware sequence learning for activity recognition and anticipation, arXiv preprint arXiv 1611 05520
- al-rfou, 2016, Theano: A python framework for fast computation of mathematical expressions, arXiv 1605 02688
- 10.1109/ICIP.2015.7351168 · doi:10.1109/icip.2015.7351168
- 10.1109/ICCV.2013.171 · DOI
- 10.1145/2522848.2532592 · DOI
- 10.1145/2647868.2654889 · DOI · Paper page
- baccouche, 2011, Sequential deep learning for human action recognition, HBU, 29
- 10.1109/CVPR.2015.7298599 · DOI · Paper page
- ji, 2010, 3D convolutional neural networks for human action recognition, ICML, 495
- ibrahim, 2015, A hierarchical deep temporal model for group activity recognition, arXiv preprint arXiv 1511 05271
- 10.1109/ICME.2015.7177428 · DOI · Paper page
- jain, 2014, University of amsterdam at thumos challenge 2014, ECCV THUMOS Challenge 2014 Zürich Switzerland
- 10.1007/978-3-319-46475-6_22 · DOI
- 10.1109/CVPRW.2016.21 · DOI
- 10.1145/2522848.2532589 · DOI
- 10.1109/TPAMI.2016.2537340 · DOI · Paper page
- 10.1109/CVPR.2011.5995586 · DOI · Paper page
- 10.1109/CVPRW.2016.142 · DOI
- xu, 2015, Uts-cmu at THUMOS 2015, CVPR THUMOS challenge
- ni, 2013, Integrating multi-stage depth-induced contextual information for human action recognition and localization, FG, 1
- 10.1109/CVPR.2014.299 · DOI · Paper page
- 10.1109/TITS.2014.2337331 · DOI · Paper page
- oneata, 2014, The LEAR submission at Thumos 2014
- 10.1109/CVPR.2015.7299101 · DOI · Paper page
- 10.1109/CVPR.2016.297 · DOI · Paper page
- yu, 2014, An introduction to computational networks and the computational network toolkit, MSR Technical Report
- peng, 2015, Encoding feature maps of cnns for action recognition, CVPR
- 10.1109/CVPR.2016.337 · DOI · Paper page
- peng, 2016, Multi-region two-stream r-cnn for action detection, ECCV, 744
- 10.1016/j.neucom.2016.04.007 · DOI
- zhao, 2015, Pooling the convolutional layers in deep convnets for action recognition, arXiv preprint arXiv 1511 05271
- 10.1109/CVPR.2016.219 · DOI · Paper page
- 10.1109/CVPR.2015.7298953 · DOI
- 10.1109/CVPR.2016.333 · DOI · Paper page
- zhu, 2016, Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks, reprint arXiv 1603 07772
- 10.1007/978-3-319-46454-1_27 · DOI
- mansimov, 2015, Initialization strategies of spatio-temporal convolutional neural networks, CoRR abs/1503 07274
- 10.1109/CVPRW.2015.7301342 · DOI · Paper page
- 10.1109/CVPR.2016.456 · DOI · Paper page
- montes, 2016, Nieto. Temporal activity detection in untrimmed videos with recurrent neural networks, arXiv preprint arXiv 1608 08128
- nasrollahi, 2015, Deep learning based super-resolution for improved action recognition, Ipta, 67
- 10.1109/ICCVW.2013.69 · DOI · Paper page
- neverova, 2014, Multi-scale deep learning for gesture detection and localization, ECCVW, 8925, 474
- abu-ei-haija, 2016, Youtube-8m A large-scale video classification benchmark CoRR abs/1609 08675
- abadi, 2015, TensorFlow Large-Scale Machine Learning on Heterogeneous Systems
- wang, 2015, A robust and efficient video representation for action recognition, IJCV, 1
- 10.1109/CVPR.2016.119 · DOI · Paper page
- 10.1016/B978-0-08-051584-7.50037-1 · DOI · Paper page
- 10.1007/s11263-015-0861-6 · DOI
- vondrick, 2011, Video annotation and tracking with active learning, NIPS
- shahroudy, 2016, Deep multimodal feature analysis for action recognition in rgb+ d videos, arXiv preprint arXiv 1603 02895
- 10.1109/ICCV.2015.460 · DOI · Paper page
- 10.1109/CVPR.2016.115 · DOI · Paper page
- varol, 2016, Long-term temporal convolutions for action recognition, CoRR abs/1604 04494
- 10.5244/C.30.58 · DOI · Paper page
- 10.1109/CVPR.2008.4587733 · DOI · Paper page
- 10.1109/ICSMC.2011.6083628 · DOI
- 10.1109/ICCV.2015.522 · DOI · Paper page
- wang, 2014, Action recognition and detection by combining motion and appearance features, THUMOS Action Recognition challenge, 1
- 10.1109/ICIP.2016.7532632 · doi:10.1109/icip.2016.7532632
- 10.1109/CVPR.2016.287 · DOI · Paper page
- shu, 2015, Action Detection with Improved Dense Trajectories and Sliding Window, 541
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, NIPS, 568
- 10.1109/CVPR.2016.331 · DOI · Paper page
- carngoz, 2016, Using convolutional 3d neural networks for user-independent continuous gesture recognition, ICPR W
- cao, 0, Action recognition with joints-pooled 3d deep convolutional descriptors
- 10.1109/ICPR.2016.7899603 · DOI
- chaudhry, 2013, Bioinspired dynamic 3d discriminative skeletal features for human action recognition, cvprw, 471
- 10.1109/ICSMC.2011.6084090 · DOI
- 10.1109/ICPR.2016.7899600 · DOI
- chen, 2016, Action recognition from depth sequences using weighted fusion of 2d and 3d auto-correlation of gradients features, Multimedia Tools and Applications, 1
- peng, 2014, Action recognition with stacked fisher vectors, ECCV, 581
- wang, 2017, Large-scale isolated gesture recognition using convolutional neural networks, arXiv preprint arXiv 1701 03308
- 10.1109/ICCV.2015.377 · DOI
- peng, 2013, Hybrid super vector with improved dense trajectories for action recognition, ICCV Workshops, 13
- qiu, 2015, Msr asia msm at thumos challenge 2015, CVPR Workshop, 8
- wang, 2015, Actions - transformations, CoRR abs/1512 00795
- deng, 2015, Deep structured models for group activity recognition, arXiv preprint arXiv 1506 01070
- wang, 2015, CUHK&SIAT submission for thumos15 action recognition challenge, THUMOS Action Recognition challenge, 1
- 10.1109/CVPR.2016.296 · DOI
- wang, 2015, Deep convolutional neural networks for action recognition using depth map sequences, CoRR abs/1501 04686
- peng, 2015, Action and Gesture Temporal Spotting with Super Vector Representation, 518
- wang, 2016, Gool. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition, ECCV
- 10.1109/CVPR.2016.295 · DOI
- 10.1109/PERCOMW.2011.5766860 · DOI
- wang, 2015, Exploring fisher vector and deep networks for action spotting, cvprw, 10
- 10.1109/ICCV.2015.362 · DOI · Paper page
- 10.1016/j.cviu.2014.06.014 · DOI
- 10.1109/CVPR.2016.167 · DOI · Paper page
- 10.1109/CVPR.2015.7298860 · DOI
- 10.1109/CVPR.2016.69 · DOI
Cited by
Vision-based human activity recognition: a survey
Beddiar, D. R., Nini, B., Sabokrou, M., & Hadid, A. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09004-3 · Cited by 505
BAT: Deep Learning Methods on Network Intrusion Detection Using NSL-KDD Dataset
Su, T., Sun, H., Zhu, J., Wang, S., & Li, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2972627 · Cited by 434
RGB-D-based human motion recognition with deep learning: A survey
Wang, P., Li, W., Ogunbona, P., Wan, J., & Escalera, S. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.04.007 · Cited by 402
Self-citationAutoLoc: Weakly-Supervised Temporal Action Localization in Untrimmed Videos
Shou, Z., Gao, H., Zhang, L., Miyazawa, K., & Chang, S.-F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_10 · Cited by 288
Deep learning based 2D human pose estimation: A survey
Dang, Q., Yin, J., Wang, B., & Zheng, W. (2019) · Tsinghua Science and Technology · doi:10.26599/tst.2018.9010100 · Cited by 281
Metadata
- Authors
- Asadi-Aghbolaghi, M., Clapes, A., Bellantonio, M., Escalante, H. J., Ponce-Lopez, V., Baro, X., Guyon, I., Kasaei, S., & Escalera, S.
- Venue
- 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 476-483
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W2728972335
- Cited by
- 173
- References
- 19
- FWCI
- 8.17
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.