The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM
Open accessAbstract
Gesture recognition aims to recognize meaningful movements of human bodies, and is of utmost importance in intelligent human-computer/robot interactions. In this paper, we present a multimodal gesture recognition method based on 3-D convolution and convolutional long-short-term-memory (LSTM) networks. The proposed method first learns short-term spatiotemporal features of gestures through the 3-D convolutional neural network, and then learns long-term spatiotemporal features by convolutional LSTM networks based on the extracted short-term spatiotemporal features. In addition, fine-tuning among multimodal data is evaluated, and we find that it can be considered as an optional skill to prevent overfitting when no pre-trained models exist. The proposed method is verified on the ChaLearn LAP large-scale isolated gesture data set (IsoGD) and the Sheffield Kinect gesture (SKIG) data set. The results show that our proposed method can obtain the state-of-the-art recognition accuracy (51.02% on the validation set of IsoGD and 98.89% on SKIG).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_2 · Cited by 3,958
Mitra, S., & Acharya, T. (2007) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.893280 · Cited by 1,981
The Visual Analysis of Human Movement: A Survey
Gavrila, D. M. (1999) · Computer Vision and Image Understanding · doi:10.1006/cviu.1998.0716 · Cited by 1,782
Vision based hand gesture recognition for human computer interaction: a survey
Rautaray, S. S., & Agrawal, A. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9356-9 · Cited by 1,526
DeepID3: Face Recognition with Very Deep Neural Networks
Sun, Y. J., Liang, D., Wang, X., & Tang, X. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00873 · Cited by 842
Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., & Kautz, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.456 · Cited by 695
Learning human activities and object affordances from RGB-D videos
Koppula, H. S., Gupta, R., & Saxena, A. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913478446 · Cited by 683
On the Complexity of Best Arm Identification in Multi-Armed Bandit Models
Kaufmann, E., Cappé, O., & Garivier, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.4443 · Cited by 661
VideoLSTM convolves, attends and flows for action recognition
Li, Z., Gavrilyuk, K., Gavves, E., Jain, M., & Snoek, C. G. M. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.10.011 · Cited by 466
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290
ChaLearn Looking at People RGB-D Isolated and Continuous Datasets for Gesture Recognition
Wan, J., Li, S. Z., Zhao, Y., Zhou, S., Guyon, I., & Escalera, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.100 · Cited by 276
Learning discriminative representations from RGB-D video data
Liu, L., & Shao, L. (2013) · Cited by 263
One shot learning gesture recognition from RGBD images
Wu, D., Zhu, F., & Shao, L. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239179 · Cited by 141
Priyal, S. P., & Bora, P. K. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.01.033 · Cited by 131
The ChaLearn gesture dataset (CGD 2011)
Guyon, I., Athitsos, V., Jangyodsuk, P., & Escalante, H. J. (2014) · Machine Vision and Applications · doi:10.1007/s00138-014-0596-3 · Cited by 106
Explore Efficient Local Features from RGB-D Data for One-Shot Learning Gesture Recognition
Wan, J., Guo, G., & Li, S. Z. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2513479 · Cited by 104
Large-scale Isolated Gesture Recognition using Convolutional Neural Networks
Wang, P., Li, W., Liu, S., Gao, Z., Tang, C., & Ogunbona, P. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899599 · Cited by 100
Challenges in Multi-modal Gesture Recognition
Escalera, S., Athitsos, V., & Guyon, I. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_1 · Cited by 91
Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model
Li, Y., Miao, Q., Tian, K., Fan, Y., Xu, X., Li, R., & Song, J. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899602 · Cited by 89
Self-citationChaLearn Joint Contest on Multimedia Challenges Beyond Visual Analysis: An overview
Escalante, H. J., Ponce-López, V., Wan, J., Riegler, M. A., Chen, B., Clapés, A., Escalera, S., Guyon, I., Baró, X., Halvorsen, P., et al. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899609 · Cited by 86
One-Shot-Learning Gesture Recognition Using HOG-HOF Features
Konečný, J., & Hagara, M. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_12 · Cited by 74
Large-scale Isolated Gesture Recognition using pyramidal 3D convolutional networks
Zhu, G., Zhang, L., Mei, L., Shao, J., Song, J., & Shen, P. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899601 · Cited by 71
Self-citationHuman Gesture Recognition on Product Manifolds
Lui, Y. M. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_2 · Cited by 70
Convolutional neural networks with intra-layer recurrent connections for scene labeling
Liang, M., Hu, X., & Zhang, B. (2015) · neural information processing systems · Cited by 70
Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network
Nishida, N., & Nakayama, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-29451-3_54 · Cited by 53
Depth Context: a new descriptor for human activity recognition by using sole depth sequences
Liu, M., & Liu, H. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.11.005 · Cited by 52
Wan, J., Ruan, Q.-Q., Li, W., An, G., & Zhao, R. (2014) · J. Electronic Imaging · doi:10.1117/1.jei.23.2.023017 · Cited by 49
Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture RecognitionPreprint
Duan, J., Zhou, S., Wan, J., Guo, X., & Li, S. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06689 · Cited by 40
Inertial Gesture Recognition with BLSTM-RNN
Lefebvre, G., Berlemont, S., Mamalet, F., & García, C. (2014) · Springer series in bio-/neuroinformatics · doi:10.1007/978-3-319-09903-3_19 · Cited by 33
Cirujeda, P., & Binefa, X. (2014) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2014.10 · Cited by 28
Fusing shape and spatio-temporal features for depth-based dynamic hand gesture recognition
Zheng, J., Feng, Z., Xu, C., et al. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3988-8 · Cited by 21
Elliptical density shape model for hand gesture recognition
Tung, P. T., & Ly, N. Q. (2014) · Symposium on Information and Communication Technology (SoICT) · doi:10.1145/2676585.2676600 · Cited by 17
A hierarchical structure for gesture recognition using RGB-D sensor
Choi, H., & Park, H. (2014) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/2658861.2658938 · Cited by 14
Cited by
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,591
RGB-D-based human motion recognition with deep learning: A survey
Wang, P., Li, W., Ogunbona, P., Wan, J., & Escalera, S. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.04.007 · Cited by 402
A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems
Tang, M., Liu, Y., & Durlofsky, L. J. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109456 · Cited by 394
Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor
Zhang, Z., Tian, Z., & Zhou, M. (2018) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2808688 · Cited by 333
Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
Kopuklu, O., Gunduz, A., Kose, N., & Rigoll, G. (2019) · 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) · doi:10.1109/fg.2019.8756576 · Cited by 319
Metadata
- Authors
- Zhu, G., Zhang, L., Shen, P., & Song, J.
- Venue
- IEEE Access
- Published
- 2017
- Volume
- 5
- Pages
- 4517-4524
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2595328592
- Cited by
- 270
- References
- 45
- FWCI
- 45.68
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.