The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition
Open accessAbstract
This paper describes a novel method called Deep Dynamic Neural Networks (DDNN) for multimodal gesture recognition. A semi-supervised hierarchical dynamic framework based on a Hidden Markov Model (HMM) is proposed for simultaneous gesture segmentation and recognition where skeleton joint information, depth and RGB images, are the multimodal input observations. Unlike most traditional approaches that rely on the construction of complex handcrafted features, our approach learns high-level spatio-temporal representations using deep neural networks suited to the input modality: a Gaussian-Bernouilli Deep Belief Network (DBN) to handle skeletal dynamics, and a 3D Convolutional Neural Network (3DCNN) to manage and fuse batches of depth and RGB images. This is achieved through the modeling and learning of the emission probabilities of the HMM required to infer the gesture sequence. This purely data driven approach achieves a Jaccard index score of 0.81 in the ChaLearn LAP gesture spotting challenge. The performance is on par with a variety of state-of-the-art hand-tuned feature-based approaches and other learning-based methods, therefore opening the door to the use of deep learning techniques in order to further explore multimodal time series data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Pattern Recognition and Machine Learning
(2007) · Kybernetes · doi:10.1108/03684920710743466 · Cited by 8,265
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
(2013) · Cited by 3,740
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
(2011) · Cited by 2,752
Laptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
Behavior Recognition via Sparse Spatio-Temporal Features
Dollar, P., Rabaud, V., Cottrell, G., & Belongie, S. (2006) · 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance · doi:10.1109/vspets.2005.1570899 · Cited by 2,460
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153
A Novel Connectionist System for Unconstrained Handwriting Recognition
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., & Schmidhuber, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.137 · Cited by 1,983
A Spatio-Temporal Descriptor Based on 3D-Gradients
Klaeser, A., Marszalek, M., & Schmid, C. (2008) · Procedings of the British Machine Vision Conference 2008 · doi:10.5244/c.22.99 · Cited by 1,846
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669
A 3-dimensional sift descriptor and its application to action recognition
Scovanner, P., Ali, S., & Shah, M. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291311 · Cited by 1,596
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
Mining actionlet ensemble for action recognition with depth cameras
Wang, J., Liu, Z., Wu, Y., & Yuan, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247813 · Cited by 1,512
Enhanced Computer Vision With Microsoft Kinect Sensor: A Review
Han, J., Shao, L., Xu, D., & Shotton, J. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2265378 · Cited by 1,367
Self-citationEvaluation of local spatio-temporal features for action recognition
Wang, H., Ullah, M. M., Klaser, A., Laptev, I., & Schmid, C. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.124 · Cited by 1,296
Marszalek, M., Laptev, I., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206557 · Cited by 1,168
Discriminative Unsupervised Feature Learning with Convolutional Neural NetworksPreprint
Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., & Brox, T. (2014) · arXiv (Cornell University) · Cited by 1,142
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Theano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005
Le, Q. V., Zou, W. Y., Yeung, S. Y., & Ng, A. Y. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995496 · Cited by 960
An Efficient Dense and Scale-Invariant Spatio-Temporal Interest Point Detector
Willems, G., Tuytelaars, T., & Van Gool, L. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88688-4_48 · Cited by 869
Convolutional Learning of Spatio-temporal Features
Taylor, G. W., Fergus, R., LeCun, Y., & Bregler, C. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15567-3_11 · Cited by 629
Connectionist Speech Recognition
Bourlard, H. A., & Morgan, N. (1994) · Springer eBooks · doi:10.1007/978-1-4615-3210-1 · Cited by 585
Convolutional-Recursive Deep Learning for 3D Object Classification
Socher, R., Huval, B., Bath, B., Manning, C. D., & Ng, A. Y. (2012) · neural information processing systems · Cited by 509
Sign Language Recognition Using Convolutional Neural Networks
Pigou, L., Dieleman, S., Kindermans, P.-J., & Schrauwen, B. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_40 · Cited by 499
Self-citationHidden Conditional Random Fields for Gesture Recognition
Wang, S. B., Quattoni, A., Morency, L.-P., Demirdjian, D., & Darrell, T. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.132 · Cited by 451
Learning Deep Generative Models
Salakhutdinov, R. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020120 · Cited by 436
ModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420
Instructing people for training gestural interactive systems
Fothergill, S., Mentis, H., Kohli, P., & Nowozin, S. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208303 · Cited by 368
Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., & Bajcsy, R. (2014) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2013.04.007 · Cited by 318
Multi-scale Deep Learning for Gesture Detection and Localization
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_33 · Cited by 229
Connectionist probability estimators in HMM speech recognition
Renals, S., Morgan, N., Bourlard, H., Cohen, M., & Franco, H. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.260359 · Cited by 228
Spatio-Temporal Laplacian Pyramid Coding for Action Recognition
Shao, L., Zhen, X., Tao, D., & Li, X. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2273174 · Cited by 219
Self-citationChaLearn Looking at People Challenge 2014: Dataset and Results
Escalera, S., Baró, X., Gonzàlez, J., Bautista, M. A., Madadi, M., Reyes, M., Ponce-López, V., Escalante, H. J., Shotton, J., & Guyon, I. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_32 · Cited by 211
Wu, D., & Shao, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.98 · Cited by 211
Self-citationShaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
Bio-inspired Dynamic 3D Discriminative Skeletal Features for Human Action Recognition
Chaudhry, R., Ofli, F., Kurillo, G., Bajcsy, R., & Vidal, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.153 · Cited by 179
Multi-modal gesture recognition challenge 2013: dataset and results
Escalera, S., Gonzàlez, J., Baró, X., Reyes, M., Lopes, O., Guyon, I., Athitsos, V., & Escalante, H. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2532595 · Cited by 177
Does Human Action Recognition Benefit from Pose Estimation?
Yao, A., Gall, J., Fanelli, G., & Gool, L. V. (2011) · British Machine Vision Conference (BMVC) · doi:10.5244/c.25.67 · Cited by 172
ChaLearn gesture challenge: Design and first results
Guyon, I., Athitsos, V., Jangyodsuk, P., Hamner, B., & Escalante, H. J. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239178 · Cited by 85
Multi-stream adaptive evidence combination for noise robust ASR
Morris, A., Hagen, A., Glotin, H., & Bourlard, H. A. (2001) · Speech Communication · doi:10.1016/s0167-6393(00)00044-3 · Cited by 83
Spatio-Temporal Convolutional Sparse Auto-Encoder for Sequence Classification
Baccouche, M., Mamalet, F., Wolf, C., García, C., & Baskurt, A. M. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.124 · Cited by 83
Fusing multi-modal features for gesture recognition
Wu, J., Cheng, J., Zhao, C., & Lu, H. (2013) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2522848.2532589 · Cited by 63
Action Points: A Representation for Low-latency Online Human Action Recognition
Nowozin, S. (2012) · Cited by 55
Continuous Gesture Recognition from Articulated Poses
Evangelidis, G. D., Singh, G., & Horaud, R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_42 · Cited by 52
A Multi-scale Approach to Gesture Detection and Recognition
Neverova, N., Wolf, C., Paci, G., Sommavilla, G., Taylor, G. W., & Nebout, F. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.69 · Cited by 52
A P300 BCI for the Masses: Prior Information Enables Instant Unsupervised Spelling
Kindermans, P.-J., Verschore, H., Verstraeten, D., & Schrauwen, B. (2012) · Ghent University Academic Bibliography (Ghent University) · Cited by 49
Self-citationA Multi-scale Boosted Detector for Efficient and Robust Gesture Recognition
Monnier, C. S., German, S., & Ost, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_34 · Cited by 49
Realistic action recognition via sparsely-constructed Gaussian processes
Liu, L., Shao, L., Zheng, F., & Li, X. (2014) · Pattern Recognition · doi:10.1016/j.patcog.2014.07.006 · Cited by 47
Self-citationGesture Recognition Using Template Based Random Forest Classifiers
Camgöz, N. C., Kındıroğlu, A. A., & Akarun, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_41 · Cited by 41
Action and Gesture Temporal Spotting with Super Vector Representation
Peng, X., Wang, L., Cai, Z., & Qiao, Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_36 · Cited by 40
Deep Dynamic Neural Networks for Gesture Segmentation and Recognition
Wu, D., & Shao, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_39 · Cited by 33
Self-citationNonparametric Gesture Labeling from Multi-modal Data
Chang, J. Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_35 · Cited by 28
Chen, G., Clarke, D., Giuliani, M., Gaschler, A., Wu, D., Weikersdorfer, D., & Knoll, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_43 · Cited by 19
Self-citationMulti-modal Gesture Recognition Using Skeletal Joints and Motion Trail Model
Liang, B., & Zheng, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_44 · Cited by 14
(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1
Cited by
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752
Metadata
- Authors
- Wu, D., Pigou, L., Kindermans, P.-J., Le, N. D.-H., Shao, L., Dambre, J., & Odobez, J.-M.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-08-01
- Volume
- 38
- Issue
- 8
- Pages
- 1583-1597
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2295038166
- Cited by
- 453
- References
- 66
- FWCI
- 95.97
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.