ModDrop: Adaptive Multi-Modal Gesture Recognition
Open accessThe DOI matches its Crossref record; 71 of the 77 references it cites were found in a registry or an index; 6 of the 18 sources its claims cite were checked for support.
Abstract
We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Furthermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.
Overview
CoChat Summary
TL;DR
The paper presents ModDrop, a multi-modal and multi-scale deep learning method for gesture detection and localization that placed first in the ChaLearn 2014 LAP Challenge track 3. ModDrop trains shared fusion layers with random modality dropping, improving recognition and making predictions robust to missing or corrupted channels.
Key findings
- The method placed first out of 17 teams in the ChaLearn 2014 LAP Challenge track 3, with the submitted hybrid model scoring 0.850 Jaccard index and surpassing the second-best score by 1.61 percentage points; post-challenge improvements raised the score to 0.870.
- On ChaLearn 2014 LAP augmented with audio, ModDrop improved robustness to missing channels: with mocap missing, accuracy rose from 38.41% to 92.82% and Jaccard from 0.306 to 0.859; with both hands missing, accuracy rose from 53.13% to 73.28% and Jaccard from 0.466 to 0.680.
- On an MNIST multi-modal sanity check, ModDrop reduced test error under occlusion: 1 segment covered 2.30% vs 10.74%, 2 segments covered 7.19% vs 35.91%, and 3 segments covered 24.88% vs 68.03%.
- Adding audio improved Jaccard by 1.3 points, and learned audio representations outperformed a phoneme-recognition baseline (F-measure 79.71% vs 56.50%; Jaccard 0.545 vs 0.256).
Methods
- A multi-scale, multi-modal convolutional architecture uses parallel single-scale paths over dynamic poses of 5 frames sampled with a given temporal stride, integrating depth and intensity video, articulated pose (183-dimensional per frame; 915-dimensional dynamic pose), and optionally audio.
- Training proceeds by per-modality pretraining, initialization of shared fusion layers to a normalized geometric mean of per-model outputs, and gradual fusion with ModDrop, which randomly drops separate modality channels during training.
- Experiments use the ChaLearn 2014 Looking at People Challenge track 3 dataset (13,858 gesture instances, 20 gesture categories), augmented with audio from the ChaLearn 2013 Multi-modal Challenge, plus MNIST sanity checks; performance is measured mainly by Jaccard index.
Limitations
- Stated: audio alone does not allow robust gesture start/end localization because of natural gesture-speech desynchronization; annotations are video-based while spoken words/phrases are typically shorter, and vocalization may be delayed or ahead of the gesture, yielding poor audio-only Jaccard scores.
- Evident from the text: evaluation centers on the ChaLearn 2014 LAP dataset plus MNIST sanity checks and ChaLearn 2013 audio augmentation; hyperparameters and ModDrop/dropout rates were optimized on validation data, and hidden-unit dropout degraded performance due to the model’s compactness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
5 Valid · 4 Not valid · 13 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward [51].”
What the sources say
- Partly supportedcites [51] [51] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, 'Maxout Networks,' in arXiv:1302.4389v4 , 2013. - sourceExperimental — accuracy not yet measured
Maxout Networks (p. 2) says it is not obvious how to compute the arithmetic mean of predictions from the exponentially many models trained by dropout. Section 7 (pp. 6–7) discusses approximate model averaging in deep networks, but does not address early fusion or establish the claim specifically for non-linear shared layers. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
What the sources say
- Partly supportedcites [53] [53] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, 'Improving neural networks by preventing coadaptation of feature detectors,' in arXiv:1207.0580 , 2012. - sourceExperimental — accuracy not yet measured
Appendix A.1 (p. 8) lists the 784-1200-1200-10 MNIST architecture and specifies 50% dropout for hidden units and 20% for input units. The main text (p. 2) reports about 110 test errors with those dropout rates, but our copy does not establish exactly 107 errors for that architecture or specify tanh activations. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
What the sources say
- Supportedcites [52] [52] P. Baldi and P. Sadowski, 'The dropout learning algorithm,' Journal of Artificial Intelligence , vol. 210, pp. 78-122, 2014. - source
Section 11.4 gives an approximate dropout regularization term proportional to the sigmoid derivative and the squared weights. It explicitly says the derivative adaptively scales the magnitude of that term. The result is presented as an approximation for a sigmoidal unit. (LLM, conf=0.88)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From our observations, inter-modality fusion is effective at early stages if both channels are of the same nature and convey complementary information.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams.”
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
“The deep learning method described in this paper placed first in the 2014 version of this competition [14].”
plainn = 2,014- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
46 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“The practitioner should be careful when determining whether or not 2 modalities are correlated under the warning that choosing a fusion strategy blindly for 2 essentially different modalities at an early stage has a high risk of learning fake cross-modality correlations and interactions (Neverova et al. 2014).”
From A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding (2024) · cites this paper as Neverova et al. 2014
The cited source says what the claim attributes to it.
In the citing paper
“Studies have also shown that fusing highly correlated features in earlier layers and less correlated features in deeper layers improve model performance50,51.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > Effect of ModDrop on ChaLearn 2014+audio. · p. 13
“In all cases, dropout (20%) and ModDrop (10%) are applied to the input signal.”
§ of parameters > Effect of ModDrop training under occlusion and noise. · p. 12
“| no | no | no | no | 91 . 94 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > 8 CONCLUSION · p. 13
“Each of the visual modalities captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at two temporal scales.”
§ 3 GESTURE CLASSIFICATION · p. 3
“Fig. 2. The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
(2011) · Cited by 2,752
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620
A Spatio-Temporal Descriptor Based on 3D-Gradients
Klaeser, A., Marszalek, M., & Schmid, C. (2008) · Procedings of the British Machine Vision Conference 2008 · doi:10.5244/c.22.99 · Cited by 1,846
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · Journal of Machine Learning Research · doi:10.5555/2627435.2697059 · Cited by 1,807
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669
Mining actionlet ensemble for action recognition with depth cameras
Wang, J., Liu, Z., Wu, Y., & Yuan, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247813 · Cited by 1,512
Multiple kernel learning, conic duality, and the SMO algorithm
Bach, F. R., Lanckriet, G. R. G., & Jordan, M. I. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015424 · Cited by 1,453
Evaluation of local spatio-temporal features for action recognition
Wang, H., Ullah, M. M., Klaser, A., Laptev, I., & Schmid, C. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.124 · Cited by 1,296
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,086
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Le, Q. V., Zou, W. Y., Yeung, S. Y., & Ng, A. Y. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995496 · Cited by 960
Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1302.4389 · Cited by 935
Efficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801
On feature combination for multiclass object classification
Gehler, P., & Nowozin, S. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459169 · Cited by 797
Elements of Large-Sample Theory
(1999) · Springer Texts in Statistics · doi:10.1007/b98855 · Cited by 677
Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.254 · Cited by 605
Unstructured human activity detection from RGBD images
Sung, J., Ponce, C., Selman, B., & Saxena, A. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224591 · Cited by 511
Sign Language Recognition Using Convolutional Neural Networks
Pigou, L., Dieleman, S., Kindermans, P.-J., & Schrauwen, B. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_40 · Cited by 499
Realtime and Robust Hand Tracking from Depth
Qian, C., Sun, X., Wei, Y., Tang, X., & Sun, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.145 · Cited by 470
Zanfir, M., Leordeanu, M., & Sminchisescu, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.342 · Cited by 438
Real-Time Human Pose Recognition in Parts from Single Depth Images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2013) · Studies in Computational Intelligence · doi:10.1007/978-3-642-28661-2_5 · Cited by 411
Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture
Tang, D., Chang, H. J., Tejani, A., & Kim, T.-K. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.490 · Cited by 398
Combining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391
Indoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1301.3572 · Cited by 372
The dropout learning algorithm
Baldi, P., & Sadowski, P. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2014.02.004 · Cited by 340
Hierarchical Aligned Cluster Analysis for Temporal Clustering of Human Motion
Zhou, F., De la Torre, F., & Hodgins, J. K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.137 · Cited by 337
(2013) · Cited by 326
Real Time Hand Pose Estimation Using Depth Sensors
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2013) · Consumer Depth Cameras for Computer Vision · doi:10.1007/978-1-4471-4640-7_7 · Cited by 288
Multi-scale Deep Learning for Gesture Detection and Localization
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_33 · Cited by 229
Self-citationReal-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219
ChaLearn Looking at People Challenge 2014: Dataset and Results
Escalera, S., Baró, X., Gonzàlez, J., Bautista, M. A., Madadi, M., Reyes, M., Ponce-López, V., Escalante, H. J., Shotton, J., & Guyon, I. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_32 · Cited by 211
Real time hand pose estimation using depth sensors
Keskin, C., Kirac, F., Kara, Y. E., & Akarun, L. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130391 · Cited by 190
Robust late fusion with rank minimization
Ye, G., Liu, D., Jhuo, I.-H., & Chang, S.-F. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248032 · Cited by 189
Multimodal feature fusion for robust event detection in web videos
Natarajan, P., Wu, S., Vitaladevuni, S., Zhuang, X., Tsakalidis, S., Park, U., Prasad, R., & Natarajan, P. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247814 · Cited by 178
Multi-modal gesture recognition challenge 2013: dataset and results
Escalera, S., Gonzàlez, J., Baró, X., Reyes, M., Lopes, O., Guyon, I., Athitsos, V., & Escalante, H. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2532595 · Cited by 177
Wu, Z., Jiang, Y., Wang, J., Pu, J., & Xue, X. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654931 · Cited by 156
Beyond Physical Connections: Tree Models in Human Pose Estimation
Wang, F., & Li, Y. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.83 · Cited by 152
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Jain, A., Tompson, J., LeCun, Y., & Bregler, C. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16808-1_21 · Cited by 142
Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models
Fang, G., Gao, W., & Zhao, D. (2006) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2006.886347 · Cited by 136
On combining classifiers using sum and product rules
Alexandre, L. A., Campilho, A. C., & Kamel, M. (2001) · Pattern Recognition Letters · doi:10.1016/s0167-8655(01)00073-3 · Cited by 117
Spatio-Temporal Convolutional Sparse Auto-Encoder for Sequence Classification
Baccouche, M., Mamalet, F., Wolf, C., García, C., & Baskurt, A. M. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.124 · Cited by 95
Self-citationDetect What You Can: Detecting and Representing Objects using Holistic Models and Body PartsPreprint
Chen, X., Mottaghi, R., Liu, X., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2031 · Cited by 91
Continuous Gesture Recognition from Articulated Poses
Evangelidis, G. D., Singh, G., & Horaud, R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_42 · Cited by 63
A Multi-scale Boosted Detector for Efficient and Robust Gesture Recognition
Monnier, C. S., German, S., & Ost, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_34 · Cited by 61
Hernández-Vela, A., Bautista, M. Á., Perez-Sala, X., Ponce-López, V., Escalera, S., Baró, X., Pujol, O., & Ángulo, C. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.09.009 · Cited by 57
A Multi-scale Approach to Gesture Detection and Recognition
Neverova, N., Wolf, C., Paci, G., Sommavilla, G., Taylor, G. W., & Nebout, F. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.69 · Cited by 52
Self-citationDeep learning of invariant spatio-temporal features from video
Chen, B. (2010) · cIRcle (University of British Columbia) · doi:10.14288/1.0051916 · Cited by 51
Gesture Recognition Using Template Based Random Forest Classifiers
Camgöz, N. C., Kındıroğlu, A. A., & Akarun, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_41 · Cited by 49
Action and Gesture Temporal Spotting with Super Vector Representation
Peng, X., Wang, L., Cai, Z., & Qiao, Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_36 · Cited by 45
Online RGB-D gesture recognition with extreme learning machines
Chen, X., & Koskela, M. (2013) · ACM International Conference on Multimodal Interaction (ICMI) · doi:10.1145/2522848.2532591 · Cited by 44
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose EstimationPreprint
Jain, A., Tompson, J., LeCun, Y., & Bregler, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.7963 · Cited by 36
Deep Dynamic Neural Networks for Gesture Segmentation and Recognition
Wu, D., & Shao, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_39 · Cited by 33
Hand Segmentation with Structured Convolutional Learning
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16811-1_45 · Cited by 32
Self-citationNonparametric Gesture Labeling from Multi-modal Data
Chang, J. Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_35 · Cited by 28
Chen, G., Clarke, D., Giuliani, M., Gaschler, A., Wu, D., Weikersdorfer, D., & Knoll, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_43 · Cited by 21
3D hand pose estimation and classification using depth sensors
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2012) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu.2012.6204611 · Cited by 2
Hand segmentation with structured convolutional learning
Neverova, N., Wolf, C., Taylor, G. W., et al. (2014) · doi:10.13140/2.1.3826.9766
Self-citationA multi-modal gesture recognition system using audio, video, and skeletal joint data
Nandakumar, K., Wan, K. W., Chan, S. M. A., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2532593
Research works printed but not linked: 3
- lee, 0, Julius—an open source real-time large vocabulary recognition engine, Proc INTERSPEECH
- bergstra, 0, Theano: A CPU and GPU math expression compiler, Proc Scipy Conf
- 10.1109/5.726791 · DOI · Paper page
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Self-citationA Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Metadata
- Authors
- Neverova, N., Wolf, C., Taylor, G., & Nebout, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- University of Guelph (CA)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-08-01
- Volume
- 38
- Issue
- 8
- Pages
- 1692-1706
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W1533025524
- Cited by
- 420
- References
- 77
- Citations by year
- 2026: 89 · 2025: 28 · 2024: 28 · 2023: 35 · 2022: 24 · 2021: 44 · 2020: 41 · 2019: 44 · 2018: 34 · 2017: 34 · 2016: 14
- FWCI
- 55.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Investissement's d’Avenir
- Briques Génériques du Logiciel Embarqué
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- gesture recognition, multimodal fusion, multi-scale deep learning, spatial and temporal scales, missing modality robustness, upper body movement, hand movement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.