ModDrop: Adaptive Multi-Modal Gesture Recognition
Open accessThe DOI matches its Crossref record; 71 of the 77 references it cites were found in a registry or an index; 6 of the 18 sources its claims cite were checked for support.
Abstract
We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Furthermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.
Overview
CoChat Summary
TL;DR
The paper presents ModDrop, a multi-modal and multi-scale deep learning method for gesture detection and localization that placed first in the ChaLearn 2014 LAP Challenge track 3. ModDrop trains shared fusion layers with random modality dropping, improving recognition and making predictions robust to missing or corrupted channels.
Key findings
- The method placed first out of 17 teams in the ChaLearn 2014 LAP Challenge track 3, with the submitted hybrid model scoring 0.850 Jaccard index and surpassing the second-best score by 1.61 percentage points; post-challenge improvements raised the score to 0.870.
- On ChaLearn 2014 LAP augmented with audio, ModDrop improved robustness to missing channels: with mocap missing, accuracy rose from 38.41% to 92.82% and Jaccard from 0.306 to 0.859; with both hands missing, accuracy rose from 53.13% to 73.28% and Jaccard from 0.466 to 0.680.
- On an MNIST multi-modal sanity check, ModDrop reduced test error under occlusion: 1 segment covered 2.30% vs 10.74%, 2 segments covered 7.19% vs 35.91%, and 3 segments covered 24.88% vs 68.03%.
- Adding audio improved Jaccard by 1.3 points, and learned audio representations outperformed a phoneme-recognition baseline (F-measure 79.71% vs 56.50%; Jaccard 0.545 vs 0.256).
Methods
- A multi-scale, multi-modal convolutional architecture uses parallel single-scale paths over dynamic poses of 5 frames sampled with a given temporal stride, integrating depth and intensity video, articulated pose (183-dimensional per frame; 915-dimensional dynamic pose), and optionally audio.
- Training proceeds by per-modality pretraining, initialization of shared fusion layers to a normalized geometric mean of per-model outputs, and gradual fusion with ModDrop, which randomly drops separate modality channels during training.
- Experiments use the ChaLearn 2014 Looking at People Challenge track 3 dataset (13,858 gesture instances, 20 gesture categories), augmented with audio from the ChaLearn 2013 Multi-modal Challenge, plus MNIST sanity checks; performance is measured mainly by Jaccard index.
Limitations
- Stated: audio alone does not allow robust gesture start/end localization because of natural gesture-speech desynchronization; annotations are video-based while spoken words/phrases are typically shorter, and vocalization may be delayed or ahead of the gesture, yielding poor audio-only Jaccard scores.
- Evident from the text: evaluation centers on the ChaLearn 2014 LAP dataset plus MNIST sanity checks and ChaLearn 2013 audio augmentation; hyperparameters and ModDrop/dropout rates were optimized on validation data, and hidden-unit dropout degraded performance due to the model’s compactness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
5 Valid · 4 Not valid · 13 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward [51].”
What the sources say
- Partly supportedcites [51] [51] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, 'Maxout Networks,' in arXiv:1302.4389v4 , 2013. - sourceExperimental — accuracy not yet measured
Maxout Networks (p. 2) says it is not obvious how to compute the arithmetic mean of predictions from the exponentially many models trained by dropout. Section 7 (pp. 6–7) discusses approximate model averaging in deep networks, but does not address early fusion or establish the claim specifically for non-linear shared layers. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
What the sources say
- Partly supportedcites [53] [53] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, 'Improving neural networks by preventing coadaptation of feature detectors,' in arXiv:1207.0580 , 2012. - sourceExperimental — accuracy not yet measured
Appendix A.1 (p. 8) lists the 784-1200-1200-10 MNIST architecture and specifies 50% dropout for hidden units and 20% for input units. The main text (p. 2) reports about 110 test errors with those dropout rates, but our copy does not establish exactly 107 errors for that architecture or specify tanh activations. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
What the sources say
- Supportedcites [52] [52] P. Baldi and P. Sadowski, 'The dropout learning algorithm,' Journal of Artificial Intelligence , vol. 210, pp. 78-122, 2014. - source
Section 11.4 gives an approximate dropout regularization term proportional to the sigmoid derivative and the squared weights. It explicitly says the derivative adaptively scales the magnitude of that term. The result is presented as an approximation for a sigmoidal unit. (LLM, conf=0.88)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From our observations, inter-modality fusion is effective at early stages if both channels are of the same nature and convey complementary information.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams.”
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
“The deep learning method described in this paper placed first in the 2014 version of this competition [14].”
plainn = 2,014- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
46 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“The practitioner should be careful when determining whether or not 2 modalities are correlated under the warning that choosing a fusion strategy blindly for 2 essentially different modalities at an early stage has a high risk of learning fake cross-modality correlations and interactions (Neverova et al. 2014).”
From A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding (2024) · cites this paper as Neverova et al. 2014
The cited source says what the claim attributes to it.
In the citing paper
“Studies have also shown that fusing highly correlated features in earlier layers and less correlated features in deeper layers improve model performance50,51.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > Effect of ModDrop on ChaLearn 2014+audio. · p. 13
“In all cases, dropout (20%) and ModDrop (10%) are applied to the input signal.”
§ of parameters > Effect of ModDrop training under occlusion and noise. · p. 12
“| no | no | no | no | 91 . 94 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > 8 CONCLUSION · p. 13
“Each of the visual modalities captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at two temporal scales.”
§ 3 GESTURE CLASSIFICATION · p. 3
“Fig. 2. The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Research works printed but not linked: 2
- [60] A. Lee, T. Kawahara, and K. Shikano, 'Julius - an open source realtime large vocabulary recognition engine,' in Interspeech , 2001. -
- [65] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' in Proceedings of the IEEE , vol. 86(11), 1998, pp. 2278-2324. <!-- image --> Natalia Neverova is a PhD student at INSA de Lyon and LIRIS (CNRS, France) working in the area of gesture and action recognition with emphasis on multi-modal aspects and deep learning methods. She is advised by Christian Wolf and Graham Taylor, and her research is a part of Interabot project in partnership with Awabot SAS. She was a visiting researcher at University of Guelph in 2014 and at Google in 2015. <!-- image --> 0 <!-- image --> <!-- image --> Christian Wolf is assistant professor at INSA de Lyon and LIRIS, CNRS, since 2005. He is interested in computer vision and machine learning, especially the visual analysis of complex scenes in motion, structured models, graphical models and deep learning. He received his MSc in computer science from Vienna University of Technology in 2000, and a PhD from the National Institute of Applied Science (INSA de Lyon), in 2003. In 2012 he obtained the habilitation diploma. Graham Taylor is an assistant professor at University of Guelph. He is interested in statistical machine learning and biologicallyinspired computer vision, with an emphasis on unsupervised learning and time series analysis. He completed his PhD at the University of Toronto in 2009, where his thesis co-advisors were Geoffrey Hinton and Sam Roweis. He did a postdoc at NYU with Chris Bregler, Rob Fergus, and Yann LeCun. Florian Nebout is a project manager at Awabot, leading different robotics projects, which are mostly dedicated to remote presence applications. He is especially interested in human robot interaction and how robotics can change our daily life. He graduated from INSA de Lyon, France and worked at CSIRO Human Computer Interaction team and ANU Information and Human Centred Computing Research Group in Canberra, Australia. · DOI · Paper page
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Self-citationA Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., & Kautz, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.456 · Cited by 695
Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition
Wu, D., Pigou, L., Kindermans, P.-J., Le, N. D.-H., Shao, L., Dambre, J., & Odobez, J.-M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2537340 · Cited by 453
Núñez, J. C., Cabido, R., Pantrigo, J. J., Montemayor, A. S., & Vélez, J. F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.033 · Cited by 419
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412
Skeleton-Based Dynamic Hand Gesture Recognition
De Smedt, Q., Wannous, H., & Vandeborre, J.-P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.153 · Cited by 321
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290
Skeleton Aware Multi-modal Sign Language Recognition
Jiang, S., Sun, B., Wang, L., Bai, Y., Li, K., & Fu, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00380 · Cited by 290
BlockSecIoTNet: Blockchain-based decentralized security architecture for IoT network
Rathore, S., Wook Kwon, B., & Park, J. H. (2019) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.06.019 · Cited by 274
AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods
Sincan, O. M., & Keles, H. Y. (2020) · IEEE Access · doi:10.1109/access.2020.3028072 · Cited by 252
A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition
Adaloglou, N., Chatzis, T., Papastratis, I., Stergioulas, A., Papadopoulos, G. T., Zacharopoulou, V., Xydopoulos, G. J., Atzakas, K., Papazachariou, D., & Daras, P. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3070438 · Cited by 242
Hierarchical LSTM for Sign Language Translation
Guo, D., Zhou, W., Li, H., & Wang, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12235 · Cited by 203
Deep Learning for Hand Gesture Recognition on Skeletal Data
Devineau, G., Moutarde, F., Xi, W., & Yang, J. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00025 · Cited by 203
MFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193
Shaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points
Baradel, F., Wolf, C., Mille, J., & Taylor, G. W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00056 · Cited by 173
Self-citationDeep Multimodal Subspace Clustering Networks
Abavisani, M., & Patel, V. M. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2875385 · Cited by 181
Avola, D., Bernardi, M., Cinque, L., Foresti, G. L., & Massaroni, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2856094 · Cited by 179
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences
Asadi-Aghbolaghi, M., Clapes, A., Bellantonio, M., Escalante, H. J., Ponce-Lopez, V., Baro, X., Guyon, I., Kasaei, S., & Escalera, S. (2017) · 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) · doi:10.1109/fg.2017.150 · Cited by 173
Temporal multimodal fusion for video emotion classification in the wild
Vielzeuf, V., Pateux, S., & Jurie, F. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143011 · Cited by 170
Learning Human Identity From Motion Patterns
Neverova, N., Wolf, C., Lacey, G., Fridman, L., Chandra, D., Barbello, B., & Taylor, G. (2016) · IEEE Access · doi:10.1109/access.2016.2557846 · Cited by 172
Self-citationMulti-Stream Multi-Class Fusion of Deep Networks for Video Classification
Wu, Z., Jiang, Y.-G., Wang, X., Ye, H., & Xue, X. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964328 · Cited by 171
CentralNet: A Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166
Towards Robust Pattern Recognition: A Review
Zhang, X.-Y., Liu, C., & Suen, C. Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2989782 · Cited by 200
Deep Learning Based Hand Gesture Recognition and UAV Flight Controls
Hu, B., & Wang, J. (2019) · Machine Intelligence Research · doi:10.1007/s11633-019-1194-7 · Cited by 171
Multimodal Machine Learning: A Survey and TaxonomyPreprint
Baltrušaitis, T., Ahuja, C., & Morency, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1705.09406 · Cited by 5,220
Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
Yu, Z., Zhou, B., Wan, J., Wang, P., Chen, H., Liu, X., Li, S. Z., & Zhao, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3087348 · Cited by 156
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136
Human Action Recognition: Pose-Based Attention Draws Focus to Hands
Baradel, F., Wolf, C., & Mille, J. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.77 · Cited by 155
Self-citationContinuous Gesture Segmentation and Recognition Using 3DCNN and Convolutional LSTM
Zhu, G., Zhang, L., Shen, P., Song, J., Shah, S. A. A., & Bennamoun, M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2869278 · Cited by 125
Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human–Robot Interaction
Gao, Q. H., Chen, Y., Ju, Z., & Liang, Y. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3059685 · Cited by 143
Sarma, D., & Bhuyan, M. K. (2021) · SN Computer Science · doi:10.1007/s42979-021-00827-x · Cited by 138
Multi-source heterogeneous data fusion
Zhang, L., Xie, Y., Xidao, L., & Zhang, X. (2018) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd.2018.8396165 · Cited by 160
Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset
Miah, A. S. M., Hasan, M. A. M., Nishimura, S., & Shin, J. (2024) · IEEE Access · doi:10.1109/access.2024.3372425 · Cited by 136
Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors
Khalid, H., Kim, M., Tariq, S., & Woo, S. S. (2021) · Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection · doi:10.1145/3476099.3484315 · Cited by 110
Soleymani, S., Dabouei, A., Kazemi, H., Dawson, J., & Nasrabadi, N. M. (2018) · 2018 24th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2018.8545061 · Cited by 107
Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition
Li, Y., He, Z., Ye, X., He, Z., & Han, K. (2019) · EURASIP Journal on Image and Video Processing · doi:10.1186/s13640-019-0476-x · Cited by 109
A Concise Yet Effective Model for Non-Aligned Incomplete Multi-View and Missing Multi-Label Learning
Li, X., & Chen, S. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3086895 · Cited by 136
Deep gesture interaction for augmented anatomy learning
Karambakhsh, A., Kamel, A., Sheng, B., Li, P., Yang, P., & Feng, D. D. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.03.004 · Cited by 118
Deep Multimodal Representation Learning from Temporal Data
Yang, X., Ramesh, P., Chitta, R., Madhvanath, S., Bernal, E. A., & Luo, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.538 · Cited by 104
Multimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101
Multi-source knowledge fusion: a survey
Zhao, X., Jia, Y., Li, A., Jiang, R., & Song, Y. (2020) · World Wide Web · doi:10.1007/s11280-020-00811-0 · Cited by 120
Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition
Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2016) · 2016 23rd International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2016.7899606 · Cited by 92
A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles
Nie, J., Yan, J., Yin, H., Ren, L., & Meng, Q. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.3027319 · Cited by 103
Chen, X., Guo, H., Wang, G., & Zhang, L. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296809 · Cited by 129
MFA-Net: Motion Feature Augmented Network for Dynamic Hand Gesture Recognition from Skeletal Data
Chen, X., Wang, G., Guo, H., et al. (2019) · Sensors · doi:10.3390/s19020239 · Cited by 87
Learning with Privileged Information via Adversarial Discriminative Modality Distillation
Garcia, N. C., Morerio, P., & Murino, V. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929038 · Cited by 105
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 95
Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection
Singhal, S., Pandey, T., Mrig, S., Shah, R. R., & Kumaraguru, P. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524650 · Cited by 127
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
3D Convolutional Neural Networks for Dynamic Sign Language Recognition
Liang, Z., Liao, S.-B., & Hu, B.-Z. (2018) · The Computer Journal · doi:10.1093/comjnl/bxy049 · Cited by 83
Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensation
Canuto, C., Samatelo, J. L. A., & Vassallo, R. F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.038 · Cited by 87
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition
Tang, Y., Wang, Z., Lu, J., Feng, J., & Zhou, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2875441 · Cited by 71
Progress and Prospects of Multimodal Fusion Methods in Physical Human–Robot Interaction: A Review
Xue, T., Wang, W., Ma, J., Liu, W., Pan, Z., & Han, M. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2995271 · Cited by 87
Hierarchical Recurrent Deep Fusion Using Adaptive Clip Summarization for Sign Language Translation
Guo, D., Zhou, W., Li, A., Li, H., & Wang, M. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2941267 · Cited by 83
Pose-conditioned Spatio-Temporal Attention for Human Action RecognitionPreprint
Baradel, F., Wolf, C., & Mille, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10106 · Cited by 111
Self-citationSMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State Analysis
Chen, H., Shi, H., Liu, X., Li, X., & Zhao, G. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01761-6 · Cited by 128
Heterogeneous hand gesture recognition using 3D dynamic skeletal data
de Smedt, Q., Wannous, H., & Vandeborre, J. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.01.008 · Cited by 81
Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations
Xu, J. J., Li, Z., Huang, F., Li, C., & Yu, P. S. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3005405 · Cited by 78
Sentiment analysis of social images via hierarchical deep fusion of content and links
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.04.010 · Cited by 76
Gesture and Sign Language Recognition with Temporal Residual Networks
Pigou, L., Herreweghe, M. V., & Dambre, J. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.365 · Cited by 59
Online Early-Late Fusion Based on Adaptive HMM for Sign Language Recognition
Guo, D., Zhou, W., Li, H., & Wang, M. (2017) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3152121 · Cited by 72
Joint Discriminative Sparse Coding for Robust Hand-Based Multimodal Recognition
Li, S., & Zhang, B. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3074315 · Cited by 60
Multi-Modal Deep Analysis for Multimedia
Zhu, W., Wang, X. Y., & Li, H. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2940647 · Cited by 67
Review of dynamic gesture recognition
Shi, Y., Li, Y., Xiaolong, F. U., Kaibin, M., & Miao, Q. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2021.05.001 · Cited by 70
(2020) · doi:10.1145/3380999 · Cited by 57
Recurrent Encoder–Decoder Networks for Vessel Trajectory Prediction With Uncertainty Estimation
Capobianco, S., Forti, N., Millefiori, L. M., Braca, P., & Willett, P. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3216823 · Cited by 66
Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey
Asadi, M., Clapés, A., Bellantonio, M., Escalante, H. J., Ponce-López, V., Baró, X., Guyon, I., Kasaei, S., & Escalera, S. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_19 · Cited by 67
Comparing random forest approaches to segmenting and classifying gestures
Joshi, A., Monnier, C. S., Betke, M., & Sclaroff, S. (2016) · Image and Vision Computing · doi:10.1016/j.imavis.2016.06.001 · Cited by 56
Multiscale Representation Learning for Image Classification: A Survey
Jiao, L., Gao, J., Liu, X., Liu, F., Yang, S., & Hou, B. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3135248 · Cited by 65
Lu, X., Xie, L., Xu, L., Mao, R., Xu, X., & Chang, S. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.030 · Cited by 73
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17260 · Cited by 52
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · arXiv · doi:10.48550/arxiv.2102.02340 · Cited by 52
Continuous hand gesture recognition based on trajectory shape information
Yang, C., Han, D. K., & Ko, H. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.05.016 · Cited by 50
Guo, D., Tang, S., & Wang, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/106 · Cited by 63
Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation
Liu, G.-H., Siravuru, A., Prabhakar, S., Veloso, M., & Kantor, G. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10422 · Cited by 49
Yuan, Q. (2020) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-189033 · Cited by 53
RetractedParking Prediction in Smart Cities: A Survey
Xiao, X., Peng, Z., Lin, Y., Jin, Z., Shao, W., Chen, R., Cheng, N., & Mao, G. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3279024 · Cited by 62
Yao, C., Wu, S., Liu, Z., & Li, P. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.11.011 · Cited by 50
Xiao, Q., Qin, M., Guo, P. W., & Zhao, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2925654 · Cited by 50
Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition
Ma, Y., Xu, T., & Kim, K. (2022) · Sensors · doi:10.3390/s22165959 · Cited by 58
A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding
Montesinos-López, O. A., Chavira-Flores, M., Kiasmiantini, Crespo-Herrera, L., Saint Piere, C., Li, H., Fritsche-Neto, R., Al-Nowibet, K., Montesinos-López, A., & Crossa, J. (2024) · GENETICS · doi:10.1093/genetics/iyae161 · Cited by 47
Sun, B., Cao, S., He, J. Y. D., & Yu, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.11.021 · Cited by 52
Roy, D. G., & Srirama, S. N. (2021) · Software Practice and Experience · doi:10.1002/spe.2972 · Cited by 58
Miah, A. S. M., Hasan, M. A. M., Jang, S.-W., Lee, H.-S., & Shin, J. (2023) · Electronics · doi:10.3390/electronics12132841 · Cited by 64
3D Skeletal Gesture Recognition via Hidden States Exploration
Liu, X., Shi, H., Hong, X., Chen, H., Tao, D., & Zhao, G. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2974061 · Cited by 44
Modality Dropout for Improved Performance-driven Talking Faces
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418840 · Cited by 42
Hussain, A., Amin, S. U., Fayaz, M., & Seo, S. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.037258 · Cited by 56
Kuo, C.-J., Ting, K.-C., Chen, Y., Yang, D., & Chen, H. (2017) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2017.10.007 · Cited by 48
Liu, C., Li, X., Li, Q., Xue, Y., Liu, H., & Gao, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.016 · Cited by 44
EmoNets: Multimodal deep learning approaches for emotion recognition in videoPreprint
Kahou, S. E., Bouthillier, X., Lamblin, P., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01800 · Cited by 41
Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 56
Cui, C., Liu, H., Liu, Q., Deng, R., Asad, Z., Wang, Y., Zhao, S., Yang, H., Landman, B. A., & Huo, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_60 · Cited by 65
Deep Multimodal Fusion: A Hybrid Approach
Amer, M. R., Shields, T. J., Siddiquie, B., Tamrakar, A., Divakaran, A., & Chai, S. M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-0997-7 · Cited by 45
Real-Time Hand Gesture Tracking for Human–Computer Interface Based on Multi-Sensor Data Fusion
Li, J., Liu, X., Wang, Z., Zhang, T., Qiu, S., Zhao, H., Zhou, X., Cai, H., Ni, R., & Cangelosi, A. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3122236 · Cited by 46
MSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 91
Metadata
- Authors
- Neverova, N., Wolf, C., Taylor, G., & Nebout, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- University of Guelph (CA)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-08-01
- Volume
- 38
- Issue
- 8
- Pages
- 1692-1706
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W1533025524
- Cited by
- 420
- References
- 77
- Citations by year
- 2026: 89 · 2025: 28 · 2024: 28 · 2023: 35 · 2022: 24 · 2021: 44 · 2020: 41 · 2019: 44 · 2018: 34 · 2017: 34 · 2016: 14
- FWCI
- 55.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Investissement's d’Avenir
- Briques Génériques du Logiciel Embarqué
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- gesture recognition, multimodal fusion, multi-scale deep learning, spatial and temporal scales, missing modality robustness, upper body movement, hand movement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.