ModDrop: Adaptive Multi-Modal Gesture Recognition
Open accessThe DOI matches its Crossref record; 71 of the 77 references it cites were found in a registry or an index; 6 of the 18 sources its claims cite were checked for support.
Abstract
We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Furthermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.
Overview
CoChat Summary
TL;DR
The paper presents ModDrop, a multi-modal and multi-scale deep learning method for gesture detection and localization that placed first in the ChaLearn 2014 LAP Challenge track 3. ModDrop trains shared fusion layers with random modality dropping, improving recognition and making predictions robust to missing or corrupted channels.
Key findings
- The method placed first out of 17 teams in the ChaLearn 2014 LAP Challenge track 3, with the submitted hybrid model scoring 0.850 Jaccard index and surpassing the second-best score by 1.61 percentage points; post-challenge improvements raised the score to 0.870.
- On ChaLearn 2014 LAP augmented with audio, ModDrop improved robustness to missing channels: with mocap missing, accuracy rose from 38.41% to 92.82% and Jaccard from 0.306 to 0.859; with both hands missing, accuracy rose from 53.13% to 73.28% and Jaccard from 0.466 to 0.680.
- On an MNIST multi-modal sanity check, ModDrop reduced test error under occlusion: 1 segment covered 2.30% vs 10.74%, 2 segments covered 7.19% vs 35.91%, and 3 segments covered 24.88% vs 68.03%.
- Adding audio improved Jaccard by 1.3 points, and learned audio representations outperformed a phoneme-recognition baseline (F-measure 79.71% vs 56.50%; Jaccard 0.545 vs 0.256).
Methods
- A multi-scale, multi-modal convolutional architecture uses parallel single-scale paths over dynamic poses of 5 frames sampled with a given temporal stride, integrating depth and intensity video, articulated pose (183-dimensional per frame; 915-dimensional dynamic pose), and optionally audio.
- Training proceeds by per-modality pretraining, initialization of shared fusion layers to a normalized geometric mean of per-model outputs, and gradual fusion with ModDrop, which randomly drops separate modality channels during training.
- Experiments use the ChaLearn 2014 Looking at People Challenge track 3 dataset (13,858 gesture instances, 20 gesture categories), augmented with audio from the ChaLearn 2013 Multi-modal Challenge, plus MNIST sanity checks; performance is measured mainly by Jaccard index.
Limitations
- Stated: audio alone does not allow robust gesture start/end localization because of natural gesture-speech desynchronization; annotations are video-based while spoken words/phrases are typically shorter, and vocalization may be delayed or ahead of the gesture, yielding poor audio-only Jaccard scores.
- Evident from the text: evaluation centers on the ChaLearn 2014 LAP dataset plus MNIST sanity checks and ChaLearn 2013 audio augmentation; hyperparameters and ModDrop/dropout rates were optimized on validation data, and hidden-unit dropout degraded performance due to the model’s compactness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
5 Valid · 4 Not valid · 13 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward [51].”
What the sources say
- Partly supportedcites [51] [51] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, 'Maxout Networks,' in arXiv:1302.4389v4 , 2013. - sourceExperimental — accuracy not yet measured
Maxout Networks (p. 2) says it is not obvious how to compute the arithmetic mean of predictions from the exponentially many models trained by dropout. Section 7 (pp. 6–7) discusses approximate model averaging in deep networks, but does not address early fusion or establish the claim specifically for non-linear shared layers. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
What the sources say
- Partly supportedcites [53] [53] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, 'Improving neural networks by preventing coadaptation of feature detectors,' in arXiv:1207.0580 , 2012. - sourceExperimental — accuracy not yet measured
Appendix A.1 (p. 8) lists the 784-1200-1200-10 MNIST architecture and specifies 50% dropout for hidden units and 20% for input units. The main text (p. 2) reports about 110 test errors with those dropout rates, but our copy does not establish exactly 107 errors for that architecture or specify tanh activations. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
What the sources say
- Supportedcites [52] [52] P. Baldi and P. Sadowski, 'The dropout learning algorithm,' Journal of Artificial Intelligence , vol. 210, pp. 78-122, 2014. - source
Section 11.4 gives an approximate dropout regularization term proportional to the sigmoid derivative and the squared weights. It explicitly says the derivative adaptively scales the magnitude of that term. The result is presented as an approximation for a sigmoidal unit. (LLM, conf=0.88)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From our observations, inter-modality fusion is effective at early stages if both channels are of the same nature and convey complementary information.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams.”
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
“The deep learning method described in this paper placed first in the 2014 version of this competition [14].”
plainn = 2,014- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
46 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“The practitioner should be careful when determining whether or not 2 modalities are correlated under the warning that choosing a fusion strategy blindly for 2 essentially different modalities at an early stage has a high risk of learning fake cross-modality correlations and interactions (Neverova et al. 2014).”
From A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding (2024) · cites this paper as Neverova et al. 2014
The cited source says what the claim attributes to it.
In the citing paper
“Studies have also shown that fusing highly correlated features in earlier layers and less correlated features in deeper layers improve model performance50,51.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > Effect of ModDrop on ChaLearn 2014+audio. · p. 13
“In all cases, dropout (20%) and ModDrop (10%) are applied to the input signal.”
§ of parameters > Effect of ModDrop training under occlusion and noise. · p. 12
“| no | no | no | no | 91 . 94 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > 8 CONCLUSION · p. 13
“Each of the visual modalities captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at two temporal scales.”
§ 3 GESTURE CLASSIFICATION · p. 3
“Fig. 2. The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Research works printed but not linked: 2
- [60] A. Lee, T. Kawahara, and K. Shikano, 'Julius - an open source realtime large vocabulary recognition engine,' in Interspeech , 2001. -
- [65] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' in Proceedings of the IEEE , vol. 86(11), 1998, pp. 2278-2324. <!-- image --> Natalia Neverova is a PhD student at INSA de Lyon and LIRIS (CNRS, France) working in the area of gesture and action recognition with emphasis on multi-modal aspects and deep learning methods. She is advised by Christian Wolf and Graham Taylor, and her research is a part of Interabot project in partnership with Awabot SAS. She was a visiting researcher at University of Guelph in 2014 and at Google in 2015. <!-- image --> 0 <!-- image --> <!-- image --> Christian Wolf is assistant professor at INSA de Lyon and LIRIS, CNRS, since 2005. He is interested in computer vision and machine learning, especially the visual analysis of complex scenes in motion, structured models, graphical models and deep learning. He received his MSc in computer science from Vienna University of Technology in 2000, and a PhD from the National Institute of Applied Science (INSA de Lyon), in 2003. In 2012 he obtained the habilitation diploma. Graham Taylor is an assistant professor at University of Guelph. He is interested in statistical machine learning and biologicallyinspired computer vision, with an emphasis on unsupervised learning and time series analysis. He completed his PhD at the University of Toronto in 2009, where his thesis co-advisors were Geoffrey Hinton and Sam Roweis. He did a postdoc at NYU with Chris Bregler, Rob Fergus, and Yann LeCun. Florian Nebout is a project manager at Awabot, leading different robotics projects, which are mostly dedicated to remote presence applications. He is especially interested in human robot interaction and how robotics can change our daily life. He graduated from INSA de Lyon, France and worked at CSIRO Human Computer Interaction team and ANU Information and Human Centred Computing Research Group in Canberra, Australia. · DOI · Paper page
Cited by
Items 301–400 of 418
Applying Image Classification Model to Spectrograms for Speech Emotion Recognition
Zhou, Y., Yang, L., & Mao, J. (2023) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait60137.2023.10528443 · Cited by 1
Skeleton-based Human Action Recognition using Ricci Curvature and Graphs Neural Networks
Ribeiro, M. V. L., Vassallo, R. F., & Samatelo, J. L. A. (2025) · Journal of Control Automation and Electrical Systems · doi:10.1007/s40313-025-01177-2 · Cited by 1
AV-CPL: Continuous Pseudo-labeling for Audio-Visual Speech Recognition
Rouditchenko, A., Collobert, R., & Likhomanenko, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93806-1_18 · Cited by 6
Islam, M. M., Haque, M. R., Raju, S. M. T. U., et al. (2025) · IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw69036.2025.00517 · Cited by 1
Multi-Level Sensor Fusion with Deep Learning
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2018) · HAL · doi:10.48550/arxiv.1811.02447 · Cited by 1
Gaze and face-to-face interaction: From multimodal data to behavioral models
Bailly, G., Mihoub, A., Wolf, C., & Elisei, F. (2018) · Advances in interaction studies · doi:10.1075/ais.10.07bai · Cited by 2
Self-citationMultimodal Deep Kernel Learning: Application to Disaster Response in Social Media Network
Yang, X., Jiu, M., & Sun, Y. M. (2021) · International Conference on Information Communication and Signal Processing (ICICSP) · doi:10.1109/icicsp54369.2021.9611988 · Cited by 1
Relational structure predictive neural architecture search for multimodal fusion
Yao, X. L., Fang, L., & Zeng, Y. (2022) · Soft Computing · doi:10.1007/s00500-022-06772-y · Cited by 1
Action-Affect Classification and Morphing using Multi-Task Representation LearningPreprint
Shields, T. J., Amer, M. R., Ehrlich, M., & Tamrakar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06554 · Cited by 2
Dense Multimodal Fusion for Hierarchically Joint RepresentationPreprint
Hu, D., Nie, F., & Li, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03414 · Cited by 1
Towards Robust Pattern Recognition: A ReviewPreprint
Zhang, X.-Y., Liu, C., & Suen, C. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06976 · Cited by 1
Multi-modal Residual Perceptron Network for Audio-Video Emotion RecognitionPreprint
Chang, X., & Skarbek, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10742 · Cited by 1
Wi-ID: WiFi-Based Identification System Using Rock-Paper-Scissors Hand GesturesPreprint
Zheng, Z., Yu, N. H., Zhang, J., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-823750/v1 · Cited by 1
Multimodal Conversational Events Estimation in Complex Social Scenes
Li, L. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750821
Martin, P.-E. (2020) · theses.fr (ABES) · doi:10.70675/7375e560zacacz497fz9547z463f0ecf46f1 · Cited by 9
A novel method of feature extraction for minor crack identification
Fan, P., & Liu, X. (2016) · International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) · doi:10.1109/cisp-bmei.2016.7852911
“Is This an Example Image?” – Predicting the Relative Abstractness Level of Image and Text
Otto, C., Holzki, S., & Ewerth, R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15712-8_46 · Cited by 2
"Is this an example image?" -- Predicting the Relative Abstractness Level of Image and TextPreprint
Otto, C., Holzki, S., & Ewerth, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.07878
RECONHECIMENTO DE GESTOS DINÂMICO PARA A INTERAÇÃO HUMANO-ROBÔ
Canuto, C., Samatelo, J. L. A., & Vassallo, R. F. (2019) · Anais do 14º Simpósio Brasileiro de Automação Inteligente · doi:10.17648/sbai-2019-111567
Liu, C., Li, X., Niu, W., et al. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961553
Zhou, J., & Komuro, T. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00006
A Multimodal Fusion Model Based on Hybrid Attention Mechanism for Gesture Recognition
Li, Y., Chen, Y., Gu, Y., & Ouyang, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73973-7_29
Deep Models for Improving the Performance and Reliability of Person Recognition
Soleymani, S. (2021) · The Research Repository @ WVU (West Virginia University) · doi:10.33915/etd.8028
CODA: Constructivism Learning for Instance-Dependent Dropout Architecture ConstructionPreprint
Li, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08444
Condat, R., Rogozan, A., Aïnouz, S., & Bensrhair, A. (2022) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9922424
Recognition of Facial and Vocal Emotional Expressions by SOAR Model
Shahrestani, M. R., Motamed, S., & Yamaghani, M. (2023) · Journal of Information Systems and Telecommunication (JIST) · doi:10.61186/jist.39828.11.43.209
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_1
Gate tunable MoS2 memristive neuron for early fusion multimodal spiking neural networkPreprint
Tian, H., Liu, Y., Peng, Y., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4190143/v1
Multimodal Disentanglement and Distillation Representation for Gesture RecognitionPreprint
Shi, Y., Li, Y., Chen, H., Liang, S., & Miao, Q. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4963605
Local and global mortality experience: A novel hierarchical model for regional mortality riskPreprint
Nalmpatian, A., Heumann, C., Alkaya, L., et al. (2024) · medRxiv · doi:10.1101/2024.10.17.24315673
FROM HAND WAVES TO COMMANDS: AI-ENHANCED GESTURE RECOGNITION IN AR/MR SYSTEMS
Mrs. Madalambika, K., & R, Y. S. (2025) · EPRA International Journal of Research & Development (IJRD) · doi:10.36713/epra19839
GestureFlow: Advanced Hand Gesture Control System
Naveen, Y., & Sree, C. N. (2025) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem44540
BiSyncFusion: Dual-Stream Bidirectional Synchronization for BEV Multimodal Fusion
You, L., Yang, C., Wang, Z., & Chen, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9869-1_21
Dhayal, P., Singh, P., Sharma, K., Sarkar, S., Rajak, D. R., Bhardwaj, A., & Raman, B. (2025) · Remote Sensing Applications Society and Environment · doi:10.1016/j.rsase.2025.101732
MRF: A Modality-Resilient Framework for Handling Missing Modalities in Multimodal Learning
Du, P., Huang, Y., Chang, X., & Li, R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_2
Demo Abstract: ECO: Low Power Context-Aware Multimodal AI on NPUs
Das, A., Agarwal, Y., Ghosh, S. K., Raha, A., & Raghunathan, V. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261815 · Cited by 2
Sparse Coding for Multimodal Hand Biometrics
Zhang, B., Zhao, S., Fei, L., & Li, S. (2026) · Springer eBooks · doi:10.1007/978-981-95-0970-6_16
Integrating psychological profiling with deep learning for enhanced boxing action recognition
Li, T., Yan, Y., & Wen, L. (2026) · Scientific Reports · doi:10.1038/s41598-025-34771-0
Self-Attention Fusion in Multimodal Behavior Recognition
Bai, B., Zhang, C., Xiaorui, S., & Yang, L. (2025) · IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) · doi:10.1109/imcec66174.2025.11332200
Hurtado-Sánchez, C., Cisneros, R. R., Cárdenas-Valdez, J. R., Téllez, A. C., & Inzunza-González, E. (2026) · Future Internet · doi:10.3390/fi18010061
Leveraging Federated Self-Supervised Approach for Stress Recognition to Mitigate Label Leakage
Benouis, M., Andre, E., & Can, Y. S. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3797882 · Cited by 1
Kim, B., Lee, D., Shyamala Devi, M., Natarajan, Y., Preethaa, K. R. S., Park, J.-Y., & Park, C.-J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6485046
Deep learning for human motion analysis
Neverova, N. (2016) · theses.fr (ABES) · doi:10.70675/a8cda552zdc96z48c1z979bzac140403e58f · Cited by 23
Self-citationDynamic hand gesture recognition : from traditional handcrafted to recent deep learning approaches
de Smedt, Q. (2017) · theses.fr (ABES) · doi:10.70675/68ddb06cz1c43z41c3z828fzc58326157b3c · Cited by 13
Approches jointes texte/image pour la compréhension multimodale de documents
Delecraz, S. (2018) · theses.fr (ABES) · doi:10.70675/1dbcf5c1zf620z42d0za4dfzdccac062b8dc · Cited by 1
Apprentissage neuronal profond pour l'analyse de contenus multimodaux et temporels
Vielzeuf, V. (2019) · theses.fr (ABES) · doi:10.70675/cd4ec871zb372z492fz94bfz25086c5fda4a · Cited by 1
Estimation de pose humaine et reconnaissance d’action par un système multi-robots
Doğan, E. (2017) · theses.fr (ABES) · doi:10.70675/c539f499z55f1z44cazba44zecef96b6d165
Granger, N. (2019) · theses.fr (ABES) · doi:10.70675/9f7530bez3094z498fzbaf7zbfac62e25c00
Face off: A survey on multimodal fusion for secure and realistic facial de-identification
Zhu, H., Jain, D. K., Zhao, X., Li, M., Gan, C., & Zhang, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104343
Structured deep learning for video analysis
Baradel, F. (2020) · theses.fr (ABES) · doi:10.70675/2a8e4991z61e3z4030z8b4cz030a297ed81c
Clickbait detection using multimodel fusion and transfer learning
Rajapaksha, P. (2020) · theses.fr (ABES) · doi:10.70675/862230d2z8e4bz431dz90a9z76ed3351df0f · Cited by 5
Registration of egocentric views for collaborative localization in security applications
Chen, H. (2021) · theses.fr (ABES) · doi:10.70675/07287733z64d8z4367za00fz8a15d64beb7f
Devineau, G. (2020) · theses.fr (ABES) · doi:10.70675/f79154a4z5b19z4ca7zaad7zbaea74138b63
Duc, P. N. H. (2020) · theses.fr (ABES) · doi:10.70675/17b51e87z4d64z4a42z834cz06bdd762a37a · Cited by 6
Voillemin, T. (2021) · theses.fr (ABES) · doi:10.70675/dfefad89z222fz4df2z8717z6f090aa100f7
Condat, R. (2022) · theses.fr (ABES) · doi:10.70675/1d38c68cz49a7z44b6z8f23z7a2cc73c7bac
Multimodal transformers for emotion recognition
Vazquez-Rodriguez, J. (2023) · theses.fr (ABES) · doi:10.70675/a1927be2zdba6z405azbe6fz6856529b46bb · Cited by 3
Robust Deep learning methods inspired by signal processing algorithms
Neacşu, A.-A. (2023) · theses.fr (ABES) · doi:10.70675/cac42885zff54z47eaz848czcfc9a22ac938
Real-Time Hand Gesture Recognition Techniques for Virtual Mouse Systems
Kumar, V., Garg, P., Gill, S., Naresh, P., Madhukar, & Tyagi, A. (2026) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm69057.2026.11465407
AI-Driven Hand Gesture Recognition with Adaptive Zoom for Smart Proximity-Based Computer Interaction
Gyaneshwar, D., & Gorrela, S. L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-18477-1_32
Varghese, E. B., Al‐Thani, D., & Qaraqe, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462949
Human–Robot Interaction via hand gestures: System design and analysis
Choudhury, A., Johnvictor, A. C., Nanthakumar, P. S., et al. (2026) · EPJ Web of Conferences · doi:10.1051/epjconf/202636701004
Alam, M. T., & Roy, K. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491709
Akalya, K., Gopikha., S., & G, K. R. (2026) · doi:10.1109/icsedis68157.2026.11518400
UniSafe: Modality-Agnostic Hateful Content Detection via Shared-Space Projection
Antar, S. S., Ador, S. S., Ding, S. H. H., & Fung, B. C. M. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795455
Edge-Deployable RGB–Thermal UAV Monitoring for Wildfires in Power Transmission Corridors
Wang, B., Huang, D., Lin, Y., et al. (2026) · Remote Sensing · doi:10.3390/rs18121869
CRISPR-MBTF: a multi-branch transformer fusion framework for CRISPR-Cas9 off-target prediction
Jahangiri-Sisakht, A., Safari, L., & Alipanahi, R. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag216 · Cited by 1
Gomaa, A., Gani, K., Feld, M., & Krüger, A. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3820490
Ali, M. B. G., & Smiti, A. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115716
Early Fusion-Based Multimodal Cardiac MRI Segmentation with Domain-Aware Augmentation
Lin, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-16271-7_19
Data-Centric Multimodal Pavement Distress Detection Using Intensity–Range Imagery
Tao, W., Bai, Y., Wang, F., et al. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810611
Descriptor: RGB Video of Natural Hand Gestures Dataset (RGB-NHG)
Rahat, M. T., Sarker, D., & Nur, K. M. (2026) · IEEE data descriptions. · doi:10.1109/ieeedata.2026.3706641
FuDensityNet: Occlusion-Aware Multimodal Activation for Robust Object Detection
Ouardirhi, Z., Zbakh, M., Benjelloun, M., & Mahmoudi, S. A. (2026) · Electronics · doi:10.3390/electronics15132783
RGB-D Dynamic Gesture Recognition with a Strong RGB Prior and Depth-Guided Residual EnhancementPreprint
Zhou, Y., & Liu, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7011193
Li, H., Sun, W., Zhang, J., et al. (2026) · Applied Sciences · doi:10.3390/app16136672
WormVIO: Deep visual-inertial odometry via hierarchical instinct-bias sensor fusion
Zheng, J., Zhao, J., Zhao, M. Y., Zhang, F., Cheng, X., Lu, B., & Liu, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104614
Ma, Y., Wang, X., Ouyang, Q., et al. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3541-4_17
ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning
Nizam, N. B., Liu, F., Kwak, S., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38260-3_49
Kaya, H., Ekinci, F., Selçuk, A. Y., Koçkar, N., DEMİR, Ö., Ugurlu, G., Güzel, M. S., & Güzel, M. S. (2026) · Physica B Condensed Matter · doi:10.1016/j.physb.2026.419022 · Cited by 1
FedGMM: Robust Temporal Multimodal Federated Learning Under Missing Modalities
Zhao, Q., & Feng, S. L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3520-9_47 · Cited by 1
Multimodal learning with thermal imagery and pose cues for privacy-preserving behaviour recognition
Pandey, P., & Poulose, A. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116659
Adaptive Multi-Level Fusion via Expert Routing for Medical Visual Question AnsweringPreprint
Nguyen, B.-D., Hoang, V.-D., & Nguyen, H. D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7156850
Multimodal Fusion and Adaptive Learning for Cardiopulmonary Classification
Abed, M. T., & Zhao, X. (2026) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3807503.3819467
Nirmala, F. J., Madhuvanthi, J. S., Viknesh, R., et al. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-31216-7_43
Liu, Y., Wu, J., & González, E. D. R. S. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133914
Li, Z., Jia, H., Huang, W., Zhao, K., & Chen, M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818168
Huang, C., Nihey, F., Kosaka, Y., Nakahara, K., Tamada, Y., & Murashita, K. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00098
Balanced Multimodal Training Through Unified Forward-Backward Modulation Strategy
Do, T. M., & Nguyen, C.-V. T. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-2597-2_12
Feng, J., Xu, N., Li, X., Fu, Z., Xiao, Z., & Huang, Z. (2026) · Biomimetics · doi:10.3390/biomimetics11080582
SivaKumari, B., Vijayaraghavan, V., & Gayathri, U. (2026) · Information systems engineering and management · doi:10.1007/978-3-032-32709-3_26
Adaptive Cross-Modal Fusion With Instance-Level Gating for Vision-Language Understanding
Patel, U. A., Patel, S., Nanavati, J., et al. (2026) · International Conference on Circuit, Power and Computing Technologies · doi:10.1109/iccpct70290.2026.11655009
LMF-CP: An Interpretable Multimodal Late-Fusion Framework for Compound Carcinogenicity Prediction
Zhu, Y., He, L., & Liang, X. (2026) · International Journal of Molecular Sciences · doi:10.3390/ijms27167499
Lv, Y., Li, Y., & Han, Y. (2026) · Food Research International · doi:10.1016/j.foodres.2026.120297
Varghese, E. B., Al‐Thani, D., & Qaraqe, M. (2026) · Array · doi:10.1016/j.array.2026.101194
Chen, J., Seng, K. P., Lim, C. S., Smith, J., & Ang, L.-M. (2026) · Electronics · doi:10.3390/electronics15174050
Multi Signer Spatio Temporal Framework for Isolated Sign Language Recognition
Renjith, S., & Varghese, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30438-4_22
Kulicki, M., Wielgosz, M., Puliti, S., Będkowski, J., & Stereńczak, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7435257
Chen, Z., Pan, Z., Li, S., Liu, Q., Gao, H., & Guan, L. (2026) · Applied Sciences · doi:10.3390/app16188921
Zhang, Y. (2026) · Frontiers in Neurology · doi:10.3389/fneur.2026.1908175
Metadata
- Authors
- Neverova, N., Wolf, C., Taylor, G., & Nebout, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- University of Guelph (CA)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-08-01
- Volume
- 38
- Issue
- 8
- Pages
- 1692-1706
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W1533025524
- Cited by
- 420
- References
- 77
- Citations by year
- 2026: 89 · 2025: 28 · 2024: 28 · 2023: 35 · 2022: 24 · 2021: 44 · 2020: 41 · 2019: 44 · 2018: 34 · 2017: 34 · 2016: 14
- FWCI
- 55.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Investissement's d’Avenir
- Briques Génériques du Logiciel Embarqué
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- gesture recognition, multimodal fusion, multi-scale deep learning, spatial and temporal scales, missing modality robustness, upper body movement, hand movement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.