ModDrop: Adaptive Multi-Modal Gesture Recognition
Open accessThe DOI matches its Crossref record; 71 of the 77 references it cites were found in a registry or an index; 6 of the 18 sources its claims cite were checked for support.
Abstract
We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Furthermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.
Overview
CoChat Summary
TL;DR
The paper presents ModDrop, a multi-modal and multi-scale deep learning method for gesture detection and localization that placed first in the ChaLearn 2014 LAP Challenge track 3. ModDrop trains shared fusion layers with random modality dropping, improving recognition and making predictions robust to missing or corrupted channels.
Key findings
- The method placed first out of 17 teams in the ChaLearn 2014 LAP Challenge track 3, with the submitted hybrid model scoring 0.850 Jaccard index and surpassing the second-best score by 1.61 percentage points; post-challenge improvements raised the score to 0.870.
- On ChaLearn 2014 LAP augmented with audio, ModDrop improved robustness to missing channels: with mocap missing, accuracy rose from 38.41% to 92.82% and Jaccard from 0.306 to 0.859; with both hands missing, accuracy rose from 53.13% to 73.28% and Jaccard from 0.466 to 0.680.
- On an MNIST multi-modal sanity check, ModDrop reduced test error under occlusion: 1 segment covered 2.30% vs 10.74%, 2 segments covered 7.19% vs 35.91%, and 3 segments covered 24.88% vs 68.03%.
- Adding audio improved Jaccard by 1.3 points, and learned audio representations outperformed a phoneme-recognition baseline (F-measure 79.71% vs 56.50%; Jaccard 0.545 vs 0.256).
Methods
- A multi-scale, multi-modal convolutional architecture uses parallel single-scale paths over dynamic poses of 5 frames sampled with a given temporal stride, integrating depth and intensity video, articulated pose (183-dimensional per frame; 915-dimensional dynamic pose), and optionally audio.
- Training proceeds by per-modality pretraining, initialization of shared fusion layers to a normalized geometric mean of per-model outputs, and gradual fusion with ModDrop, which randomly drops separate modality channels during training.
- Experiments use the ChaLearn 2014 Looking at People Challenge track 3 dataset (13,858 gesture instances, 20 gesture categories), augmented with audio from the ChaLearn 2013 Multi-modal Challenge, plus MNIST sanity checks; performance is measured mainly by Jaccard index.
Limitations
- Stated: audio alone does not allow robust gesture start/end localization because of natural gesture-speech desynchronization; annotations are video-based while spoken words/phrases are typically shorter, and vocalization may be delayed or ahead of the gesture, yielding poor audio-only Jaccard scores.
- Evident from the text: evaluation centers on the ChaLearn 2014 LAP dataset plus MNIST sanity checks and ChaLearn 2013 audio augmentation; hyperparameters and ModDrop/dropout rates were optimized on validation data, and hidden-unit dropout degraded performance due to the model’s compactness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
5 Valid · 4 Not valid · 13 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward [51].”
What the sources say
- Partly supportedcites [51] [51] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, 'Maxout Networks,' in arXiv:1302.4389v4 , 2013. - sourceExperimental — accuracy not yet measured
Maxout Networks (p. 2) says it is not obvious how to compute the arithmetic mean of predictions from the exponentially many models trained by dropout. Section 7 (pp. 6–7) discusses approximate model averaging in deep networks, but does not address early fusion or establish the claim specifically for non-linear shared layers. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
What the sources say
- Partly supportedcites [53] [53] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, 'Improving neural networks by preventing coadaptation of feature detectors,' in arXiv:1207.0580 , 2012. - sourceExperimental — accuracy not yet measured
Appendix A.1 (p. 8) lists the 784-1200-1200-10 MNIST architecture and specifies 50% dropout for hidden units and 20% for input units. The main text (p. 2) reports about 110 test errors with those dropout rates, but our copy does not establish exactly 107 errors for that architecture or specify tanh activations. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
What the sources say
- Supportedcites [52] [52] P. Baldi and P. Sadowski, 'The dropout learning algorithm,' Journal of Artificial Intelligence , vol. 210, pp. 78-122, 2014. - source
Section 11.4 gives an approximate dropout regularization term proportional to the sigmoid derivative and the squared weights. It explicitly says the derivative adaptively scales the magnitude of that term. The result is presented as an approximation for a sigmoidal unit. (LLM, conf=0.88)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From our observations, inter-modality fusion is effective at early stages if both channels are of the same nature and convey complementary information.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams.”
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
“The deep learning method described in this paper placed first in the 2014 version of this competition [14].”
plainn = 2,014- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
46 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“The practitioner should be careful when determining whether or not 2 modalities are correlated under the warning that choosing a fusion strategy blindly for 2 essentially different modalities at an early stage has a high risk of learning fake cross-modality correlations and interactions (Neverova et al. 2014).”
From A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding (2024) · cites this paper as Neverova et al. 2014
The cited source says what the claim attributes to it.
In the citing paper
“Studies have also shown that fusing highly correlated features in earlier layers and less correlated features in deeper layers improve model performance50,51.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > Effect of ModDrop on ChaLearn 2014+audio. · p. 13
“In all cases, dropout (20%) and ModDrop (10%) are applied to the input signal.”
§ of parameters > Effect of ModDrop training under occlusion and noise. · p. 12
“| no | no | no | no | 91 . 94 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ of parameters > 8 CONCLUSION · p. 13
“Each of the visual modalities captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at two temporal scales.”
§ 3 GESTURE CLASSIFICATION · p. 3
“Fig. 2. The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Research works printed but not linked: 2
- [60] A. Lee, T. Kawahara, and K. Shikano, 'Julius - an open source realtime large vocabulary recognition engine,' in Interspeech , 2001. -
- [65] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' in Proceedings of the IEEE , vol. 86(11), 1998, pp. 2278-2324. <!-- image --> Natalia Neverova is a PhD student at INSA de Lyon and LIRIS (CNRS, France) working in the area of gesture and action recognition with emphasis on multi-modal aspects and deep learning methods. She is advised by Christian Wolf and Graham Taylor, and her research is a part of Interabot project in partnership with Awabot SAS. She was a visiting researcher at University of Guelph in 2014 and at Google in 2015. <!-- image --> 0 <!-- image --> <!-- image --> Christian Wolf is assistant professor at INSA de Lyon and LIRIS, CNRS, since 2005. He is interested in computer vision and machine learning, especially the visual analysis of complex scenes in motion, structured models, graphical models and deep learning. He received his MSc in computer science from Vienna University of Technology in 2000, and a PhD from the National Institute of Applied Science (INSA de Lyon), in 2003. In 2012 he obtained the habilitation diploma. Graham Taylor is an assistant professor at University of Guelph. He is interested in statistical machine learning and biologicallyinspired computer vision, with an emphasis on unsupervised learning and time series analysis. He completed his PhD at the University of Toronto in 2009, where his thesis co-advisors were Geoffrey Hinton and Sam Roweis. He did a postdoc at NYU with Chris Bregler, Rob Fergus, and Yann LeCun. Florian Nebout is a project manager at Awabot, leading different robotics projects, which are mostly dedicated to remote presence applications. He is especially interested in human robot interaction and how robotics can change our daily life. He graduated from INSA de Lyon, France and worked at CSIRO Human Computer Interaction team and ANU Information and Human Centred Computing Research Group in Canberra, Australia. · DOI · Paper page
Cited by
Items 101–200 of 418
A Blockchain-Powered Decentralized and Secure Computing Paradigm
Mendis, G. J., Wu, Y., Wei, J., Sabounchi, M., & Roche, R. (2020) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2020.2983007 · Cited by 62
Isolated Sign Language Recognition with Multi-scale Features using LSTM
Sincan, Ö. M., Tur, A. O., & Keleş, H. Y. (2019) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu.2019.8806467 · Cited by 39
Improved Lite Audio-Visual Speech Enhancement
Chuang, S.-Y., Wang, H., & Tsao, Y. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153265 · Cited by 57
Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
Maheshwari, H., Liu, Y.-C., & Kira, Z. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00106 · Cited by 37
A naïve Bayes baseline for early gesture recognition
Escalante, H. J., Morales, E. F., & Sucar, L. E. (2016) · Pattern Recognition Letters · doi:10.1016/j.patrec.2016.01.013 · Cited by 38
ChaLearn looking at people: A review of events and resources
Escalera, S., Baró, X., Escalante, H. J., & Guyon, I. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966041 · Cited by 38
Multi-modal metric learning for vehicle re-identification in traffic surveillance environment
Tang, Y., Wu, D., Jin, Z. J., Zou, W., & Li, X. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296683 · Cited by 39
Multi-Modal Cross Learning for Improved People Counting using Short-Range FMCW Radar
Aydogdu, C. Y., Hazra, S., Santra, A., et al. (2020) · IEEE International Radar Conference · doi:10.1109/radar42522.2020.9114871 · Cited by 35
Multilevel Sensor Fusion With Deep Learning
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2018) · IEEE Sensors Letters · doi:10.1109/lsens.2018.2878908 · Cited by 38
Visual Scene-aware Hybrid Neural Network Architecture for Video-based Facial Expression Recognition
Lee, M. K., Choi, D. Y., Kim, D. H., & Song, B. C. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756551 · Cited by 45
Voskou, A., Panousis, K. P., Kosmopoulos, D., Metaxas, D., & Chatzis, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01173 · Cited by 43
Hierarchical attention-based multimodal fusion for video captioning
Wu, C., Wei, Y., Chu, X., Sun, W., Su, F., & Wang, L. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.029 · Cited by 38
Learning to Plan with Uncertain Topological Maps
Beeching, E., Dibangoye, J., Simonin, O., & Wolf, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_28 · Cited by 58
Self-citationAlfasly, S., Lu, J., Xu, C., & Zou, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01957 · Cited by 47
Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from VideosPreprint
Escalante, H. J., Kaya, H., Salah, A. A., Escalera, S., Güçlütürk, Y., Güçlü, U., Baró, X., Guyon, I., Jacques, J. C. S., Madadi, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00745 · Cited by 49
Multimodal Distillation for Egocentric Action Recognition
Radevski, G., Grujicic, D., Blaschko, M. B., Moens, M., & Tuytelaars, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00481 · Cited by 62
Alshamsi, H., Këpuska, V., Alshamsi, H., & Meng, H. (2018) · IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) · doi:10.1109/iemcon.2018.8614831 · Cited by 33
Perspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies
Liu, W., Chen, J., Wang, H., Fu, Z., Peijnenburg, W. J. G. M., & Hong, H. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.4c03088 · Cited by 54
Liu, X., & Zhao, G. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3003783 · Cited by 37
Ramachandram, D., Lisicki, M., Shields, T. J., Amer, M. R., & Taylor, G. W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.11.071 · Cited by 39
Self-citationLi, Y., Wang, B., Shao, G., & Shao, S. (2020) · IEEE Access · doi:10.1109/access.2020.3046124 · Cited by 31
Guarrasi, V., & Soda, P. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106625 · Cited by 28
Skeleton Aware Multi-modal Sign Language RecognitionPreprint
Jiang, S., Sun, B., Wang, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08833 · Cited by 28
Pande, S., Banerjee, A., Kumar, S., Banerjee, B., & Chaudhuri, S. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00558 · Cited by 29
Multimodal Fusion Remote Sensing Image–Audio Retrieval
Yang, R., Wang, S., Sun, Y., Zhang, H., Liao, Y., Gu, Y., Hou, B., & Jiao, L. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3194076 · Cited by 31
3D Hand Gesture Recognition by Analysing Set-of-Joints Trajectories
de Smedt, Q., Wannous, H., & Vandeborre, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-91863-1_7 · Cited by 28
Sarma, D., Kavyasree, V., & Bhuyan, M. K. (2022) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-022-00477-z · Cited by 28
Literature review of vision‐based dynamic gesture recognition using deep learning techniques
Jain, R., Karsh, R. K., & Barbhuiya, A. A. (2022) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.7159 · Cited by 30
Learning wave fields evolution in North West Pacific with deep neural networks
Gao, Z., Liu, X., Yv, F., Wang, J., & Xing, C. (2022) · Applied Ocean Research · doi:10.1016/j.apor.2022.103393 · Cited by 26
Fu, Z., Liu, F., Wang, H., Qi, J., Fu, X., Zhou, A., & Li, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02172 · Cited by 49
Multi-Modal Residual Perceptron Network for Audio–Video Emotion Recognition
Chang, X., & Skarbek, W. (2021) · Sensors · doi:10.3390/s21165452 · Cited by 31
On-the-Fly Modulation for Balanced Multimodal Learning
Wei, Y., Hu, D., Du, H., & Wen, J.-R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3468315 · Cited by 46
SCNET: A Novel UGI Cancer Screening Framework Based on Semantic-Level Multimodal Data Fusion
Ding, S., Huang, H. M., Li, Z., Liu, X., & Yang, S. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2983126 · Cited by 30
Liu, H., Fan, Y., Li, H., Wang, J., Hu, D., Cui, C., Lee, H. H., Zhang, H., & Oğuz, İ. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_43 · Cited by 32
Angle based hand gesture recognition using graph convolutional network
Aiman, U., & Ahmad, T. (2023) · Computer Animation and Virtual Worlds · doi:10.1002/cav.2207 · Cited by 26
Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM
Do, N.-T., Kim, S.-H., Yang, H.-J., & Lee, G.-S. (2020) · Applied Sciences · doi:10.3390/app10186293 · Cited by 25
Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion
El-Ghaish, H., Hussien, M. E., Shoukry, A., & Onai, R. (2018) · IEEE Access · doi:10.1109/access.2018.2868319 · Cited by 26
Language-Guided Music Recommendation for Video via Prompt Analogies
McKee, D., Salamon, J., Šivic, J., & Russell, B. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01420 · Cited by 45
TACFN: Transformer-Based Adaptive Cross-Modal Fusion Network for Multimodal Emotion Recognition
Liu, F., Fu, Z., Wang, Y., & Zheng, Q. (2023) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150019 · Cited by 50
Research on basketball players’ action recognition based on interactive system and machine learning
Jin, L., & Gu, D. (2020) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-189205 · Cited by 28
Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation
Hosseini, B., Montagné, R., & Hammer, B. (2019) · Data Science and Engineering · doi:10.1007/s41019-020-00123-3 · Cited by 28
Bastida, J. O., Gallego, A., Rico-Juan, J. R., & Albarrán, P. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107693 · Cited by 27
Zhai, B., Guan, Y., Catt, M., & Plötz, T. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3494961 · Cited by 25
Real-Time Human-UAV Interaction: New Dataset and Two Novel Gesture-Based Interacting Systems
Kassab, M. A., Ahmed, M., Maher, A., & Zhang, B. (2020) · IEEE Access · doi:10.1109/access.2020.3033157 · Cited by 21
Improving Unimodal Object Recognition with Multimodal Contrastive Learning
Meyer, J., Eitel, A., Brox, T., & Burgard, W. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341029 · Cited by 26
Learning Adaptive Hidden Layers for Mobile Gesture Recognition
Hu, T.-K., Lin, Y., & Hsiu, P.-C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12279 · Cited by 20
Dyhand: dynamic hand gesture recognition using BiLSTM and soft attention methods
Singh, R. P., & Singh, L. D. (2024) · The Visual Computer · doi:10.1007/s00371-024-03307-4 · Cited by 25
Interweaving deep learning and semantic techniques for emotion analysis in human-machine interaction
Kollias, D., Marandianos, G., Raouzaiou, A. T., & Stafylopatis, A. (2015) · International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP) · doi:10.1109/smap.2015.7370086 · Cited by 21
Human‐action recognition using a multi‐layered fusion scheme of Kinect modalities
Seddik, B., Gazzah, S., & Amara, N. E. B. (2017) · IET Computer Vision · doi:10.1049/iet-cvi.2016.0326 · Cited by 17
Audio-Visual Kinship Verification in the Wild
Wu, X., Granger, É., Kinnunen, T., Feng, X., & Hadid, A. (2019) · International Conference on Biometrics · doi:10.1109/icb45273.2019.8987241 · Cited by 21
Altmann, M., Ott, P., Stache, N. C., & Waldschmidt, C. (2021) · IEEE Access · doi:10.1109/access.2021.3056878 · Cited by 19
Evaluation of hidden Markov models using deep CNN features in isolated sign recognition
Tur, A. O., & Keleş, H. Y. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10593-w · Cited by 17
DeSIRe: Deep Signer-Invariant Representations for Sign Language Recognition
Ferreira, P. M., Pernes, D., Rebelo, A., & Cardoso, J. S. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2957347 · Cited by 18
Multi-View Discriminant Framework for Automatic Modulation Open Set Recognition
Hou, J., Xu, D., Song, F., Chen, Z., Xuan, Q., Zheng, S., Lin, Y., & Yang, X. (2024) · IEEE Transactions on Communications · doi:10.1109/tcomm.2024.3506934 · Cited by 18
mXception and dynamic image for hand gesture recognition
Karsh, B., Laskar, R. H., & Karsh, R. K. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09509-0 · Cited by 16
Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond
Li, J., Li, C., Wu, Y., & Qian, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3375641 · Cited by 38
Chen, G., Chen, J., Lienen, M., Conradt, J., Röhrbein, F., & Knoll, A. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00073 · Cited by 19
Skeleton Based Dynamic Hand Gesture Recognition using LSTM and CNN
Ikram, A., & Liu, Y. (2020) · International Conference on Image Processing and Machine Vision (IPMV) · doi:10.1145/3421558.3421568 · Cited by 16
A unified representation network for segmentation with missing modalitiesPreprint
Lau, K., Adler, J., & Sjölund, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06683 · Cited by 33
EMG-Based Automatic Gesture Recognition Using Lipschitz-Regularized Neural Networks
Neacşu, A., Pesquet, J.-C., & Burileanu, C. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3635159 · Cited by 15
Are mid-air dynamic gestures applicable to user identification?
Liu, H., Dai, L.-L., Hou, S., Han, J., & Liu, H. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.026 · Cited by 16
Ge, Y., Li, B., Yan, W., et al. (2018) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci.2018.8377532 · Cited by 15
Progression Modelling for Online and Early Gesture Detection
Gupta, V., Dwivedi, S. K., Dabral, R., & Jain, A. (2019) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2019.00040 · Cited by 23
Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces
Gomaa, A. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3557034 · Cited by 21
UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion
Chen, G., Lin, J., & Qin, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3254650 · Cited by 14
Ousmane, A. M., Djara, T., Walis, F. J. Z., & Vianou, A. C. (2019) · International Conference on Bio-engineering for Smart Technologies (BioSMART) · doi:10.1109/biosmart.2019.8734245 · Cited by 17
Scanning, attention, and reasoning multimodal content for sentiment analysis
Liu, Y., Li, Z., Zhou, K., Zhang, L., Li, L., Tian, P., & Shen, S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110467 · Cited by 16
El-Din, D. M., Hassanein, A. E., & Hassanien, E. E. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02150-2 · Cited by 19
Constructivism Learning: A Learning Paradigm for Transparent Predictive Analytics
Li, X., & Huan, J. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3097994 · Cited by 18
Hand Gesture Recognition: A Survey
Yashas, J., & Shivakumar, G. (2019) · International Conference on Applied Machine Learning (ICAML) · doi:10.1109/icaml48257.2019.00009 · Cited by 14
Adaptive cross-fusion learning for multi-modal gesture recognition
Zhou, B., Wan, J., Liang, Y., & Guo, G. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2021.05.003 · Cited by 13
Modout: Learning Multi-Modal Architectures by Stochastic Regularization
Li, F., Neverova, N., Wolf, C., & Taylor, G. W. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.59 · Cited by 14
Self-citationTemporal Hierarchical Dictionary Guided Decoding for Online Gesture Segmentation and Recognition
Chen, H., Liu, X., Shi, J., & Zhao, G. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3028962 · Cited by 19
Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers
Zhang, X., & Yin, L. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667030 · Cited by 13
Deep Multimodal Representation Learning from Temporal DataPreprint
Yang, X., Ramesh, P. S., Chitta, R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03152 · Cited by 12
Daly, K., & Olukoya, O. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107959 · Cited by 19
Modout: Learning to Fuse Modalities via Stochastic Regularization
Li, F., Neverova, N., Wolf, C., & Taylor, G. W. (2016) · Journal of Computational Vision and Imaging Systems · doi:10.15353/vsnl.v2i1.103 · Cited by 11
Self-citationExploring Coherence in Visual Explanations
Alikhani, M., & Stone, M. (2018) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr.2018.00063 · Cited by 14
Gesture Recognition Using Enhanced Depth Motion Map and Static Pose Map
Zhang, Z., Wei, S., Song, Y., & Zhang, Y. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.38 · Cited by 11
Hidden States Exploration for 3D Skeleton-Based Gesture Recognition
Liu, X., Shi, H., Hong, X., Chen, H., Tao, D., & Zhao, G. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00201 · Cited by 13
Classification of human activity using radar and video multimodal learning
de Jong, R. J., de Wit, J. J. M., & Uysal, F. (2021) · IET Radar Sonar & Navigation · doi:10.1049/rsn2.12064 · Cited by 11
Learning deep and compact models for gesture recognition
Mullick, K., & Namboodiri, A. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8297033 · Cited by 12
Estimation of Ideal Binary Mask for Audio-Visual Monaural Speech Enhancement
Balasubramanian, S., Rajavel, R., & Kar, A. (2023) · Circuits Systems and Signal Processing · doi:10.1007/s00034-023-02340-3 · Cited by 10
Auxiliary audio–textual modalities for better action recognition on vision-specific annotated videos
Alfasly, S., Lu, J. J., Chen, X., Yu, L., & Zou, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110808 · Cited by 12
Chameleon: A Multimodal Learning Framework Robust to Missing Modalities
Liaqat, M. I., Nawaz, S., Zaheer, M. Z., Saeed, M. S., Sajjad, H., Schepper, T. D., Nandakumar, K., Khan, M. H., Gallo, I., & Schedl, M. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00370-y · Cited by 23
CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks
Jiang, R., Han, J., Xue, Y., Wang, P., & Tang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8067-3_12 · Cited by 14
Sarma, D., Kavyasree, V., & Bhuyan, M. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08847 · Cited by 15
Wearable Human–Machine Gesture Interaction Based on Fabric Piezoelectric Sensor
Lu, Y., Li, Z., Wang, X., Jiang, J., Zhu, M., & Xie, M. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3416188 · Cited by 11
Resource-Efficient Federated Multimodal Learning via Layer-Wise and Progressive Training
Tun, Y. L., Thwal, C. M., Nguyen, M. N. H., & Zhang, C. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3554541 · Cited by 12
Two Stage Continuous Gesture Recognition Based on Deep Learning
Wang, H. (2021) · Electronics · doi:10.3390/electronics10050534 · Cited by 10
Multi-Modal Fusion Sign Language Recognition Based on Residual Network and Attention Mechanism
Chu, C., Xiao, Q., Zhang, Y., & Liu, X. (2022) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001422500367 · Cited by 12
Input Dropout for Spatially Aligned Modalities
de Blois, S., Garon, M., Gagné, C., & Lalonde, J. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9190858 · Cited by 14
Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse
Sincan, Ö. M., Camgöz, N. C., & Bowden, R. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00210 · Cited by 24
Chou, J.-C., Chien, C.-M., & Livescu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446625 · Cited by 8
Uni-Modal Event-Agnostic Knowledge Distillation for Multimodal Fake News Detection
Liu, G., Zhang, J., Liu, Q., Wu, J., Wu, S., & Wang, L. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3477977 · Cited by 17
Multimodal Time Series Data Fusion Based on SSAE and LSTM
Zhu, Q., Zhang, S., Zhang, Y., Yu, C., Dang, M., & Zhang, L. (2021) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc49053.2021.9417410 · Cited by 12
Multi-source Knowledge Fusion: A Survey
Zhao, X., Jia, Y., Li, A., Jiang, R., Xie, H., Song, Y., & Han, W. (2019) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc.2019.00026 · Cited by 12
Development of Eye Blink Rate Level Classification System Utilizing Sitting Postural Behavior Data
Lee, H., Yoo, T., Hyun, S., Beck, D., & Park, W. (2021) · IEEE Access · doi:10.1109/access.2021.3121288 · Cited by 8
Context-Aware Automatic Sign Language Video Transcription in Psychiatric Interviews
Pikoulis, E., Bifis, A., Τρίγκα, Μ., Constantinopoulos, C., & Kosmopoulos, D. (2022) · Sensors · doi:10.3390/s22072656 · Cited by 9
Accelerometer-based gesture recognition using dynamic time warping and sparse representation
Wang, H., & Li, Z. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2775-2 · Cited by 8
Metadata
- Authors
- Neverova, N., Wolf, C., Taylor, G., & Nebout, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- University of Guelph (CA)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-08-01
- Volume
- 38
- Issue
- 8
- Pages
- 1692-1706
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W1533025524
- Cited by
- 420
- References
- 77
- Citations by year
- 2026: 89 · 2025: 28 · 2024: 28 · 2023: 35 · 2022: 24 · 2021: 44 · 2020: 41 · 2019: 44 · 2018: 34 · 2017: 34 · 2016: 14
- FWCI
- 55.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Investissement's d’Avenir
- Briques Génériques du Logiciel Embarqué
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- gesture recognition, multimodal fusion, multi-scale deep learning, spatial and temporal scales, missing modality robustness, upper body movement, hand movement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.