Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training
Open accessAbstract
We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Abavisani and colleagues developed a fusion approach that derives knowledge from multiple modalities within individual unimodal 3D CNN networks.
“Likewise, Abavisani et al. [200] developed a fusion approach to derive knowledge from multiple modalities in individual unimodal 3D CNN networks.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [200]
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2× · see citing works
“agent by repurposing the learned features or transferring them to improve its learning on a target task [28,37]. This technique has been shown to be successful in many different types of applications [2,27,16,14,43]. While our method is closely related to transfer learning, our learning agents (i.e. modality networks)…”
MethodHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Deep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757 · see citing works
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 3,711 · see citing works
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00675 · Cited by 3,672 · see citing works
Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · see citing works
Two-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45103-x_50 · Cited by 3,010 · see citing works
(2011) · Cited by 2,752 · see citing works
Torrey, L. A., & SHAVLIK, J. W. (2010) · IGI Global eBooks · doi:10.4018/978-1-60566-766-9.ch011 · Cited by 1,887 · see citing works
Vision based hand gesture recognition for human computer interaction: a survey
Rautaray, S. S., & Agrawal, A. (2012) · Artificial Intelligence Review · doi:10.1007/s10462-012-9356-9 · Cited by 1,535 · see citing works
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 2× · see citing works
“A multimodal recognition system is trained with multiple streams of data and classifies the multimodal observations during testing [35, 4] (Figure 1 (a)).”
BackgroundMethodDeep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 885 · see citing works
Molchanov, P. A., Yang, X., Gupta, S., Kim, K., Tyree, S., & Kautz, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.456 · Cited by 698 · see citing works
Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers
Huang, J.-T., Li, J., Yu, D., Deng, L., & Gong, Y. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639081 · Cited by 612 · see citing works
Hand gesture recognition with 3D convolutional neural networks
Molchanov, P. A., Gupta, S., Kim, K., & Kautz, J. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301342 · Cited by 548 · see citing works
Robust 3D Action Recognition with Random Occupancy Patterns
Wang, J., Liu, Z., Chorowski, J., Chen, Z., & Wu, Y. N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_62 · Cited by 471 · see citing works
Ohn-Bar, E., & Trivedi, M. M. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2337331 · Cited by 454 · see citing works
Correlation Alignment for Unsupervised Domain Adaptation
Sun, B., Feng, J., & Saenko, K. (2017) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-58347-1_8 · Cited by 423 · see citing works
EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition
Zhang, Y., Cao, C., Cheng, J., & Lu, H. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2808769 · Cited by 377 · see citing works
SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition
Camgöz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.332 · Cited by 373 · see citing works
Cui, R., Liu, H., & Zhang, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.175 · Cited by 347 · see citing works
A Robust and Efficient Video Representation for Action Recognition
Wang, H., Oneaţă, D., Verbeek, J., & Schmid, C. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0846-5 · Cited by 344 · see citing works
Touch-less interactive augmented reality game on vision-based wearable device
Lv, Z., Halawani, A., Feng, S., Réhman, S. U., & Li, H. (2015) · Personal and Ubiquitous Computing · doi:10.1007/s00779-015-0844-1 · Cited by 296 · see citing works
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM
Zhu, G., Zhang, L., Shen, P., & Song, J. (2017) · IEEE Access · doi:10.1109/access.2017.2684186 · Cited by 269 · see citing works
Multi-sensor system for driver's hand-gesture recognition
Molchanov, P. A., Gupta, S., Kim, K., & Pulli, K. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163132 · Cited by 243 · see citing works
Learning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition
Zhang, L., Zhu, G., Shen, P., Song, J., Shah, S. A. A., & Bennamoun, M. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.369 · Cited by 241 · see citing works
Multi-scale Deep Learning for Gesture Detection and Localization
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_33 · Cited by 239 · see citing works
Deep Multimodal Subspace Clustering Networks
Abavisani, M., & Patel, V. M. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2875385 · Cited by 184 · see citing works
Self-citationMultimodal Gesture Recognition Based on the ResC3D Network
Miao, Q., Li, Y., Ouyang, W., Ma, Z., Xu, X., Shi, W., & Cao, X. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.360 · Cited by 168 · see citing works
Tamrakar, A., Ali, S., Yu, Q., Liu, J., Javed, O., Divakaran, A., Cheng, H., & Sawhney, H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248114 · Cited by 115 · see citing works
Cao, C., Zhang, Y., Wu, Y., Lu, H., & Cheng, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.406 · Cited by 104 · see citing works
Deep Transfer Learning for Multiple Class Novelty Detection
Perera, P., & Patel, V. M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01181 · Cited by 101 · see citing works
Self-citationMultimodal sparse and low-rank subspace clustering
Abavisani, M., & Patel, V. M. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.05.002 · Cited by 95 · see citing works
Self-citationUsing Convolutional 3D Neural Networks for User-independent continuous gesture recognition
Camgöz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899606 · Cited by 92 · see citing works
Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields
Shen, X., Hua, G., Williams, L., et al. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.11.003 · Cited by 83 · see citing works
Large-Scale Multimodal Gesture Recognition Using Heterogeneous Networks
Wang, H., Wang, P., Song, Z., & Li, W. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.370 · Cited by 67 · see citing works
LPSNet: A Novel Log Path Signature Feature Based Hand Gesture Recognition Framework
Li, C., Zhang, X., & Jin, L. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.80 · Cited by 45 · see citing works
Deep CNN-based Multi-task Learning for Open-Set RecognitionPreprint
Oza, P., & Patel, V. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03161 · Cited by 33 · see citing works
Self-citationIn2I: Unsupervised Multi-Image-to-Image Translation Using Generative Adversarial Networks
Perera, P., Abavisani, M., & Patel, V. M. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545464 · Cited by 32 · see citing works
Self-citationMulti-Modal Distance Metric Learning: ABayesian Non-parametric Approach
Babagholami-Mohamadabadi, B., Roostaiyan, S. M., Zarghami, A., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16199-0_5 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., Wang, F., Zhang, D., Lei, Z., Li, H., & Jin, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01952 · Cited by 53 · see citing works
A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., & Wang, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274783 · Cited by 38 · see citing works
- Authors
- Abavisani, M., Joze, H. R. V., & Patel, V. M.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Microsoft (United States) (US)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1165-1174
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2904106524
- Cited by
- 190
- References
- 51
- Citations by year
- 2026: 7 · 2025: 22 · 2024: 31 · 2023: 36 · 2022: 31 · 2021: 36 · 2020: 23 · 2019: 3 · 2018: 1
- FWCI
- 18.77
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- dynamic hand gesture recognition, multimodal training, knowledge transfer, 3D convolutional neural networks, cross-modal feature alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
