Real-Time Gesture Recognition Using 3D Sensory Data and a Light Convolutional Neural Network
Abstract
In this work, we propose an end-to-end system that provides both hardware and software support for real-time gesture recognition. We apply a convolutional neural network over 3D rotation data of finger joints rather than over vision-based data, in order to extract high-level intentions (features) users are trying to convey. A pair of customized motion capturing gloves are designed with inertial measurement unit (IMU) sensors to obtain gestural datasets for network training and real-time recognition. A network reduction strategy has been developed to appropriately reduce a network's complexity in both depth and width dimensions while maintaining a high recognition accuracy with the classification model produced by the network. The classification model is able to classify new data samples by scanning a real-time stream of joint rotations during the use of the gloves. Our evaluation results expose the relationships between the network reduction hyperparameters and the change of recognition accuracy. Based on the evaluation, we are able to determine an appropriate version of the light network and achieve 98% accuracy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data gloves with integrated IMU sensors reached a speed of 8.94 milliseconds per frame and an accuracy of 98% with 0.1 s of predicting.
“With 0.1 s of predicting, the system reached a speed of 8.94 milliseconds per frame and an accuracy of 98% [183].”
From Review of Wearable Devices and Data Collection Considerations for Connected Health (2021) · cites this paper as [183]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
The Effectiveness of Data Augmentation in Image Classification using Deep Learning
Perez, L., & Wang, J. (2017) · arXiv · doi:10.48550/arxiv.1712.04621 · Cited by 2,499 · see citing works
Grammar, Gesture, and Meaning in American Sign Language
Liddell, S. K. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511615054 · Cited by 1,390 · see citing works
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278 · see citing works
Gesture recognition with a Wii controller
Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/1347390.1347395 · Cited by 463 · see citing works
Hand gesture recognition with leap motion and kinect devices
Marin, G., Dominio, F., & Zanuttigh, P. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025313 · Cited by 447 · see citing works
2019 The 19th International Conference on Control, Automation and Systems
(2019) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas47443.2019.8971658 · Cited by 380 · see citing works
Data Augmentation for Deep Neural Network Acoustic Modeling
Cui, X., Goel, V., & Kingsbury, B. (2015) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2015.2438544 · Cited by 376 · see citing works
18th International Conference on Pattern Recognition (ICPR'06)
(2006) · doi:10.1109/icpr11944.2006 · Cited by 280 · see citing works
Dynamic Hand Gesture Recognition With Leap Motion Controller
Lu, W., Zheng, T., & Chu, J. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2590470 · Cited by 237 · see citing works
Lang, C. E., & Schieber, M. H. (2004) · Journal of Neurophysiology · doi:10.1152/jn.00480.2004 · Cited by 232 · see citing works
Hand gesture recognition with jointly calibrated Leap Motion and depth sensor
Marin, G., Dominio, F., & Zanuttigh, P. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2451-6 · Cited by 220 · see citing works
Evaluation of the CyberGlove as a whole-hand input device
Kessler, G. D., Hodges, L. F., & Walker, N. (1995) · ACM Transactions on Computer-Human Interaction · doi:10.1145/212430.212431 · Cited by 191 · see citing works
(2005) · ePrints Soton (University of Southampton) · Cited by 170 · see citing works
Continuous body and hand gesture recognition for natural human-computer interaction
Song, Y., Demirdjian, D., & Davis, R. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2133366.2133371 · Cited by 149 · see citing works
Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing
Georgi, M., Amma, C., & Schultz, T. (2015) · International Conference on Bio-inspired Systems and Signal Processing · doi:10.5220/0005276900990108 · Cited by 135 · see citing works
Xu, D. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.109 · Cited by 117 · see citing works
Navigation using inertial sensors [Tutorial]
Groves, P. D. (2015) · IEEE Aerospace and Electronic Systems Magazine · doi:10.1109/maes.2014.130191 · Cited by 114 · see citing works
A soft stretchable bending sensor and data glove applications
Shen, Z., Yi, J., Li, X., Lo, M. H. P., Chen, M. Z. Q., Hu, Y. S., & Wang, Z. (2016) · Robotics and Biomimetics · doi:10.1186/s40638-016-0051-1 · Cited by 97 · see citing works
Dispelling the Gorilla Arm Syndrome: The Viability of Prolonged Gesture Interactions
Hansberger, J. T., Peng, C. Y., Mathis, S. L., Shanthakumar, V. A., Meacham, S., Cao, L., & Blakely, V. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-57987-0_41 · Cited by 79 · see citing works
Self-citationReal-time and continuous hand gesture spotting: An approach based on artificial neural networks
Neto, P., Pereira, D., Pires, J. N., & Moreira, A. P. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630573 · Cited by 74 · see citing works
Mechanical model of flex sensors used to sense finger movements
Saggio, G. (2012) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2012.07.023 · Cited by 73 · see citing works
A sliding window approach to natural hand gesture recognition using a custom data glove
Luzhnica, G., Simon, J., Lex, E., et al. (2016) · IEEE Symposium on 3D User Interfaces (3DUI) · doi:10.1109/3dui.2016.7460035 · Cited by 53 · see citing works
Hand gesture recognition with convolutional neural networks for the multimodal UAV control
Ma, Y., Liu, Y., Jin, R., Yuan, X., Sekha, R., Wilson, S., & Vaidyanathan, R. (2017) · Workshop on Research, Education and Development of Unmanned Aerial Systems (RED-UAS) · doi:10.1109/red-uas.2017.8101666 · Cited by 50 · see citing works
Quaternion-Based Gesture Recognition Using Wireless Wearable Motion Capture Sensors
Alavi, S., Arsenault, D., & Whitehead, A. (2016) · Sensors · doi:10.3390/s16050605 · Cited by 47 · see citing works
MEMS for automotive and consumer electronics
Finkbeiner, S. (2013) · doi:10.1109/esscirc.2013.6649059 · Cited by 41 · see citing works
Data-driven glove calibration for hand motion capture
Wang, Y., & Neff, M. (2013) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/2485895.2485901 · Cited by 37 · see citing works
CPCDN: Content Delivery Powered by Context and User Intelligence
Wang, Z., Zhu, W., Chen, M., et al. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2365364 · Cited by 36 · see citing works
Gesture-recognition with Non-referenced Tracking
Keir, P., Payne, J., Elgoyhen, J., et al. (2006) · doi:10.1109/vr.2006.64 · Cited by 35 · see citing works
The Effect of Flexor Digitorum Profundus Quadriga on Grip Strength
HORTON, T. C., Sauerland, S., & Davis, T. R. C. (2007) · Journal of Hand Surgery (European Volume) · doi:10.1016/j.jhsb.2006.11.005 · Cited by 26 · see citing works
Implementation of Real-Time Static Hand Gesture Recognition Using Artificial Neural Network
Yusnita, L., Rosalina, R., Roestam, R., et al. (2017) · CommIT (Communication and Information Technology) Journal · doi:10.21512/commit.v11i2.2282 · Cited by 25 · see citing works
A Multimodal Interface for Virtual Information Environments
Hansberger, J. T., Peng, C. Y., Blakely, V., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-21607-8_5 · Cited by 17 · see citing works
Self-citationSaggio, G. (2011) · International Symposium on Applied Sciences in Biomedical and Communication Technologies · doi:10.1145/2093698.2093703 · Cited by 16 · see citing works
A Comparative Evaluation of Bend Sensors for Wearable Applications
Dunne, L. E., Smyth, B., & Caulfield, B. M. (2007) · Proceedings · doi:10.1109/iswc.2007.4373797 · Cited by 9 · see citing works
HandMagic: Towards user interaction with inertial measuring units
Calella, J. C., Ortega, F. R., Rishe, N. D., et al. (2016) · Proceedings of IEEE Sensors/Proceedings of IEEE Sensors ... · doi:10.1109/icsens.2016.7808524 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Review of Wearable Devices and Data Collection Considerations for Connected Health
Vijayan, V., Connolly, J. P., Condell, J., McKelvey, N., & Gardiner, P. (2021) · Sensors · doi:10.3390/s21165589 · Cited by 435 · see citing works
- Authors
- Diliberti, N., Peng, C., Kaufman, C., Dong, Y., & Hansberger, J. T.
- Affiliations
- University of Alabama in Huntsville (US)
- Rochester Institute of Technology (US)
- DEVCOM Army Research Laboratory (US)
- Venue
- Proceedings of the 27th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2019-10-15
- Pages
- 401-410
- OpenAlex
- W2981867264
- Cited by
- 26
- References
- 41
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 3 · 2023: 1 · 2022: 3 · 2021: 10 · 2020: 2
- FWCI
- 3.04
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- gesture recognition, real-time recognition, 3D motion data, network compression, network pruning, inertial measurement unit, gesture classification, wearable sensors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
