Gesture recognition with the linear optical sensor and recurrent neural networks
Abstract
In this paper, the optical linear sensor, a representative of low-resolution sensors, was investigated in the multiclass recognition of near-field hand gestures. The recurrent neural network (RNN) with a gated recurrent unit (GRU) memory cell was utilized as a gestures classifier. A set of 27 gestures was collected from a group of volunteers. The 27 000 sequences obtained were divided into training, validation, and test subsets. The primary research goal was to define the most appropriate model architecture in terms of the accurate recognition of each gesture. An additional aim of the research was to investigate the kind of input data, i.e., raw data or preprocessed (feature) data, which generally produces better results. Therefore, three datasets were generated: raw data, simple features data, and high-level features data. (This includes information about hand poses which are already recognized.) The random search method was applied to achieve hyperparameter optimization to find the best possible topology for the neural network. The analysis performed shows that selected models were characterized by a test score at a level of 96.89% for the raw data, 95.75% for simple features, and 93.38% for high-level features. Results indicate that the direct use of raw data obtained from the optical linear sensor evaluated on the RNN with GRU memory cells allows for the reliable recognition of even complex gestures. Therefore, such solutions may have the potential to serve as a support to or as an alternative to video-based sensors especially for mobile devices.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ref. [197] showed that training an RNN with raw data and with simple custom features yielded similar performance for gesture recognition (96.89% vs 93.38%).
“Ref. [197] showed that training an RNN with raw data and with simple custom features yielded similar performance for gesture recognition (96.89% vs 93.38%).”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [197]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
(2012) · Cited by 8,009 · see citing works
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · see citing works
Self-Normalizing Neural NetworksPreprint
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02515 · Cited by 2,031 · see citing works
User-defined gestures for surface computing
Wobbrock, J. O., Morris, M. R., & Wilson, A. D. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518866 · Cited by 1,339 · see citing works
Hyperopt: a Python library for model selection and hyperparameter optimization
Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., & Cox, D. D. (2015) · Computational Science & Discovery · doi:10.1088/1749-4699/8/1/014008 · Cited by 1,002 · see citing works
DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., & Zhang, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11941 · Cited by 779 · see citing works
Visual Recognition of American Sign Language Using Hidden Markov Models.
Starner, T. E. (1995) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 659 · see citing works
Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks
Wang, P., Li, Z., Hou, Y., & Li, W. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967191 · Cited by 391 · see citing works
Gesture recognition using recurrent neural networks
Murakami, K., & Taguchi, H. (1991) · doi:10.1145/108844.108900 · Cited by 359 · see citing works
Tsironi, E., Barros, P. V. A., Weber, C., & Wermter, S. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.088 · Cited by 249 · see citing works
American Sign Language Recognition Using Leap Motion Sensor
Chuan, C., Regina, E., & Guardino, C. (2014) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2014.110 · Cited by 225 · see citing works
Hand gesture recognition using input-output hidden Markov models
Marcel, S., Bernier, O., Viallet, J. E., & Collobert, D. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840674 · Cited by 134 · see citing works
Sign language recognition with long short-term memory
Liu, T., Zhou, W., & Li, H. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532884 · Cited by 130 · see citing works
A Hand Gesture Recognition Sensor Using Reflected Impulses
Kim, S. Y., Han, H. G., Kim, J. W., et al. (2017) · IEEE Sensors Journal · doi:10.1109/jsen.2017.2679220 · Cited by 128 · see citing works
Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition
Gałka, J., Mąsior, M., Zaborski, M., et al. (2016) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2583542 · Cited by 118 · see citing works
Accelerometer-Based Hand Gesture Recognition by Neural Network and Similarity Matching
Xie, R., & Cao, J. (2016) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2546942 · Cited by 117 · see citing works
Recognition of Arabic Sign Language (ArSL) Using Recurrent Neural Networks
Maraqa, M. R., Al-Zboun, F., Dhyabat, M., et al. (2012) · Journal of Intelligent Learning Systems and Applications · doi:10.4236/jilsa.2012.41004 · Cited by 104 · see citing works
Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition
Chai, X., Liu, Z., Yin, F., Liu, Z., & Chen, X. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899603 · Cited by 87 · see citing works
Recognition of Arabic Sign Language (ArSL) using recurrent neural networks
Maraqa, M. R., & Zitar, R. A. (2008) · International Conference on the Applications of Digital Information and Web Technologies (ICADIWT) · doi:10.1109/icadiwt.2008.4664396 · Cited by 76 · see citing works
Real time gesture recognition using continuous time recurrent neural networks
Bailador, G., Roggen, D., Tröster, G., et al. (2007) · doi:10.4108/bodynets.2007.149 · Cited by 71 · see citing works
Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks
Shin, S., & Sung, W. (2016) · 2016 IEEE International Symposium on Circuits and Systems (ISCAS) · doi:10.1109/iscas.2016.7539037 · Cited by 59 · see citing works
Withana, A. I., Peiris, R. L., Samarasekara, N., et al. (2015) · doi:10.1145/2702123.2702371 · Cited by 54 · see citing works
A Multi-scale Approach to Gesture Detection and Recognition
Neverova, N., Wolf, C., Paci, G., Sommavilla, G., Taylor, G. W., & Nebout, F. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.69 · Cited by 52 · see citing works
Constructing finite state machines for fast gesture recognition
Hong, P., Turk, M., & Huang, T. S. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2000.903639 · Cited by 49 · see citing works
Rapid recognition of dynamic hand gestures using leap motion
Chen, Y., Ding, Z., Chen, Y.-L., et al. (2015) · IEEE International Conference on Information and Automation (ICIA) · doi:10.1109/icinfa.2015.7279509 · Cited by 41 · see citing works
Gesture Recognition with a Convolutional Long Short-Term Memory Recurrent Neural Network.
Tsironi, E., Barros, P. V. A., & Wermter, S. (2016) · The European Symposium on Artificial Neural Networks · Cited by 37 · see citing works
Local Feature Based Online Mode Detection with Recurrent Neural Networks
Otte, S., Krechel, D., Liwicki, M., et al. (2012) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2012.229 · Cited by 32 · see citing works
Centroid tracking based dynamic hand gesture recognition using discrete Hidden Markov Models
Premaratne, P., Yang, S., Vial, P. J., et al. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.06.075 · Cited by 29 · see citing works
Real time gesture recognition system using posture classifier and Jordan recurrent neural network
Araga, Y., Shirabayashi, M., Kaida, K., et al. (2012) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252595 · Cited by 24 · see citing works
Gesture recognition via pose classification
Ng, C. W., & Ranganath, S. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2000.903641 · Cited by 23 · see citing works
bukhari, D., Wang, Y., & Wang, H. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.03.179 · Cited by 20 · see citing works
Recognition and anticipation of hand motions using a recurrent neural network
Vamplew, P. W., & Adams, A. H. (2005) · doi:10.1109/icnn.1995.487241 · Cited by 15 · see citing works
Pose classification in the gesture recognition using the linear optical sensor
Czuszyński, K., Rumiński, J., & Wtorek, J. (2017) · Conference on Human System Interactions · doi:10.1109/hsi.2017.8004989 · Cited by 14 · see citing works
Low-power motion gesture sensor with a partially open cavity package
Kim, J. S., Yun, S. J., & Kim, Y. S. (2016) · Optics Express · doi:10.1364/oe.24.010537 · Cited by 9 · see citing works
Action Recognition Based on Joint Trajectory Maps with Convolutional Neural NetworksPreprint
Wang, P., Li, W., Li, C., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.09401 · Cited by 8 · see citing works
Virtual Mouse: A Low Cost Proximity-Based Gestural Pointing Device
Tang, S. K., Tseng, W. C., Luo, W. W., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21605-3_54 · Cited by 7 · see citing works
The Passive Operating Mode of the Linear Optical Gesture Sensor
Czuszyński, K., Rumiński, J., & Wtorek, J. (2018) · Advances in Electrical and Computer Engineering · doi:10.4316/aece.2018.01018 · Cited by 7 · see citing works
Semi complex navigation with an active optical gesture sensor
Czuszyński, K., Rumiński, J., Bujnowski, A., et al. (2016) · doi:10.1145/2968219.2971375 · Cited by 6 · see citing works
A recurrent neural network based Schaeffer gesture recognition system
Oprea, S., García-García, A., García‐Rodríguez, J., et al. (2017) · doi:10.1109/ijcnn.2017.7965885 · Cited by 5 · see citing works
Analysis of Properties of an Active Linear Gesture Sensor
Czuszyński, K., Rumiński, J., & Wtorek, J. (2017) · Metrology and Measurement Systems · doi:10.1515/mms-2017-0052 · Cited by 4 · see citing works
Gesture Recognition System for Isolated Sign Language Signs
Stefanov, K., & Beskow, J. (2016) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625 · see citing works
“[78] showed that training a RNN with raw data and with simple custom features yielded similar performance for gesture recognition (96.”
BackgroundMethodTrustworthy AI Development Guidelines for Human System Interaction
Wickramasinghe, C. S., Marino, D. L., Grandio, J., & Manic, M. (2020) · 2020 13th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi49210.2020.9142644 · Cited by 71 · see citing works
- Authors
- Czuszynski, K., Ruminski, J., & Kwasniewska, A.
- Venue
- IEEE Sensors Journal
- Published
- 2018-07-01
- Volume
- 18
- Issue
- 13
- Pages
- 5429-5438
- ISSN
- 1530-437X (ISSN-L)
- 1558-1748
- 2379-9153
- OpenAlex
- W2802617992
- Cited by
- 36
- References
- 45
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2023: 3 · 2022: 8 · 2021: 3 · 2020: 8 · 2019: 4 · 2018: 4
- FWCI
- 4.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Sensors Journal (publishedVersion)
- Funders
- NCBiR, FWF, SNSF, ANR, and FNR in the framework of ERA-NET CHIST-ERA II through Project eGLASSES—The interactive eyeglasses for mobile, perceptual computing
- Statutory Funds of the Electronics, Telecommunications and Informatics Faculty, Gdansk University of Technology
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- gesture recognition, low-resolution sensors, recurrent neural networks, gated recurrent unit, raw sensor data, feature extraction, hyperparameter optimization, random search, mobile devices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
