Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning
Open accessAbstract
Developing a user interface (UI) suitable for headset environments is one of the challenges in the field of augmented reality (AR) technologies. This study proposes a hands-free UI for an AR headset that exploits facial gestures of the wearer to recognize user intentions. The facial gestures of the headset wearer are detected by a custom-designed sensor that detects skin deformation based on infrared diffusion characteristics of human skin. We designed a deep neural network classifier to determine the user’s intended gestures from skin-deformation data, which are exploited as user input commands for the proposed UI system. The proposed classifier is composed of a spatiotemporal autoencoder and deep embedded clustering algorithm, trained in an unsupervised manner. The UI device was embedded in a commercial AR headset, and several experiments were performed on the online sensor data to verify operation of the device. We achieved implementation of a hands-free UI for an AR headset with average accuracy of 95.4% user-command recognition, as determined through tests by participants.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inertial Measurement Unit (IMU) sensors are now widely deployed in professional sports.
“Athletes have several specific sensor-based applications, and Inertial Measurement Unit (IMU) sensors are now widely deployed in professional sports [15].”
From Review of Wearable Devices and Data Collection Considerations for Connected Health (2021) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
(1967) · Cited by 23,719 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687 · see citing works
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · see citing works
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · see citing works
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409 · see citing works
Unsupervised deep embedding for clustering analysis
Xie, J., Girshick, R., & Farhadi, A. (2015) · arXiv · doi:10.48550/arxiv.1511.06335 · Cited by 2,476 · see citing works
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,975 · see citing works
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
Learning Temporal Regularity in Video Sequences
Hasan, M., Choi, J., Neumann, J., Roy–Chowdhury, A. K., & Davis, L. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.86 · Cited by 1,510 · see citing works
Farrell, T. J., Patterson, M. S., & Wilson, B. C. (1992) · Medical Physics · doi:10.1118/1.596777 · Cited by 1,380 · see citing works
Understanding of Internal Clustering Validation Measures
Liu, Y., Li, Z., Xiong, H., Gao, X., & Wu, J. (2010) · Proceedings · doi:10.1109/icdm.2010.35 · Cited by 1,031 · see citing works
Adaptive Control Processes: A Guided Tour
Morgenstern, D., & Bellman, R. (1962) · Econometrica · doi:10.2307/1909901 · Cited by 1,018 · see citing works
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 666 · see citing works
Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder
Chong, Y. S., & Tay, Y. H. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59081-3_23 · Cited by 613 · see citing works
Brain-computer interfaces for communication and control
McFarland, D. J., & Wolpaw, J. R. (2011) · Communications of the ACM · doi:10.1145/1941487.1941506 · Cited by 523 · see citing works
Light-Emitting Diodes (LEDs) in Dermatology
Barolet, D. (2008) · Seminars in Cutaneous Medicine and Surgery · doi:10.1016/j.sder.2008.08.003 · Cited by 439 · see citing works
On the anatomy and physiology of the skin
Langer, K. (1978) · British Journal of Plastic Surgery · doi:10.1016/0007-1226(78)90003-6 · Cited by 301 · see citing works
Discriminatively boosted image clustering with fully convolutional auto-encoders
Li, F., Qiao, H., & Zhang, B. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.05.019 · Cited by 273 · see citing works
(1998) · doi:10.1093/oso/9780195104462.001.0001 · Cited by 223 · see citing works
Unsupervised Learning of Long-Term Motion Dynamics for Videos
Luo, Z., Peng, B., Huang, D.-A., Alahi, A., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.751 · Cited by 189 · see citing works
Facial performance sensing head-mounted display
Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., & Ma, C. (2015) · ACM Transactions on Graphics · doi:10.1145/2766939 · Cited by 180 · see citing works
Anisotropy of light propagation in human skin
Nickell, S., Hermann, M., Essenpreis, M., Farrell, T. J., Krämer, U., & Patterson, M. S. (2000) · Physics in Medicine and Biology · doi:10.1088/0031-9155/45/10/310 · Cited by 167 · see citing works
(2019) · doi:10.1109/wacv.2019.00178 · Cited by 151 · see citing works
High-fidelity facial and speech animation for VR HMDs
Olszewski, K., Lim, J. J., Saito, S., & Li, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980252 · Cited by 141 · see citing works
Anisotropy of light propagation in biological tissue
Kienle, A., Forster, F. K., & Hibst, R. (2004) · Optics Letters · doi:10.1364/ol.29.002617 · Cited by 123 · see citing works
Scheirer, J., Fernandez, R., & Picard, R. W. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632878 · Cited by 115 · see citing works
Hand-free motion interaction on Google Glass
Lv, Z., Feng, L., Li, H., & Feng, S. (2014) · doi:10.1145/2669062.2669066 · Cited by 93 · see citing works
Facial Expression Recognition in Daily Life by Embedded Photo Reflective Sensors on Smart Eyewear
Masai, K., Sugiura, Y., Ogata, M., Kunze, K., Inami, M., & Sugimoto, M. (2016) · International Conference on Intelligent User Interfaces · doi:10.1145/2856767.2856770 · Cited by 85 · see citing works
Light propagation in dry and wet softwood
Kienle, A., D’Andrea, C., Foschum, F., Taroni, P., & Pifferi, A. (2008) · Optics Express · doi:10.1364/oe.16.009895 · Cited by 77 · see citing works
Learning Temporal Regularity in Video SequencesPreprint
Hasan, M., Choi, J., Neumann, J., Roy–Chowdhury, A. K., & Davis, L. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.04574 · Cited by 66 · see citing works
A model of facial biomechanics for speech production
Lucero, J. C., & Munhall, K. G. (1999) · The Journal of the Acoustical Society of America · doi:10.1121/1.428108 · Cited by 56 · see citing works
Suzuki, K., Nakamura, F., Otsuka, J., Masai, K., Itoh, Y., Sugiura, Y., & Sugimoto, M. (2017) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2017.7892245 · Cited by 51 · see citing works
Kienle, A., Wetzel, C., Bassi, A. L., Comelli, D., Taroni, P., & Pifferi, A. (2007) · Journal of Biomedical Optics · doi:10.1117/1.2709864 · Cited by 48 · see citing works
Unsupervised Learning of Long-Term Motion Dynamics for VideosPreprint
Luo, Z., Peng, B., Huang, D.-A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.01821 · Cited by 31 · see citing works
Hong, I., Bong, K., Shin, D., Park, S., Lee, K. J., Kim, Y., & Yoo, H. (2015) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2015.7063058 · Cited by 17 · see citing works
Hand-free natural user interface for VR HMD with IR based facial gesture tracking sensor
Kim, J., Cha, J., Lee, H., & Kim, S. (2017) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3139131.3143420 · Cited by 16 · see citing works
Self-citationGugenheimer, J., Dobbelstein, D., Winkler, C., Haas, G., & Rukzio, E. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2890242 · Cited by 13 · see citing works
An IR-based facial expression tracking sensor for head-mounted displays
Cha, J., Kim, J., & Kim, S. (2016) · Proceedings of IEEE Sensors/Proceedings of IEEE Sensors ... · doi:10.1109/icsens.2016.7808913 · Cited by 12 · see citing works
Self-citationCha, J., Kim, J., & Kim, S. (2019) · Measurement · doi:10.1016/j.measurement.2019.04.065 · Cited by 9 · see citing works
Self-citationCyberTouch - Touch and Cursor Interface for VR HMD
Lee, S.-H., Ha, G., Cha, J., Kim, J. H., Lee, H., & Kim, S. (2015) · Communications in computer and information science · doi:10.1007/978-3-319-21380-4_85 · Cited by 8 · see citing works
Self-citationFace-Off: A Face Reconstruction Technique for Virtual Reality (VR) Scenarios
Khan, M. S. L., Réhman, S. U., Söderström, U., Halawani, A., & Li, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46604-0_35 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [18]Gugenheimer J. Dobbelstein D. Winkler C. Haas G. Rukzio E. FaceTouch: Touch Interaction for Mobile Virtual Reality Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems San Jose, CA, USA 7–12 May 2016 3679 3682
- [19]Lazar J. Feng J.H. Hochheiser H. Research Methods in Human-Computer Interaction 2nd ed. Morgan Kaufmann Cambridge, MA, USA 2017 9780128093436
- [20]Helander M.G. Landauer T.K. Prabhu P.V. Handbook of Human-Computer Interaction Elsevier Amsterdam, The Netherlands 1997
- [44]Cha J. Kim S. Kim J. A Hands-free Natural User Interface (NUI) for AR/VR Head-Mounted Displays Exploiting Wearer’s Facial Gestures, Demonstration in NeurIPS Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology Gothenburg, Sweden 8–10 November 2018
Cited by
Sorted by citations
Review of Wearable Devices and Data Collection Considerations for Connected Health
Vijayan, V., Connolly, J. P., Condell, J., McKelvey, N., & Gardiner, P. (2021) · Sensors · doi:10.3390/s21165589 · Cited by 435 · see citing works
Deep Learning in Computational Design Synthesis: A Comprehensive Review
Singh, S., Rai, R., Khawale, R. P., Patel, D., Bielecki, D., Nguyen, R., Wang, J., & Zhang, Z. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4064215 · Cited by 22 · see citing works
Zhang, Y., & Doyle, T. E. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1233328 · Cited by 20 · see citing works
Privacy-Preserved In-Cabin Monitoring System for Autonomous Vehicles
Mishra, A., Cha, J., & Kim, S. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/5389359 · Cited by 11 · see citing works
Self-citationShen, X., Yan, Y., Yu, C., & Shi, Y. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3550327 · Cited by 11 · see citing works
Difference-attention graph convolutional network for skeleton-based gesture recognition
Wang, Y., Yu, M., Li, N., Gao, E., Wang, G., & Zhu, L. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00385-9 · Cited by 4 · see citing works
Hands-Free User Interface for VR Headsets Based on In Situ Facial Gesture Sensing
Kim, J., Cha, J., & Kim, S. (2020) · Sensors · doi:10.3390/s20247206 · Cited by 4 · see citing works
Self-citationAI and Personal Data in the Metaverse: A Risk Assessment
Simoes, S., del Rincón, J. M., & McWilliams, G. (2024) · IEEE Smart World Congress (SWC) · doi:10.1109/swc62898.2024.00340 · Cited by 3 · see citing works
TegakARi: Augmenting Creative Drawing With Audio and Visual Cues
Iyer, D., Uhde, A., Karlgren, K., & Kuzuoka, H. (2024) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3726986.3727899
Automatic learning of next generation human-computer interactions
Debard, Q. (2020) · theses.fr (ABES) · doi:10.70675/8aa26513zedabz4492zbfacz4c4613e74be6
Wan, F. C., & Teng, J. (2026) · Computer Animation and Virtual Worlds · doi:10.1002/cav.70164
AI_MDD Approaches for Generating Adaptive and Intelligent UIs: A Systematic Literature Review
Aljabri, A., Hijazi, H., Salamin, S., & Khemaja, M. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p1620
- Authors
- Cha, J., Kim, J., & Kim, S.
- Affiliations
- Yonsei University (KR)
- Corresponding author
- Kim, Shiho
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2019-10-14
- Volume
- 19
- Issue
- 20
- Pages
- 4441
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC6832972
- OpenAlex
- W2979298412
- Cited by
- 12
- References
- 45
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 1 · 2020: 2
- FWCI
- 1.18
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Information and Communications Technology Promotion (2017-0-00244)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Augmented Reality Applications
- Keywords
- augmented reality headsets, facial gesture recognition, deep embedded clustering, unsupervised deep learning, user intention recognition
- MeSH terms
- Virtual Reality, Deep Learning, Algorithms, Face, Gestures, Humans, Skin, User-Computer Interface, Cluster Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
