Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch

Abstract

Smartwatch is becoming one of the most popular wearable device with many major smartphone manufacturers such as Samsung and Apple releasing their smartwatches recently. Apart from the fitness applications, the smartwatch provides a rich user interface that has enabled many applications like instant messaging and email. Since the smartwatch is worn on the wrist, it introduces a unique opportunity to understand user's arm, hand and possibly finger movements using its accelerometer and gyroscope sensors. Although user's arm and hand gestures are likely to be identified with ease using the smartwatch sensors, it is not clear how much of user's finger gestures can be recognized. In this paper, we show that motion energy measured at the smartwatch is sufficient to uniquely identify user's hand and finger gestures. We identify essential features of accelerometer and gyroscope data that reflect the movements of tendons (passing through the wrist) when performing a finger or a hand gesture. With these features, we build a classifier that can uniquely identify 37 (13 finger, 14 hand and 10 arm) gestures with an accuracy of 98\%. We further extend our gesture recognition to identify the characters written by the user with her index finger on a surface, and show that such finger-writing can also be accurately recognized with nearly 95% accuracy. Our presented results will enable many novel applications like remote control and finger-writing-based input to devices using smartwatch.