When Video meets Inertial Sensors: Zero-shot Domain Adaptation for Finger Motion Analytics with Inertial Sensors
Open accessAbstract
Ubiquitous finger motion tracking enables a number of exciting applications in augmented reality, sports analytics, rehabilitation-healthcare etc. While finger motion tracking with cameras is very mature, largely due to availability of massive training datasets, there is a dearth of training data for developing robust machine learning (ML) models for wearable IoT devices with Inertial Measurement Unit (IMU) sensors. Towards addressing this problem, this paper presents ZeroNet, a system that shows the feasibility of developing ML models for IMU sensors with zero training overhead. ZeroNet harvests training data from publicly available videos for performing inferences on IMU. The difference in data among video and IMU domains introduces a number of challenges due to differences in sensor-camera coordinate systems, body sizes of users, speed/orientation changes during gesturing, sensor position variations etc. ZeroNet addresses these challenges by systematically extracting motion data from videos and transforming them into acceleration and orientation information measured by IMU sensors. Furthermore, data-augmentation techniques are exploited that create synthetic variations in the harvested training data to enhance the generalizability and robustness of the ML models to user diversity. Evaluation with 10 users demonstrates a top-1 accuracy of 82.4% and a top-3 accuracy of 94.8% for recognition of 50 finger gestures thus indicating promise. While we have only scratched the surface, we outline a number of interesting possibilities for extending this work in the cross-disciplinary areas of computer vision, machine learning, and wearable IoT for enabling novel applications in finger motion tracking.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing repositories of videos of human activities can be utilized to generate virtual IMU training data.
“Recently, a paradigm change to data collection for supervised recognizer training has been introduced where existing repositories of videos of human activities are utilized to generate virtual IMU training data [14,15,16,17].”
From Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables (2021) · cites this paper as [16]
Virtual IMU data from videos can be used for American Sign Language (ASL) recognition tasks.
“Recently, Liu et al. [16] demonstrated that virtual IMU data from videos could be used for American Sign Language (ASL) recognition tasks.”
From Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables (2021) · cites this paper as [16]
Systems like IMUTube can be used to successfully generate training data for more complex activities than the free-weight gym exercises studied in this paper.
“The reported preliminary results indicate that systems like IMUTube can be used to successfully generate training data for more complex activities than the ones studied in this paper.”
From Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables (2021) · cites this paper as [16]
Several works extracted 3D motion information from videos and transferred the knowledge to synthesize virtual on-body IMU sensor data.
“Several works extracted 3D motion information from videos and transferred the knowledge to synthesize virtual on-body IMU sensor data [265,266].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [266]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Adversarial Discriminative Domain Adaptation
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.316 · Cited by 5,154 · see citing works
AutoAugment: Learning Augmentation Strategies From Data
Cubuk, E. D., Zoph, B., Mané, D., Vasudevan, V., & Le, Q. V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00020 · Cited by 4,026 · see citing works
Multi-column deep neural networks for image classification
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248110 · Cited by 3,760 · see citing works
Deep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757 · see citing works
(1994) · Knowledge Discovery and Data Mining · Cited by 3,137 · see citing works
Best practices for convolutional neural networks applied to visual document analysis
Simard, P., Steinkraus, D., & Platt, J. C. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2003.1227801 · Cited by 2,805 · see citing works
(2013) · Cited by 2,056 · see citing works
Self-citationIntroducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 1× · see citing works
“In contrast, for wearables, Daphnet [14] gait dataset has 5 hours of walking data from 10 subjects and PAMP2 dataset[54] has 7.”
Whole-home gesture recognition using wireless signals
Pu, Q., Gupta, S., Gollakota, S., & Patel, S. (2013) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/2500423.2500436 · Cited by 1,120 · see citing works
Zero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · see citing works
Wang, H., Sen, S., Elgohary, A., Farid, M., Youssef, M. A., & Choudhury, R. R. (2012) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2307636.2307655 · Cited by 828 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky (2012) · Medical Entomology and Zoology · Cited by 799 · see citing works
Um, T. T., Pfister, F. M. J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., & Kulić, D. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136817 · Cited by 729 · see citing works
Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom
Bachlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J. M., Giladi, N., & Troster, G. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036165 · Cited by 693 · cited 2× · see citing works
“In contrast, for wearables, Daphnet [14] gait dataset has 5 hours of walking data from 10 subjects and PAMP2 dataset[54] has 7.”
InfluentialOpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity FieldsPreprint
Cao, Z., Hidalgo, G., Simon, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08008 · Cited by 672 · see citing works
Um, T. T. (n.d.) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 602 · see citing works
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00013 · Cited by 522 · see citing works
Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally
Zhou, Z., Shin, J., Zhang, L., Gurudu, S., Gotway, M. B., & Liang, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.506 · Cited by 430 · see citing works
MEMS inertial sensors: A tutorial overview
Shaeffer, D. K. (2013) · IEEE Communications Magazine · doi:10.1109/mcom.2013.6495768 · Cited by 376 · see citing works
Emotion recognition using wireless signals
Zhao, M., Adib, F., & Katabi, D. (2016) · Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking · doi:10.1145/2973750.2973762 · Cited by 370 · see citing works
Deep CORAL: Correlation Alignment for Deep Domain AdaptationPreprint
Sun, B., & Saenko, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.01719 · Cited by 361 · see citing works
Huang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., & Pons-Moll, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275108 · Cited by 342 · see citing works
Times-series data augmentation and deep learning for construction equipment activity recognition
Rashid, K. M., & Louis, J. (2019) · Advanced Engineering Informatics · doi:10.1016/j.aei.2019.100944 · Cited by 341 · see citing works
Han, X., Zhong, Y., Cao, L., & Zhang, L. (2017) · Remote Sensing · doi:10.3390/rs9080848 · Cited by 334 · see citing works
Monocular Total Capture: Posing Face, Body, and Hands in the Wild
Xiang, D., Joo, H., & Sheikh, Y. A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01122 · Cited by 332 · see citing works
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P. A., Breuel, T. M., Gall, J., & Kautz, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316 · see citing works
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images
Cai, Y., Ge, L., Cai, J., & Yuan, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_41 · Cited by 289 · see citing works
Parate, A., Chiu, M., Chadowitz, C., Ganesan, D., & Kalogerakis, E. (2014) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2594368.2594379 · Cited by 282 · see citing works
Towards 3D human pose construction using wifi
Jiang, W., Xue, H., Miao, C., Wang, S., Lin, S., Tian, C., Murali, S., Hu, H., Sun, Z. Y., & Su, L. (2020) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3372224.3380900 · Cited by 279 · see citing works
CHI '86 Conference on Human Factors in Computing Systems
(1985) · Computer · doi:10.1109/mc.1985.1662775 · Cited by 272 · see citing works
Dropout Training as Adaptive RegularizationPreprint
Wager, S., Wang, S., & Liang, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1493 · Cited by 260 · see citing works
Ahmed, M. A., Zaidan, B. B., Zaidan, A. A., Salih, M. M., & Lakulu, M. M. (2018) · Sensors · doi:10.3390/s18072208 · Cited by 255 · see citing works
BERT with History Answer Embedding for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Croft, W. B., Zhang, Y., & Iyyer, M. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331341 · Cited by 220 · see citing works
Zhou, P., Li, M., & Shen, G. (2014) · Proceedings of the 28th Annual International Conference on Mobile Computing And Networking · doi:10.1145/2639108.2639110 · Cited by 204 · see citing works
I am a Smartwatch and I can Track my User's Arm
Shen, S., Wang, H., & Choudhury, R. R. (2016) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906407 · Cited by 185 · see citing works
I am a smartphone and i can tell my user's walking direction
Roy, N., Wang, H., & Choudhury, R. R. (2014) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2594368.2594392 · Cited by 184 · see citing works
The Stability of Inverse Problems
Bertero, M., Mol, C. D., & Viano, G. A. (1980) · Topics in current physics · doi:10.1007/978-3-642-81472-3_5 · Cited by 157 · see citing works
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411841 · Cited by 152 · see citing works
Susanto, E. A., Tong, R. K., Ockenfeld, C., & Ho, N. S. K. (2015) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-015-0033-5 · Cited by 144 · see citing works
A Systematic Study of Unsupervised Domain Adaptation for Robust Human-Activity Recognition
Chang, Y., Mathur, A., Isopoussu, A., Song, J., & Kawsar, F. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3380985 · Cited by 140 · see citing works
Learning Local Shape Descriptors from Part Correspondences with Multiview Convolutional Networks
Huang, H., Kalogerakis, E., Chaudhuri, S., Ceylan, D., Kim, V. G., & Yumer, E. (2017) · ACM Transactions on Graphics · doi:10.1145/3137609 · Cited by 139 · see citing works
Deep learning-based smart task assistance in wearable augmented reality
Park, K.-B., Kim, M., Choi, S. H., & Lee, J. Y. (2019) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2019.101887 · Cited by 139 · see citing works
Through-Wall Human Mesh Recovery Using Radio Signals
Zhao, M., Liu, Y., Raghu, A., Zhao, H., Li, T., Torralba, A. B., & Katabi, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01021 · Cited by 132 · see citing works
Self-citationImpairment of individual finger movements in Parkinson's disease
Agostino, R., Currà, A., Giovannelli, M., Modugno, N., Manfredi, M., & Berardelli, A. (2003) · Movement Disorders · doi:10.1002/mds.10313 · Cited by 130 · see citing works
Elastic spectral distortion for low resource speech recognition with deep neural networks
Kanda, N., Takeda, R., & Obuchi, Y. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707748 · Cited by 115 · see citing works
Nirjon, S., Gummeson, J., Gelb, D., & Kim, K.-H. (2015) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742665 · Cited by 113 · see citing works
Classification Of Breast Cancer Histology Images Using ALEXNET
Nawaz, W., Ahmed, S., Tahir, A., & Khan, H. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93000-8_99 · Cited by 111 · see citing works
APAC: Augmented PAttern Classification with Neural NetworksPreprint
Sato, I., Nishimura, H., & Yokoi, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.03229 · Cited by 108 · see citing works
IMUSim: A simulation environment for inertial sensing algorithm design and evaluation
Young, A. D., Ling, M. J., & Arvind, D. K. (2011) · Information Processing in Sensor Networks · Cited by 56 · see citing works
Brain-Computer Interface for a Prosthetic Hand Using Local Machine Control and Haptic Feedback
Ramos‐Murguialday, A., Aggarwal, V., Chatterjee, A., Cho, Y., Rasmussen, R. G., O'Rourke, B., Acharya, S., & Thakor, N. V. (2007) · International Conference on Rehabilitation Robotics · doi:10.1109/icorr.2007.4428487 · Cited by 45 · see citing works
Mariakakis, A. T., Parsi, S., Patel, S., & Wobbrock, J. O. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173808 · Cited by 43 · see citing works
Rey, V. F., Hevesi, P., Kovalenko, O., & Lukowicz, P. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3345590 · Cited by 43 · cited 2× · see citing works
“Although in a similar spirit to recent works [38, 55, 66, 74] showing the feasibility of harvesting training data from videos for identifying upto ten classes of human activities, ZeroNet differs from the above works in following ways: (i) Shows the feasibility of harvesting training data from videos for a gesture rec…”
BackgroundGolf Swing Segmentation from a Single IMU Using Machine Learning
Kim, M., & Park, S. (2020) · Sensors · doi:10.3390/s20164466 · Cited by 43 · see citing works
Patient performance evaluation using Kinect and Monte Carlo-based finger tracking
Cordella, F., Corato, F. D., Zollo, L., Siciliano, B., & van der Smagt, P. (2012) · IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) · doi:10.1109/biorob.2012.6290794 · Cited by 37 · see citing works
Finger Gesture Tracking for Interactive Applications
Liu, Y., Jiang, F., & Gowda, M. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3414117 · Cited by 33 · see citing works
Self-citationAnnapurna: An automated smartwatch-based eating detection and food journaling system
Sen, S., Subbaraju, V., Misra, A., Balan, R. K., & Lee, Y. (2020) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2020.101259 · Cited by 26 · see citing works
Deep generative cross-modal on-body accelerometer data synthesis from videos
Zhang, S., & Alshurafa, N. I. (2020) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3410530.3414329 · Cited by 21 · see citing works
Self-citationTowards a Brain-Computer Interface for Dexterous Control of a Multi-Fingered Prosthetic Hand
Acharya, S., Aggarwal, V., Tenore, F. V., Shin, H., Etienne‐Cummings, R., Schieber, M. H., & Thakor, N. V. (2007) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/cne.2007.369646 · Cited by 19 · see citing works
Zhang, H., Guo, G., Song, C., Xu, C., Cheung, K. Y.-W., Alexis, J., Li, H., Li, D., Wang, K., & Xu, W. (2020) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3372224.3380889 · Cited by 19 · see citing works
A Multi-Sensor Setting Activity Recognition Simulation Tool
Takeda, S., Okita, T., Lago, P., & Inoue, S. (2018) · Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers · doi:10.1145/3267305.3267509 · Cited by 18 · see citing works
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Ploetz, T. (2020) · arXiv · doi:10.48550/arxiv.2006.05675 · Cited by 14 · see citing works
A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable SensorsPreprint
Xiao, F., Pei, L., Chu, L., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01874 · Cited by 8 · see citing works
Unsupervised Domain Adaptation for Human Activity Recognition
Barbosa, P., Garcia, K. D., Mendes‐Moreira, J., & de Carvalho, A. C. P. L. F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03493-1_65 · Cited by 7 · see citing works
Application Informed Motion Signal Processing for Finger Motion Tracking Using Wearable Sensors
Liu, Y., Jiang, F., & Gowda, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053466 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
3 of the 72 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- [28][28] Hesse, S., et al. A new electromechanical trainer for sensorimotor rehabilitation of paralysed fingers: a case series in chronic and acute stroke patients. Journal of neuroengineering and rehabilitation 5 , 1 (2008), 21. -
- [30][30] Ionescu, C., et al. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence 36 , 7 (2013), 1325-1339. -
- [32][32] Jaitly, N., et al. Vocal tract length perturbation improves speech recognition. In Proc. ICML Workshop on Deep Learning for Audio, Speech and Language (2013). -
- [45][45] Nirjon, S., et al. Typingring: A wearable ring platform for text input. In ACM MobiSys (2015). -
- [46][46] Oura ring review. https://www.wareable.com/health-and-wellbeing/oura-ring2018-review-6628, 2018. -
- [48][48] Oura ring review -the early adopter catches the worm. https://www. androidauthority.com/oura-ring-2-review-933935/, 2019. -
- [49][49] Parate, A., et al. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In ACM MobiSys (2014), pp. 149-161. -
- [76][76] Zhang, H., et al. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243 (2019). - · arXiv
- [77][77] Zhang, H., et al. Pdlens: smartphone knows drug effectiveness among parkinson's via daily-life activity fusion. In ACM MobiCom (2020). -
- [80][80] Zhao, M., et al. Learning sleep stages from radio signals: A conditional adversarial architecture. In ICML (2017), pp. 4100-4109. -
- [82][82] Zhou, P., et al. Use it free: Instantly knowing your phone attitude. In ACM MobiCom (2014), pp. 605-616. -
Cited by
Sorted by citations
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625 · see citing works
“Several works extracted 3D motion information from videos and transferred the knowledge to synthesize virtual on-body IMU sensor data [264, 265].”
BackgroundSelf-citationCosmo: contrastive fusion learning with small data for multimodal human activity recognition
Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J. (2022) · Proceedings of the 28th Annual International Conference on Mobile Computing And Networking · doi:10.1145/3495243.3560519 · Cited by 94 · see citing works
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3678545 · Cited by 61 · see citing works
Zhou, H., Lu, T., Liu, Y., Zhang, S., Liu, R., & Gowda, M. (2023) · IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) · doi:10.1145/3576842.3582382 · Cited by 53 · see citing works
Self-citationSynthesized Millimeter-Waves for Human Motion Sensing
Zhang, X., Li, Z., & Zhang, J. (2022) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3560905.3568542 · Cited by 51 · see citing works
Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings
Tong, C., Ge, J., & Lane, N. D. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3494995 · Cited by 41 · see citing works
Zhou, H., Lu, T.-T., Liu, Y.-L., Zhang, S.-J., & Gowda, M. K. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3534587 · Cited by 24 · see citing works
Self-citationKwon, H., Abowd, G. D., & Plötz, T. (2021) · Sensors · doi:10.3390/s21248337 · Cited by 30 · see citing works
Liu, Y., Zhang, S., Gowda, M., & Nelakuditi, S. (2022) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3570613 · Cited by 30 · see citing works
Self-citationSynthetic Smartwatch IMU Data Generation from In-the-wild ASL Videos
Santhalingam, P. S., Pathak, P. H., Rangwala, H., & Košecká, J. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3596261 · Cited by 28 · see citing works
Li, J., Huang, L., Shah, S., Jones, S. J., Jin, Y., Wang, D., Russell, A., Choi, S., Gao, Y., Yuan, J., et al. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610881 · Cited by 28 · see citing works
Optimizing Sensor Position with Virtual Sensors in Human Activity Recognition System Design
Xia, C., & Sugiura, Y. (2021) · Sensors · doi:10.3390/s21206893 · Cited by 24 · see citing works
Jawthenticate: Microphone-free Speech-based Authentication using Jaw Motion and Facial Vibrations
Srivastava, T., Pan, S., Nguyen, P., & Jain, S. (2023) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3625687.3625813 · Cited by 23 · see citing works
I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart Earphones
Zhang, S., Lu, T., Zhou, H., Liu, Y., Liu, R., & Gowda, M. (2023) · ACM Transactions on Internet of Things · doi:10.1145/3614438 · Cited by 23 · see citing works
Self-citationXia, C., & Sugiura, Y. (2022) · ACM International Symposium on Wearable Computers (ISWC) · doi:10.1145/3544794.3558460 · Cited by 22 · see citing works
MARLP: Time-series Forecasting Control for Agricultural Managed Aquifer Recharge
Chen, Y., Yang, K., An, Z., Holder, B., Paloutzian, L., Bali, K. M., & Du, W. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671533 · Cited by 21 · see citing works
Xia, C., Fang, X., Arakawa, R., & Sugiura, Y. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3534576 · Cited by 19 · see citing works
Vi-Fi: Associating Moving Subjects across Vision and Wireless Sensors
Liu, H., Alali, A., Ibrahim, M. M., Cao, B. B., Meegan, N., Li, H., Gruteser, M., Jain, S., Dana, K., Ashok, A., et al. (2022) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn54338.2022.00024 · Cited by 19 · see citing works
A Survey of Datasets, Applications, and Models for IMU Sensor Signals
Saraf, A., Moon, S., & Madotto, A. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193365 · Cited by 18 · see citing works
If only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios
Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150267 · Cited by 16 · see citing works
Computing with Smart Rings: A Systematic Literature Review
Wang, Z. S., Yu, R., Wang, X., Ding, J., Tang, J., Fang, J., He, Z., Li, Z., Röddiger, T., Xu, W., et al. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749480 · Cited by 14 · see citing works
Jin, Y., Zhang, S., Gao, Y., Xu, X., Choi, S., Li, Z., Adler, H. J., & Jin, Z. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3596255 · Cited by 13 · see citing works
Self-citationSignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors
Zhou, H., Lu, T., Mckinnie, K., Palagano, J., DeHaan, K., & Gowda, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3613286 · Cited by 13 · see citing works
Self-citationTransASL: A Smart Glass based Comprehensive ASL Recognizer in Daily Life
Jin, Y., Choi, S., Gao, Y., Li, J., Li, Z., & Jin, Z. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584071 · Cited by 12 · see citing works
Interpersonal Distance Tracking with mmWave Radar and IMUs
Dai, Y., Shuai, X., Tan, R., & Xing, G. (2023) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1145/3583120.3586958 · Cited by 10 · see citing works
CROMOSim: A Deep Learning-Based Cross-Modality Inertial Measurement Simulator
Hao, Y., Lou, X., Wang, B., & Zheng, R. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3230370 · Cited by 8 · see citing works
Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs
Rey, V. F., Ray, L. S. S., Xia, Q., Wu, K., & Lukowicz, P. (2024) · ACM International Symposium on Wearable Computers (ISWC) · doi:10.1145/3675095.3676609 · Cited by 9 · see citing works
SoundSieve: Seconds-Long Audio Event Recognition on Intermittently-Powered Systems
Monjur, M., Luo, Y., Wang, Z., & Nirjon, S. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596859 · Cited by 7 · see citing works
CROMOSim: A Deep Learning-based Cross-modality Inertial Measurement Simulator
Hao, Y., Wang, B., & Zheng, R. (2022) · 2022 2nd International Workshop on Cyber-Physical-Human System Design and Implementation (CPHS) · doi:10.1109/cphs56133.2022.9804560 · Cited by 5 · see citing works
Exploiting Serialized Fine-Grained Action Recognition Using WiFi Sensing
Tong, W., Li, R., Gong, X., Zhai, S., Zheng, X., & Ye, G. (2021) · Mobile Information Systems · doi:10.1155/2021/4770143 · Cited by 4 · see citing works
Enhancing the Applicability of Sign Language Translation
Li, J., Xu, J., Liu, Y., Xu, W., & Li, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3350111 · Cited by 4 · see citing works
Self-citationSmartGlove: Robust Sign Language Recognition With Cross-Domain Generation
Feng, M., Li, K., Chen, J., Zhang, J., Liu, D., Liu, Y., Chen, S., Tao, X., Tong, X., Xie, X. N., et al. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3623388 · Cited by 2 · see citing works
Self-citationCertified Robustness against Sensor Heterogeneity in Acoustic Sensing
Nguyen, D. P., Dai, Y., Li, X., & Tan, R. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749481 · Cited by 1 · see citing works
Toward Scalable ASL Education: Egocentric Stereo Sensing with LLM Feedback for Error-Aware Learning
Cai, Y., Li, Z., Lu, T., Zhu, Y., Wu, Y.-S., Zhang, Q., Xu, X., Jin, Z. K., Gowda, M. K., & Jin, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790774 · Cited by 1 · see citing works
Self-citationPoster Abstract: Enhancing Human Motion Sensing with synthesized Millimeter-Waves
Zhang, X., Wang, K., Li, Z., & Zhang, J. (2024) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn61024.2024.00054 · Cited by 1 · see citing works
Review on Human Computer Interaction Intelligent Learning for Improved Identification Procedure
Jacob, I. J. (2022) · Journal of Trends in Computer Science and Smart Technology · doi:10.36548/jtcsst.2022.2.002
CrossHand: Multimodal 3D Hand Reconstruction via Vision and Wearable Sensor Data Fusion
Lin, S., Zhou, G., Xiao, C., Jiang, Q., Yang, D., & Yu, H. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733472
Rob2HanD: LLM-Driven Robotic Arm for IMU Interaction Dataset Generation
Liu, J., Yu, C., Chen, X., Bu, J., & Zeng, L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791740
Continuous ASL Fingerspelling Recognition from Wearable Accelerometers
Hou, Z., Xie, Y., Li, F., & Wang, H. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00096
- Authors
- Liu, Y., Zhang, S., & Gowda, M.
- Affiliations
- Pennsylvania State University (US)
- Venue
- Proceedings of the International Conference on Internet-of-Things Design and Implementation
- Publisher
- ACM
- Published
- 2021-05-18
- Pages
- 182-194
- OpenAlex
- W3163120578
- Cited by
- 39
- References
- 72
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 5 · 2023: 12 · 2022: 11 · 2021: 4
- FWCI
- 5.98
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) (publishedVersion)
- Funders
- National Science Foundation (1909479)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- inertial measurement unit, gesture recognition, wearable IoT devices, data augmentation, user heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
