IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“IMUGPT2.0 [116] introduces a motion filter (to eliminate irrelevant generated sequences) and metrics to evaluate the diversity of generated data, resulting in a 50% effort reduction for generating virtual IMU data.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [116]
In the citing paper
“Leng et al. [116,117] demonstrate how foundational models can be used to generate realistic and useful IMU data.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [116]
Citations
Cites
1 retracted
Research on Data Security and Privacy Protection of Wearable Equipment in Healthcare
Jiang, D., & Shi, G. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/6656204 · Cited by 48 · cited 1×
“However, one of the biggest challenges that the HAR community faces is the lack of large-scale labeled datasets due to the expensive annotation process, the need for domain experts, and potential issues with invasion of privacy [16, 18, 31, 35, 61].”
RetractedAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“Preliminary experiments demonstrated the promising potential of the approach, showing that the extracted virtual IMU data can lead to significant improvements in the downstream HAR performance.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“The problem is framed as a binary classification task where the input is a textual description 𝑚 , the motion caption of a human motion sequence, along with a specific activity of interest 𝑎 .”
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734 · cited 1×
“Approaches for generating virtual data through cross modality transfer from different data sources have been explored across various domains (Table 1). cross modality transfer has been actively investigated for the generation of virtual IMU data for HAR from various sources.”
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · cited 1×
“Conventional HAR applications are typically based on (some variant of) the "Activity Recognition Chain (ARC)" (e.g., [14]), which divides the analysis problem into five general processing blocks: recording, preprocessing, (sliding-window) segmentation, feature extraction, and classification.”
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 2×
“VisualAcc [83] uses photometric effect-based interrogation and an optic-to-inertia transformer that senses human motion passively and reconstructs inertial data using the Optical Motion Field (OMF).”
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 1×
“HAR finds extensive application in various domains, including fitness tracking [15, 33], health monitoring [7, 46], sign language recognition [49, 54], and in identifying critical events like vehicular accidents and falls [39, 48].”
Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom
Bachlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J. M., Giladi, N., & Troster, G. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036165 · Cited by 693 · cited 2×
“HAR finds extensive application in various domains, including fitness tracking [15, 33], health monitoring [7, 46], sign language recognition [49, 54], and in identifying critical events like vehicular accidents and falls [39, 48].”
van Hees, V. T., Fang, Z., Langford, J., Assah, F., Mohammad, A., da Silva, I. C. M., Trenell, M. I., White, T., Wareham, N. J., & Brage, S. (2014) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00421.2014 · Cited by 676 · cited 1×
“These approaches aim to convert data from other modalities, such as video [37], motion capture [82], and optic data [83], into virtual IMU data .”
Generating Diverse and Natural 3D Human Motions from Text
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00509 · Cited by 575 · cited 4×
“For the HAD-AW dataset, a positive correlation was observed, indicating that the virtual IMU data led to a drop in the per-class F1 score for more activities than it helped.”
InfluentialHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437 · cited 2×
“Our evaluations are based on three models that are widely used in the HAR research community: i) a Random Forest classifier; ii) DeepConvLSTM [58]; and iii) DeepConvLSTM with self-attention [71].”
On-body localization of wearable devices: An investigation of position-aware activity recognition
Sztyler, T., & Stuckenschmidt, H. (2016) · 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom.2016.7456521 · Cited by 369 · cited 3×
“VisualAcc [83] uses photometric effect-based interrogation and an optic-to-inertia transformer that senses human motion passively and reconstructs inertial data using the Optical Motion Field (OMF).”
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346 · cited 1×
“The research community has developed a range of approaches to specifically tackle the sparse data problem, including self-supervised learning [26–28, 67, 73], few-shot learning [21], semi-supervised learning [10], prototypical learning [9, 17], adversarial learning [8, 41], and transfer learning…”
Ravi, D., Wong, C., Lo, B., & Yang, G.-Z. (2016) · 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN) · doi:10.1109/bsn.2016.7516235 · Cited by 269 · cited 1×
“The effectiveness of these methods relies heavily on the availability of accurately annotated datasets.”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts
Guo, C., Zuo, X., Wang, S., & Cheng, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19833-5_34 · Cited by 203
Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearable Sensors
Singh, S. P., Sharma, M. K., Lay-Ekuakille, A., Gangwar, D., & Gupta, S. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3045135 · Cited by 199 · cited 1×
“This model outputs a textual description of the motion sequence.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197 · cited 1×
“VisualAcc [83] uses photometric effect-based interrogation and an optic-to-inertia transformer that senses human motion passively and reconstructs inertial data using the Optical Motion Field (OMF).”
Wearable devices in healthcare: Privacy and information security issues
Cilliers, L. (2020) · Health Information Management Journal · doi:10.1177/1833358319851684 · Cited by 169 · cited 1×
“This hinders the development of robust and generalizable models.”
Human action recognition using transfer learning with deep representations
Sargano, A. B., Wang, X., Angelov, P., & Habib, Z. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965890 · Cited by 163 · cited 1×
“The key idea is to use LLMs to generate diverse textual descriptions of the different ways that humans can perform certain activities.”
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411841 · Cited by 152
Self-citationContrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3463506 · Cited by 150
Self-citationAhuja, K., Jiang, Y., Goel, M., & Harrison, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445138 · Cited by 147 · cited 1×
“[53] Gyro Head Motion Recognition Pose Estimation & Neural Network Vid2Doppler [4] Doppler”
Masked reconstruction based self-supervision for human activity recognition
Haresamudram, H., Beedu, A., Agrawal, V., Grady, P. L., Essa, I., Hoffman, J., & Plötz, T. (2020) · Proceedings of the 2020 International Symposium on Wearable Computers · doi:10.1145/3410531.3414306 · Cited by 146 · cited 1×
“The research community has developed a range of approaches to specifically tackle the sparse data problem, including self-supervised learning [26–28, 67, 73], few-shot learning [21], semi-supervised learning [10], prototypical learning [9, 17], adversarial learning [8, 41], and transfer learning…”
Self-citationSelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
Tang, C. I., Perez-Pozuelo, I., Spathis, D., Brage, S., Wareham, N., & Mascolo, C. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3448112 · Cited by 140
Hammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 1×
“The resulting motion caption is then evaluated by an LLM, which provides a binary ‘yes’ or ‘no’ answer indicating whether the caption accurately describes the specified activity.”
Self-citationAssessing the State of Self-Supervised Human Activity Recognition Using Wearables
Haresamudram, H., Essa, I., & Plötz, T. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3550299 · Cited by 107 · cited 1×
“Leng et al. [43] piloted the concept of IMUGPT that combines large language models (LLMs), motion synthesis methods, and signal processing techniques to generate virtual IMU data with no manual effort.”
Self-citationDeep Learning for Human Activity Recognition in Mobile Computing
Plotz, T., & Guan, Y. (2018) · Computer · doi:10.1109/mc.2018.2381112 · Cited by 99
ProtoHAR: Prototype Guided Personalized Federated Learning for Human Activity Recognition
Cheng, D., Zhang, L., Bu, C., Wang, X., Wu, H., & Song, A. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3275438 · Cited by 92
Self-citationFew-shot learning-based human activity recognition
Feng, S., & Duarte, M. F. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.06.070 · Cited by 91
A review of wearable sensors based fall-related recognition systems
Liu, J., Li, X., Huang, S., Chao, R., Chao, R., Wang, S., Wang, S., & Liu, L. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105993 · Cited by 62 · cited 1×
“A novel motion filter then screens out incorrect sequences, retaining only relevant motion sequences for virtual IMU data extraction.”
A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors
Xiao, F., Pei, L., Chu, L., Zou, D., Yu, W., Zhu, Y., & Li, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69873-7_19 · Cited by 59
WearBreathing: Real World Respiratory Rate Monitoring Using Smartwatches
Liaqat, D., Abdalla, M., Abed-Esfahani, P., Gabel, M., Son, T., Wu, R., Gershon, A., Rudzicz, F., & Lara, E. D. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328927 · Cited by 58 · cited 1×
“An LLM automatically generates textual descriptions of activities, which are then converted into motion sequences by a motion synthesis model.”
Handling annotation uncertainty in human activity recognition
Kwon, H., Abowd, G. D., & Plötz, T. (2019) · Proceedings of the 23rd International Symposium on Wearable Computers · doi:10.1145/3341163.3347744 · Cited by 52 · cited 1×
“This hinders the development of robust and generalizable models.”
Self-citationSenseCollect: We Need Efficient Ways to Collect On-body Sensor-based Human Activity Data!
Chen, W., Lin, S., Thompson, E., & Stankovic, J. A. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3478119 · Cited by 51
Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data
Kwon, H., Wang, B., Abowd, G. D., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3478096 · Cited by 48
Self-citationMyoGym: introducing an open gym data set for activity recognition collected using myo armband
Koskimäki, H., Siirtola, P., & Röning, J. (2017) · Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers · doi:10.1145/3123024.3124400 · Cited by 46 · cited 3×
“HAR finds extensive application in various domains, including fitness tracking [15, 33], health monitoring [7, 46], sign language recognition [49, 54], and in identifying critical events like vehicular accidents and falls [39, 48].”
SignRing: Continuous American Sign Language Recognition Using IMU Rings and Virtual IMU Data
Li, J., Huang, L., Shah, S., Jones, S. J., Jin, Y., Wang, D., Russell, A., Choi, S., Gao, Y., Yuan, J., et al. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610881 · Cited by 44
Rey, V. F., Hevesi, P., Kovalenko, O., & Lukowicz, P. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3345590 · Cited by 43 · cited 2×
“The most notable ones are: (i) paucity of labeled data – the time-consuming and expensive nature of wearable data collection along with its inherent privacy concerns have led to datasets being relatively smaller in size; (ii) difficulty in data annotation – ambiguity in the target activities and…”
IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data
Bhalla, S., Goel, M., & Khurana, R. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3494994 · Cited by 41
Adversarial Multi-view Networks for Activity Recognition
Bai, L., Yao, L., Wang, X., Kanhere, S. S., Guo, B., & Yu, Z. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3397323 · Cited by 40 · cited 1×
“…research community has developed a range of approaches to specifically tackle the sparse data problem, including self-supervised learning [26–28, 67, 73], few-shot learning [21], semi-supervised learning [10], prototypical learning [9, 17], adversarial learning [8, 41], and transfer learning [69].”
Liu, Y., Zhang, S., & Gowda, M. (2021) · Proceedings of the International Conference on Internet-of-Things Design and Implementation · doi:10.1145/3450268.3453537 · Cited by 39 · cited 2×
“…labeled data – the time-consuming and expensive nature of wearable data collection along with its inherent privacy concerns have led to datasets being relatively smaller in size; (ii) difficulty in data annotation – ambiguity in the target activities and its context results in incorrectly labeled…”
Leng, Z., Kwon, H., & Ploetz, T. (2023) · Proceedings of the 2023 International Symposium on Wearable Computers · doi:10.1145/3594738.3611361 · Cited by 39
Self-citationZhang, P., Dutta, P., & Xing, G. (2017) · doi:10.1145/3055031 · Cited by 36
Semi-supervised learning for human activity recognition using adversarial autoencoders
Balabka, D. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3344854 · Cited by 36 · cited 1×
“…research community has developed a range of approaches to specifically tackle the sparse data problem, including self-supervised learning [26–28, 67, 73], few-shot learning [21], semi-supervised learning [10], prototypical learning [9, 17], adversarial learning [8, 41], and transfer learning [69].”
Finger Gesture Tracking for Interactive Applications: A Pilot Study with Sign Languages
Liu, Y., Jiang, F., & Gowda, M. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3414117 · Cited by 35 · cited 2×
“The recognition is typically achieved through the use of supervised learning methods to classify segmented data streams into activities of interest (or the null class).”
Els, F., & Cilliers, L. (2017) · Conference on Information Communications Technology and Society (ICTAS) · doi:10.1109/ictas.2017.7920658 · Cited by 31 · cited 1×
“The effectiveness of these methods relies heavily on the availability of accurately annotated datasets.”
Kwon, H., Abowd, G. D., & Plötz, T. (2021) · Sensors · doi:10.3390/s21248337 · Cited by 30 · cited 3×
“Sensor-based human activity recognition using wearable devices (HAR), i.e., the automated assessment of what a person is doing (and when), has come a long way and it is considered one of the central pillars of mobile, wearable, and ubiquitous computing [36].”
InfluentialSelf-citationAudioIMU: Enhancing Inertial Sensing-Based Activity Recognition with Acoustic Models
Liang, D., Li, G., Adaimi, R., Mărculescu, R., & Thomaz, E. (2022) · ACM International Symposium on Wearable Computers (ISWC) · doi:10.1145/3544794.3558471 · Cited by 29 · cited 1×
“cross modality transfer methods have recently been introduced in a number of application domains with the goal of opportunistically utilizing existing datasets from source modalities other than those targeted by a specific application.”
Synthetic Smartwatch IMU Data Generation from In-the-wild ASL Videos
Santhalingam, P. S., Pathak, P., Rangwala, H., & Kosecka, J. (2023) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3596261 · Cited by 28 · cited 4×
“To address this challenge, Santhalingam et al. [68] trained a model to convert 3D hand poses into accurate virtual IMU data for sign language recognition.”
InfluentialXia, C., & Sugiura, Y. (2022) · Proceedings of the 2022 ACM International Symposium on Wearable Computers · doi:10.1145/3544794.3558460 · Cited by 22
Proceedings of the 2023 International Symposium on Wearable Computers
(2023) · doi:10.1145/3594738 · Cited by 21
Deep generative cross-modal on-body accelerometer data synthesis from videos
Zhang, S., & Alshurafa, N. (2020) · Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers · doi:10.1145/3410530.3414329 · Cited by 21 · cited 1×
“The most notable ones are: (i) paucity of labeled data – the time-consuming and expensive nature of wearable data collection along with its inherent privacy concerns have led to datasets being relatively smaller in size; (ii) difficulty in data annotation – ambiguity in the target activities and…”
Xia, C., Saito, A., & Sugiura, Y. (2022) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2022.113463 · Cited by 19 · cited 1×
“Despite considerable advancements in HAR and its numerous practical applications, there exist significant challenges that are inherent and specific to HAR using wearable sensing platforms [16, 20, 61, 63].”
Improving Cross-Subject Activity Recognition via Adversarial Learning
Leite, C. F. S., & Xiao, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2993818 · Cited by 18 · cited 1×
“Leng et al. [43] piloted the concept of IMUGPT that combines large language models (LLMs), motion synthesis methods, and signal processing techniques to generate virtual IMU data with no manual effort.”
If only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios
Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150267 · Cited by 16 · cited 3×
“HAR finds extensive application in various domains, including fitness tracking [15, 33], health monitoring [7, 46], sign language recognition [49, 54], and in identifying critical events like vehicular accidents and falls [39, 48].”
Self-citationAn LSTM-based Descriptor for Human Activities Recognition using IMU Sensors
Ashry, S., Elbasiony, R. M., & Gomaa, W. E. (2018) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0006902405040511 · Cited by 15 · cited 1×
“Another limitation of IMUGPT is that the motion synthesis model may generate motion sequences that do not accurately describe the intended activity, which leads to irrelevant virtual IMU data being extracted, potentially degrading the downstream performance.”
On the Utility of Virtual On-body Acceleration Data for Fine-grained Human Activity Recognition
Leng, Z., Jain, Y., Kwon, H., & Plötz, T. (2023) · International Symposium on Wearable Computers · doi:10.1145/3594738.3611364 · Cited by 13
Self-citationMartin, D., Leng, Z., Gemicioglu, T., Womack, J., Heath, J., Neubauer, W. C., Kwon, H., Plötz, T., & Starner, T. E. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3614491 · Cited by 10 · cited 1×
“A novel motion filter then screens out incorrect sequences, retaining only relevant motion sequences for virtual IMU data extraction.”
Self-citationCar crash detection on smartphones
Lahn, J., Peter, H., & Braun, P. G. (2015) · doi:10.1145/2790044.2790049 · Cited by 7
Comparison of Surface Models and Skeletal Models for Inertial Sensor Data Synthesis
Uhlenberg, L., & Amft, O. (2022) · doi:10.1109/bsn56160.2022.9928504 · Cited by 7 · cited 2×
“…data ("small labeled dataset problem") by automatically converting sensor readings from one modality to another, resulting in "virtual sensor data" in the target modality, thereby transferring knowledge in form of the underlying fundamentals of the human movements from one modality to another.”
Measurement of noise characteristics of MEMS accelerometers
Mohd-Yasin, F., Korman, C. E., & Nagel, D. J. (2002) · doi:10.1109/isdrs.2001.984472 · Cited by 5 · cited 1×
“These approaches aim to convert data from other modalities, such as video [37], motion capture [82], and optic data [83], into virtual IMU data .”
Research works printed but not linked: 14
- Lei Bai, Lina Yao, Xianzhi Wang, Salil S. Kanhere, and Yang Xiao. 2020. Prototype Similarity Learning for Activity Recognition. In Advances in Knowledge Discovery and Data Mining, Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng, and Sinno Jialin Pan (Eds.). Springer International Publishing, Cham, 649--661.
- Richard O. Duda, Peter E. Hart, and David G. Stork. 2000. Pattern Classification (2nd Edition) (2 ed.). Wiley-Interscience.
- Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. MotionGPT: Human Motion as a Foreign Language. arXiv preprint arXiv:2306.14795 (2023). · arXiv
- Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset. Advances in Neural Information Processing Systems (2023).
- MinYen Lu, ChenHao Chen, Shigemi Ishida, Yugo Nakamura, and Yutaka Arakawa. 2022. A study on estimating the accurate head IMU motion from Video. Proceedings of the Symposium on Multimedia, Distributed, Cooperative, and Mobile (DICOMO) 2022 2022 (07 2022), 918--923. https://cir.nii.ac.jp/crid/1050011771467456512
- OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat Red Avila Igor Babuschkin Suchir Balaji Valerie Balcom et al. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors (2016).
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, and Ioannis Antonoglou others. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2307.09288 [cs.CL] · arXiv
- 10.5555/3295222.3295378 · doi:10.5555/3295222.3295378
- Chenhan Xu, Huining Li, Zhengxiong Li, Xingyu Chen, Aditya Singh Rathore, Hanbin Zhang, Kun Wang, and Wenyao Xu. 2022. The Visual Accelerometer: A High-fidelity Optic-to-Inertial Transformation Framework for Wearable Health Computing. In 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI). IEEE, 319--329.
- Hyungjun Yoon, Hyeongheon Cha, Canh Hoang Nguyen, Taesik Gong, and Sung-Ju Lee. 2022. IMG2IMU: Applying Knowledge from Large-Scale Images to IMU Applications via Contrastive Learning. arXiv preprint arXiv:2209.00945 (2022). · arXiv
- Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 2023. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023. ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. arXiv preprint arXiv:2304.01116 (2023). · arXiv
- Mi Zhang and Alexander A. Sawchuk. 2012. USC-HAD: A Daily Activity Dataset for Ubiquitous Activity Recognition Using Wearable Sensors. Association for Computing Machinery.
Cited by
Transfer Learning in Sensor-Based Human Activity Recognition: A Survey
Dhekane, S. G., & Ploetz, T. (2025) · ACM Computing Surveys · doi:10.1145/3717608 · Cited by 42
Self-citationHaresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729467 · Cited by 33
Self-citationLarge Language Models Are Zero-Shot Recognizers for Activities of Daily Living
Civitarese, G., Fiori, M., Choudhary, P., & Bettini, C. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3725856 · Cited by 67
“Another line of research leverages a pre-trained model generating 3D motion sequences from text [51] to generate synthetic labeled data from wearable/mobile devices [34].”
Large Language Models are Zero-Shot Recognizers for Activities of Daily Living
Civitarese, G., Fiori, M., Choudhary, P., & Bettini, C. (2024) · arXiv · doi:10.48550/arxiv.2407.01238 · Cited by 28
“Another line of research leverages a pre-trained model generating 3D motion sequences from text [51] to generate synthetic labeled data from wearable/mobile devices [34].”
LLMs and IoT: A Comprehensive Survey on Large Language Models and the Internet of Things
Sarhaddi, F., Nguyen, N. T., Zuniga, A., Khan, M., Hui, P., Tarkoma, S., Flores, H., & Nurmi, P. (2026) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2026.3680064 · Cited by 33
Metadata
- Authors
- Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Emory University (US)
- ORCID
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2024-08-22
- Volume
- 8
- Issue
- 3
- Pages
- 1-32
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W4402350391
- Cited by
- 61
- References
- 58
- Citations by year
- 2026: 26 · 2025: 29 · 2024: 6
- FWCI
- 8.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human activity recognition, cross-modality transfer, large language models, inertial measurement unit, data scarcity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.