IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“IMUGPT2.0 [116] introduces a motion filter (to eliminate irrelevant generated sequences) and metrics to evaluate the diversity of generated data, resulting in a 50% effort reduction for generating virtual IMU data.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [116]
In the citing paper
“Leng et al. [116,117] demonstrate how foundational models can be used to generate realistic and useful IMU data.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [116]
Citations
Cites
1 retracted
Research on Data Security and Privacy Protection of Wearable Equipment in Healthcare
Jiang, D., & Shi, G. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/6656204 · Cited by 48 · cited 1×
“However, one of the biggest challenges that the HAR community faces is the lack of large-scale labeled datasets due to the expensive annotation process, the need for domain experts, and potential issues with invasion of privacy [16, 18, 31, 35, 61].”
RetractedAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“Preliminary experiments demonstrated the promising potential of the approach, showing that the extracted virtual IMU data can lead to significant improvements in the downstream HAR performance.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“The problem is framed as a binary classification task where the input is a textual description 𝑚 , the motion caption of a human motion sequence, along with a specific activity of interest 𝑎 .”
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734 · cited 1×
“Approaches for generating virtual data through cross modality transfer from different data sources have been explored across various domains (Table 1). cross modality transfer has been actively investigated for the generation of virtual IMU data for HAR from various sources.”
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · cited 1×
“Further details about how the comparative diversity is utilized to halt the generation process is highlighted in the subsequent sections.”
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · cited 1×
“Conventional HAR applications are typically based on (some variant of) the "Activity Recognition Chain (ARC)" (e.g., [14]), which divides the analysis problem into five general processing blocks: recording, preprocessing, (sliding-window) segmentation, feature extraction, and classification.”
Research works printed but not linked: 14
- Lei Bai, Lina Yao, Xianzhi Wang, Salil S. Kanhere, and Yang Xiao. 2020. Prototype Similarity Learning for Activity Recognition. In Advances in Knowledge Discovery and Data Mining, Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng, and Sinno Jialin Pan (Eds.). Springer International Publishing, Cham, 649--661.
- Richard O. Duda, Peter E. Hart, and David G. Stork. 2000. Pattern Classification (2nd Edition) (2 ed.). Wiley-Interscience.
- Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. MotionGPT: Human Motion as a Foreign Language. arXiv preprint arXiv:2306.14795 (2023). · arXiv
- Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset. Advances in Neural Information Processing Systems (2023).
- MinYen Lu, ChenHao Chen, Shigemi Ishida, Yugo Nakamura, and Yutaka Arakawa. 2022. A study on estimating the accurate head IMU motion from Video. Proceedings of the Symposium on Multimedia, Distributed, Cooperative, and Mobile (DICOMO) 2022 2022 (07 2022), 918--923. https://cir.nii.ac.jp/crid/1050011771467456512
- OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat Red Avila Igor Babuschkin Suchir Balaji Valerie Balcom et al. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors (2016).
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, and Ioannis Antonoglou others. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2307.09288 [cs.CL] · arXiv
- 10.5555/3295222.3295378 · doi:10.5555/3295222.3295378
- Chenhan Xu, Huining Li, Zhengxiong Li, Xingyu Chen, Aditya Singh Rathore, Hanbin Zhang, Kun Wang, and Wenyao Xu. 2022. The Visual Accelerometer: A High-fidelity Optic-to-Inertial Transformation Framework for Wearable Health Computing. In 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI). IEEE, 319--329.
- Hyungjun Yoon, Hyeongheon Cha, Canh Hoang Nguyen, Taesik Gong, and Sung-Ju Lee. 2022. IMG2IMU: Applying Knowledge from Large-Scale Images to IMU Applications via Contrastive Learning. arXiv preprint arXiv:2209.00945 (2022). · arXiv
- Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 2023. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023. ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. arXiv preprint arXiv:2304.01116 (2023). · arXiv
- Mi Zhang and Alexander A. Sawchuk. 2012. USC-HAD: A Daily Activity Dataset for Ubiquitous Activity Recognition Using Wearable Sensors. Association for Computing Machinery.
Cited by
Transfer Learning in Sensor-Based Human Activity Recognition: A Survey
Dhekane, S. G., & Ploetz, T. (2025) · ACM Computing Surveys · doi:10.1145/3717608 · Cited by 42
Self-citationHaresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729467 · Cited by 33
Self-citationLarge Language Models Are Zero-Shot Recognizers for Activities of Daily Living
Civitarese, G., Fiori, M., Choudhary, P., & Bettini, C. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3725856 · Cited by 67
“Another line of research leverages a pre-trained model generating 3D motion sequences from text [51] to generate synthetic labeled data from wearable/mobile devices [34].”
Large Language Models are Zero-Shot Recognizers for Activities of Daily Living
Civitarese, G., Fiori, M., Choudhary, P., & Bettini, C. (2024) · arXiv · doi:10.48550/arxiv.2407.01238 · Cited by 28
“Another line of research leverages a pre-trained model generating 3D motion sequences from text [51] to generate synthetic labeled data from wearable/mobile devices [34].”
LLMs and IoT: A Comprehensive Survey on Large Language Models and the Internet of Things
Sarhaddi, F., Nguyen, N. T., Zuniga, A., Khan, M., Hui, P., Tarkoma, S., Flores, H., & Nurmi, P. (2026) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2026.3680064 · Cited by 33
Metadata
- Authors
- Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Emory University (US)
- ORCID
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2024-08-22
- Volume
- 8
- Issue
- 3
- Pages
- 1-32
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W4402350391
- Cited by
- 61
- References
- 58
- Citations by year
- 2026: 26 · 2025: 29 · 2024: 6
- FWCI
- 8.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human activity recognition, cross-modality transfer, large language models, inertial measurement unit, data scarcity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition. https://app.cochat.ai/papers/10.1145/3678545
- The paper
- Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024). IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. https://doi.org/10.1145/3678545
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.