Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The development of robust, generalized models for human activity recognition (HAR) has been hindered by the scarcity of large-scale, labeled data sets. Recent work has shown that virtual IMU data extracted from videos using computer vision techniques can lead to substantial performance improvements when training HAR models combined with small portions of real IMU data. Inspired by recent advances in motion synthesis from textual descriptions and connecting Large Language Models (LLMs) to various AI models, we introduce an automated pipeline that first uses ChatGPT to generate diverse textual descriptions of activities. These textual descriptions are then used to generate 3D human motion sequences via a motion synthesis model, T2M-GPT, and later converted to streams of virtual IMU data. We benchmarked our approach on three HAR datasets (RealWorld, PAMAP2, and USC-HAD) and demonstrate that the use of virtual IMU training data generated using our new approach leads to significantly improved HAR model performance compared to only using real IMU data. Our approach contributes to the growing field of cross-modality transfer methods and illustrate how HAR models can be improved through the generation of virtual training data that do not require any manual effort.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Leng et al. [116,117] demonstrate how foundational models can be used to generate realistic and useful IMU data.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [117]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“Also, we will also study the effect of using prompt weighting (often used in text-to-image generation [20]), giving more weights to the activityrelated parts of the prompt, which allows the motion synthesis model to focus more on the activity.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Rank transformations as a bridge between parametric and nonparametric statistics
Conover, W. J., & Iman, R. L. (1981) · The American Statistician · doi:10.1080/00031305.1981.10479327 · Cited by 3,922 · cited 1×
“Following [11], we perform a distribution mapping be-tween the virtual IMU data and the real IMU data using the rank transformation approach [7].”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 2×
“Yet, when only virtual IMU dataset was used for DeepConvLSTM, the performance significantly drop due to too small training dataset.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5×
“Large Language Models: Large Language Models (LLMs) such as PaLM [6], LLaMA [26], GPT-3 [3], and ChatGPT (built upon InstructGPT [17]) have attracted enormous attention for their superior performances in many natural language processing (NLP) tasks.”
BackgroundMethodInfluentialRank Transformations as a Bridge Between Parametric and Nonparametric Statistics
Conover, W. J., & Iman, R. L. (1981) · The American Statistician · doi:10.2307/2683975 · Cited by 1,481
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 7×
“Figure 2: Model performance on RealWorld [21], PAMAP2 [17], and USC-HAD [34] datasets when different amount of real”
MethodInfluentialThe Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 1×
“The development of accurate and robust predictive models for human activity recognition (HAR) is essential for, e.g., monitoring fitness, analyzing health-related behavior, and improving industrial processes [2, 4, 13, 20].”
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · cited 1×
“Also, we will also study the effect of using prompt weighting (often used in text-to-image generation [20]), giving more weights to the activityrelated parts of the prompt, which allows the motion synthesis model to focus more on the activity.”
MethodWearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom
Bachlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J. M., Giladi, N., & Troster, G. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036165 · Cited by 693 · cited 1×
“3611361 fitness, analyzing health-related behavior, and improving industrial processes [2, 4, 14, 23].”
BackgroundGenerating Diverse and Natural 3D Human Motions from Text
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00509 · Cited by 575
USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors
Zhang, M., & Sawchuk, A. A. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370438 · Cited by 554
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477 · cited 1×
“Recently, Visual ChatGPT [27] and HuggingGPT [22] were introduced to tackle complex multi-modal tasks.”
BackgroundHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
On-body localization of wearable devices: An investigation of position-aware activity recognition
Sztyler, T., & Stuckenschmidt, H. (2016) · 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom.2016.7456521 · Cited by 369
Wearable Activity Tracking in Car Manufacturing
Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., & Tr, G. (2008) · IEEE Pervasive Computing · doi:10.1109/mprv.2008.40 · Cited by 367 · cited 1×
“3611361 fitness, analyzing health-related behavior, and improving industrial processes [2, 4, 14, 23].”
BackgroundGenerating Human Motion from Textual Descriptions with Discrete Representations
Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., & Ying, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01415 · Cited by 328
Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., & Bermano, A. H. (2022) · arXiv · doi:10.48550/arxiv.2209.14916 · Cited by 290
Decoupling Human and Camera Motion from Videos in the Wild
Ye, V., Pavlakos, G., Malik, J., & Kanazawa, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02033 · Cited by 210 · cited 1×
“A cost-effective method for extension would be to use computer vision techniques such as 3D human pose estimation [31] on existing videos to extract the human motion sequence for the new activities.”
MethodNatural motion animation through constraining and deconstraining at will
Yamane, K., & Nakamura, Y. (2003) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2003.1207443 · Cited by 203 · cited 2×
“Using the motion sequence, the joint rotations and translations are estimated through inverse kinematics [29].”
MethodVisual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197
Unpaired motion style transfer from video to animation
Aberman, K., Weng, Y., Lischinski, D., Cohen-Or, D., & Chen, B. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392469 · Cited by 175 · cited 1×
“We will test diffusion-based motion synthesis models [22, 28, 33], which generate more diverse motion sequences for similar prompts and use motion style transfer [1] to apply different motion styles to the generated motion sequences.”
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411841 · Cited by 152
Self-citationHammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 1×
“For the Random Forest classifier, ECDF features [9] (15 components) are extracted from the windows for training.”
MethodSelf-citationWearBreathing: Real World Respiratory Rate Monitoring Using Smartwatches
Liaqat, D., Abdalla, M., Abed-Esfahani, P., Gabel, M., Son, T., Wu, R., Gershon, A., Rudzicz, F., & Lara, E. D. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328927 · Cited by 58 · cited 1×
“3611361 fitness, analyzing health-related behavior, and improving industrial processes [2, 4, 14, 23].”
BackgroundSenseCollect: We Need Efficient Ways to Collect On-body Sensor-based Human Activity Data!
Chen, W., Lin, S., Thompson, E., & Stankovic, J. A. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3478119 · Cited by 51
Abdelzaher, T., Sweden, I. '. T. 9. I. C. O. I. P. I. S. N. 2. S., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Embedded Systems, & SIGs, A. (2010) · Cited by 49
Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data
Kwon, H., Wang, B., Abowd, G. D., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3478096 · Cited by 48
Self-citationWearables in the wet lab: a laboratory system for capturing and guiding experiments
Scholl, P. M., Wille, M., & Van Laerhoven, K. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2807547 · Cited by 45 · cited 1×
“Although detecting locomotion is important due to its relevance to individuals’ health, daily activities also involve sporadic and complex motions, such as washing dishes, bed making, etc [13, 15], and even rare activities, for example in wet labs [21].”
BackgroundKwon, H., Abowd, G. D., & Plötz, T. (2021) · Sensors · doi:10.3390/s21248337 · Cited by 30 · cited 1×
“Previous studies [10, 12] have shown that the extracted virtual IMU data lead to improved model performance when mixed with the real IMU data and allowed for effective training of more complex models.”
Self-citationXia, C., & Sugiura, Y. (2022) · Proceedings of the 2022 ACM International Symposium on Wearable Computers · doi:10.1145/3544794.3558460 · Cited by 22 · cited 2×
“[27] proposed a spring-joint model to augment the extracted virtual acceleration signal and trained a classifier on the augmented virtual IMU data to recognize reverse lunge, warm up, and high knee tap.”
Proceedings of the 2023 International Symposium on Wearable Computers
(2023) · doi:10.1145/3594738 · Cited by 21
On specialized window lengths and detector based human activity recognition
Li, H., Abowd, G. D., & Plötz, T. (2018) · Proceedings of the 2018 ACM International Symposium on Wearable Computers · doi:10.1145/3267242.3267246 · Cited by 19
Self-citationYoung, A. D., Ling, M. J., & Arvind, D. K. (2010) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1145/1791212.1791243 · Cited by 18
An LSTM-based Descriptor for Human Activities Recognition using IMU Sensors
Ashry, S., Elbasiony, R. M., & Gomaa, W. E. (2018) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0006902405040511 · Cited by 15 · cited 1×
“Although detecting locomotion is important due to its relevance to individuals’ health, daily activities also involve sporadic and complex motions, such as washing dishes, bed making, etc [13, 15], and even rare activities, for example in wet labs [21].”
Research works printed but not linked: 4
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates , Inc ., 1877 –1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1877–1901.
- Sara Mohammed Reda Elbasiony and Walid Gomaa. 2018. An LSTM-based Descriptor for Human Activities Recognition using IMU Sensors. 504-511. https://doi.org/10.5220/0006902405040511 · DOI
- Alec Radford , Jong Wook Kim , Chris Hallacy , Aditya Ramesh , Gabriel Goh , Sandhini Agarwal , Girish Sastry , Amanda Askell , Pamela Mishkin , Jack Clark , Gretchen Krueger , and Ilya Sutskever . 2021 . Learning Transferable Visual Models From Natural Language Supervision . In Proceedings of the 38th International Conference on Machine Learning, Vol. 139 . PMLR, 8748–8763. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139. PMLR, 8748–8763.
- Chen Xin , Biao Jiang , Wen Liu , Zilong Huang , Bin Fu , Tao Chen , Jingyi Yu , and Gang Yu . 2023 . Executing your Commands via Motion Diffusion in Latent Space . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Chen Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. 2023. Executing your Commands via Motion Diffusion in Latent Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Cited by
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3678545 · Cited by 61
Self-citationTransfer Learning in Sensor-Based Human Activity Recognition: A Survey
Dhekane, S. G., & Ploetz, T. (2025) · ACM Computing Surveys · doi:10.1145/3717608 · Cited by 42
Haresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729467 · Cited by 33
GOAT: A Generalized Cross-Dataset Activity Recognition Framework with Natural Language Supervision
Miao, S., & Chen, L. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699736 · Cited by 27
ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models
Arrotta, L., Bettini, C., Civitarese, G., & Fiori, M. (2024) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp61445.2024.00029 · Cited by 19
Metadata
- Authors
- Leng, Z., Kwon, H., & Ploetz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Georgia Institute of Technology, United States
- Venue
- Proceedings of the 2023 International Symposium on Wearable Computers
- Publisher
- ACM
- Published
- 2023-10-08
- Pages
- 39-43
- OpenAlex
- W4387321395
- Cited by
- 39
- References
- 21
- Citations by year
- 2026: 11 · 2025: 15 · 2024: 12
- FWCI
- 4.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Symposium on Wearable Computers (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human activity recognition, large language models, ChatGPT, cross-modality transfer, synthetic training data, wearable sensor data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.