Translating Videos into Synthetic Training Data for Wearable Sensor-Based Activity Recognition Systems Using Residual Deep Convolutional Networks
Open accessThe DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
Human activity recognition (HAR) using wearable sensors has benefited much less from recent advances in Deep Learning than fields such as computer vision and natural language processing. This is, to a large extent, due to the lack of large scale (as compared to computer vision) repositories of labeled training data for sensor-based HAR tasks. Thus, for example, ImageNet has images for around 100,000 categories (based on WordNet) with on average 1000 images per category (therefore up to 100,000,000 samples). The Kinetics-700 video activity data set has 650,000 video clips covering 700 different human activities (in total over 1800 h). By contrast, the total length of all sensor-based HAR data sets in the popular UCI machine learning repository is less than 63 h, with around 38 of those consisting of simple mode of locomotion activities like walking, standing or cycling. In our research we aim to facilitate the use of online videos, which exist in ample quantities for most activities and are much easier to label than sensor data, to simulate labeled wearable motion sensor data. In previous work we already demonstrated some preliminary results in this direction, focusing on very simple, activity specific simulation models and a single sensor modality (acceleration norm). In this paper, we show how we can train a regression model on generic motions for both accelerometer and gyro signals and then apply it to videos of the target activities to generate synthetic Inertial Measurement Units (IMU) data (acceleration and gyro norms) that can be used to train and/or improve HAR models. We demonstrate that systems trained on simulated data generated by our regression model can come to within around 10% of the mean F1 score of a system trained on real sensor data. Furthermore, we show that by either including a small amount of real sensor data for model calibration or simply leveraging the fact that (in general) we can easily generate much more simulated data from video than we can collect its real version, the advantage of the latter can eventually be equalized.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Cross-modality transfer approaches [54,108,109,161,162] are instrumental for extracting virtual IMU data from 2D RGB videos of activities.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [54]
In the citing paper
“In recent years, there has been a growing interest in the combination of data obtained from videobased systems together with the time-series data originating from on-body sensors Rey et al. (2019); Kwon et al. (2020); Fortes Rey et al. (2021).”
From Opportunity++: A Multimodal Dataset for Video- and Wearable, Object and Ambient Sensors-Based Human Activity Recognition (2021) · cites this paper as Fortes Rey et al. (2021)
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Residual connections help train deeper networks by adding the output of convolutions to an identity function [43], while dilated convolutions increase the receptive field by orders of magnitude without increasing the computational cost.”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. ImageNet Large Scale Visual Recognition Challenge.”
BackgroundInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.10.045 · Cited by 20,384 · cited 4×
“It may also be necessary to introduce onto the model mechanisms for explicitly encoding physical boundary conditions as proposed, e.g., by the concept of Physically Informed Neural Networks [45].”
MethodInfluentialRealtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · cited 2×
“The application of standard tools (specifically Open Pose [21,41]) to identify humans in videos and extraction of their 2D poses.”
MethodOpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536 · cited 4×
“Obtaining 3D poses is possible with a calibrated system of more than one camera [21].”
BackgroundMethodInfluentialA Survey on Human Activity Recognition using Wearable Sensors
Lara, O. D., & Labrador, M. A. (2013) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2012.110112.00192 · Cited by 2,662 · cited 1×
“Human activity recognition (HAR) using wearable sensors has been a successful research field [1,2] for nearly two decades.”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 1×
“There has been the expectation that as large amounts of sensor data become available, HAR will be able to benefit from the advances in Deep Learning techniques in the same way as fields, such as computer vision and speech recognition [3,4].”
BackgroundDGM: A deep learning algorithm for solving partial differential equations
Sirignano, J., & Spiliopoulos, K. (2018) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.08.029 · Cited by 2,208 · cited 1×
“There has recently been increased interest in exploiting the neural networks capability as universal function approximators for the prediction of the dynamics of physical systems, in particular the solution of differential equations [34] replacing traditional methods such as FEMs.”
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014 · cited 2×
“Human activity recognition (HAR) using wearable sensors has been a successful research field [1,2] for nearly two decades.”
BackgroundEnd-to-End Recovery of Human Shape and Pose
Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00744 · Cited by 1,870
Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., & Black, M. J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_34 · Cited by 1,498 · cited 2×
“Extracting those poses from a monocular camera is no easy task due to occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D [29].”
BackgroundMethodIntroducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 1×
“Probably the most popular benchmark datasets in the wearable sensing HAR community are PAMAP2 [8,9] and Opportunity [10,11] with, respectively, less than 8 h of labeled data and less than 14 even if we count all activity granularities.”
BackgroundVNect: real-time 3D human pose estimation with a single RGB camera
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073596 · Cited by 1,113 · cited 1×
“Some works [24,27] predict the 2D joints and then fit the 3D skeleton using different strategies, others [23,25,26] are trained to predict 3D joints directly from the image.”
BackgroundCollecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
Self-citationThe Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 2×
“Probably the most popular benchmark datasets in the wearable sensing HAR community are PAMAP2 [8,9] and Opportunity [10,11] with, respectively, less than 8 h of labeled data and less than 14 even if we count all activity granularities.”
BackgroundNeural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation
Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., & Schiele, B. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00062 · Cited by 498
Pose-Normalized Image Generation for Person Re-identification
Qian, X., Fu, Y., Xiang, T., Wang, W., Qiu, J., Wu, Y., Jiang, Y.-G., & Xue, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_40 · Cited by 468
Creating and benchmarking a new dataset for physical activity monitoring
Reiss, A., & Stricker, D. (2012) · Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments · doi:10.1145/2413097.2413148 · Cited by 295 · cited 1×
“Probably the most popular benchmark datasets in the wearable sensing HAR community are PAMAP2 [8,9] and Opportunity [10,11] with, respectively, less than 8 h of labeled data and less than 14 even if we count all activity granularities.”
BackgroundLCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
Rogez, G., Weinzaepfel, P., & Schmid, C. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2892985 · Cited by 272 · cited 1×
“Some works [24,27] predict the 2D joints and then fit the 3D skeleton using different strategies, others [23,25,26] are trained to predict 3D joints directly from the image.”
BackgroundKnoll, F., Murrell, T., Sriram, A., Yakubova, N., Zbontar, J., Rabbat, M., Defazio, A., Muckley, M. J., Sodickson, D. K., Zitnick, C. L., et al. (2020) · Magnetic Resonance in Medicine · doi:10.1002/mrm.28338 · Cited by 263
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411841 · Cited by 152 · cited 2×
“Recent work that follows this approach is [38], which uses off the shelf methods to predict 3D human poses for monocular images and then”
BackgroundOn the effect of data set size on bias and variance in classification learning
Brain, D., & Webb, G. I. (1999) · Cited by 84
Li, X., Luo, J., & Younes, R. (2020) · Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers · doi:10.1145/3410530.3414367 · Cited by 71 · cited 1×
“While more known in the computer vision community, GANs have shown great promise also in mobile sensors where they have been used to augment datasets and even improve models using semi-supervised learning [30,31].”
BackgroundSummary of the sussex-huawei locomotion-transportation recognition challenge 2020
Wang, L., Gjoreski, H., Ciliberto, M., Lago, P., Murao, K., Okita, T., & Roggen, D. (2020) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3410530.3414341 · Cited by 61
IMUSim: A simulation environment for inertial sensing algorithm design and evaluation
Young, A. D., Ling, M. J., & Arvind, D. K. (2011) · Information Processing in Sensor Networks · Cited by 56
SenseGAN: Enabling Deep Learning for Internet of Things with a Semi-Supervised Framework
Yao, S., Zhao, Y., Shao, H., Zhang, C., Zhang, A., Hu, S., Liu, D., Liu, S., Su, L., & Abdelzaher, T. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264954 · Cited by 54
Rey, V. F., Hevesi, P., Kovalenko, O., & Lukowicz, P. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3345590 · Cited by 43 · cited 3×
“As one of the first published approaches in this direction we have previously shown [14] initial results on how regression models can be trained to simulate motion sensor data from videos of the respective activities.”
BackgroundMethodResultSelf-citationSimulation of foot-mounted IMU signals for the evaluation of PDR algorithms
Zampella, F., Jiménez, A. R., Seco, F. A., Prieto, J. C., & Guevara, J. I. (2011) · International Conference on Indoor Positioning and Indoor Navigation · doi:10.1109/ipin.2011.6071930 · Cited by 35 · cited 1×
“For example, there is ample work in simulating foot IMU data for pedestrian dead reckoning [18].”
BackgroundBanos, O., Calatroni, A., Damas, M., Pomares, H., Rojas, I., Sagha, H., del R. Mill´n, J., Troster, G., Chavarriaga, R., & Roggen, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.17 · Cited by 33 · cited 2×
“Other approaches rely on on-body markers [36,37] or systems with multiple cameras such as kinect [22].”
BackgroundMethodVision2Sensor: Knowledge Transfer Across Sensing Modalities for Human Activity Recognition
Radu, V., & Henne, M. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3351242 · Cited by 33 · cited 16×
“A big HAR related video data set, the Kinetics-700 [13] has 650,000 video clips covering 700 different human activities.”
BackgroundInfluentialFully Automatic Multi-person Human Motion Capture for VR Applications
Elhayek, A., Kovalenko, O., Murthy, P., Malik, J. N., & Stricker, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01790-3_3 · Cited by 24 · cited 1×
“Some works [24,27] predict the 2D joints and then fit the 3D skeleton using different strategies, others [23,25,26] are trained to predict 3D joints directly from the image.”
BackgroundA Multi-Sensor Setting Activity Recognition Simulation Tool
Takeda, S., Okita, T., Lago, P., & Inoue, S. (2018) · Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers · doi:10.1145/3267305.3267509 · Cited by 18 · cited 1×
“Other approaches rely on on-body markers [36,37] or systems with multiple cameras such as kinect [22].”
BackgroundLago, P., Takeda, S., Okita, T., & Inoue, S. (2019) · Springer series in adaptive environments · doi:10.1007/978-3-030-13001-5_10 · Cited by 10
Asare, P., Dickerson, R. F., Wu, X., Lach, J., & Stankovic, J. A. (2013) · Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2517351.2517392 · Cited by 8 · cited 2×
“The most obvious choice is between (1) using existing ML methods for estimating 3D poses from 2D videos [23–27] and then using bio mechanical models [15] to generate the sensor data and (2) training an end-to-end ML model that goes directly from 2D video to sensor data.”
MethodNew Pedestrian Trajectory Simulator to Study Innovative Yaw Angle Constraints
Ascher, C., Kessler, C., Maier, A., Crocoll, P., & Trommer, G. F. (2010) · Proceedings of the Satellite Division's International Technical Meeting (Online)/Proceedings of the Satellite Division's International Technical Meeting (CD-ROM) · Cited by 5
Domain Adaptation of IMU sensors using Generative Adversarial Networks
Radhakrishnan, S. (2020) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 1
Research works printed but not linked: 7
- Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., and Zisserman, A. (2020). A Short Note on the Kinetics-700-2020 Human Action Dataset. arXiv.
- Smith, M., Moore, T., Hill, C., Noakes, C., and Hide, C. (2003, January 22–23). Simulation of GNSS/IMU measurements. Proceedings of the ISPRS International Workshop. Working Group I/5: Theory, Technology and Realities of Inertial/GPS Sensor Orientation, Castelldefels, Spain.
- Parés, M., Rosales, J., and Colomina, I. (February, January 30). Yet another IMU simulator: Validation and applications. Proceedings of the Eurocow, Castelldefels, Spain.
- Murthy, P., Kovalenko, O., Elhayek, A., Gava, C.C., and Stricker, D. (2017). 3D Human Pose Tracking inside Car using Single RGB Spherical Camera. Proceedings of the ACM Chapters Computer Science in Cars Symposium (CSCS), ACM.
- Raissi, M., Perdikaris, P., and Karniadakis, G.E. (2017). Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv.
- Bai, S., Kolter, J.Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv.
- Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
Cited by
Ciliberto, M., Fortes Rey, V., Calatroni, A., Lukowicz, P., & Roggen, D. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.792065 · Cited by 35
“In recent years, there has been a growing interest in the combination of data obtained from video-based systems together with the time-series data originating from on-body sensors Rey et al. (2019); Kwon et al. (2020); Fortes Rey et al. (2021).”
BackgroundSelf-citationHaresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729467 · Cited by 33
“Virtual IMU data generation has emerged as a viable solution to overcome data scarcity, with cross-modality transfer approaches [53, 103, 104, 153, 154] being instrumental in extracting virtual IMU data from 2D RGB videos of human activities.”
Self-citationSynthetic Smartwatch IMU Data Generation from In-the-wild ASL Videos
Santhalingam, P. S., Pathak, P., Rangwala, H., & Kosecka, J. (2023) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3596261 · Cited by 28
“Generative methods proposed in [14, 18] apply machine learning to learn a function that can derive IMU data from videos.”
BackgroundMethodOptimizing Sensor Position with Virtual Sensors in Human Activity Recognition System Design
Xia, C., & Sugiura, Y. (2021) · Sensors · doi:10.3390/s21206893 · Cited by 29
“Additionally, another scheme enabling the end-to-end ML network has also been designed [35–37].”
MethodIf only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios
Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150267 · Cited by 16
Metadata
- Authors
- Fortes Rey, V., Garewal, K. K., & Lukowicz, P.
- Affiliations
- Embedded Intelligence, German Research Center for Artificial Intelligence (DFKI), 67663 Kaiserslautern, Germany
- German Research Centre for Artificial Intelligence (DE)
- University of Kaiserslautern (DE)
- Corresponding author
- Garewal, Kamalveer Kaur
- Lukowicz, Paul
- ORCID
- Lukowicz, Paul
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-03-31
- Volume
- 11
- Issue
- 7
- Pages
- 3094
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3146293857
- Cited by
- 17
- References
- 38
- Citations by year
- 2025: 5 · 2024: 1 · 2023: 5 · 2022: 4 · 2021: 2
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Bundesministerium für Bildung und Forschung (01IW18001)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Physical Activity and Health
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human activity recognition, wearable sensors, synthetic training data, inertial measurement unit, data augmentation, model calibration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.