CoChat
Sign Up
PreprintCited by 3Context-Aware Activity Recognition SystemsHuman Pose and Action RecognitionHand Gesture Recognition Systems

The DOI matches its DataCite record.

Yet it moves: Learning from Generic Motions to Generate IMU data from YouTube videos

Open access
Rey, Vitor Fortes, Garewal, Kamalveer Kaur, Lukowicz, Paul·2020·arXiv
DOI 10.48550/arxiv.2011.11600arXiv 2011.11600
Open this paper in CoChat — free accountOpen PDF

Abstract

Human activity recognition (HAR) using wearable sensors has benefited much less from recent advances in Machine Learning than fields such as computer vision and natural language processing. This is to a large extent due to the lack of large scale repositories of labeled training data. In our research we aim to facilitate the use of online videos, which exists in ample quantity for most activities and are much easier to label than sensor data, to simulate labeled wearable motion sensor data. In previous work we already demonstrate some preliminary results in this direction focusing on very simple, activity specific simulation models and a single sensor modality (acceleration norm)\cite{10.1145/3341162.3345590}. In this paper we show how we can train a regression model on generic motions for both accelerometer and gyro signals and then apply it to videos of the target activities to generate synthetic IMU data (acceleration and gyro norms) that can be used to train and/or improve HAR models. We demonstrate that systems trained on simulated data generated by our regression model can come to within around 10% of the mean F1 score of a system trained on real sensor data. Furthermore we show that by either including a small amount of real sensor data for model calibration or simply leveraging the fact that (in general) we can easily generate much more simulated data from video than we can collect in terms of real sensor data the advantage of real sensor data can be eventually equalized.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.11600

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. [1][1] Vitor Fortes Rey, Peter Hevesi, Onorina Kovalenko, and Paul Lukowicz. Let there be imu data: Generating training data for wearable, motion sensor based activity recognition from monocular rgb videos. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, UbiComp/ISWC '19 Adjunct, page 699–708, New York, NY, USA, 2019. Association for Computing Machinery.
  2. [2][2] Oscar D Lara and Miguel A Labrador. A survey on human activity recognition using wearable sensors. IEEE communications surveys & tutorials, 15(3):1192–1209, 2012.
  3. [3][3] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters, 119:3–11, 2019.
  4. [4][4] Lin Wang, Hristijan Gjoreski, Mathias Ciliberto, Paula Lago, Kazuya Murao, Tsuyoshi Okita, and Daniel Roggen. Summary of the sussex-huawei locomotion-transportation recognition challenge 2020. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, UbiComp-ISWC '20, page 351–358, New York, NY, USA, 2020. Association for Computing Machinery.
  5. [5][5] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  6. [7][7] Philip Asare, Robert F Dickerson, Xianyue Wu, John Lach, and John A Stankovic. Bodysim: a multi-domain modeling and simulation framework for body sensor networks research and design. In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems, pages 1–2, 2013. 27 A PREPRINT, SUBMITTED TO IMWUT IN NOVEMBER 2020
  7. [8][8] C Ascher, C Kessler, A Maier, P Crocoll, and GF Trommer. New pedestrian trajectory simulator to study innovative yaw angle constraints. In Proceedings of the 23rd International Technical Meeting of The Satellite Division of the Institute of Navigation (ION GNSS 2010), pages 504–510, 2010.
  8. [9][9] Alexander D Young, Martin J Ling, and Damal K Arvind. Imusim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks, pages 199–210. IEEE, 2011.
  9. [10][10] Francisco J Zampella, Antonio R Jiménez, Fernando Seco, J Carlos Prieto, and Jorge I Guevara. Simulation of foot-mounted imu signals for the evaluation of pdr algorithms. In 2011 International Conference on Indoor Positioning and Indoor Navigation, pages 1–7. IEEE, 2011.
  10. [11][11] MJ Smith, T Moore, CJ Hill, CJ Noakes, and C Hide. Simulation of gnss/imu measurements. In ISPRS International Workshop. Working Group I/5: Theory, Technology and Realities of Inertial/GPS Sensor Orientation, pages 22–23, 2003.
  11. [12][12] ME Parés, JJ Rosales, and I Colomina. Yet another imu simulator: validation and applications. Proceedings of the Eurocow, Castelldefels, Spain, 30, 2008.
  12. [13][13] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008, 2018. · arXiv
  13. [14][14] O. Banos, A. Calatroni, M. Damas, H. Pomares, I. Rojas, H. Sagha, J. del R. Milln, G. Troster, R. Chavarriaga, and D. Roggen. Kinect=imu? learning mimo signal mappings to automatically translate activity recognition systems across sensor modalities. In 2012 16th International Symposium on Wearable Computers, pages 92–99, 2012.
  14. [15][15] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7122–7131, 2018.
  15. [16][16] Ahmed Elhayek, Onorina Kovalenko, Pramod Murthy, Jameel Malik, and Didier Stricker. Fully automatic multi-person human motion capture for vr applications. In International Conference on Virtual Reality and Augmented Reality - EuroVR, pages 28–47. Springer, 2018.
  16. [17][17] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (TOG), 36(4):44, 2017.
  17. [18][18] Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. Lcr-net++: Multi-person 2d and 3d pose detection in natural images. IEEE transactions on pattern analysis and machine intelligence, 2019.
  18. [20][20] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In 2018 international conference on 3D vision (3DV), pages 484–494. IEEE, 2018.
  19. [21][21] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In European Conference on Computer Vision, pages 561–578. Springer, 2016.
  20. [22][22] Shingo Takeda, Tsuyoshi Okita, Paula Lago, and Sozo Inoue. A multi-sensor setting activity recognition simulation tool. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, UbiComp '18, pages 1444–1448, New York, NY, USA, 2018. ACM.
  21. [23][23] Paula Lago, Shingo Takeda, Tsuyoshi Okita, and Sozo Inoue. Measured: Evaluating sensor-based activity recognition scenarios by simulating accelerometer measures from motion capture. In Human Activity Sensing, pages 135–149. Springer, 2019.
  22. [24][24] Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D Abowd, Nicholas D Lane, and Thomas Ploetz. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. arXiv preprint arXiv:2006.05675, 2020. · arXiv
  23. [25][25] Valentin Radu and Maximilian Henne. Vision2sensor: Knowledge transfer across sensing modalities for human activity recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3(3), September 2019.
  24. [26][26] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017. 28 http://arxiv.org/abs/1812.08008 http://arxiv.org/abs/2006.05675 A PREPRINT, SUBMITTED TO IMWUT IN NOVEMBER 2020 · arXiv
  25. [27][27] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. · arXiv
  26. [28][28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  27. [29][29] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. · arXiv
  28. [30][30] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019. 29 http://arxiv.org/abs/1803.01271 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Rey, V. F., Garewal, K. K., & Lukowicz, P.
ORCID
Lukowicz, Paul
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.11600
arXiv
2011.11600
OpenAlex
W3108536583
Cited by
3
References
0
Citations by year
2023: 1 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
human activity recognition, wearable sensors, synthetic sensor data, YouTube videos, regression models, data augmentation, model calibration, limited training data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review