Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.

IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

Kwon, Hyeokhyen, Tong, Catherine, Haresamudram, Harish, Gao, Yan, Abowd, Gregory D., Lane, Nicholas D., Plötz, Thomas · 2020 · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Abstract

The lack of large-scale, labeled data sets impedes progress in developing robust and generalized predictive models for on-body sensor-based human activity recognition (HAR). Labeled data in human activity recognition is scarce and hard to come by, as sensor data collection is expensive, and the annotation is time-consuming and error-prone. To address this problem, we introduce IMUTube, an automated processing pipeline that integrates existing computer vision and signal processing techniques to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets. Our initial results are very promising, but the greater promise of this work lies in a collective approach by the computer vision, signal processing, and activity recognition communities to extend this work in ways that we outline. This should lead to on-body, sensor-based HAR becoming yet another success story in large-dataset breakthroughs in recognition.

Overview

Integrity signals

Registry

Crossref · 10.1145/3411841

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 49 references

Research works printed but not linked: 40

  1. S. Alireza Golestaneh and L. Karam . 2017. Spatially-varying blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5800--5809 . S. Alireza Golestaneh and L. Karam. 2017. Spatially-varying blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5800--5809.
  2. T. Alldieck , M. Magnor , B. Bhatnagar , C. Theobalt , and G. Pons-Moll . 2019. Learning to reconstruct people in clothing from a single RGB camera . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1175--1186 . T. Alldieck, M. Magnor, B. Bhatnagar, C. Theobalt, and G. Pons-Moll. 2019. Learning to reconstruct people in clothing from a single RGB camera. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1175--1186.
  3. M. Andriluka , L. Pishchulin , P. Gehler , and B. Schiele . 2014. 2D Human Pose Estimation: New Benchmark and State of the Art Analysis . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2014. 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  4. O. Bogdan , V. Eckstein , F. Rameau , and J. Bazin . 2018 . DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras . In Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production. ACM , 6 : 1 - 6 :10. O. Bogdan, V. Eckstein, F. Rameau, and J. Bazin. 2018. DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras. In Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production. ACM, 6:1-6:10.
  5. Z. Cao , T. Simon , S. Wei , and Y. Sheikh . 2017. Realtime multi-person 2d pose estimation using part affinity fields . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7291--7299 . Z. Cao, T. Simon, S. Wei, and Y. Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7291--7299.
  6. Blender Online Community . 2018. Blender - a 3D modelling and rendering package . Blender Foundation, Stichting Blender Foundation, Amsterdam . http://www.blender.org Blender Online Community. 2018. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam. http://www.blender.org
  7. C. Gu , C. Sun , D. Ross , C. Vondrick , C. Pantofaru , Y. Li , S. Vijayanarasimhan , G. Toderici , S. Ricco , R. Sukthankar , C. Schmid , and J. Malik . 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6047--6056 . C. Gu, C. Sun, D. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, C. Schmid, and J. Malik. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6047--6056.
  8. S. Haradal , H. Hayashi , and S. Uchida . 2018. Biosignal data augmentation based on generative adversarial networks . In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 368--371 . S. Haradal, H. Hayashi, and S. Uchida. 2018. Biosignal data augmentation based on generative adversarial networks. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 368--371.
  9. K. He , X. Zhang , S. Ren , and J. Sun . 2016. Deep residual learning for image recognition . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778 . K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778.
  10. I. Joel , A.and Stergios. 2011 . A Direct Least-Squares (DLS) method for PnP . In IEEE International Conference on Computer Vision (ICCV). IEEE. I. Joel, A.and Stergios. 2011. A Direct Least-Squares (DLS) method for PnP. In IEEE International Conference on Computer Vision (ICCV). IEEE.
  11. A. Kanazawa , M. Black , D. Jacobs , and J. Malik . 2018. End-to-end recovery of human shape and pose . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7122--7131 . A. Kanazawa, M. Black, D. Jacobs, and J. Malik. 2018. End-to-end recovery of human shape and pose. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7122--7131.
  12. C. Kang , H. Jung , and Y. Lee . 2019 . Towards Machine Learning with Zero Real-World Data. In The ACM Workshop on Wearable Systems and Applications. 41--46 . C. Kang, H. Jung, and Y. Lee. 2019. Towards Machine Learning with Zero Real-World Data. In The ACM Workshop on Wearable Systems and Applications. 41--46.
  13. P. Karlsson , B. Lo , and G. Z. Yang . 2014. Inertial sensing simulations using modified motion capture data . In Proceedings of the International Conference on Wearable and Implantable Body Sensor Networks (BSN). 16--19 . P. Karlsson, B. Lo, and G. Z. Yang. 2014. Inertial sensing simulations using modified motion capture data. In Proceedings of the International Conference on Wearable and Implantable Body Sensor Networks (BSN). 16--19.
  14. D. Kingma and J. Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
  15. N. Lane , Y. Xu , H. Lu , S. Hu , T. Choudhury , A. Campbell , and F. Zhao . 2011. Enabling Large-Scale Human Activity Inference on Smartphones Using Community Similarity Networks . In Proceedings of the International Conference on Ubiquitous Computing. ACM, 355--364 . N. Lane, Y. Xu, H. Lu, S. Hu, T. Choudhury, A. Campbell, and F. Zhao. 2011. Enabling Large-Scale Human Activity Inference on Smartphones Using Community Similarity Networks. In Proceedings of the International Conference on Ubiquitous Computing. ACM, 355--364.
  16. A. Le Guennec , S. Malinowski , and R. Tavenard . 2016. Data Augmentation for Time Series Classification using Convolutional Neural Networks . In ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data. A. Le Guennec, S. Malinowski, and R. Tavenard. 2016. Data Augmentation for Time Series Classification using Convolutional Neural Networks. In ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data.
  17. N. Mahmood , N. Ghorbani , N. Troje , G. Pons-Moll , and M. Black . 2019. AMASS: Archive of motion capture as surface shapes . In IEEE International Conference on Computer Vision (ICCV). 5442--5451 . N. Mahmood, N. Ghorbani, N. Troje, G. Pons-Moll, and M. Black. 2019. AMASS: Archive of motion capture as surface shapes. In IEEE International Conference on Computer Vision (ICCV). 5442--5451.
  18. A. Mathur , T. Zhang , S. Bhattacharya , P. Velickovic , L. Joffe , N. Lane , F. Kawsar , and P. Lió . 2018. Using deep data augmentation training to address software and hardware heterogeneities in wearable and smartphone sensing devices . In IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 200--211 . A. Mathur, T. Zhang, S. Bhattacharya, P. Velickovic, L. Joffe, N. Lane, F. Kawsar, and P. Lió. 2018. Using deep data augmentation training to address software and hardware heterogeneities in wearable and smartphone sensing devices. In IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 200--211.
  19. V. Nair and G. Hinton . 2010. Rectified linear units improve restricted boltzmann machines . In Proceedings of the international conference on machine learning (ICML). 807--814 . V. Nair and G. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the international conference on machine learning (ICML). 807--814.
  20. F. Ofli , R. Chaudhry , G. Kurillo , R. Vidal , and R. Bajcsy . 2013. Berkeley mhad: A comprehensive multimodal human action database . In IEEE Workshop on Applications of Computer Vision (WACV). IEEE, 53--60 . F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy. 2013. Berkeley mhad: A comprehensive multimodal human action database. In IEEE Workshop on Applications of Computer Vision (WACV). IEEE, 53--60.
  21. A. Oord , S. Dieleman , H. Zen , K. Simonyan , O. Vinyals , A. Graves , N. Kalchbrenner , A. Senior , and K. Kavukcuoglu . 2016 . Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016). A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016). · arXiv
  22. D. Pavllo , C. Feichtenhofer , D. Grangier , and M. Auli . 2019. 3D human pose estimation in video with temporal convolutions and semi-supervised training . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7753--7762 . D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 2019. 3D human pose estimation in video with temporal convolutions and semi-supervised training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7753--7762.
  23. T. Plötz , C. Chen , N. Hammerla , and G. Abowd . 2012. Automatic synchronization of wearable sensors and video-cameras for ground truth annotation-a practical approach . In Proceedings of the ACM International Symposium on Wearable Computers. IEEE, 100--103 . T. Plötz, C. Chen, N. Hammerla, and G. Abowd. 2012. Automatic synchronization of wearable sensors and video-cameras for ground truth annotation-a practical approach. In Proceedings of the ACM International Symposium on Wearable Computers. IEEE, 100--103.
  24. G. Ramponi , P. Protopapas , M. Brambilla , and R. Janssen . 2018 . T-cgan: Conditional generative adversarial network for data augmentation in noisy time series with irregular sampling. arXiv preprint arXiv:1811.08295 (2018). G. Ramponi, P. Protopapas, M. Brambilla, and R.Janssen. 2018. T-cgan: Conditional generative adversarial network for data augmentation in noisy time series with irregular sampling. arXiv preprint arXiv:1811.08295 (2018). · arXiv
  25. J. Redmon , S. Divvala , R. Girshick , and A. Farhadi . 2016. You only look once: Unified, real-time object detection . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779--788 . J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You only look once: Unified, real-time object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779--788.
  26. A. Reiss and D. Stricker . 2012. Introducing a new benchmarked dataset for activity monitoring . In Proceedings of the ACM International Symposium on Wearable Computers. IEEE, 108--109 . A. Reiss and D. Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the ACM International Symposium on Wearable Computers. IEEE, 108--109.
  27. A. Reiss and D. Stricker . 2013. Personalized mobile physical activity recognition . In Proceedings of the ACM International Symposium on Wearable Computers. 25--28 . A. Reiss and D. Stricker. 2013. Personalized mobile physical activity recognition. In Proceedings of the ACM International Symposium on Wearable Computers. 25--28.
  28. M. Rosca B. Lakshminarayanan and S. Mohamed. 2018. Distribution matching in variational inference. arXiv preprint arXiv:1802.06847 (2018). M. Rosca B. Lakshminarayanan and S. Mohamed. 2018. Distribution matching in variational inference. arXiv preprint arXiv:1802.06847 (2018). · arXiv
  29. A. Saeed , T. Ozcelebi , and J. Lukkien . 2019. Multi-task Self-Supervised Learning for Human Activity Detection . Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 3 , 2 ( 2019 ), 1--30. A. Saeed, T. Ozcelebi, and J. Lukkien. 2019. Multi-task Self-Supervised Learning for Human Activity Detection. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 3, 2 (2019), 1--30.
  30. Z. Shen , W. Wang , X. Lu , J. Shen , H. Ling , T. Xu , and L. Shao . 2019 . Human-Aware Motion Deblurring. In IEEE International Conference on Computer Vision (ICCV). 5572--5581 . Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao. 2019. Human-Aware Motion Deblurring. In IEEE International Conference on Computer Vision (ICCV). 5572--5581.
  31. J. Shi , L. Xu , and J. Jia . 2014. Discriminative blur detection features . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2965--2972 . J. Shi, L. Xu, and J. Jia. 2014. Discriminative blur detection features. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2965--2972.
  32. G. Sigurdsson , G. Varol , X. Wang , I. Laptev , A. Farhadi , and A. Gupta . 2016 . Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. arXiv preprint arXiv:1604.01753 (2016). G. Sigurdsson, G. Varol, X. Wang, I. Laptev, A. Farhadi, and A. Gupta. 2016. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. arXiv preprint arXiv:1604.01753 (2016). · arXiv
  33. E. Thomaz , I. Essa , and G. Abowd . 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing . In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 1029--1040 . E. Thomaz, I. Essa, and G. Abowd. 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 1029--1040.
  34. C. Tong , S. Tailor , and N. Lane . 2020. Are Accelerometers for Activity Recognition a Dead-End? . In Proceedings of the International Workshop on Mobile Computing Systems and Applications. ACM, 39--44 . C. Tong, S. Tailor, and N. Lane. 2020. Are Accelerometers for Activity Recognition a Dead-End?. In Proceedings of the International Workshop on Mobile Computing Systems and Applications. ACM, 39--44.
  35. F. Xiao L. Pei L. Chu D. Zou W. Yu Y. Zhu and T. Li. 2020. A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors. arXiv preprint arXiv:2003.01874 (2020). F. Xiao L. Pei L. Chu D. Zou W. Yu Y. Zhu and T. Li. 2020. A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors. arXiv preprint arXiv:2003.01874 (2020). · arXiv
  36. S. Yao , Y. Zhao , H. Shao , C. Zhang , A. Zhang , S. Hu , D. Liu , S. Liu , Lu Su , and T. Abdelzaher . 2018. Sensegan: Enabling deep learning for internet of things with a semi-supervised framework . Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 2 , 3 ( 2018 ), 1--21. S. Yao, Y. Zhao, H. Shao, C. Zhang, A. Zhang, S. Hu, D. Liu, S. Liu, Lu Su, and T. Abdelzaher. 2018. Sensegan: Enabling deep learning for internet of things with a semi-supervised framework. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 2, 3 (2018), 1--21.
  37. A. Young , M. Ling , and D. Arvind . 2011. IMUSim: A simulation environment for inertial sensing algorithm design and evaluation . In Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 199--210 . A. Young, M. Ling, and D. Arvind. 2011. IMUSim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 199--210.
  38. J. Yu and R. Ramamoorthi . 2019. Robust Video Stabilization by Optimization in CNN Weight Space . In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3800--3808 . J. Yu and R. Ramamoorthi. 2019. Robust Video Stabilization by Optimization in CNN Weight Space. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3800--3808.
  39. M. Zhang and A. A. Sawchuk . 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors . In Proceedings of the International Conference on Ubiquitous Computing. M. Zhang and A. A. Sawchuk. 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the International Conference on Ubiquitous Computing.
  40. Q. Zhang and R. Pless . 2004. Extrinsic calibration of a camera and laser range finder (improves camera calibration) . In IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE. Q. Zhang and R. Pless. 2004. Extrinsic calibration of a camera and laser range finder (improves camera calibration). In IEEE International Conference on Intelligent Robots and Systems (IROS). IEEE.

Cited by

See all 156

Metadata

Authors
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T.
Affiliations
  • Georgia Institute of Technology (US)
  • University of Oxford (GB)
Venue
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Published
2020-09-04
Volume
4
Issue
3
Pages
1-29
ISSN
2474-9567 (ISSN-L)
OpenAlex
W3083323811
Cited by
152
References
49
Citations by year
2026: 20 · 2025: 29 · 2024: 24 · 2023: 32 · 2022: 23 · 2021: 24
FWCI
7.68
Type
journal article
Language
en
Open access
closed
Locations
Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion)
Funders
Engineering and Physical Sciences Research Council (1892895)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
Keywords
human activity recognition, on-body sensors, inertial measurement unit, automatic annotation, accelerometry, wearable sensors, labeled data scarcity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition. https://app.cochat.ai/papers/10.1145/3411841
The paper
Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., & Plötz, T. (2020). IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. https://doi.org/10.1145/3411841

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.