CoChat

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Open access

The DOI matches its Crossref record; 48 of the 87 references it cites were found in a registry or an index, and 3 are not checked yet; 6 of the 50 sources its claims cite were checked for support.

Abstract

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

Verified 2026-10-05 · 3 references not checked yet

Overview

CoChat Summary

TL;DR

The paper shows that large-scale virtual IMU data generated by IMUTube can train much more complex HAR models than previously feasible, using a model with about 1100× more trainable parameters than a standard ConvNet. On 13 dumbbell exercises, the best model reaches 80.2% mean F1, improving by 17.6 absolute points over a CNN baseline and 6.2 over DeepConvLSTM.

Key findings

  • The proposed model trained with virtual IMU data achieved 80.2% mean F1, a +17.6% absolute improvement over the CNN baseline and +6.2% absolute over DeepConvLSTM.
  • The full architecture contains approximately 1100× more trainable parameters than the convolutional baseline, with reported gains of 12.64% absolute F1 using real IMU only and 17.63% absolute F1 using real + virtual IMU.
  • Adaptive trimming alone increased parameters by about 10× and improved F1 by 4.33% on average; multi-window kernels added another ~10× complexity and 3.44% absolute F1.
  • Multi-view kernels gave 3.58% absolute F1 on average, skip connections 2.65% with virtual IMU, while recurrent layers and uncertainty modeling gave only marginal, non-significant improvements (0.19% and 0.27% absolute F1, respectively).

Methods

  • The model combines adaptive sensor-window trimming, multi-scale hourglass convolutions, multi-window kernels, multi-view time-channel kernels, skip connections, multi-scale recurrent aggregation, and heteroscedastic uncertainty modeling for noisy samples.
  • Virtual IMU data were generated with IMUTube from YouTube videos for 13 dumbbell exercises, yielding >41 h of virtual data; real IMU data came from MyoGym (10 users, 30 free-weight activities, 2.81 min per activity on average, about 36 min total for calibration).
  • Evaluation used 30 Hz data, 4-second windows with 1-second overlap, subjects 1–5 for training, 6–7 for validation, 8–10 for testing, mean F1 score, and Wilson score intervals from 15 runs; models were pretrained on calibrated virtual IMU data and fine-tuned on real IMU data.

Limitations

  • The paper states the virtual dataset is still relatively small compared with datasets like ImageNet, and that increasing virtual data may introduce noise that could counteract performance gains.
  • The paper notes the studied gym exercises are relatively constrained, the architecture components were manually designed rather than learned via NAS, and uncertainty modeling had only marginal effect.
  • Evident from the text: evaluation uses a single wrist sensor, 13 exercises, only 10 real users, weak labels from YouTube search terms, and a domain gap between virtual and real IMU data that requires calibration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s21248337

Cites

48 of 87 references verified0 citation errors · 36 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Deriving human activity recognition systems from large scale virtual IMU data generated through IMUTube [14]. Videos for specific activities are retrieved from public repositories such as YouTube through keyword-based queries. These search terms serve as weak labels for all generated sensor data. The IMUTube system then processes the retrieved videos to (i) suppress irrelevant videos; (ii) eliminate portions of the videos that are of insufficient quality (motion blur, occlusions, etc.); (iii) generates virtual IMU data for on-body sensor positions as specified by the query; and (iv) calibrates the virtual IMU data towards more realistic sensor data. The resulting, large-scale virtual IMU dataset is then used for HAR model training.
Figure 1 Deriving human activity recognition systems from large scale virtual IMU data generated through IMUTube [14]. Videos for specific activities are retrieved from public repositories such as YouTube through keyword-based queries. These search terms serve as weak labels for all generated sensor data. The IMUTube system then processes the retrieved videos to (i) suppress irrelevant videos; (ii) eliminate portions of the videos that are of insufficient quality (motion blur, occlusions, etc.); (iii) generates virtual IMU data for on-body sensor positions as specified by the query; and (iv) calibrates the virtual IMU data towards more realistic sensor data. The resulting, large-scale virtual IMU dataset is then used for HAR model training.CC BY · PubMed Central
Figure 2 Overview of the novel model architecture for deep learning-based human activity recognition using wearables. An input sensor frame is first processed with Adaptive Trimming (AdapTrimm or AT) to determine the sub-window region that contains the core motion of the target activity. The feature representation is then extracted for this core motion sub-window (green segment). Non-linear multi-scale convolution (MS Conv) captures temporal dynamics at multiple temporal scales. Multi-kernel window convolution (MW Conv) captures varying lengths of motion signals from multi-scale features. Multi-view convolution (MV Conv) uses diverse shapes of convolutional kernels to capture time-channel correlations. The feature representation from intermediate convolutional layers is used as input to the final classifier in the form of auxiliary inputs with skip-connections to enhance gradient flow in the bottom layers of the recognition network. The convolutional feature representations are aggregated by a non-linear multi-scale recurrent neural network (MS RNN), which is used to predict the activity class of the input sensor frame. Along with the class label, the confidence for the prediction is generated by the model, which is for loss calculation to regularize the contribution of noisy inputs for model updates during training.
Figure 2 Overview of the novel model architecture for deep learning-based human activity recognition using wearables. An input sensor frame is first processed with Adaptive Trimming (AdapTrimm or AT) to determine the sub-window region that contains the core motion of the target activity. The feature representation is then extracted for this core motion sub-window (green segment). Non-linear multi-scale convolution (MS Conv) captures temporal dynamics at multiple temporal scales. Multi-kernel window convolution (MW Conv) captures varying lengths of motion signals from multi-scale features. Multi-view convolution (MV Conv) uses diverse shapes of convolutional kernels to capture time-channel correlations. The feature representation from intermediate convolutional layers is used as input to the final classifier in the form of auxiliary inputs with skip-connections to enhance gradient flow in the bottom layers of the recognition network. The convolutional feature representations are aggregated by a non-linear multi-scale recurrent neural network (MS RNN), which is used to predict the activity class of the input sensor frame. Along with the class label, the confidence for the prediction is generated by the model, which is for loss calculation to regularize the contribution of noisy inputs for model updates during training.CC BY · PubMed Central
Figure 3 Non-linear multi-scale feature extraction (MS-Conv) with rescaling factors of [1/2,1/4,1/8]. To capture low dimension temporal representation at each scaling factor, we adopt hourglass networks, which are often used to capture low dimensional spatial representations of 2D images in computer vision scenarios [72]. The hourglass network first downsamples input with the convolutional model and then upsamples back to the original scale with the transposed convolutional model with ReLU activation. We apply separate hourglass network models to capture each temporal scale information independently, which is then aggregated and remixed with the bottleneck convolutional layer for final output.
Figure 3 Non-linear multi-scale feature extraction (MS-Conv) with rescaling factors of [1/2,1/4,1/8]. To capture low dimension temporal representation at each scaling factor, we adopt hourglass networks, which are often used to capture low dimensional spatial representations of 2D images in computer vision scenarios [72]. The hourglass network first downsamples input with the convolutional model and then upsamples back to the original scale with the transposed convolutional model with ReLU activation. We apply separate hourglass network models to capture each temporal scale information independently, which is then aggregated and remixed with the bottleneck convolutional layer for final output.CC BY · PubMed Central
Figure 4 Multi-kernel window convolution (MW-Conv) with kernel sizes of [0.15,0.5,1,1.5,2] seconds, respectively, to capture short and long motions in human activities. We used different colors for each kernels to show that they are independent and not sharing the weights. Features from each kernel are concatenated along the feature map axis and recombined through the bottleneck layer. We have explored other feature aggregation approaches, such as summation or bilinear pooling [82], but concatenation was most effective over others.
Figure 4 Multi-kernel window convolution (MW-Conv) with kernel sizes of [0.15,0.5,1,1.5,2] seconds, respectively, to capture short and long motions in human activities. We used different colors for each kernels to show that they are independent and not sharing the weights. Features from each kernel are concatenated along the feature map axis and recombined through the bottleneck layer. We have explored other feature aggregation approaches, such as summation or bilinear pooling [82], but concatenation was most effective over others.CC BY · PubMed Central
Figure 5 Multi-view convolution (MV-Conv) with kernel shapes t×1, 1×c, t×c, and 1×1 are used to capture the time and channel correlations. We used different colors for each kernels to show that they are independent and not sharing the weights. Features from each kernel are concatenated along the the feature map axis and recombined through the bottleneck layer. We have explored other feature aggregation approaches, such as summation or bilinear pooling [82], but concatenation was most effective over others.
Figure 5 Multi-view convolution (MV-Conv) with kernel shapes t×1, 1×c, t×c, and 1×1 are used to capture the time and channel correlations. We used different colors for each kernels to show that they are independent and not sharing the weights. Features from each kernel are concatenated along the the feature map axis and recombined through the bottleneck layer. We have explored other feature aggregation approaches, such as summation or bilinear pooling [82], but concatenation was most effective over others.CC BY · PubMed Central
Figure 6 MSMVMW-Conv model with skip connections and temporal aggregation through multi-scale recurrent neural network. To handle vanishing gradients at the bottom layers of the MSMVMW-Conv model, skip connections are introduced to directly feed feature representations from intermediate layers as auxiliary inputs to the classifier (blue). In this example, feature representations from each intermediate layer are processed by the multi-layer convolutional neural network with max-pooling to match the size of the feature representation from the final layer. The fused features from all convolutional layers are passed to the multi-scale recurrent neural network model for temporal aggregation and classification (orange).
Figure 6 MSMVMW-Conv model with skip connections and temporal aggregation through multi-scale recurrent neural network. To handle vanishing gradients at the bottom layers of the MSMVMW-Conv model, skip connections are introduced to directly feed feature representations from intermediate layers as auxiliary inputs to the classifier (blue). In this example, feature representations from each intermediate layer are processed by the multi-layer convolutional neural network with max-pooling to match the size of the feature representation from the final layer. The fused features from all convolutional layers are passed to the multi-scale recurrent neural network model for temporal aggregation and classification (orange).CC BY · PubMed Central
Figure 7 Comparison between model training with (right) and without considering sample uncertainty (left). Information flows related to outlier and relevant samples are colored in red and blue, respectively. The size of the backward arrows illustrates the relative contribution of the samples to the overall gradient signal as it is distributed to the model parameters through backpropagation. (a) For standard model training, the contributions from outliers and relevant samples are the same for the model update. (b) Our uncertainty assessment model quantifies the degree of uncertainty for a given sample and then regularizes model training accordingly.
Figure 7 Comparison between model training with (right) and without considering sample uncertainty (left). Information flows related to outlier and relevant samples are colored in red and blue, respectively. The size of the backward arrows illustrates the relative contribution of the samples to the overall gradient signal as it is distributed to the model parameters through backpropagation. (a) For standard model training, the contributions from outliers and relevant samples are the same for the model update. (b) Our uncertainty assessment model quantifies the degree of uncertainty for a given sample and then regularizes model training accordingly.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 7 Not valid · 27 Not checked · 75 Own results and interpretations

60 more claims

Create a free account to see all 115 claims and to verify the 44 references still open.

See all claims — free account
  • Probably not a substantial claim. 35
  • It may state two separate claims as one. 19
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 87 references

Research works printed but not linked: 17

  1. Hammerla N.Y. Halloran S. Plötz T. Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables AAAI Press Palo Alto, CA, USA 2016 1533 1540
  2. Fawaz H. Forestier G. Weber J. Idoumghar L. Muller P. Data augmentation using synthetic data for time series classification with deep residual networks arXiv 2018 1808.02455
  3. Ramponi G. Protopapas P. Brambilla M. Janssen R. T-cgan: Conditional generative adversarial network for data augmentation in noisy time series with irregular sampling arXiv 2018 1811.08295
  4. Yosinski J. Clune J. Bengio Y. Lipson H. How transferable are features in deep neural networks? arXiv 2014 1411.1792
  5. Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Kaiser L. Polosukhin I. Attention is all you need arXiv 2017 1706.03762
  6. Redmon J. Farhadi A. YOLOv3: An Incremental Improvement arXiv 2018 1804.02767
  7. Joel A. Stergios I. A Direct Least-Squares (DLS) method for PnP Proceedings of the 2011 International Conference on Computer Vision Barcelona, Spain 6–13 November 2011
  8. Vankadari M. Garg S. Majumder A. Kumar S. Behera A. Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation Proceedings of the European Conference on Computer Vision Springer Cham, Switzerland 2020 443 459
  9. Kingma D. Ba J. Adam: A method for stochastic optimization arXiv 2014 1412.6980
  10. Li P. Lin Y. Schultz-Fellenz E. Contextual hourglass network for semantic segmentation of high resolution aerial imagery arXiv 2018 1810.12813
  11. Oñoro-Rubio D. Niepert M. Contextual Hourglass Networks for Segmentation and Density Estimation arXiv 2018 1806.04009
  12. Chung J. Ahn S. Bengio Y. Hierarchical multiscale recurrent neural networks arXiv 2016 1609.01704
  13. Kádár A. Côté M. Chrupała G. Alishahi A. Revisiting the hierarchical multiscale lstm arXiv 2018 1807.03595
  14. Kendall A. Gal Y. What uncertainties do we need in bayesian deep learning for computer vision? arXiv 2017 1703.04977
  15. Rolnick D. Veit A. Belongie S. Shavit N. Deep learning is robust to massive label noise arXiv 2017 1705.10694
  16. Misra I. Zitnick C.L. Hebert M. Shuffle and learn: Unsupervised learning using temporal order verification Proceedings of the European Conference on Computer Vision Springer Cham, Switzerland 2016 527 544
  17. Devlin J. Chang M. Lee K. Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding arXiv 2018 1810.04805

Cited by

Metadata

Authors
Kwon, H., Abowd, G. D., & Plötz, T.
Affiliations
  • Georgia Institute of Technology (US)
  • Northeastern University (US)
Corresponding author
Plötz, Thomas
Venue
Sensors
Publisher
MDPI AG
Published
2021-12-13
Volume
21
Issue
24
Pages
8337
ISSN
1424-8220 (ISSN-L)
OpenAlex
W4200504110
Cited by
30
References
86
Citations by year
2026: 4 · 2025: 10 · 2024: 4 · 2023: 10 · 2022: 2
FWCI
1.44
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Samsung Scholarship Program (N/A)
  • Oracle for Research (N/A)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
Keywords
human activity recognition, inertial measurement unit, wearable sensors, cross-modality transfer, deep neural network, model complexity, dumbbell exercise, transfer learning
MeSH terms
Wearable Electronic Devices, Humans, Human Activities, Neural Networks, Computer, Recognition, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.