CoChat

MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices

Strömbäck, David, Huang, Sangxia, Radu, Valentin · 2020 · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Cited by 73

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Fitness tracking devices have risen in popularity in recent years, but limitations in terms of their accuracy and failure to track many common exercises presents a need for improved fitness tracking solutions. This work proposes a multimodal deep learning approach to leverage multiple data sources for robust and accurate activity segmentation, exercise recognition and repetition counting. For this, we introduce the MM-Fit dataset; a substantial collection of inertial sensor data from smartphones, smartwatches and earbuds worn by participants while performing full-body workouts, and time-synchronised multi-viewpoint RGB-D video, with 2D and 3D pose estimates. We establish a strong baseline for activity segmentation and exercise recognition on the MM-Fit dataset, and demonstrate the effectiveness of our CNN-based architecture at extracting modality-specific spatial temporal features from inertial sensor and skeleton sequence data. We compare the performance of unimodal and multimodal models for activity recognition across a number of sensing devices and modalities. Furthermore, we demonstrate the effectiveness of multimodal deep learning at learning cross-modal representations for activity recognition, which achieves 96% accuracy across all sensing modalities on unseen subjects in the MM-Fit dataset; 94% using data from the smartwatch only; 85% from the smartphone only; and 82% on data from the earbud device. We strengthen single-device performance by using the zeroing-out training strategy, which phases out the other sensing modalities. Finally, we implement and evaluate a strong repetition counting baseline on our MM-Fit dataset. Collectively, these tasks contribute to recognising, segmenting and timing exercise and non-exercise activities for automatic exercise logging.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3432701

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 36

  1. 10.1016/j.patrec.2014.04.011 · DOI · Paper page
  2. Almaslukh B, 2017, International Journal of Computer Science and Network Security 17 (04
  3. 10.3390/s140406474 · DOI · Paper page
  4. Bender C. G., 2017 IEEE Sensors Applications Symposium (SAS). 1--6.
  5. Catalin Ionescu Cristian Sminchisescu, 2011, Latent Structured Models for Human Pose Estimation. In International Conference on Computer Vision.
  6. Chen Liming, 2012, Sensor-Based Activity, Recognition. Trans. Sys. Man Cyber Part C, 42, 6
  7. Choi K. S., 2013 IEEE International Conference on Consumer Electronics (ICCE). 436--437
  8. 10.1109/ACPR.2015.7486569 · DOI · Paper page
  9. Du Yong, 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  10. Evangelidis Georgios Dimitrios, 2014, Skeletal Quads: Human Action Recognition Using Joint Quadruples. 2014 22nd International Conference on Pattern Recognition
  11. Glorot Xavier, 2011, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.), 15
  12. Guo Xiaonan, 2018, Device-free personalized fitness assistant using WiFi, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2, 1
  13. Ha S., Multi-modal Convolutional Neural Networks for Activity Recognition. In 2015 IEEE International Conference on Systems, Man, and Cybernetics. 3017--3022
  14. Chang Keng, 2007, Canny
  15. 10.1016/j.imavis.2017.01.010 · DOI · Paper page
  16. 10.1145/3214269 · DOI · Paper page
  17. Khurana Rushil, 2018, Detecting, Recognizing and Tracking Simultaneous Exercises in Unconstrained Scenes. IMWUT 2
  18. Kingma Diederick P, 2015, International Conference on Learning Representations (ICLR).
  19. LeCun Yann, 2015, Deep learning. Nature 521, 7553 (27 5
  20. Levy O., Live Repetition Counting. In 2015 IEEE International Conference on Computer Vision (ICCV). 3020--3028
  21. Li Chao, 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)
  22. Maosen Li Siheng Chen Xu Chen Ya Zhang Yanfeng Wang and Qi Tian. 2019. Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition. In CVPR. Maosen Li Siheng Chen Xu Chen Ya Zhang Yanfeng Wang and Qi Tian. 2019. Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition. In CVPR.
  23. Martinez Julieta, 2017, Little
  24. Dan Morris T. Scott Saponas Andrew Guillory and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find recognize and count repetitive exercises. In CHI. Dan Morris T. Scott Saponas Andrew Guillory and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find recognize and count repetitive exercises. In CHI. · DOI · Paper page
  25. Mortazavi Bobak, 2014, Determining the Single Best Axis for Exercise Repetition Recognition and Counting on SmartWatches. 2014 11th International Conference on Wearable and Implantable Body Sensor Networks
  26. Ngiam Jiquan, 2011, Ng
  27. Riza Alp Guler Iasonas Kokkinos, 2018, DensePose: Dense Human Pose Estimation In The Wild. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  28. 10.1109/CVPR.2018.00939 · DOI
  29. 10.3390/s18092967 · DOI · Paper page
  30. 10.1021/ac60214a047 · DOI · Paper page
  31. Soro Andrea, 2019, Recognition and Repetition Counting for Complex Physical Exercises with Deep Learning. Sensors 19, 3
  32. Um Terry Taewoong, 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
  33. Eduardo Velloso Andreas Bulling Hans-Werner Gellersen Wallace Ugulino and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In AH. Eduardo Velloso Andreas Bulling Hans-Werner Gellersen Wallace Ugulino and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In AH. · DOI · Paper page
  34. 10.1109/CVPR.2014.82 · DOI · Paper page
  35. 10.1109/TPAMI.2013.198 · DOI · Paper page
  36. Yang Jianbo, 2015, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy.

Cited by

Metadata

Authors
Strömbäck, D., Huang, S., & Radu, V.
Venue
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Published
2020-12-17
Volume
4
Issue
4
Pages
1-22
ISSN
2474-9567 (ISSN-L)
OpenAlex
W3114040862
Cited by
73
References
17
FWCI
2.92
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.