MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Fitness tracking devices have risen in popularity in recent years, but limitations in terms of their accuracy and failure to track many common exercises presents a need for improved fitness tracking solutions. This work proposes a multimodal deep learning approach to leverage multiple data sources for robust and accurate activity segmentation, exercise recognition and repetition counting. For this, we introduce the MM-Fit dataset; a substantial collection of inertial sensor data from smartphones, smartwatches and earbuds worn by participants while performing full-body workouts, and time-synchronised multi-viewpoint RGB-D video, with 2D and 3D pose estimates. We establish a strong baseline for activity segmentation and exercise recognition on the MM-Fit dataset, and demonstrate the effectiveness of our CNN-based architecture at extracting modality-specific spatial temporal features from inertial sensor and skeleton sequence data. We compare the performance of unimodal and multimodal models for activity recognition across a number of sensing devices and modalities. Furthermore, we demonstrate the effectiveness of multimodal deep learning at learning cross-modal representations for activity recognition, which achieves 96% accuracy across all sensing modalities on unseen subjects in the MM-Fit dataset; 94% using data from the smartwatch only; 85% from the smartphone only; and 82% on data from the earbud device. We strengthen single-device performance by using the zeroing-out training strategy, which phases out the other sensing modalities. Finally, we implement and evaluate a strong repetition counting baseline on our MM-Fit dataset. Collectively, these tasks contribute to recognising, segmenting and timing exercise and non-exercise activities for automatic exercise logging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“Convolutional Neural Networks (CNNs) [30] are particularly well suited for working with sensor data, as they efficiently capture local dependencies in the data, and can learn scale-invariant features, important for dealing with activities performed at different rates of frequency [18, 54, 55].”
BackgroundRealtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · cited 2×
“RNNs are able to efficiently leverage time-series information, while GNNs are able to incorporate the spatial constraints and relationships of human joints in their models.”
BackgroundHuman3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 1×
“6M [8, 24].”
A Survey on Human Activity Recognition using Wearable Sensors
Lara, O. D., & Labrador, M. A. (2013) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2012.110112.00192 · Cited by 2,662
A survey on vision-based human action recognition
Poppe, R. (2010) · Image and Vision Computing · doi:10.1016/j.imavis.2009.11.014 · Cited by 2,242
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668
Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors
Zeng, M., Nguyen, L. T., Yu, B., Mengshoel, O. J., Zhu, J., Wu, P., & Zhang, J. (2014) · Proceedings of the 6th International Conference on Mobile Computing, Applications and Services · doi:10.4108/icst.mobicase.2014.257786 · Cited by 876 · cited 1×
“Convolutional Neural Networks (CNNs) [30] are particularly well suited for working with sensor data, as they efficiently capture local dependencies in the data, and can learn scale-invariant features, important for dealing with activities performed at different rates of frequency [18, 54, 55].”
BackgroundA New Representation of Skeleton Sequences for 3D Action Recognition
Ke, Q., Bennamoun, M., An, S., Sohel, F., & Boussaid, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.486 · Cited by 793 · cited 13×
“Cylindrical coordinates is demonstrated to improve performance over Cartesian coordinates [25].”
BackgroundInfluentialStisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714
Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606
Transfer learning for activity recognition: a survey
Cook, D., Feuz, K. D., & Krishnan, N. C. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-013-0665-3 · Cited by 447 · cited 1×
“Surprisingly, the effect of model pretraining was not significant for improving model performance on our exercise recognition task, unlike what is currently understood in the transfer learning community [11].”
BackgroundMultimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281 · cited 3×
“explore a number of fusing strategies for multimodal deep learning methods [41], comparing early feature concatenation and modality-specific architectures, on activity and context recognition tasks using inertial sensor data.”
BackgroundMethodSelf-citationTowards multimodal deep learning for activity recognition on mobile devices
Radu, V., Lane, N. D., Bhattacharya, S., Mascolo, C., Marina, M. K., & Kawsar, F. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct · doi:10.1145/2968219.2971461 · Cited by 147 · cited 2×
“demonstrate their superior HAR performance compared to traditional shallow-learning approaches, by using a multimodal-RBM network to learn a shared representation for accelerometer and gyroscope data [40].”
MethodSelf-citationAdversarial Multi-view Networks for Activity Recognition
Bai, L., Yao, L., Wang, X., Kanhere, S. S., Guo, B., & Yu, Z. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3397323 · Cited by 40 · cited 1×
“user, adversarial networks have been used [3], which rely on Siamese networks to decrease the variants between the representations of different subjects.”
MethodHwang, I., Cha, G., & Oh, S. (2017) · 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2017.8170441 · Cited by 28 · cited 6×
“Wang et al., explore the integration of video and inertial sensors from a wrist worn device, using CNNs to extract features independently on each modality, and performing classification using a RNN[23].”
MethodInfluential
Research works printed but not linked: 36
- 10.1016/j.patrec.2014.04.011 · DOI · Paper page
- Almaslukh B, 2017, International Journal of Computer Science and Network Security 17 (04
- 10.3390/s140406474 · DOI · Paper page
- Bender C. G., 2017 IEEE Sensors Applications Symposium (SAS). 1--6.
- Catalin Ionescu Cristian Sminchisescu, 2011, Latent Structured Models for Human Pose Estimation. In International Conference on Computer Vision.
- Chen Liming, 2012, Sensor-Based Activity, Recognition. Trans. Sys. Man Cyber Part C, 42, 6
- Choi K. S., 2013 IEEE International Conference on Consumer Electronics (ICCE). 436--437
- 10.1109/ACPR.2015.7486569 · DOI · Paper page
- Du Yong, 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Evangelidis Georgios Dimitrios, 2014, Skeletal Quads: Human Action Recognition Using Joint Quadruples. 2014 22nd International Conference on Pattern Recognition
- Glorot Xavier, 2011, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.), 15
- Guo Xiaonan, 2018, Device-free personalized fitness assistant using WiFi, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2, 1
- Ha S., Multi-modal Convolutional Neural Networks for Activity Recognition. In 2015 IEEE International Conference on Systems, Man, and Cybernetics. 3017--3022
- Chang Keng, 2007, Canny
- 10.1016/j.imavis.2017.01.010 · DOI · Paper page
- 10.1145/3214269 · DOI · Paper page
- Khurana Rushil, 2018, Detecting, Recognizing and Tracking Simultaneous Exercises in Unconstrained Scenes. IMWUT 2
- Kingma Diederick P, 2015, International Conference on Learning Representations (ICLR).
- LeCun Yann, 2015, Deep learning. Nature 521, 7553 (27 5
- Levy O., Live Repetition Counting. In 2015 IEEE International Conference on Computer Vision (ICCV). 3020--3028
- Li Chao, 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)
- Maosen Li Siheng Chen Xu Chen Ya Zhang Yanfeng Wang and Qi Tian. 2019. Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition. In CVPR. Maosen Li Siheng Chen Xu Chen Ya Zhang Yanfeng Wang and Qi Tian. 2019. Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition. In CVPR.
- Martinez Julieta, 2017, Little
- Dan Morris T. Scott Saponas Andrew Guillory and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find recognize and count repetitive exercises. In CHI. Dan Morris T. Scott Saponas Andrew Guillory and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find recognize and count repetitive exercises. In CHI. · DOI · Paper page
- Mortazavi Bobak, 2014, Determining the Single Best Axis for Exercise Repetition Recognition and Counting on SmartWatches. 2014 11th International Conference on Wearable and Implantable Body Sensor Networks
- Ngiam Jiquan, 2011, Ng
- Riza Alp Guler Iasonas Kokkinos, 2018, DensePose: Dense Human Pose Estimation In The Wild. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- 10.1109/CVPR.2018.00939 · DOI
- 10.3390/s18092967 · DOI · Paper page
- 10.1021/ac60214a047 · DOI · Paper page
- Soro Andrea, 2019, Recognition and Repetition Counting for Complex Physical Exercises with Deep Learning. Sensors 19, 3
- Um Terry Taewoong, 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Eduardo Velloso Andreas Bulling Hans-Werner Gellersen Wallace Ugulino and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In AH. Eduardo Velloso Andreas Bulling Hans-Werner Gellersen Wallace Ugulino and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In AH. · DOI · Paper page
- 10.1109/CVPR.2014.82 · DOI · Paper page
- 10.1109/TPAMI.2013.198 · DOI · Paper page
- Yang Jianbo, 2015, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy.
Cited by
Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data
Kwon, H., Wang, B., Abowd, G. D., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3478096 · Cited by 48
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-26297-2 · Cited by 26
Oishi, N., Birch, P., Lago, P., & Roggen, D. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116367 · Cited by 1
Metadata
- Authors
- Strömbäck, D., Huang, S., & Radu, V.
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2020-12-17
- Volume
- 4
- Issue
- 4
- Pages
- 1-22
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W3114040862
- Cited by
- 73
- References
- 17
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.