Design of Novel Deep Learning Models for Real-time Human Activity Recognition with Mobile Phones
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
In this paper we present deep learning based techniques for human activity classification that are designed to run in real time on mobile devices. Our methods minimize the size of the model and computational overhead in order to run on the embedded processor and preserve battery life. Prior work shows that the inertial measurement unit (IMU) data from waist-mounted mobile phones can be used to develop accurate classification models for various human activities such as walking, running, stair-climbing, etc. However, these models have largely been based on hand crafted features derived from temporal and spectral statistics. More recently, deep learning has been applied to IMU sensor data, but have not been optimized for resourceconstrained devices. We present a detailed study of the traditional hand-crafted features used for shallow/statistical models that consist of a over 561 manually chosen set of dimensions. We show, through principal component analysis (PCA) and application of a published support vector machine (SVM) pipeline, that the number of features can be significantly reduced - less than 100 features that give the same performance. In addition, we show that features derived from frequency-domain transformations do not contribute to the accuracy of these models. Finally, we provide details of our learning technique which creates 2D signal images from windowed samples of IMU data. Our pipeline includes a convolutional neural network (CNN) with several layers (1 convolutional layer and 1 averaging layer and a fully connected layer). We show that by removing the steps in the pipeline and layers in the CNN, we can still achieve 0.98 F1 score but with a much smaller memory footprint and corresponding computational cost. To increase the classification accuracy of our pipeline we added a hybrid bi-class support vector machine (SVM) that was trained using the labeled and flattened convolutional layer after each training image was processed. The learned feature set is almost half the size of the original hand crafted feature set and combining the CNN with the SVM results in 0.99 F1 score. We also investigate a novel application of transfer learning by using the time series 2D signal images to re-train two different publicly available networks, Inception/ImageNet and MobileNet. We find that re-trained ImageNet networks could be created $<; 5.5$ MB (suitable for mobile phones) and classification accuracy ranging from 0.83 to 0.93 (F1 score), thus indicating that retraining can be a useful future direction to build new classifiers for continuously evolving activities quickly while also being applicable to mobile device classification. Finally, we show that these deep learning models may be generalizable enough such that classifiers built from a given set of users for a specified set of activities can be used for a new user/subject as well.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Science, technology and the future of small autonomous drones
Floreano, D., & Wood, R. J. (2015) · Nature · doi:10.1038/nature14542 · Cited by 1,476
Handbook of Unmanned Aerial Vehicles
(2015) · doi:10.1007/978-90-481-9707-1 · Cited by 1,103
Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks
Jiang, W., & Yin, Z. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2806333 · Cited by 676 · cited 5×
“The rows component represents the number permutations for stacked IMU signals, as created in [8].”
BackgroundMethodInfluentialDeep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606 · cited 2×
“A comparison of the use of both CNNs and RNNs as well as a study on the effects of hyperparameters for wearables used to classify complex human activity is presented in [3].”
BackgroundMethodDeep Activity Recognition Models with Triaxial Accelerometers
Alsheikh, M. A., Selim, A., Niyato, D., Doyle, L., Lin, S., & Tan, H.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.04664 · Cited by 126
Cited by
Deep Learning for Sensor-based Human Activity Recognition
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2022) · ACM Computing Surveys · doi:10.1145/3447744 · Cited by 690
“s’ prediction of activities. Table 4. Deep Learning Works for Interpretability of Deep Learning Models in Sensory Data Method Subclass References Traditional - Li et al. 2015 [85]; Nutter et al. 2018 [103]; Xue et al. 2018 [163]; Brophy et al. 2018 [19]; Attetion Mechanism Soft Attention Tang et al. 2016 [142]; Muraha…”
BackgroundRamanujam, E., Perumal, T., & Padmavathi, S. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3069927 · Cited by 455
Demrozi, F., Pravadelli, G., Bihorac, A., & Rashidi, P. (2020) · IEEE Access · doi:10.1109/access.2020.3037715 · Cited by 368
Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2020) · arXiv · doi:10.48550/arxiv.2001.07416 · Cited by 314
“s’ prediction of activities. Table 4. Deep Learning Works for Interpretability of Deep Learning Models in Sensory Data Method Subclass References Traditional - Li et al. 2015 [85]; Nutter et al. 2018 [103]; Xue et al. 2018 [163]; Brophy et al. 2018 [19]; Attetion Mechanism Soft Attention Tang et al. 2016 [142]; Muraha…”
BackgroundKumar, P., & Chauhan, S. (2021) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-021-00063-5 · Cited by 13
Retracted
Metadata
- Authors
- Nutter, M., Crawford, C. H., & Ortiz, J.
- Venue
- 2018 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2018-07
- Pages
- 1-8
- OpenAlex
- W2896722540
- Cited by
- 24
- References
- 5
- FWCI
- 0.85
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.