Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
The Sussex-Huawei Locomotion-Transportation Recognition Challenge presented a unique opportunity to the activity-recognition community to test their approaches on a large, real-life benchmark dataset with activities different from those typically recognized. The goal of the challenge was to recognize, as accurately as possible, eight locomotion activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway) using smartphone sensor data. This paper describes the method we developed to win this challenge, and provides an analysis of the effectiveness of its components. We used complex feature extraction and selection methods to train classical machine learning models. In addition, we trained deep learning models using a novel end-to-end architecture for deep multimodal spectro-temporal fusion. All the models were fused into an ensemble with the final predictions smoothed by a hidden Markov model to account for temporal dependencies of the activities. The presented method achieved an F1 score of 94.9% on the challenge test data. We tested different sampling frequencies, window sizes, feature types, classification models and the importance of stand-alone sensors and their fusion for the task. Finally, we present an energy-efficient smartphone implementation of the method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 2×
“In 2016, Ordonez et al. introduced the DeepConvLSTM [15] , an architecture that stacks CNN layers over a multimodal long short-term memory (LSTM) recurrent neural network (RNN) for AR [16] .”
BackgroundTime series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299
PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS
ROSENBLATT, F. (1961) · doi:10.21236/ad0256582 · Cited by 1,691
Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors
Zeng, M., Nguyen, L. T., Yu, B., Mengshoel, O. J., Zhu, J., Wu, P., & Zhang, J. (2014) · Proceedings of the 6th International Conference on Mobile Computing, Applications and Services · doi:10.4108/icst.mobicase.2014.257786 · Cited by 876
Collecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
A deep learning approach to on-node sensor data analytics for mobile or wearable devices
Ravi, D., Wong, C., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2633287 · Cited by 439
Blood Pressure Estimation from Photoplethysmogram Using a Spectro-Temporal Deep Neural Network
Slapničar, G., Mlakar, N., & Luštrek, M. (2019) · Sensors · doi:10.3390/s19153420 · Cited by 387
Self-citationBhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
Optimising sampling rates for accelerometer-based human activity recognition
Khan, A., Hammerla, N., Mellor, S., & Plötz, T. (2016) · Pattern Recognition Letters · doi:10.1016/j.patrec.2016.01.001 · Cited by 171
On attention models for human activity recognition
Murahari, V. S., & Plötz, T. (2018) · Proceedings of the 2018 ACM International Symposium on Wearable Computers · doi:10.1145/3267242.3267287 · Cited by 157 · cited 1×
“Most recently, Murahari et al. [18] experimented with attention mechanisms in the DeepConvLSTM, which improved its performance in the AR domain by a few percentage points.”
Activity recognition from on-body sensors by classifier fusion: sensor scalability and robustness
Zappi, P., Stiefmeier, T., Farella, E., Roggen, D., Benini, L., & Troster, G. (2007) · 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information · doi:10.1109/issnip.2007.4496857 · Cited by 126
Guo, H., Chen, L., Peng, L., & Chen, G. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971708 · Cited by 83
Research works printed but not linked: 20
- Happiness and Satisfaction with Work Commute, (https://link.springer.com/article/10.1007/s11205-012-0003-2). · DOI · Paper page
- Gjoreski, 2018, The university of sussex-huawei locomotion and transportation dataset for multimodal analytics with mobile devices, IEEE Access, 10.1109/ACCESS.2018.2858933 · DOI · Paper page
- Janko, 2017, E-gibalec: mobile application to monitor and encourage physical activity in schoolchildren, J. Ambient Intell. Smart Environ., 9, 595, 10.3233/AIS-170453 · DOI
- Cvetković, 2018, Real-time activity monitoring with a wristband and a smartphone, Inf. Fusion, 43, 77, 10.1016/j.inffus.2017.05.004 · DOI
- Krishna, 2018, An lstm based system for prediction of human activities with durations, Proc. ACM Interact. Mob. Wearable Ubiquit. Technol., 1, 147, 10.1145/3161201 · DOI · Paper page
- Martin, 2017, Methods for real-time prediction of the mode of travel using smartphone-based gps and accelerometer data, Sensors, 17, 10.3390/s17092058 · DOI
- Fang, 2016, Transportation modes classification using sensors on smartphones, Sensors, 16, 1324, 10.3390/s16081324 · DOI
- Reddy, 2010, Using mobile phones to determine transportation modes, ACM Trans. Sensor Netw. (TOSN), 6, 13
- Lee, 2016, Energy-efficient real-time human activity recognition on smart mobile devices, Mob. Inf. Syst., 2016
- Su, 2014, Activity recognition with smartphone sensors, Tsinghua Sci. Technol., 19, 235, 10.1109/TST.2014.6838194 · DOI · Paper page
- Quinlan, 1996, Improved use of continuous attributes in c4. 5, J. Artif. Intell. Res., 4, 77, 10.1613/jair.279 · DOI · Paper page
- Aha, 1991, Instance-based learning algorithms, Mach Learn, 6, 37, 10.1023/A:1022689900470 · DOI · Paper page
- Breiman, 1996, Bagging predictors, Mach. Learn., 24, 123, 10.1023/A:1018054314350 · doi:10.1023/a:1018054314350 · Paper page
- Martínez-Muñoz, 2008, Class-switching neural network ensembles, Neurocomputing, 71, 2521, 10.1016/j.neucom.2007.11.041 · DOI
- Martínez-Muñoz, 2005, Switching class labels to generate classification ensembles, Pattern Recognit., 38, 1483, 10.1016/j.patcog.2005.02.020 · DOI
- Ioffe, 2015, Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv Preprint arXiv:1502.03167 · arXiv
- Lane, 2017, Squeezing deep learning into mobile and embedded devices, IEEE Pervasive Comput., 16, 82, 10.1109/MPRV.2017.2940968 · DOI · Paper page
- Deb, 2002, A fast and elitist multiobjective genetic algorithm: nsga-ii, IEEE Trans. Evol. Comput., 6, 182, 10.1109/4235.996017 · DOI · Paper page
- Cho, 2014, Learning phrase representations using rnn encoder-decoder for statistical machine translation, arXiv preprint arXiv:1406.1078 · arXiv
- Abavisani, 2018, Deep multimodal subspace clustering networks, IEEE J. Sel. Top. Signal Process., 12, 1601, 10.1109/JSTSP.2018.2875385 · DOI · Paper page
Cited by
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800
“Deep learning has seen great success in a wide variety of applications, such as natural language processing, speech recognition, medical applications, computer vision, and intelligent transportation system [1, 2, 3, 4].”
BackgroundCross-dataset deep transfer learning for activity recognition
Gjoreski, M., Kalabakov, S., Luštrek, M., Gams, M., & Gjoreski, H. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3344865 · Cited by 28
Self-citationDigging deeper: towards a better understanding of transfer learning for human activity recognition
Hoelzemann, A., & Van Laerhoven, K. (2020) · Proceedings of the 2020 International Symposium on Wearable Computers · doi:10.1145/3410531.3414311 · Cited by 20
“By using an architecture called MultiResNet [5], that transfers the data into frequency domain and uses residual blocks they achieved promising results when transferring from Skoda Mini Checkpoint [23] to OPPORTUNITY [18], PAMAP2 [17] or JSI-FOS [4].”
Method
Metadata
- Authors
- Gjoreski, M., Janko, V., Slapničar, G., Mlakar, M., Reščič, N., Bizjak, J., Drobnič, V., Marinko, M., Mlakar, N., Luštrek, M., & Gams, M.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2020-10
- Volume
- 62
- Pages
- 47-62
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W3019253063
- Cited by
- 96
- References
- 17
- FWCI
- 6.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- Funders
- Javna Agencija za Raziskovalno Dejavnost RS (U2-AG-16/0672, U2-RP-19/0441)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.