Human Activity Recognition With Smartphone and Wearable Sensors Using Deep Learning Techniques: A Review
The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
Human Activity Recognition (HAR) is a field that infers human activities from raw time-series signals acquired through embedded sensors of smartphones and wearable devices. It has gained much attraction in various smart home environments, especially to continuously monitor human behaviors in ambient assisted living to provide elderly care and rehabilitation. The system follows various operation modules such as data acquisition, pre-processing to eliminate noise and distortions, feature extraction, feature selection, and classification. Recently, various state-of-the-art techniques have proposed feature extraction and selection techniques classified using traditional Machine learning classifiers. However, most of the techniques use rustic feature extraction processes that are incapable of recognizing complex activities. With the emergence and advancement of high computational resources, Deep Learning techniques are widely used in various HAR systems to retrieve features and classification efficiently. Thus, this review paper focuses on providing profound concise of deep learning techniques used in smartphone and wearable sensor-based recognition systems. The proposed techniques are categorized into conventional and hybrid deep learning models described with its uniqueness, merits, and limitations. The paper also discusses various benchmark datasets used in existing techniques. Finally, the paper lists certain challenges and issues that require future research and improvements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“Ordóñez and Roggen [69] have proposed a DeepConvLSTM model based on the success of recurrent neural networks to recognize multimodal activities and to explicitly model the temporal dynamics of feature activations.”
BackgroundDeep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014 · cited 4×
“The wearable devices, especially fitness trackers and smartwatches, are mostly used for recognition as it has various inbuilt/integrated sensors such as Accelerometer, Gyroscope, and Orientation sensors [3] available at preferable cost.”
BackgroundMethodInfluential(2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 1,959
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456
Human activity recognition with smartphone sensors using deep learning neural networks
Ronao, C. A., & Cho, S.-B. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.04.032 · Cited by 1,215
Nweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930
LSTM-CNN Architecture for Human Activity Recognition
Xia, K., Huang, J., & Wang, H. (2020) · IEEE Access · doi:10.1109/access.2020.2982225 · Cited by 851 · cited 1×
“Xia et al. [55] have proposed a deep hybrid model by integrating LSTM models connected with Convolutional Layers and Global Pooling layers (GAP).”
MethodReal-time human activity recognition from accelerometer data using Convolutional Neural Networks
Ignatov, A. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2017.09.027 · Cited by 814
Physical Human Activity Recognition Using Wearable Sensors
Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi, F., Oukhellou, L., & Amirat, Y. (2015) · Sensors · doi:10.3390/s151229858 · Cited by 796
The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742
Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714
UniMiB SHAR: A Dataset for Human Activity Recognition Using Acceleration Data from Smartphones
Micucci, D., Mobilio, M., & Napoletano, P. (2017) · Applied Sciences · doi:10.3390/app7101101 · Cited by 596
UniMiB SHAR: A Dataset for Human Activity Recognition Using Acceleration Data from Smartphones
Micucci, D., Mobilio, M., & Napoletano, P. (2017) · doi:10.20944/preprints201706.0033.v2 · Cited by 596
UniMiB SHAR: A Dataset for Human Activity Recognition Using Acceleration Data from Smartphones
Micucci, D., Mobilio, M., & Napoletano, P. (2017) · doi:10.20944/preprints201706.0033.v1 · Cited by 596
Deep Learning Models for Real-time Human Activity Recognition with Smartphones
Wan, S., Qi, L., Xu, X., Tong, C., & Gu, Z. (2020) · Mobile Networks and Applications · doi:10.1007/s11036-019-01445-x · Cited by 594
mHealthDroid: A Novel Framework for Agile Development of Mobile Health Applications
Banos, O., Garcia, R., Holgado-Terriza, J. A., Damas, M., Pomares, H., Rojas, I., Saez, A., & Villalonga, C. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-13105-4_14 · Cited by 583
Deep-Learning-Enhanced Human Activity Recognition for Internet of Healthcare Things
Zhou, X., Liang, W., Wang, K. I.-K., Wang, H., Yang, L. T., & Jin, Q. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2985082 · Cited by 556
Nweke, H. F., Teh, Y. W., Mujtaba, G., & Al-garadi, M. A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.002 · Cited by 394
A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer
Chen, Y., & Xue, Y. (2015) · 2015 IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2015.263 · Cited by 378 · cited 1×
“Chen and Xue [23] have proposed a CNN model to recognize the simple activities through tri-axial accelerometer signals of a smartphone.”
BackgroundSmartphone and Smartwatch-Based Biometrics Using Activities of Daily Living
Weiss, G. M., Yoneda, K., & Hayajneh, T. (2019) · IEEE Access · doi:10.1109/access.2019.2940729 · Cited by 360
Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors
Zhao, Y., Yang, R., Chevalier, G., Xu, X., & Zhang, Z. (2018) · Mathematical Problems in Engineering · doi:10.1155/2018/7316954 · Cited by 348
InnoHAR: A Deep Neural Network for Complex Human Activity Recognition
Xu, C., Chai, D., He, J., Zhang, X., & Duan, S. (2019) · IEEE Access · doi:10.1109/access.2018.2890675 · Cited by 333 · cited 1×
“Xu et al. [71], have proposed a combination of Inception neural network and recurrent neural network model to recognize multi-channel sensor inputs.”
BackgroundBanos, O., Villalonga, C., Garcia, R., Saez, A., Damas, M., Holgado-Terriza, J. A., Lee, S., Pomares, H., & Rojas, I. (2015) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-14-s2-s6 · Cited by 321
Imaging and fusing time series for wearable sensor-based human activity recognition
Qin, Z., Zhang, Y., Meng, S., Qin, Z., & Choo, K.-K. R. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.06.014 · Cited by 309
Recent trends in machine learning for human activity recognition—A survey
Ramasamy Ramamurthy, S., & Roy, N. (2018) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1254 · Cited by 299
Chung, S., Lim, J., Noh, K. J., Kim, G., & Jeong, H. (2019) · Sensors · doi:10.3390/s19071716 · Cited by 215 · cited 1×
“Chung et al. [45] have performed a pilot study with 15 participants about the on-body sensor positioning system.”
BackgroundDanHAR: Dual Attention Network for multimodal human activity recognition using wearable sensors
Gao, W., Zhang, L., Teng, Q., He, J., & Wu, H. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107728 · Cited by 214
Wang, K., He, J., & Zhang, L. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2917225 · Cited by 206
Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearable Sensors
Singh, S. P., Sharma, M. K., Lay-Ekuakille, A., Gangwar, D., & Gupta, S. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3045135 · Cited by 199
New Advances and Challenges of Fall Detection Systems: A Survey
Xu, T., Zhou, Y., & Zhu, J. (2018) · Applied Sciences · doi:10.3390/app8030418 · Cited by 179
A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices
Agarwal, P., & Alam, M. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.289 · Cited by 175
A Smartphone-Based Adaptive Recognition and Real-Time Monitoring System for Human Activities
Qi, W., Su, H., & Aliverti, A. (2020) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2020.2984181 · Cited by 169
AHAR: Adaptive CNN for Energy-Efficient Human Activity Recognition in Low-Power Edge Devices
Rashid, N., Demirel, B. U., & Abdullah Al Faruque, M. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3140465 · Cited by 169
A Hybrid Deep Learning Model for Human Activity Recognition Using Multimodal Body Sensing Data
Gumaei, A., Hassan, M. M., Alelaiwi, A., & Alsalman, H. (2019) · IEEE Access · doi:10.1109/access.2019.2927134 · Cited by 157 · cited 1×
“Gumaei et al. [63] has proposed an effective multi-modal sensor-based activity recognition using a hybrid deep learning model.”
BackgroundPosition-aware activity recognition with wearable devices
Sztyler, T., Stuckenschmidt, H., & Petrich, W. H. (2017) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2017.01.008 · Cited by 148
Tang, Y., Teng, Q., Zhang, L., Min, F., & He, J. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3015521 · Cited by 143
Cruciani, F., Vafeiadis, A., Nugent, C., Cleland, I., McCullagh, P., Votis, K., Giakoumis, D., Tzovaras, D., Chen, L., & Hamzaoui, R. (2020) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-020-00026-2 · Cited by 142
Human Activity Recognition Based on Wearable Sensor Using Hierarchical Deep LSTM Networks
Wang, L., & Liu, R. (2019) · Circuits Systems and Signal Processing · doi:10.1007/s00034-019-01116-y · Cited by 141
SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
Tang, C. I., Perez-Pozuelo, I., Spathis, D., Brage, S., Wareham, N., & Mascolo, C. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3448112 · Cited by 140
Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation
Khan, M. A. A. H., Roy, N., & Misra, A. (2018) · 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom.2018.8444585 · Cited by 139
Vision-based patient monitoring: a comprehensive review of algorithms and technologies
Sathyanarayana, S., Satzoda, R. K., Sathyanarayana, S., & Srikanthan, T. (2015) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-015-0328-1 · Cited by 137
Masoud, M., Jaradat, Y., Manasrah, A., & Jannoud, I. (2019) · Journal of Sensors · doi:10.1155/2019/6514520 · Cited by 125
Wearable Sensor-Based Human Activity Recognition Using Hybrid Deep Learning Techniques
Wang, H., Zhao, J., Li, J., Tian, L., Tu, P., Cao, T., An, Y., Wang, K., & Li, S. (2020) · Security and Communication Networks · doi:10.1155/2020/2132138 · Cited by 124
Mukherjee, D., Mondal, R., Singh, P. K., Sarkar, R., & Bhattacharjee, D. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09537-7 · Cited by 124
Reyes-Ortiz, J. L., Oneto, L., Ghio, A., Samà, A., Anguita, D., & Parra, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7_23 · Cited by 114
A Novel Semisupervised Deep Learning Method for Human Activity Recognition
Zhu, Q., Chen, Z., & Soh, Y. C. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2889315 · Cited by 97
Human Daily Activity and Fall Recognition Using a Smartphone’s Acceleration Sensor
Chatzaki, C., Pediaditis, M., Vavoulas, G., & Tsiknakis, M. (2017) · Communications in Computer and Information Science · doi:10.1007/978-3-319-62704-5_7 · Cited by 91
w-HAR: An Activity Recognition Dataset and Framework Using Low-Power Wearable Devices
Bhat, G., Tran, N., Shill, H. A., & Ogras, U. Y. (2020) · Sensors · doi:10.3390/s20185356 · Cited by 91
Um, T. T., Babakeshizadeh, V., & Kulic, D. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206051 · Cited by 89
Sun, J., Fu, Y., Li, S., He, J., Xu, C., & Tan, L. (2018) · Journal of Sensors · doi:10.1155/2018/8580959 · Cited by 88
Efficient Human Activity Recognition Solving the Confusing Activities Via Deep Ensemble Learning
Zhu, R., Xiao, Z., Li, Y., Yang, M., Tan, Y., Zhou, L., Lin, S., & Wen, H. (2019) · IEEE Access · doi:10.1109/access.2019.2922104 · Cited by 86 · cited 1×
“fluctuations and improve the sensitivity and specificity rate of confusing HAR [24] presented an ensemble of CNN with”
BackgroundCheng, X., Zhang, L., Tang, Y., Liu, Y., Wu, H., & He, J. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3149337 · Cited by 82
Deep Human Activity Recognition With Localisation of Wearable Sensors
Lawal, I. A., & Bano, S. (2020) · IEEE Access · doi:10.1109/access.2020.3017681 · Cited by 80
Coarse-Fine Convolutional Deep-Learning Strategy for Human Activity Recognition
Avilés-Cruz, C., Ferreyra-Ramírez, A., Zúñiga-López, A., & Villegas-Cortéz, J. (2019) · Sensors · doi:10.3390/s19071556 · Cited by 73
A Robust Deep Learning Approach for Position-Independent Smartphone-Based Human Activity Recognition
Almaslukh, B., Artoli, A. M., & Al-Muhtadi, J. (2018) · Sensors · doi:10.3390/s18113726 · Cited by 71
Limitations with activity recognition methodology & data sets
Lockhart, J. W., & Weiss, G. M. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication · doi:10.1145/2638728.2641306 · Cited by 70
Ashry, S., Ogawa, T., & Gomaa, W. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2985374 · Cited by 69
A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors
Xiao, F., Pei, L., Chu, L., Zou, D., Yu, W., Zhu, Y., & Li, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69873-7_19 · Cited by 59
Ascioglu, G., & Şenol, Y. (2020) · IEEE Access · doi:10.1109/access.2020.3024003 · Cited by 57
Weakly Supervised Human Activity Recognition From Wearable Sensors by Recurrent Attention Learning
He, J., Zhang, Q., Wang, L., & Pei, L. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2885796 · Cited by 56
Wang, K., He, J., & Zhang, L. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3086008 · Cited by 49
Margin-Based Deep Learning Networks for Human Activity Recognition
Lv, T., Wang, X., Jin, L., Xiao, Y., & Song, M. N. (2020) · Sensors · doi:10.3390/s20071871 · Cited by 37
An LSTM-based Descriptor for Human Activities Recognition using IMU Sensors
Ashry, S., Elbasiony, R. M., & Gomaa, W. E. (2018) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0006902404940501 · Cited by 35
HDL: Hierarchical Deep Learning Model based Human Activity Recognition using Smartphone Sensors
Su, T., Sun, H., Ma, C., Jiang, L., & Xu, T. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851889 · Cited by 33
Jeong, C. Y., & Kim, M. (2019) · Sensors · doi:10.3390/s19173688 · Cited by 26
Design of Novel Deep Learning Models for Real-time Human Activity Recognition with Mobile Phones
Nutter, M., Crawford, C. H., & Ortiz, J. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489319 · Cited by 24
Alemayoh, T. T., Lee, J. H., & Okamoto, S. (2019) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/urai.2019.8768791 · Cited by 24
Shojaedini, S. V., & Beirami, M. J. (2020) · Biomedical Engineering Letters · doi:10.1007/s13534-020-00160-x · Cited by 24
ADL Classification Based on Autocorrelation Function of Inertial Signals
Gomaa, W. E., Elbasiony, R. M., & Ashry, S. (2017) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2017.00-53 · Cited by 23
Combining Symbolic Reasoning and Deep Learning for Human Activity Recognition
Rueda, F. M., Lüdtke, S., Schröder, M., Yordanova, K., Kirste, T., & Fink, G. A. (2019) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2019.8730792 · Cited by 20
Deep Learning for Smartphone-Based Human Activity Recognition Using Multi-sensor Fusion
Buenaventura, C. V. S., Tiglao, N. M. C., & Atienza, R. (2019) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-06158-6_7 · Cited by 16
Cheng, X., Zhang, L., Tang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03259 · Cited by 3
Research works printed but not linked: 3
- 10.1109/jsen.2020.3015521 · DOI · Paper page
- 2020, UCI Machine Learning Repository
- 2020, Kaggle datasets
Cited by
Qiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
“There are some existing review papers in the literature for deep learning approaches for sensor-based human activity recognition [63, 204, 224, 291].”
BackgroundMethodDeep Learning for Time Series Classification and Extrinsic Regression: A Current Survey
Mohammadi Foumani, N., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., & Salehi, M. (2024) · ACM Computing Surveys · doi:10.1145/3649448 · Cited by 268
Wearable Sensor-Based Human Activity Recognition in the Smart Healthcare System
Serpush, F., Menhaj, M. B., Masoumi, B., & Karasfi, B. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/1391906 · Cited by 223
Islam, M. M., Nooruddin, S., Karray, F., & Muhammad, G. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106060 · Cited by 217
Metadata
- Authors
- Ramanujam, E., Perumal, T., & Padmavathi, S.
- Affiliations
- Thiagarajar College of Engineering (IN)
- Universiti Putra Malaysia (MY)
- Venue
- IEEE Sensors Journal
- Published
- 2021-06-15
- Volume
- 21
- Issue
- 12
- Pages
- 13029-13040
- ISSN
- 1530-437X (ISSN-L)
- 1558-1748
- 2379-9153
- OpenAlex
- W3139646219
- Cited by
- 455
- References
- 69
- Citations by year
- 2026: 67 · 2025: 98 · 2024: 119 · 2023: 106 · 2022: 52 · 2021: 13
- FWCI
- 26.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Sensors Journal (publishedVersion)
- Universiti Putra Malaysia Institutional Repository (Universiti Putra Malaysia) (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human activity recognition, smartphone sensors, wearable sensors, deep learning, ambient assisted living, elderly care, rehabilitation, feature extraction, feature selection, time-series signal processing, benchmark dataset, hybrid deep learning model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.