Multi-modal human action recognition using deep neural networks fusing image and inertial sensor data
Abstract
Human action recognition has been studied in many fields including computer vision and sensor networks using inertial sensors. However, there are limitations such as spatial constraints, occlusions in images, sensor unreliability, and the inconvenience of users. In order to solve these problems we suggest a sensor fusion method for human action recognition exploiting RGB images from a single fixed camera and a single wrist mounted inertial sensor. These two different domain information can complement each other to fill the deficiencies that exist in both image based and inertial sensor based human action recognition methods. We propose two convolutional neural network (CNN) based feature extraction networks for image and inertial sensor data and a recurrent neural network (RNN) based classification network with long short term memory (LSTM) units. Training of deep neural networks and testing are done with synchronized images and sensor data collected from five individuals. The proposed method results in better performance compared to single sensor-based methods with an accuracy of 86.9% in cross-validation. We also verify that the proposed algorithm robustly classifies the target action when there are failures in detecting body joints from images.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The combination setting outperformed camera-only and IMU-only systems for action classification.
“The cross-validated accuracy for action classification showed that the combination setting outperformed camera-only and IMU-only systems.”
From Review of Wearable Devices and Data Collection Considerations for Connected Health (2021) · cites this paper as [172]
A NN system based on IMU sensor data can help physicians in situations when action recognition approaches based on cameras fail to reliably forecast the precise human position.
“According to the findings, a NN system based on IMU sensor data can help physicians in situations when action recognition approaches based on cameras fail to reliably forecast the precise human position [172].”
From Review of Wearable Devices and Data Collection Considerations for Connected Health (2021) · cites this paper as [172]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,288 · see citing works
Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · see citing works
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 3× · see citing works
“Furthermore, a neural network is exploited for better classification without putting the effort of humans for designing proper features [4].”
BackgroundA Survey on Human Activity Recognition using Wearable Sensors
Lara, O. D., & Labrador, M. A. (2013) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2012.110112.00192 · Cited by 2,662 · see citing works
Articulated pose estimation with flexible mixtures-of-parts
Yang, Y., & Ramanan, D. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995741 · Cited by 999 · see citing works
Towards Understanding Action Recognition
Jhuang, H., Gall, J., Zuffi, S., Schmid, C., & Black, M. J. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.396 · Cited by 906 · see citing works
Collecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788 · see citing works
The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 3× · see citing works
“However, the IMU-based action recognition shows meaningful performance only with multiple sensors mounted on the whole body which hinders the patients’ convenience [3].”
BackgroundBerkeley MHAD: A comprehensive Multimodal Human Action Database
Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., & Bajcsy, R. K. (2013) · Proceedings · doi:10.1109/wacv.2013.6474999 · Cited by 490 · see citing works
Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers
Ward, J. A., Lukowicz, P., Troster, G., & Starner, T. E. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.197 · Cited by 431 · see citing works
Activity recognition with smartphone sensors
Su, X. Y., Tong, H., & Ji, P. (2014) · Tsinghua Science & Technology · doi:10.1109/tst.2014.6838194 · Cited by 383 · see citing works
Morris, D., Saponas, T. S., Guillory, A., & Kelner, I. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557116 · Cited by 189 · see citing works
Jatobá, L., Grossmann, U., Kunze, C., Ottenbacher, J., & Stork, W. (2008) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2008.4650398 · Cited by 114 · see citing works
Alvarez-Alvarez, A., Alonso, J., & Triviño, G. (2013) · Information Sciences · doi:10.1016/j.ins.2013.01.029 · Cited by 43 · see citing works
Structural Support Vector Machine
Xue, H., Chen, S., & Yang, Q. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87732-5_56 · Cited by 35 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Review of Wearable Devices and Data Collection Considerations for Connected Health
Vijayan, V., Connolly, J. P., Condell, J., McKelvey, N., & Gardiner, P. (2021) · Sensors · doi:10.3390/s21165589 · Cited by 435 · see citing works
Robust Human Activity Recognition Using Multimodal Feature-Level Fusion
Ehatisham-ul-Haq, M., Javed, A., Azam, M. A., Malik, H. M. A., Irtaza, A., Lee, I. H., & Mahmood, M. T. (2019) · IEEE Access · doi:10.1109/access.2019.2913393 · Cited by 183 · see citing works
Arikumar, K., Prathiba, S., Alazab, M., Gadekallu, T., Pandya, S., Khan, J., & Moorthy, R. (2022) · Sensors · doi:10.3390/s22041377 · Cited by 168 · see citing works
RetractedNadeem, A., Jalal, A., & Kim, K. (2020) · Symmetry · doi:10.3390/sym12111766 · Cited by 95 · see citing works
Strömbäck, D., Huang, S., & Radu, V. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3432701 · Cited by 73 · see citing works
Ahmad, Z., & Khan, N. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2947446 · Cited by 71 · see citing works
Inertial Sensor Data to Image Encoding for Human Action Recognition
Ahmad, Z., & Khan, N. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3062261 · Cited by 56 · see citing works
3D Human Action Recognition: Through the eyes of researchers
Sarkar, A., Banerjee, A., Singh, P. K., & Sarkar, R. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116424 · Cited by 43 · see citing works
Ahmad, Z., & Khan, N. (2018) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2018.000-2 · Cited by 37 · see citing works
A novel multi-modal rehabilitation monitoring over human motion intention recognition
Kamal, S., Alshehri, M. S., Alqahtani, Y., Alshahrani, A., Almujally, N. A., Jalal, A., & Liu, H. (2025) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2025.1568690 · Cited by 22 · see citing works
Male, J., & Martínez-Hernández, U. (2021) · IEEE International Conference on Industrial Technology (ICIT) · doi:10.1109/icit46573.2021.9453672 · Cited by 15 · see citing works
Empowering Supply Chain Management System with Machine Learning and Blockchain Technology
Alshurideh, M. T., Hamadneh, S., Alzoubi, H. M., Kurdi, B. H. A., Nuseir, M. T., & AlHamad, A. (2024) · Studies in big data · doi:10.1007/978-3-031-31801-6_21 · Cited by 13 · see citing works
Human activity recognition based on multi-modal fusion
Zhang, C., Zu, T., Hou, Y., He, J., Yang, S., & Dong, R. (2023) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-023-00132-x · Cited by 7 · see citing works
Enhancing Sensor-Based Human Activity Recognition using Efficient Channel Attention
Jitpattanakul, A., & Mekruksavanich, S. (2023) · Proceedings of IEEE Sensors/Proceedings of IEEE Sensors ... · doi:10.1109/sensors56945.2023.10325157 · Cited by 7 · see citing works
Improved Spatiotemporal Framework for Human Activity Recognition in Smart Environment
Salem, Z., & Weiss, A. P. (2022) · Sensors · doi:10.3390/s23010132 · Cited by 7 · see citing works
A Basic Study on Ballroom Dance Figure Classification with LSTM Using Multi-modal Sensor
Matsuyama, H., Hiroi, K., Kaji, K., Yonezawa, T., & Kawaguchi, N. (2020) · Smart innovation, systems and technologies · doi:10.1007/978-981-15-8944-7_13 · Cited by 5 · see citing works
Low-cost multi-view pose tracking using active markers
Ruppel, P., Hendrich, N., & Zhang, J. (2019) · IEEE International Conference on Industrial Cyber-Physical Systems (ICPS) · doi:10.1109/icphys.2019.8780297 · Cited by 3 · see citing works
Multi-modal fusion in ergonomic health: bridging visual and pressure for sitting posture detection
Quan, Q., Gao, Y., Bai, Y., & Jin, Z. (2024) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-024-00164-x · Cited by 2 · see citing works
Khan, N., & Ahmad, Z. (2023) · doi:10.32920/22734272.v1 · Cited by 2 · see citing works
Human Activity Recognition using Convolutional Neural Network
Dugar, M. (2023) · Computer Applications & Technological Solutions (CATS) · doi:10.1109/cats58046.2023.10424362 · Cited by 1 · see citing works
Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial SensorsPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734263.v1 · Cited by 1 · see citing works
Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial SensorsPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734263 · Cited by 1 · see citing works
Khan, N., & Ahmad, Z. (2023) · doi:10.32920/22734272
Inertial Sensor Data To Image Encoding For Human Action RecognitionPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734293
Inertial Sensor Data To Image Encoding For Human Action RecognitionPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734293.v1
Enhancing Human Activity Recognition with IMU-based Motion Analysis
Singh, A., Verma, A., & Cherukuri, B. (2023) · International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) · doi:10.1109/icac3n60023.2023.10541396
Review on the application of Artificial Neural Networks in gait analysis
Lin, Y. (2025) · Advances in Engineering Innovation · doi:10.54254/2977-3903/2025.29746
- Authors
- Hwang, I., Cha, G., & Oh, S.
- Affiliations
- Seoul National University (KR)
- Venue
- 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)
- Publisher
- IEEE
- Published
- 2017-11
- Pages
- 278-283
- OpenAlex
- W2775529075
- Cited by
- 28
- References
- 21
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 10 · 2022: 3 · 2021: 3 · 2020: 3 · 2019: 3 · 2018: 1
- FWCI
- 0.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) (publishedVersion)
- Funders
- Samsung
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human action recognition, multimodal sensor fusion, inertial sensors, RGB images, wrist-worn sensors, convolutional neural networks, recurrent neural networks, LSTM, wearable sensors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
