Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; none of the 62 sources its claims cite is checked for support yet.

A Survey on Multimodal Wearable Sensor-Based Human Action Recognition

Open access
Ni, Jianyuan, Guo, Bohan, Haque, Syed Tousiful, Ngu, Anne H. H., Ding, Cheng · 2026 · IEEE Internet of Things Journal

Abstract

The combination of increased life expectancy and falling birth rates is resulting in an aging population. Wearable Sensor-based Human Activity Recognition (WSHAR) emerges as a promising assistive technology to support the daily lives of older individuals, unlocking vast potential for human-centric applications. However, recent surveys in WSHAR have been limited, focusing either solely on deep learning approaches or on a single sensor modality. In real life, our human interact with the world in a multi-sensory way, where diverse information sources are intricately processed and interpreted to accomplish a complex and unified sensing system. To give machines similar intelligence, multimodal machine learning, which merges data from various sources, has become a popular research area with recent advancements. In this study, we present a comprehensive survey from a novel perspective on how to leverage multimodal learning to WSHAR domain for newcomers and researchers. We begin by presenting the recent sensor modalities as well as deep learning approaches in HAR. Subsequently, we explore the techniques used in present multimodal systems for WSHAR. This includes inter-multimodal systems which utilize sensor modalities from both visual and non-visual systems and intra-multimodal systems that simply take modalities from non-visual systems. After that, we focus on current multimodal learning approaches that have applied to solve some of the challenges existing in WSHAR. Specifically, we make extra efforts by connecting the existing multimodal literature from other domains, such as computer vision and natural language processing, with current WSHAR area. Finally, we identify the corresponding challenges and potential research direction in current WSHAR area for further improvement.

Overview

CoChat Summary

## TL;DR

This paper is a survey of multimodal wearable sensor-based human action recognition (WSHAR), organizing the field around visual and non-visual data, multimodal fusion, and challenges such as scarce labels, class imbalance, feature alignment, and deployment constraints. It argues that multimodal learning can improve robustness and accuracy, and highlights future directions including foundation models, unified multimodal systems, personalization, privacy, and concurrent activity segmentation.

## Key findings

  • The survey states that WSHAR is motivated by aging-related health needs and that wearable IMU-based approaches can preserve privacy, but they often perform worse than visual methods; it cites a prior fall detection study with 86% accuracy using wrist-worn accelerometer data and DL.
  • For multimodal HAR, the paper reports that combining depth and IMUs improved accuracy by 2–23% compared with using them individually in one cited study.
  • The paper identifies several persistent WSHAR challenges: annotation scarcity, class imbalance, distribution discrepancy, computational cost, heterogeneous feature alignment, and limited multimodal deployment on constrained devices.
  • On deployment, it cites a lightweight multimodal temporal segment attention network tested on a Raspberry Pi 4B and reports that TensorRT compression reduced latency by about 10% on AGX and 50% on iPhone XR.

## Methods

  • This is a narrative survey; it does not report a new experimental dataset or a new model, but systematically reviews prior work on visual and non-visual HAR, multimodal fusion, and challenge-oriented solutions.
  • The survey covers visual modalities (RGB frame, egocentric video, skeleton sequence) and non-visual modalities (audio, WiFi/CSI, inertial sensors), then reviews inter-multimodal and intra-multimodal HAR approaches.
  • Its analysis is a literature synthesis that organizes prior methods by modality, multimodal learning strategy, and challenge category, and it connects WSHAR work to advances in CV and NLP.

## Limitations

  • The paper explicitly states that existing WSHAR multimodal research is still in its infancy and that several issues are not fully explored, including heterogeneous feature alignment and lightweight model deployment.
  • It notes that many multimodal datasets contain limited labeled data, are mostly small- or medium-scale, and that only a few include IMU data; this is a limitation of the field that the paper highlights.
  • Evident from the text: the paper is a survey, so it does not provide a unified benchmark, a controlled empirical comparison across methods, or new experimental results of its own.

Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/jiot.2026.3716688

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 3 Not valid · 47 Not checked · 23 Own results and interpretations

29 more claims

Create a free account to see all 73 claims and to verify the 62 references still open.

See all claims — free account
  • Probably not a substantial claim. 28
  • It may state two separate claims as one. 15

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 242

  1. 2017, World Population Prospects: Key Findings and Advance Tables
  2. Ding, 2024, A survey of LLMs on biosignal applications, 10.36227/techrxiv.173155915.55479649/v1 · DOI
  3. 10.1109/TIP.2021.3086590 · DOI
  4. 10.1109/ICASSP.2016.7472168 · DOI
  5. 10.1016/j.pmcj.2016.05.002 · DOI
  6. 10.3390/s23031105 · DOI
  7. 10.1016/j.bios.2024.117073 · DOI
  8. 10.1016/j.knosys.2021.106970 · DOI · Paper page
  9. 10.1109/TPAMI.2022.3183112 · DOI · Paper page
  10. 10.1016/j.knosys.2019.104939 · DOI
  11. 10.1145/3503161.3548238 · DOI
  12. 10.1007/s12668-013-0088-3 · DOI · Paper page
  13. 10.1109/SURV.2012.110112.00192 · DOI · Paper page
  14. 10.1016/j.artmed.2024.102900 · DOI · Paper page
  15. 10.1007/s11042-020-09004-3 · DOI · Paper page
  16. 10.1145/3472290 · DOI · Paper page
  17. 10.1145/3447744 · DOI · Paper page
  18. 10.1109/JSEN.2021.3069927 · DOI · Paper page
  19. 10.3390/s22041476 · DOI · Paper page
  20. 10.1007/s11263-022-01594-9 · DOI · Paper page
  21. 10.1145/3717608 · DOI · Paper page
  22. 10.1145/3648469 · DOI
  23. 10.1109/IJCNN60899.2024.10650060 · DOI
  24. 10.3390/s24155045 · DOI
  25. 10.1145/3699767 · DOI
  26. 10.1007/s10462-025-11275-x · DOI
  27. 10.1145/3729467 · DOI · Paper page
  28. 10.1109/ACCESS.2025.3590073 · DOI
  29. 10.1109/THMS.2014.2362520 · DOI
  30. 10.1109/TMM.2018.2875510 · DOI
  31. 10.1109/TPAMI.2016.2522398 · DOI · Paper page
  32. 10.3390/s19173680 · DOI
  33. 10.1109/JSEN.2019.2947446 · DOI
  34. 10.3390/s21030768 · DOI
  35. 10.1109/CVPRW56347.2022.00224 · DOI
  36. 10.1109/JSEN.2023.3349191 · DOI
  37. 10.1109/ICCA.2018.8444326 · DOI
  38. 10.1109/JSEN.2020.3034614 · DOI
  39. 10.1016/j.inffus.2019.06.014 · DOI · Paper page
  40. 10.48550/arXiv.1503.02531 · DOI · Paper page
  41. 10.1109/ICCV.2019.00875 · DOI
  42. 10.1109/ICASSP43922.2022.9746752 · doi:10.1109/icassp43922.2022.9746752
  43. 10.1007/978-3-031-78354-8_27 · DOI
  44. 10.1109/JSEN.2024.3510811 · DOI
  45. 10.1016/j.neucom.2025.132517 · DOI
  46. 10.1609/aaai.v34i04.5963 · DOI · Paper page
  47. Tian, Learning MLPs on graphs: A unified view of effectiveness, robustness, and efficiency
  48. 10.1145/3711121 · DOI
  49. 10.1109/FG59268.2024.10581969 · DOI
  50. 10.1109/IMSA65733.2025.11167801 · DOI
  51. 10.1109/ACCESS.2025.3556001 · DOI
  52. 10.1109/PerCom64205.2025.00020 · DOI
  53. Chen, 2025, COMODO: Cross-modal video-to-IMU distillation for efficient egocentric human activity recognition, arXiv:2503.07259 · arXiv
  54. 10.1109/ISMAR-Adjunct64951.2024.00037 · DOI
  55. 10.1109/EMBC.2012.6346340 · DOI · Paper page
  56. 10.1109/WACV.2013.6474999 · DOI · Paper page
  57. 10.1007/978-3-642-38682-4_25 · DOI
  58. 10.1109/JSEN.2014.2306094 · DOI · Paper page
  59. 10.1109/ICIP.2015.7350781 · DOI · Paper page
  60. 10.1109/3DV.2017.00058 · DOI
  61. 10.1109/JSEN.2018.2872862 · DOI
  62. 10.1109/JSEN.2018.2869807 · DOI
  63. 10.3390/s19091988 · DOI · Paper page
  64. 10.1109/FG47880.2020.00018 · DOI
  65. 10.1109/CVPR46437.2021.01103 · DOI
  66. 10.3389/fcomp.2021.792065 · DOI · Paper page
  67. 10.1109/CVPR52688.2022.01842 · DOI · Paper page
  68. 10.1109/JSEN.2022.3150225 · DOI
  69. 10.1007/s11263-021-01531-2 · DOI · Paper page
  70. 10.52202/068431-1003 · doi:10.52202/068431-1003
  71. 10.1038/s41597-022-01573-2 · DOI
  72. 10.1038/s41597-023-02554-9 · DOI
  73. 10.1007/s00530-021-00875-6 · DOI
  74. 10.1016/j.patrec.2024.06.017 · DOI
  75. Jiang, 2025, A large-scale multimodal dataset and benchmarks for human activity scene understanding and reasoning, arXiv:2512.07136 · arXiv
  76. Yang, Deep convolutional neural networks on multichannel time series for human activity recognition, 15, 3995
  77. 10.1145/2971648.2971708 · DOI · Paper page
  78. 10.1109/EUVIP47703.2019.8946180 · DOI
  79. 10.1109/ICMCCE.2018.00052 · DOI
  80. 10.1109/ACCESS.2020.3027979 · DOI
  81. 10.1007/s00607-021-00928-8 · DOI · Paper page
  82. 10.1016/j.asoc.2021.107728 · DOI · Paper page
  83. 10.1109/TETCI.2021.3136642 · DOI
  84. 10.1109/TII.2022.3165875 · DOI · Paper page
  85. 10.1016/j.eswa.2023.121296 · DOI
  86. 10.1145/3130985 · DOI
  87. 10.1016/j.inffus.2017.06.004 · DOI
  88. 10.1371/journal.pone.0227039 · DOI
  89. 10.1145/3550284 · DOI
  90. 10.1109/THMS.2022.3188726 · DOI
  91. 10.1109/JIOT.2024.3400022 · DOI
  92. 10.1109/PerComWorkshops59983.2024.10502797 · doi:10.1109/percomworkshops59983.2024.10502797
  93. 10.1016/j.eswa.2024.125564 · DOI
  94. 10.1109/JIOT.2025.3555799 · DOI
  95. 10.1109/TIM.2024.3365155 · DOI
  96. 10.1109/TIM.2025.3571487 · DOI
  97. 10.52202/085713-1313 · DOI
  98. 10.1145/3638550.3641130 · DOI
  99. 10.1145/3478119 · DOI
  100. 10.1109/CVPR52733.2024.01265 · DOI
  101. Li, Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation, 80
  102. 10.52202/079017-2096 · DOI
  103. 10.18653/v1/2024.acl-long.620 · DOI
  104. 10.1145/3272127.3275108 · DOI · Paper page
  105. 10.1109/TIM.2021.3111996 · DOI
  106. 10.1109/TMC.2022.3230370 · DOI
  107. 10.1109/DASC/PiCom/CBDCom/Cy55231.2022.9928021 · DOI
  108. 10.3233/JIFS-213174 · DOI
  109. 10.1145/3596711.3596800 · DOI · Paper page
  110. 10.1145/3411841 · DOI · Paper page
  111. 10.1145/3544793.3560337 · DOI
  112. 10.1109/BSN58485.2023.10331242 · DOI
  113. 10.1145/3610881 · DOI
  114. 10.1145/3596261 · DOI
  115. 10.1145/3594738.3611361 · DOI · Paper page
  116. 10.1145/3678545 · DOI · Paper page
  117. Akbari, VATT: Transformers for multimodal self-supervised learning from raw video, audio and text, 34, 24206
  118. Choi, 2023, Multimodal contrastive learning with hard negative sampling for human activity recognition, arXiv:2309.01262 · arXiv
  119. 10.1109/IJCNN55064.2022.9892522 · DOI
  120. Deldari, 2023, CroSSL: Cross-modal self-supervised learning for time-series through latent masking, arXiv:2307.16847 · arXiv
  121. 10.1109/TETCI.2021.3079966 · DOI
  122. 10.1109/WACV48630.2021.00280 · DOI
  123. 10.1109/CVPR.2019.01227 · DOI · Paper page
  124. 10.1109/CVPR52688.2022.00298 · DOI · Paper page
  125. 10.1007/s13042-024-02443-6 · DOI · Paper page
  126. Liu, 2023, Large language models are few-shot health learners, arXiv:2305.15525 · arXiv
  127. Li, Frozen language model helps ECG zero-shot learning, 402
  128. 10.48550/arXiv.1312.6114 · DOI · Paper page
  129. Goodfellow, Generative adversarial nets, 27, 2672
  130. Ho, 2020, Denoising diffusion probabilistic models, NeurIPS, 33, 6840
  131. 10.1109/IJCNN.2018.8489106 · DOI · Paper page
  132. 10.1109/ICCI51257.2020.9247670 · DOI
  133. 10.1016/j.neucom.2021.12.097 · DOI
  134. 10.1145/3715014.3722089 · DOI
  135. 10.1145/3410530.3414329 · DOI
  136. Cubuk, 2018, AutoAugment: Learning augmentation policies from data, arXiv:1805.09501 · arXiv
  137. 10.48550/arxiv.1710.09412 · DOI · Paper page
  138. 10.1109/ICCV.2019.00612 · DOI · Paper page
  139. 10.21437/Interspeech.2018-2456 · DOI
  140. 10.1109/CVPRW56347.2022.00504 · DOI
  141. 10.1109/MIS.2020.3026715 · DOI
  142. 10.1016/j.asoc.2023.110125 · DOI
  143. 10.1145/3675094.3678464 · DOI
  144. Asano, Labelling unlabelled videos from scratch with multi-modal self-supervision, 33, 4660
  145. 10.1109/CVPR42600.2020.00990 · DOI · Paper page
  146. 10.1609/aaai.v36i2.20035 · doi:10.1609/aaai.v36i2.20035
  147. Alayrac, Self-supervised multimodal versatile networks, 33, 25
  148. 10.1145/3805622.3810732 · doi:10.1145/3805622.3810732
  149. 10.1145/3328932 · DOI · Paper page
  150. 10.1145/3448112 · DOI · Paper page
  151. 10.1038/s41746-024-01062-3 · DOI · Paper page
  152. 10.1145/3517246 · DOI · Paper page
  153. 10.1109/JBHI.2024.3373019 · DOI · Paper page
  154. 10.1145/3746252.3761215 · DOI · Paper page
  155. 10.1109/JSEN.2024.3390755 · DOI
  156. 10.24963/ijcai.2025/952 · DOI · Paper page
  157. 10.1109/JSEN.2024.3491933 · DOI · Paper page
  158. 10.1145/3715071.3750416 · DOI
  159. Sheng, 2024, Consistency based weakly self-supervised learning for human activity recognition with wearables, arXiv:2408.07282 · arXiv
  160. 10.1109/JIOT.2024.3379019 · DOI
  161. 10.1145/3675094.3677562 · DOI
  162. 10.1145/3659597 · DOI · Paper page
  163. 10.1016/j.neucom.2026.133963 · DOI
  164. 10.1145/3704921 · DOI
  165. 10.1109/TETCI.2025.3526504 · DOI
  166. Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
  167. 10.1145/3631445 · DOI · Paper page
  168. 10.1145/3265689.3265705 · DOI
  169. 10.1016/j.neucom.2021.04.124 · DOI
  170. 10.1109/CVPR46437.2021.00692 · DOI · Paper page
  171. Geng, 2022, Multimodal masked autoencoders learn transferable representations, arXiv:2205.14204 · arXiv
  172. 10.1145/3534589 · DOI
  173. 10.1109/ICIP.2019.8802909 · DOI
  174. 10.1109/JSEN.2023.3337367 · DOI
  175. 10.1109/TNNLS.2025.3556317 · DOI
  176. 10.1038/s41598-024-63195-5 · DOI
  177. 10.1038/s41598-025-26297-2 · DOI · Paper page
  178. 10.1613/jair.953 · DOI · Paper page
  179. 10.1145/3543848 · DOI
  180. 10.1016/j.cmpb.2023.107495 · DOI
  181. 10.1016/j.eswa.2017.09.030 · DOI · Paper page
  182. 10.1109/TKDE.2022.3182327 · DOI
  183. 10.1007/s11042-020-09227-4 · DOI
  184. 10.1007/978-3-031-66538-7_21 · DOI
  185. 10.1109/ICFTIC64248.2024.10912869 · DOI
  186. Bao, 2023, A recent survey of heterogeneous transfer learning, arXiv:2310.08459 · arXiv
  187. Yu, 2025, CMD-HAR: Cross-modal disentanglement for wearable human activity recognition, arXiv:2503.21843 · arXiv
  188. 10.1145/3616855.3635795 · DOI
  189. 10.1145/3699779 · DOI
  190. 10.1109/TPAMI.2024.3367412 · DOI
  191. 10.1109/TIM.2025.3588988 · DOI
  192. 10.1109/CVPR.2016.96 · DOI · Paper page
  193. 10.1007/978-3-030-01237-3_7 · DOI · Paper page
  194. 10.1109/CVPR52688.2022.01594 · DOI
  195. 10.15625/1813-9663/22770 · DOI
  196. 10.1145/3428666 · DOI
  197. 10.1162/tacl_a_00704 · DOI · Paper page
  198. 10.1371/journal.pone.0335047 · DOI
  199. 10.1016/j.inffus.2023.101953 · DOI
  200. 10.3390/electronics13183612 · DOI

200 of 242 shown

Cited by

See all 17

Metadata

Authors
Ni, J., Guo, B., Haque, S. T., Ngu, A. H. H., & Ding, C.
Affiliations
  • Texas State University (US)
  • University of Malaya (MY)
  • Georgia Institute of Technology (US)
Venue
IEEE Internet of Things Journal
Published
2026-10
Volume
13
Issue
19
Pages
44221-44236
ISSN
2327-4662 (ISSN-L)
OpenAlex
W4395477635
Cited by
17
References
0
Citations by year
2026: 8
FWCI
22.50
Type
journal article
Language
en
Open access
Locations
  • IEEE Internet of Things Journal (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
NSF (SCH-21223749)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
wearable sensors, human activity recognition, multimodal learning, assistive technology, population aging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Survey on Multimodal Wearable Sensor-Based Human Action Recognition. https://app.cochat.ai/papers/10.1109/jiot.2026.3716688
The paper
Ni, J., Guo, B., Haque, S. T., Ngu, A. H. H., & Ding, C. (2026). A Survey on Multimodal Wearable Sensor-Based Human Action Recognition. IEEE Internet of Things Journal. https://doi.org/10.1109/jiot.2026.3716688

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.