CoChat

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Open access

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 5 of the 32 sources its claims cite were checked for support.

Abstract

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities.

Overview

CoChat Summary

TL;DR

The paper builds an eight-IMU testbed and LSTM framework to study on-body sensor placement, sampling rate, and multimodal fusion for human activity recognition. It finds that 10 Hz data from four sensors—both wrists, right ankle, and waist—is sufficient for ADLs, and classifier-level fusion improves recognition accuracy up to 94.47%.

Key findings

  • Activity data with sampling rate as low as 10 Hz from four sensors at both wrists, right ankle, and waist was sufficient for recognizing ADLs including eating and driving; a 2.5 s sliding window outperformed other settings, with 25 samples covering 2.5 s.
  • The four-sensor baseline achieved 93.0% recognition accuracy. Among three-sensor combinations, right wrist, right ankle, and waist was most competitive; with two sensors, placing one on the upper half and one on the lower half performed better than both on the upper or both on the lower half. Right wrist tended to outperform left wrist, and right ankle played a more critical role than waist.
  • Single-modality accuracy was 92.18% for accelerometer, 85.22% for magnetometer, and 78.33% for gyroscope. Multimodal fusion reached up to 93.07% with stacking/voting, while soft-voting achieved 94.47% versus hard-voting at 93.48%, a 2.48% improvement over single-modal recognition.
  • Activity-specific weights increased accuracy by 7.65% for walking upstairs and 36.97% for walking downstairs. Static activities were best detected by accelerometer, and fusing additional sensors could degrade sitting/standing accuracy; stair walking was often confused with plain walking.

Methods

  • A testbed with eight body-worn IMU sensors (MPU-9250) and an Android mobile device collected data from five researchers (two males, three females; ages 34–48; heights 160–175 cm; all right-handed) over three workdays, about three hours per subject, in real-world and controlled environments. Final configuration used four sensors on left/right wrist, right ankle, and waist.
  • Raw sensor data were fed to an LSTM network with a 2.5 s sliding window and 50% overlap; parameters included batch size 150, one stacked LSTM layer, hidden neuron size 32, 25 steps, and keep probability 0.8. Evaluation used 10-fold cross-validation and leave-one-subject-out cross-validation with jittering data augmentation.
  • Sensor fusion used classifier-level two-level ensembles: LSTM base learners per modality, stacking with Random Forest/kNN/SVM meta-learners (Random Forest adopted), and soft/hard voting with weights based on sensor accuracy or activity-specific characteristics.

Limitations

  • The paper states that experiments used a small number of test subjects within the institution, that the meta-learner was evaluated only with classical algorithms (RF, kNN, SVM), and that future work should include large-scale/long-term experiments, additional sensing modalities, and online weight learning.
  • Evident from the text: the pilot involved only five right-handed subjects with limited demographic variation, gyroscope data were used without calibration, and stair walking versus plain walking produced equivocal results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s19071716

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Testbed configuration. (a) A body-worn Inertial Measurement Unit (IMU) device consists of (1) a 9-axis IMU sensor (i.e., 3-axis accelerometer, gyroscope, and magnetometer, respectively), (2) a Micro Controller Unit (MCU), and (3) a micro-USB rechargeable Li-Po battery module; (b) Among eight body-worn sensors (in circles), only four sensors at left and right wrists, waist, and ankle (in red circles) are adopted for the data analysis.
Figure 1 Testbed configuration. (a) A body-worn Inertial Measurement Unit (IMU) device consists of (1) a 9-axis IMU sensor (i.e., 3-axis accelerometer, gyroscope, and magnetometer, respectively), (2) a Micro Controller Unit (MCU), and (3) a micro-USB rechargeable Li-Po battery module; (b) Among eight body-worn sensors (in circles), only four sensors at left and right wrists, waist, and ankle (in red circles) are adopted for the data analysis.CC BY · PubMed Central
Figure 2 Testbed system consists of a mobile device, a relay module, and multiple IMU sensor devices. (1) Request for soft-AP setting, (2) Radio Frequency (RF) connection request, (3) Response on connection establishment, (4)–(5) Activity data transmission.
Figure 2 Testbed system consists of a mobile device, a relay module, and multiple IMU sensor devices. (1) Request for soft-AP setting, (2) Radio Frequency (RF) connection request, (3) Response on connection establishment, (4)–(5) Activity data transmission.CC BY · PubMed Central
Figure 3 Recognition accuracy according to various length of sliding window.
Figure 3 Recognition accuracy according to various length of sliding window.CC BY · PubMed Central
Figure 4 Many-to-one Long Short-Term Memory (LSTM) network architecture used for activity classification with nine classes. n stands for the number of samples included in a 2.5 s window.
Figure 4 Many-to-one Long Short-Term Memory (LSTM) network architecture used for activity classification with nine classes. n stands for the number of samples included in a 2.5 s window.CC BY · PubMed Central
Figure 5 Classifier-level ensemble for multimodal sensor fusion. Prediction results from each sensor modality are used as meta-features for the meta-learner.
Figure 5 Classifier-level ensemble for multimodal sensor fusion. Prediction results from each sensor modality are used as meta-features for the meta-learner.CC BY · PubMed Central
Figure 6 Recognition accuracy and execution time according to different sampling rate of the sensor.
Figure 6 Recognition accuracy and execution time according to different sampling rate of the sensor.CC BY · PubMed Central
Figure 7 Recognition performance according to different numbers and combinations of body-worn sensors. Both accuracy and F1-score are adopted as the performance measure. (a) 10-fold cross-validation result; (b) Leave-one-subject-out cross-validation result.
Figure 7 Recognition performance according to different numbers and combinations of body-worn sensors. Both accuracy and F1-score are adopted as the performance measure. (a) 10-fold cross-validation result; (b) Leave-one-subject-out cross-validation result.CC BY · PubMed Central
Figure 8 Performance and execution time of meta-learner models used in stacking ensemble.
Figure 8 Performance and execution time of meta-learner models used in stacking ensemble.CC BY · PubMed Central
Figure 9 Recognition accuracy and F1-score measure of each sensor modality using the LSTM network.
Figure 9 Recognition accuracy and F1-score measure of each sensor modality using the LSTM network.CC BY · PubMed Central
Figure 10 Recognition accuracy of a group of activities using multiple sensor modalities. (a) A group of combinatorial activities of sitting, including driving, moving in a car (as a passenger), and eating; (b) A group of static activities including sitting, standing, and lying; (c) A group of activities that involves walking, including walking upstairs and walking downstairs.
Figure 10 Recognition accuracy of a group of activities using multiple sensor modalities. (a) A group of combinatorial activities of sitting, including driving, moving in a car (as a passenger), and eating; (b) A group of static activities including sitting, standing, and lying; (c) A group of activities that involves walking, including walking upstairs and walking downstairs.CC BY · PubMed Central
Figure 11 Confusion matrix for each activity. The x-axis stands for the ground truth and the y-axis denotes predicted labels through voting ensemble.
Figure 11 Confusion matrix for each activity. The x-axis stands for the ground truth and the y-axis denotes predicted labels through voting ensemble.CC BY · PubMed Central
Figure 12 Proportion of weights of each sensor is computed based on the recognition accuracy.
Figure 12 Proportion of weights of each sensor is computed based on the recognition accuracy.CC BY · PubMed Central
Figure 13 Recognition accuracy increases when applying weights that are customized to each activity.
Figure 13 Recognition accuracy increases when applying weights that are customized to each activity.CC BY · PubMed Central
Figure 14 Recognition accuracy and F1-score measure of stacking ensemble using various combinations of sensors, and two voting ensemble techniques adopting all sensor modalities.
Figure 14 Recognition accuracy and F1-score measure of stacking ensemble using various combinations of sensors, and two voting ensemble techniques adopting all sensor modalities.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 0 Not valid · 13 Not checked · 50 Own results and interpretations

41 more claims

Create a free account to see all 67 claims and to verify the 27 references still open.

See all claims — free account
  • Probably not a substantial claim. 9
  • It may state two separate claims as one. 12
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 49 references

Cited by

Metadata

Authors
Chung, S., Lim, J., Noh, K. J., Kim, G., & Jeong, H.
Affiliations
Electronics and Telecommunications Research Institute (KR)
Corresponding author
Chung, Seungeun
Venue
Sensors
Publisher
MDPI AG
Published
2019-04-10
Volume
19
Issue
7
Pages
1716
ISSN
1424-8220 (ISSN-L)
OpenAlex
W2940096514
Cited by
215
References
48
Citations by year
2026: 22 · 2025: 33 · 2024: 35 · 2023: 36 · 2022: 31 · 2021: 32 · 2020: 21 · 2019: 5
FWCI
9.94
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Electronics and Telecommunications Research Institute
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords
human activity recognition, inertial measurement unit, multimodal sensor fusion, ensemble learning, LSTM networks, activities of daily living, sampling frequency, sensor placement, eating
MeSH terms
Wearable Electronic Devices, Deep Learning, Standing Position, Activities of Daily Living, Algorithms, Automobile Driving, Humans, Human Activities, Monitoring, Physiologic, Biosensing Techniques, Walking, Multimodal Imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.