CoChat

Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition

Open access

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 8 of the 23 sources its claims cite were checked for support.

Abstract

Introduces DeepConvLSTM, a deep learning architecture combining convolutional and LSTM recurrent units for multimodal wearable activity recognition, outperforming non-recurrent deep networks and handcrafted feature baselines by 4% to 9%.

Overview

CoChat Summary

TL;DR

The paper proposes DeepConvLSTM, a deep framework combining four convolutional layers with two LSTM recurrent layers for wearable activity recognition from raw multimodal sensor data. It reports improved F1 scores over a baseline CNN and prior results on the OPPORTUNITY and Skoda datasets, including 0.915 F1 for OPPORTUNITY gestures with the Null class and 0.958 F1 on Skoda.

Key findings

  • On OPPORTUNITY, DeepConvLSTM achieved F1 scores of 0.930 for modes of locomotion without Null, 0.895 for modes with Null, 0.866 for gestures without Null, and 0.915 for gestures with Null; the baseline CNN scored 0.912, 0.878, 0.783, and 0.883, respectively. The paper reports a 6% average improvement over the best OPPORTUNITY challenge submissions and more than 9% improvement on gesture recognition without Null.
  • On Skoda, DeepConvLSTM reached 0.958 F1, compared with 0.884 for the baseline CNN, 0.893 for CNN [43], and 0.861 for CNN [23], improving the best reported result by 6%.
  • Multimodal fusion improved gesture recognition on OPPORTUNITY without Null: 0.689 F1 with accelerometers only, 0.611 with gyroscopes only, 0.745 with accelerometers + gyroscopes, 0.839 with accelerometers + gyroscopes + magnetic sensors, and 0.864 with the full OPPORTUNITY sensor set.
  • DeepConvLSTM had 999,122 parameters versus 7,445,458 for the baseline CNN, a 600% increase for the baseline; training took 340.3 min versus 282.2 min, and classification took 6.68 s versus 5.43 s, with DeepConvLSTM classifying almost 15 min of data per second and achieving 900× real-time on the GPU used.

Methods

  • DeepConvLSTM uses four convolutional layers with 64 feature maps each, followed by two dense LSTM layers with 128 cells and a softmax output; the baseline CNN shares the convolutional layers but uses two non-recurrent dense layers with 128 units. Models were trained supervised with cross-entropy loss, mini-batch gradient descent with RMSProp, learning rate 10e-3, decay factor 0.9, mini-batches of 100 segments, and dropout p=0.5 on dense-layer inputs.
  • Inputs were sliding windows of 500 ms with 250 ms step size, using raw sensor data with minimal preprocessing: linear interpolation for missing values and per-channel normalization to [0,1]. Evaluation used weighted F1 to address class imbalance.
  • Experiments used the OPPORTUNITY dataset (on-body sensors, 113 channels, 30 Hz; Task A: 5-class modes of locomotion/postures; Task B: 18-class sporadic gestures) and the Skoda dataset (one subject, 10 right-arm 3D accelerometers, decimated to 30 Hz, 10 manipulative gestures, about 3 h and about 70 repetitions per gesture). Analyses included comparison to baseline CNN and published results, confusion matrices, sensor-modality fusion, and hyperparameter evaluations of sequence length and number of convolutional layers.

Limitations

  • The paper states that DeepConvLSTM has not yet been implemented on a wearable device; the GPU used clearly outperforms current high-end wearable computational power, and training is best envisioned server-side. It also states that pooling was omitted because the sliding-window setup limits downsampling, though pooling could be useful without that requirement, and that training time increases with the number of layers, creating a performance–computational budget trade-off.
  • Evident from the text: Skoda evaluation uses one subject and only right-arm sensors, and OPPORTUNITY testing is limited to ADL4 and ADL5 for Subjects 2 and 3, which constrains generalizability. The text does not report cross-validation or statistical significance testing.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s16010115

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Different types of units in neural networks. (a) MLP with three dense layers; (b) recurrent neural network (RNN) with two dense layers. The activation and hidden value of the unit in layer (l+1) are computed in the same time step t; (c) The recurrent LSTM cell is an extension of RNNs, where the internal memory can be updated, erased or read out.
Figure 1 Different types of units in neural networks. (a) MLP with three dense layers; (b) recurrent neural network (RNN) with two dense layers. The activation and hidden value of the unit in layer (l+1) are computed in the same time step t; (c) The recurrent LSTM cell is an extension of RNNs, where the internal memory can be updated, erased or read out.CC BY · PubMed Central
Figure 2 Representation of a temporal convolution over a single sensor channel in a three-layer convolutional neural network (CNN). Layer (l-1) defines the sensor data at the input. The next layer (l) is composed of two feature maps (a1^l(τ) and a2^l(τ)) extracted by two different kernels (K11^(l-1) and K21^(l-1)). The deepest layer (layer (l+1)) is composed by a single feature map, resulting from temporal convolution in layer l of a two-dimensional kernel K1^l. The time axis (which is convolved over) is horizontal.
Figure 2 Representation of a temporal convolution over a single sensor channel in a three-layer convolutional neural network (CNN). Layer (l-1) defines the sensor data at the input. The next layer (l) is composed of two feature maps (a1^l(τ) and a2^l(τ)) extracted by two different kernels (K11^(l-1) and K21^(l-1)). The deepest layer (layer (l+1)) is composed by a single feature map, resulting from temporal convolution in layer l of a two-dimensional kernel K1^l. The time axis (which is convolved over) is horizontal.CC BY · PubMed Central
Figure 3 Architecture of the DeepConvLSTM (Conv, convolutional) framework for activity recognition. From the left, the signals coming from the wearable sensors are processed by four convolutional layers, which allow learning features from the data. Two dense layers then perform a non-linear transformation, which yields the classification outcome with a softmax logistic regression output layer on the right. Input at Layer 1 corresponds to sensor data of size D×S¹, where D denotes the number of sensor channels and S^l the length of features maps in layer l. Layers 2–5 are convolutional layers. K^l denotes the kernels in layer l (depicted as red squares). F^l denotes the number of feature maps in layer l. In convolutional layers, ai^l denotes the activation that defines the feature map i in layer l. Layers 6 and 7 are dense layers. In dense layers, at,i^l denotes the activation of the unit i in hidden layer l at time t. The time axis is vertical.
Figure 3 Architecture of the DeepConvLSTM (Conv, convolutional) framework for activity recognition. From the left, the signals coming from the wearable sensors are processed by four convolutional layers, which allow learning features from the data. Two dense layers then perform a non-linear transformation, which yields the classification outcome with a softmax logistic regression output layer on the right. Input at Layer 1 corresponds to sensor data of size D×S¹, where D denotes the number of sensor channels and S^l the length of features maps in layer l. Layers 2–5 are convolutional layers. K^l denotes the kernels in layer l (depicted as red squares). F^l denotes the number of feature maps in layer l. In convolutional layers, ai^l denotes the activation that defines the feature map i in layer l. Layers 6 and 7 are dense layers. In dense layers, at,i^l denotes the activation of the unit i in hidden layer l at time t. The time axis is vertical.CC BY · PubMed Central
Figure 4 Placement of on-body sensors used in the OPPORTUNITYdataset (left: inertial measurements units; right: 3-axis accelerometers) [7].
Figure 4 Placement of on-body sensors used in the OPPORTUNITYdataset (left: inertial measurements units; right: 3-axis accelerometers) [7].CC BY · PubMed Central
Figure 5 Sequence labelling after segmenting the data with a sliding window. The sensor signals are segmented by a jumping window. The activity class within each sequence is considered to be the ground truth label annotated at the sample T of that window.
Figure 5 Sequence labelling after segmenting the data with a sliding window. The sensor signals are segmented by a jumping window. The activity class within each sequence is considered to be the ground truth label annotated at the sample T of that window.CC BY · PubMed Central
Figure 6 Output class probabilities for a ~25 s-long fragment of sensor signals in the test set of the OPPORTUNITY dataset, which comprises 10 annotated gestures. Each point in the plot represents the class probabilities obtained from processing the data within a sequence of 500 ms obtained from a sliding window ending at that point. The dashed line represents the Null class. DeepConvLSTM offers a better performance identifying the start and ending of gestures.
Figure 6 Output class probabilities for a ~25 s-long fragment of sensor signals in the test set of the OPPORTUNITY dataset, which comprises 10 annotated gestures. Each point in the plot represents the class probabilities obtained from processing the data within a sequence of 500 ms obtained from a sliding window ending at that point. The dashed line represents the Null class. DeepConvLSTM offers a better performance identifying the start and ending of gestures.CC BY · PubMed Central
Figure 7 F1 score performance of DeepConvLSTM on the OPPORTUNITY dataset. Classification performance is displayed individually per gesture, for different lengths of the input sensor data segments. Experiments carried out with sequences of length of 400 ms, 500 ms, 1400 ms and 2750 ms. The horizontal axis represents the ratio between the gesture length and the sequence length (ratios under one represent performance for gestures whose durations are shorter than the sequence duration).
Figure 7 F1 score performance of DeepConvLSTM on the OPPORTUNITY dataset. Classification performance is displayed individually per gesture, for different lengths of the input sensor data segments. Experiments carried out with sequences of length of 400 ms, 500 ms, 1400 ms and 2750 ms. The horizontal axis represents the ratio between the gesture length and the sequence length (ratios under one represent performance for gestures whose durations are shorter than the sequence duration).CC BY · PubMed Central
Figure 8 Performance of Skoda and OPPORTUNITY (recognizing gestures and with the Null class) datasets with different numbers of convolutional layers.
Figure 8 Performance of Skoda and OPPORTUNITY (recognizing gestures and with the Null class) datasets with different numbers of convolutional layers.CC BY · PubMed Central

Verification

Claims the paper makes

5 Valid · 5 Not valid · 13 Not checked · 99 Own results and interpretations

84 more claims

Create a free account to see all 122 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 23

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

20 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 49 references

Research works printed but not linked: 1

  1. LeCun, Y., and Bengio, Y. (1998). The Handbook of Brain Theory and Neural Networks, MIT Press.

Cited by

See all 2,806

Metadata

Authors
Ordóñez, F., & Roggen, D.
Affiliations
University of Sussex (GB)
Corresponding author
Ordóñez, Francisco
Venue
Sensors
Publisher
MDPI AG
Published
2016-01-18
Volume
16
Issue
1
Pages
115
ISSN
1424-8220 (ISSN-L)
OpenAlex
W2270470215
Cited by
2,761
References
49
Citations by year
2026: 284 · 2025: 256 · 2024: 284 · 2023: 270 · 2022: 319 · 2021: 379 · 2020: 347 · 2019: 288 · 2018: 219 · 2017: 93 · 2016: 21
FWCI
140.52
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Sussex Research Online (University of Sussex) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
  • Google Faculty Research Award (Is deep learning useful for wearable activity recognition?)
  • EPSRC (EP/N007816/1)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
Keywords
human activity recognition, wearable sensors, multimodal sensor fusion, convolutional neural networks, LSTM neural network, temporal dynamics, automatic feature extraction, hyperparameter optimization
MeSH terms
Machine Learning, Clothing, Humans, Human Activities, Signal Processing, Computer-Assisted, Databases, Factual, Neural Networks, Computer, Monitoring, Ambulatory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. https://app.cochat.ai/papers/10.3390/s16010115
The paper
Ordóñez, F., & Roggen, D. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors. https://doi.org/10.3390/s16010115

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.