CoChat
Sign Up
Cited by 293Emotion and Mood RecognitionContext-Aware Activity Recognition SystemsSentiment Analysis and Opinion Mining

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.

Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection

Open access
Kanjo, Eiman, Younis, Eman M.G., Ang, Chee Siang·2019·Information Fusion
DOI 10.1016/j.inffus.2018.09.001
Open this paper in CoChat — free accountOpen PDF

Abstract

The detection and monitoring of emotions are important in various applications, e.g., to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g., EEG and GSR), environmental data (e.g., audio and weather), videos (e.g., for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper evaluates deep learning on multimodal smartphone/wearable sensor data for emotion detection “in the wild” using the EnvBodySens dataset. A hybrid CNN-LSTM model on fused raw sensor channels achieved the best average accuracy and F-measure (94.7% and 0.949) compared with MLP and CNN baselines.

Key findings

  • CNN-LSTM on fused modalities achieved average accuracy 94.7% and F-measure 0.949 (Table 1; abstract rounds to 95% and 95%), outperforming MLP (72.9%, F-measure 0.729) and CNN (78.6%, F-measure 0.787).
  • Among single modalities, on-body data was strongest for CNN-LSTM (accuracy 87.3%, F-measure 0.874), while environment (59.7%) and location (64.0%) were weaker alone; fusing the other modalities with on-body data improved accuracy by approximately 7%.
  • CNN-LSTM outperformed the authors’ earlier ensemble feature-engineering approach by a 6% margin (ensemble average accuracy 83%, F-measure 82%) and exceeded MLP by more than 20%.
  • A sliding window size of 40 samples (160 ms) performed best, with F-measure 0.949, accuracy 94.7, and RMSE 0.291.

Methods

  • Dataset: EnvBodySens, collected from 40 female participants (average age 28) walking in Nottingham city centre for up to 45 minutes, yielding 550,432 sensor data frames and 5,345 self-report responses on a 5-step SAM valence scale; six users were excluded due to logging problems. Sensors included heart rate, GSR, body temperature, accelerometer, gyro, noise, UV, air pressure, and GPS, forming 20 channels across on-body, environmental, and location modalities.
  • Models: MLP with two hidden layers (64 and 32 neurons, input 20×40, 5 outputs), CNN with two convolutional layers (32 and 64 filters), two max-pooling layers, and a fully connected layer, and CNN-LSTM adding an LSTM layer. Models were trained per participant on each modality separately and on fused data, totaling 12 models per user dataset.
  • Analysis: Data were preprocessed by removing the first and last 30 seconds and segmenting with a sliding window; each subject’s data were split 70% training / 30% testing. Performance was compared across architectures and modality combinations using precision, recall, F-measure, accuracy, and RMSE, implemented in TensorFlow.

Limitations

  • Stated in the paper: many network parameters remain tunable and were not fully explored (e.g., overlapping windows and overlap width); the approach needs testing on diverse datasets with larger numbers of participants and additional modalities such as pollution levels and crowd density. Combining all participants into one dataset produced high across-subject variation and model accuracy below 50%, indicating that emotion recognition is subject-dependent and difficult to generalize across individuals.
  • Evident from the text: the sample is limited to 40 female participants, average age 28, walking fixed routes in Nottingham under similar weather conditions (average 20°C) around 11am, which restricts demographic, geographic, and contextual generalizability. Six users’ data were also excluded because of logging problems.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.inffus.2018.09.001

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 26 cited sources checked · 0 supported · 0 flagged

Show all 40 claims

  • It may state two separate claims as one. 16
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 18
29 of 29

  • The paper relies on this without support.

    No other work has applied deep learning on the combination of physiological, environmental and location data for emotion recognition.

    “Specifically, to the best of our knowledge, no other work has applied deep learning on the combination of physiological, environmental and location data for emotion recognition.”

  • The paper relies on this without support.

    The majority of studies employing deep learning on activity recognition are restricted to a handful of data channels.

    “The majority of the studies employing deep learning on activity recognition are restricted to a handful of data channels as opposed to this study, where we utilise 20 sensor channels from three different modalities to classify emotion against self-reported emotion labels.”

  • The paper relies on this without support.

    Hand-crafted features are domain-specific and often poorly generalise to other similar problem domains.

    “Although hand-crafted features have yielded promising results, they are domainspecific, and often poorly generalise to other similar problem domains.”

  • The paper relies on this without support.

    Handcrafted-based approaches involve laborious human intervention for selecting the most discriminating features and decision thresholds from sensor data.

    “Handcrafted-based approaches involve laborious human intervention for selecting the most discriminating features and decision thresholds from sensor data.”

  • The paper relies on this without support.

    Traditional machine learning and feature engineering algorithms may not be efficient enough to extract the complex and non-linear patterns generally observed in multimodal time series datasets.

    “Moreover, traditional machine learning and feature engineering algorithms may not be efficient enough to extract the complex and non-linear patterns generally observed in multimodal time series datasets.”

  • The paper relies on this without support.

    Feature extraction and feature selection are computationally expensive.

    “Furthermore, feature extraction and feature selection are computationally expensive.”

  • The paper relies on this without support.

    Traditional shallow methods, which contain only a small number of non-linear operations, do not have the capacity to accurately model variation of time series data.

    “Traditional shallow methods, which contain only a small number of non-linear operations, do not have the capacity to accurately model such variation of time series data.”

  • The paper relies on this without support.

    CNN can capture local dependencies and invariant features in accelerometer data.

    “It was shown that CNN can capture local dependencies and invariant features in the data.”

  • The paper relies on this without support.

    In some affective states examined (e.g., relaxation models built on EDA; fun and excitement models built on BVP; relaxation models built on fused EDA and BVP), deep learning without prior feature selection manages to reach or even outperform the performances of models built on ad-hoc extracted features boosted by automatic feature selection.

    “Furthermore, in some affective states examined (e.g., relaxation models built on Electrodermal Activities (EDA); fun and excitement models built on Blood Volume Pulse (BVP); relaxation models built on fused EDA and BVP), deep learning without prior feature selection manages to reach or even outperform the performances of models built on ad-hoc extracted features which are boosted by automatic feat”

  • The paper relies on this without support.

    The EnvBodySens dataset includes key components for emotion modelling and is representative of a typical affective modelling scenario.

    “While the Env-BodySens dataset includes key components for emotion modelling and is representative of a typical affective modelling scenario”

  • The paper relies on this without support.

    This is one of the few studies looking into emotional recognition of participants in their natural environment using multiple sources of time series data.

    “This is one of the few studies looking into emotional recognition of participants in their natural environment using multiple sources of time series data.”

  • The paper relies on this without support.

    Traditional emotion recognition techniques require feature engineering process to be applied to data prior to modelling, which might be challenging especially if the dataset is multimodal and large.

    “Although this type of time series data can help us understand peoples emotion, traditional emotion recognition techniques requires features engineering process to be applied to data prior to modelling, which might be challenging especially if the dataset is multimodal and large.”

  • The paper relies on this without support.

    Deep learning offers an automated way for features extraction embedded in the process.

    “Deep learning offers an automated way for features extraction embedded in the process.”

  • The paper relies on this without support.

    Others have successfully created a universal deep learning model for gesture data as gestures performed by different individuals are typically quite similar.

    “Others have successfully created a universal deep learning model for gesture data as gestures performed by different individuals are typically quite similar.”
14 of 14

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Convolution padding descriptions conflict

    Major

    The methods description says the proposed convolution uses no zero padding, but the CNN implementation specifies padding that prevents the matrix from shrinking.

  • Fused MLP F-measure disagrees with Table 1

    Minor

    The text reports an F-measure of 0.58 for the MLP trained on fused modalities, whereas Table 1 reports 0.729 for the MLP "All" model.

2 of 2

Citations

Cites

Flagged first, then by citations

Show all 51 references

Research works printed but not linked: 1

  1. M. Gadaleta, M. Rossi, IDnet: smartphone-based gait recognition with convolutional neural networks, arXiv:1606.03238 (2016). · arXiv

Cited by

Sorted by citations

Items 201–293 of 293

Related

See all

Metadata

Authors
Kanjo, E., Younis, E. M. G., & Ang, C. S.
Affiliations
  • Nottingham Trent University (GB)
  • Minia University (EG)
  • University of Kent (GB)
Corresponding author
Kanjo, Eiman
ORCID
Venue
Information Fusion
Publisher
Elsevier BV
Published
2019-09
Volume
49
Pages
46-56
ISSN
1566-2535 (ISSN-L)
DOI
10.1016/j.inffus.2018.09.001
OpenAlex
W2898242330
Cited by
293
References
51
Citations by year
2026: 18 · 2025: 23 · 2024: 43 · 2023: 43 · 2022: 52 · 2021: 42 · 2020: 54 · 2019: 16
FWCI
37.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Information Fusion (publishedVersion, cc-by-nc-nd)
  • Nottingham Trent University's Institutional Repository (Nottingham Trent Repository) (acceptedVersion)
  • Kent Academic Repository (University of Kent) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
emotion recognition, multimodal sensor data, mobile sensing, wearable devices, on-body sensors, location data, environmental sensors, CNN-LSTM, emotion classification, feature extraction, human-robot interaction, smartphone sensing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review