Extraction and Interpretation of Deep Autoencoder-based Temporal Features from Wearables for Forecasting Personalized Mood, Health, and Stress
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Continuous wearable sensor data in high resolution contain physiological and behavioral information that can be utilized to predict human health and wellbeing, establishing the foundation for developing early warning systems to eventually improve human health and wellbeing. We propose a deep neural network framework, the Locally Connected Long Short-Term Memory Denoising AutoEncoder (LC-LSTM-DAE), to automatically extract features from passively collected raw sensor data and perform personalized prediction of self-reported mood, health, and stress scores with high precision. We enabled personalized learning of features by finetuning the general representation model with participant-specific data. The framework was evaluated using wearable sensor data and wellbeing labels collected from college students (total 6391 days from N=239). Sensor data include skin temperature, skin conductance, and acceleration; wellbeing labels include self-reported mood, health and stress scored 0 - 100. Compared to the prediction performance based on hand-crafted features, the proposed framework achieved higher precision with a smaller number of features. We also provide statistical interpretation and visual explanation to the automatically learned features and the prediction models. Our results show the possibility of predicting self-reported mood, health, and stress accurately using an interpretable deep learning framework, ultimately for developing real-time health and wellbeing monitoring and intervention systems that can benefit various populations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392
scikit-image: image processing in Python
van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J. D., Yager, N., Gouillart, E., & Yu, T. (2014) · PeerJ · doi:10.7717/peerj.453 · Cited by 6,794
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557
Nonlinear principal component analysis using autoassociative neural networks
Kramer, M. A. (1991) · AIChE Journal · doi:10.1002/aic.690370209 · Cited by 3,037
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“Similarly, in essence, [66] integrated CNN and long short-term memory (LSTM) networks for human activity recognition based on 500 ms data segmentation from accelerometers and gyroscopes, and [44] also applied CNN-LSTM to recognize emotional levels using 45-minute recordings from smart-phones and…”
MethodBoucsein, W. (2012) · doi:10.1007/978-1-4614-1126-0 · Cited by 2,526
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207
Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction
Sakurada, M., & Yairi, T. (2014) · Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis · doi:10.1145/2689746.2689747 · Cited by 1,438
Mobile edge computing, Fog et al.: A survey and analysis of security threats and challenges
Roman, R., Lopez, J., & Mambo, M. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2016.11.009 · Cited by 1,338 · cited 1×
“Edge computing keeps sensitive data local, thus it can minimize the risk of being exposed to cyber attacks and misuse [74].”
BackgroundWang, R., Chen, F., Chen, Z., Li, T., Harari, G., Tignor, S., Zhou, X., Ben-Zeev, D., & Campbell, A. T. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2632048.2632054 · Cited by 1,116
A Hybrid Approach to Privacy-Preserving Federated Learning
Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., & Zhou, Y. (2019) · Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3338501.3357370 · Cited by 1,029
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802 · cited 1×
“When it comes to mental health with ubiquitous sensors, privacy and security are even more sensitive due to disagreements among researchers and lack of guidelines [61].”
BackgroundSaeb, S., Zhang, M., Karr, C. J., Schueller, S. M., Corden, M. E., Kording, K. P., & Mohr, D. C. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4273 · Cited by 782 · cited 1×
“Specifically, mood, health, and stress are three commonly studied wellbeing labels, and many studies demonstrated successful approaches to either detect or predict these labels using unobtrusively collected sensor data [39, 41, 50, 52, 57, 59, 60, 63, 76, 80, 85, 87, 88, 91, 97–99, 106, 108, 109].”
BackgroundEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752 · cited 1×
“Depending on the architecture and dynamics, the basic concept of ANN can embody in many variations (e.g. multi-layer perceptron (MLP), CNN, RNN) for a wide range of real-world solutions including activity, emotion, and other wellbeing-related predictions [34, 68, 90].”
BackgroundKrauchi, K., & Wirz-Justice, A. (1994) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.1994.267.3.r819 · Cited by 558 · cited 1×
“ST can be reflective of many health problems (e.g. fever and heat exhaustion) [13, 73] and biological process (e.g. circadian rhythm and sleep) [48].”
BackgroundStress Recognition Using Wearable Sensors and Mobile Phones
Sano, A., & Picard, R. W. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.117 · Cited by 523
Self-citationSmart health monitoring systems: An overview of design and modeling
Baig, M. M., & Gholamhosseini, H. (2013) · Journal of Medical Systems · doi:10.1007/s10916-012-9898-z · Cited by 396 · cited 1×
“The industrial and commercial development of wearable sensors has enabled non-intrusive, ubiquitous, inexpensive, and continuous high-resolution data collection [7].”
BackgroundTracking Depression Dynamics in College Students Using Mobile Phone and Wearable Sensing
Wang, R., Wang, W., daSilva, A., Huckins, J. F., Kelley, W. M., Heatherton, T. F., & Campbell, A. T. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3191775 · Cited by 340
Sano, A., Taylor, S., McHill, A. W., Phillips, A. J., Barger, L. K., Klerman, E., & Picard, R. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9410 · Cited by 333 · cited 3×
“Wearable and mobile technologies have assisted researchers to unobtrusively collect and monitor multiple body signals that can reveal one’s internal state [82].”
BackgroundMethodSelf-citationKanjo, E., Younis, E. M. G., & Ang, C. S. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.09.001 · Cited by 293 · cited 1×
“…in essence, [66] integrated CNN and long short-term memory (LSTM) networks for human activity recognition based on 500 ms data segmentation from accelerometers and gyroscopes, and [44] also applied CNN-LSTM to recognize emotional levels using 45-minute recordings from smart-phones and wearables.”
MethodRobust Mobile Crowd Sensing: When Deep Learning Meets Edge Computing
Zhou, Z., Liao, H., Gu, B., Huq, K. M. S., Mumtaz, S., & Rodriguez, J. (2018) · IEEE Network · doi:10.1109/mnet.2018.1700442 · Cited by 260 · cited 1×
“Especially with the rapid growth of interdisciplinary collaboration to leverage deep learning in affective computing, human subject data are usually shared on cloud among multiple parties, bringing up challenges for regulating sensitive data acquisition, management, and usage [115].”
BackgroundWang, R., Aung, M. S. H., Abdullah, S., Brian, R., Campbell, A. T., Choudhury, T., Hauser, M., Kane, J., Merrill, M., Scherer, E. A., et al. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971740 · Cited by 257 · cited 2×
“Specifically, mood, health, and stress are three commonly studied wellbeing labels, and many studies demonstrated successful approaches to either detect or predict these labels using unobtrusively collected sensor data [39, 41, 50, 52, 57, 59, 60, 63, 76, 80, 85, 87, 88, 91, 97–99, 106, 108, 109].”
BackgroundPersonalized Multitask Learning for Predicting Tomorrow's Mood, Stress, and Health
Taylor, S., Jaques, N., Nosakhare, E., Sano, A., & Picard, R. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2784832 · Cited by 251 · cited 4×
“[87] leveraged multi-task learning to forecast emotional wellbeing using features extracted from sensor, phone, and survey data with 78-82% classification accuracy.”
BackgroundMethodInfluentialSelf-citationMerikangas, K. R., Swendsen, J., Hickie, I. B., Cui, L., Shou, H., Merikangas, A. K., Zhang, J., Lamers, F., Crainiceanu, C., Volkow, N. D., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.3546 · Cited by 231 · cited 2×
“AC directly monitors human movement and sleep patterns, thus suitable for measuring energy and physical activities from which mental status could be inferred [59].”
BackgroundIs Alzheimer’s disease a systemic disease?
Morris, J. K., Honea, R. A., Vidoni, E. D., Swerdlow, R. H., & Burns, J. M. (2014) · Biochimica et Biophysica Acta (BBA) - Molecular Basis of Disease · doi:10.1016/j.bbadis.2014.04.012 · Cited by 219
Zeng, M., Gao, H., Yu, T., Mengshoel, O. J., Langseth, H., Lane, I., & Liu, X. (2018) · Proceedings of the 2018 ACM International Symposium on Wearable Computers · doi:10.1145/3267242.3267286 · Cited by 185 · cited 1×
“, saliency maps and attention mechanism) [86, 107], it is still very difficult to associate those deep outcomes with a standardized and human-understandable description, especially for sequential sensor data within the health and wellbeing context.”
BackgroundQuantitative analysis of wrist electrodermal activity during sleep
Sano, A., Picard, R. W., & Stickgold, R. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.09.011 · Cited by 141
Self-citationPeng (2018) · doi:10.1145/3214277 · Cited by 134
On the role of features in human activity recognition
Haresamudram, H., Anderson, D. V., & Plötz, T. (2019) · Proceedings of the 23rd International Symposium on Wearable Computers · doi:10.1145/3341163.3347727 · Cited by 90 · cited 1×
“Many studies have been carried out to explore the feasibility of distributing raw sensory data in AE so that wellbeing-related features can be automatically learned through back propagation in either unsupervised or supervised manner [29, 31].”
BackgroundJoint modeling of heterogeneous sensing data for depression assessment via multi-task learning
Lu, J., Shang, C., Yue, C., Morillo, R., Ware, S., Kamath, J., Bamis, A., Russell, A., Wang, B., & Bi, J. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3191753 · Cited by 85 · cited 1×
“Specifically, mood, health, and stress are three commonly studied wellbeing labels, and many studies demonstrated successful approaches to either detect or predict these labels using unobtrusively collected sensor data [39, 41, 50, 52, 57, 59, 60, 63, 76, 80, 85, 87, 88, 91, 97–99, 106, 108, 109].”
Background
Cited by
Explainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216
Human-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30
From Biomarkers to Behavior: Mapping the Neuroimmune Web of Pain, Mood, and Memory
Tanaka, M., & Battaglia, S. (2025) · Biomedicines · doi:10.3390/biomedicines13092226 · Cited by 13
Metadata
- Authors
- Li, B., & Sano, A.
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2020-06-15
- Volume
- 4
- Issue
- 2
- Pages
- 1-26
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W3085547322
- Cited by
- 95
- References
- 30
- FWCI
- 12.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.