Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Background: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera–based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monitoring results, a person must remain still for several minutes while a measurement is being taken. This is cumbersome and makes its use in real-life situations impractical. Objective: We proposed a system that combines PPG and thermography with the aim of improving cardiovascular signal quality and detecting stress responses quickly. Methods: Using a smartphone camera with a low-cost thermal camera added on, we built a novel system that continuously and reliably measures 2 different types of cardiovascular events: (1) blood volume pulse and (2) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 participants, involved in stress-inducing mental workload tasks, measured their physiological responses to stressors over a short time period (20 seconds) immediately after each task. Participants reported their perceived stress levels on a 10-cm visual analog scale. For the instant stress inference task, we built novel low-level feature sets representing cardiovascular variability. We then used the automatic feature learning capability of artificial neural networks to improve the mapping between the extracted features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. Results: First, we found that the measured PPG signals presented high quality cardiac cyclic information (mean pSQI: 0.755; SD 0.068). We also found that the measured thermal changes of the nose tip presented high-quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (mean pSQI: from 0.714 to 0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the cardiovascular signals (ie, heart rate variability and thermal directionality) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing precrafted features-based methods. In addition, the 17-fold leave-one-subject-out cross-validation results showed that combining both modalities produced higher accuracy than using PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art stress recognition methods that require long-term measurements. Finally, we explored effects of different data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. Conclusions: The results demonstrate the feasibility of using smartphone-based imaging for instant stress detection. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental health care solutions in the wild.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An overview of heart rate variability metrics and norms
Shaffer, F., & Ginsberg, J. P. (2017) · Frontiers in Public Health · doi:10.3389/fpubh.2017.00258 · Cited by 7,185
Kirschbaum, C., Pirke, K.-M., & Hellhammer, D. H. (2008) · Neuropsychobiology · doi:10.1159/000119004 · Cited by 6,093
Pagani, M., Lombardi, F., Guzzetti, S., Rimoldi, O., Furlan, R., Pizzinelli, P., Sandrone, G., Malfatto, G., Dell'Orto, S., & Piccaluga, E. (1986) · Circulation Research · doi:10.1161/01.res.59.2.178 · Cited by 4,130 · cited 4×
“Based on this observation, many studies have proposed to use the LF/HF ratio as a stress indicator [4,22,24,32].”
BackgroundMethodInfluentialA survey of mobile phone sensing
Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., & Campbell, A. (2010) · IEEE Communications Magazine · doi:10.1109/mcom.2010.5560598 · Cited by 2,292
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207
Shaffer, F., McCraty, R., & Zerr, C. L. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01040 · Cited by 1,866
The LF/HF ratio does not accurately measure cardiac sympatho-vagal balance
Billman, G. E. (2013) · Frontiers in Physiology · doi:10.3389/fphys.2013.00026 · Cited by 1,401
Sympathovagal Balance: A Critical Appraisal
Eckberg, D. L. (1997) · Circulation · doi:10.1161/01.cir.96.9.3224 · Cited by 1,087 · cited 2×
“literature in psychology or affective computing), there have still been arguments of such metrics in regards to the possibility of oversimplifying physiological responses [33–35].”
BackgroundHeart Rate Variability ? A Historical Perspective
Billman, G. E. (2011) · Frontiers in Physiology · doi:10.3389/fphys.2011.00086 · Cited by 786
Discriminating Stress From Cognitive Load Using a Wearable EDA Device
Setz, C., Arnrich, B., Schumm, J., La Marca, R., Troster, G., & Ehlert, U. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036164 · Cited by 742 · cited 1×
“h it has been shown that the Stroop and Math tasks lead to cognitive overload [2,59], they are limited in the amount of stress they induce due to the lack of psychosocial stressors or other stressors [2,62]. Hence, following [2,40,59,62], we also introduce further stressors: a) social evaluative threats (close observa…”
BackgroundInfluential
Cited by
Photoplethysmography signal processing and synthesis
Mejía-Mejía, E., Allen, J., Budidha, K., El-Hajj, C., Kyriacou, P. A., & Charlton, P. H. (2022) · Photoplethysmography · doi:10.1016/b978-0-12-823374-0.00015-3 · Cited by 138
Mentis, A.-F. A., Lee, D., & Roussos, P. (2024) · Molecular Psychiatry · doi:10.1038/s41380-023-02047-6 · Cited by 75
Magalhaes, C., Mendes, J., & Vardasca, R. (2021) · Applied Sciences · doi:10.3390/app11020842 · Cited by 49
Metadata
- Authors
- Cho, Y., Julier, S. J., & Bianchi-Berthouze, N.
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2019-04-09
- Volume
- 6
- Issue
- 4
- Pages
- e10140
- ISSN
- 2368-7959 (ISSN-L)
- OpenAlex
- W3105211913
- Cited by
- 102
- References
- 22
- FWCI
- 2.45
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.