Deep learning-based classification of posttraumatic stress disorder and depression following trauma utilizing visual and auditory markers of arousal and mood
The DOI matches its Crossref record, with a small difference in title or year; 115 of the 117 references it cites were found in a registry or an index.
Abstract
Background Visual and auditory signs of patient functioning have long been used for clinical diagnosis, treatment selection, and prognosis. Direct measurement and quantification of these signals can aim to improve the consistency, sensitivity, and scalability of clinical assessment. Currently, we investigate if machine learning-based computer vision (CV), semantic, and acoustic analysis can capture clinical features from free speech responses to a brief interview 1 month post-trauma that accurately classify major depressive disorder (MDD) and posttraumatic stress disorder (PTSD). Methods N = 81 patients admitted to an emergency department (ED) of a Level-1 Trauma Unit following a life-threatening traumatic event participated in an open-ended qualitative interview with a para-professional about their experience 1 month following admission. A deep neural network was utilized to extract facial features of emotion and their intensity, movement parameters, speech prosody, and natural language content. These features were utilized as inputs to classify PTSD and MDD cross-sectionally. Results Both video- and audio-based markers contributed to good discriminatory classification accuracy. The algorithm discriminates PTSD status at 1 month after ED admission with an AUC of 0.90 (weighted average precision = 0.83, recall = 0.84, and f1-score = 0.83) as well as depression status at 1 month after ED admission with an AUC of 0.86 (weighted average precision = 0.83, recall = 0.82, and f1-sc…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“From the forest plot highlights studies with notably higher AUC scores, particularly Reece et al, 52 Kirinde Gamaarachchige and Inkpen, 84 and Schultebraucks et al. 43”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [43]
In the citing paper
“The latter study incorporates a broader set of linguistic and acoustic features, including pitch frame counts, potentially enhancing predictive accuracy.”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [43]
In the citing paper
“SHAP was implemented in 4 studies spanning dementia detection [9], auditory verbal hallucination assessment [52], psychiatric disorder classification [51], and TBI [53].”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [51]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“It is an additive feature attribution method that uses kernel functions and currently the gold standard to interpret deep neural networks (Lundberg & Lee, 2017).”
MethodBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“Data were preprocessed using R package caret (Breiman, 1996; Kuhn, 2008; Kuhn & Johnson, 2013).”
MethodGauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
(2016) · Cited by 11,679
The expression of the emotions in man and animals.
Darwin, C. (1872) · J. Murray eBooks · doi:10.1037/10001-000 · Cited by 11,657
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“We used k -fold cross-validation with 10 folds in the training set to decrease the risk of overfitting (Stone, 1974).”
MethodBuilding Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863 · cited 1×
“Data were preprocessed using R package caret (Breiman, 1996; Kuhn, 2008; Kuhn & Johnson, 2013).”
MethodInsel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067 · cited 1×
“Our results also suggest implications for the fields ’ ability to put a clinical focus on discrete behavioral and physiological dimensions as metrics of risk and treatment response consistent with the research domain criteria approach (Cuthbert & Insel, 2013; Insel, 2014; Insel et al., 2010).”
Result
Research works printed but not linked: 6
- 10.1037/0021-843x.109.1.26 · doi:10.1037/0021-843x.109.1.26 · Paper page
- 2013, Diagnostic and statistical manual of mental disorders (DSM-5®)
- Eaton, 2004, The use of psychological testing for treatment planning and outcomes assessment: Instruments for adults, 363
- Pennebaker, 2015, The development and psychometric properties of LIWC2015
- Ekman, 1978, Facial action coding system: Investigator's guide
- Carmi, 2020, Management of terrorism induced stress – guideline for the golden hours, 175
Cited by
Items 101–119 of 119
Predicting post-traumatic stress disorder in relatives of critically ill patients
Dupont, T., Duchesnay, É., Pochard, F., Kentish‐Barnes, N., & Azoulay, É. (2025) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000001309
A Scoping Review on the Use of Voice Biomarkers for Emotional Assessment
Sartori, R., Marinaro, F., Tommasi, F., Buccoliero, A., Zene, M., Shah, S. A. H., & Ceschi, A. (2025) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000915
Can machine learning predict PTSD symptoms from trauma narratives of children and adolescents?
Giuliani, A., Sharp, T. H., Chideya, Y., Meiser‐Stedman, R., Tomlinson, M., & Halligan, S. L. (2025) · European Journal of Psychotraumatology · doi:10.1080/20008066.2025.2589709
Digital biomarkers of emotion processes associated with Posttraumatic Stress Disorder
Meaney, T. (2025) · UNSWorks (University of New South Wales, Sydney, Australia) · doi:10.26190/unsworks/31101
Towards Responsible Multimodal Modeling for Mental Healthcare
Kaya, H., & Soğancıoğlu, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07956-5_1 · Cited by 1
Демарева, В., Nazarov, N., Bushueva, K., & Demarev, A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-14674-8_35
Verma, M., & Beg, M. J. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1325-2.ch003
Neuroscientific Approach to Artificial Intelligence for Psychopathologies
Celik, K., & Çelik, S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1325-2.ch006
Nouveaux biomarqueurs vocaux pour la détection automatique de la somnolence
Martin, V. (2022) · theses.fr (ABES) · doi:10.70675/e8669cefz980fz4fc6zbc44z07ad3ca56810
Examining the utility of digital phenotyping for the prediction of intrusive experiences
Meaney, T., Yadav, V., Galatzer‐Levy, I. R., & Bryant, R. A. (2026) · European Journal of Psychotraumatology · doi:10.1080/20008066.2026.2646126
“These intrusion measures were partially derived from the Intrusive Memories Questionnaire (IMQ) (Baker & Goodson, 2023), but other, more specific, measures were added based on their relevance to important aspects of intrusive experiences associated with PTSD, such as vividness, imagery, and emotional intensity (Clark…”
InfluentialSelf-citationGershov, S., Hilberdink, C. E., Zhao, Y., Birnbaum, S. B., Müller, V., Wall, S. P., & Schultebraucks, K. (2026) · Journal of Medical Internet Research · doi:10.2196/87049
“Although speech-based and linguistic features have shown promise in prior work [24,37-39], we intentionally excluded them from our framework because their interpretation depends on language-specific models that may not generalize across cultural contexts, can be biased by differences in language…”
InfluentialSelf-citationIntegrated Framework for Stress Level Classification Using Eye-Tracking, GSR and Heart Rate Signals
Munjal, A., Singh, K. J., & Kapoor, D. S. (2026) · International Conference on Signal, Systems, and Computing for Next-Gen Automation (ICSSCNA) · doi:10.1109/icsscna68616.2026.11546735
“Besides, audiovisual behavioral signatures allow automated identification of both post, traumatic stress disorder and depression [12].”
Wang, W. (2026) · UNSWorks (University of New South Wales, Sydney, Australia) · doi:10.26190/unsworks/32548
Natural language processing of youth speech predicts psychopathology across adolescence
Antonacci, C., Uy, J. P., Kwan, K., Giampetruzzi, E., Jones, S., Pennebaker, J. W., & Gotlib, I. H. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00683-9 · Cited by 1
“Other research with adults has found that linguistic markers can track psychological processes following trauma 25 and predict psychiatric outcomes more broadly 26,27 .”
Multimodal computational analysis of longitudinal stress profiles in healthcare workers
Gershov, S., Cho, D., Vergyri, D., Ferguson, K., Taitz, A., Adams, C., Malgaroli, M., Gallagher, T., Marmar, C. R., & Schultebraucks, K. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03328-4
Self-citationModelling the Flow of Emotion: A Hierarchical Attention Network for Vocal Trajectory Classification
Addagarla, V., & Haritha, D. S. N. (2026) · International Journal of Drug Delivery Technology · doi:10.25258/ijddt.16.86s.124
Mueller, V., Yadav, J., Farnsworth, A., Hilberdink, C. E., Gershov, S., Oermann, E. K., Wall, S. P., Shalev, A. Y., & Schultebraucks, K. (2026) · Nature Communications · doi:10.1038/s41467-026-78768-3
Self-citation
Metadata
- Authors
- Schultebraucks, K., Yadav, V., Shalev, A. Y., Bonanno, G. A., & Galatzer-Levy, I. R.
- Affiliations
- Columbia University Irving Medical Center (US)
- AiCure (United States) (US)
- New York University (US)
- Columbia University (US)
- Corresponding author
- Schultebraucks, Katharina
- Venue
- Psychological Medicine
- Publisher
- Cambridge University Press (CUP)
- Published
- 2020-08-03
- Volume
- 52
- Issue
- 5
- Pages
- 957-967
- ISSN
- 0033-2917 (ISSN-L)
- 1469-8978
- OpenAlex
- W3047024470
- Semantic Scholar
- fccf0932af2f598ff567f7f61fed92aa7a7217d1
- Cited by
- 108
- References
- 117
- Citations by year
- 2026: 10 · 2025: 31 · 2024: 24 · 2023: 20 · 2022: 12 · 2021: 8 · 2020: 3
- FWCI
- 10.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (K01MH102415)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Psychology › Clinical Psychology › Posttraumatic Stress Disorder Research
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- post-traumatic stress disorder, major depressive disorder, trauma (physical injury), facial emotion recognition, prosody, acoustic analysis, computer vision, natural language processing, digital biomarkers, emergency department patients
- MeSH terms
- Deep Learning, Arousal, Depression, Depressive Disorder, Major, Major Depressive Disorder, Humans, Stress Disorders, Post-Traumatic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.