Validation of Visual and Auditory Digital Markers of Suicidality in Acutely Suicidal Psychiatric Inpatients: Proof-of-Concept Study
Open accessPublished version of a preprint
Abstract
Background: Multiple symptoms of suicide risk have been assessed based on visual and auditory information, including flattened affect, reduced movement, and slowed speech. Objective quantification of such symptomatology from novel data sources can increase the sensitivity, scalability, and timeliness of suicide risk assessment. Objective: We aimed to examine measurements extracted from video interviews using open-source deep learning algorithms to quantify facial, vocal, and movement behaviors in relation to suicide risk severity in recently admitted patients following a suicide attempt. Methods: We utilized video to quantify facial, vocal, and movement markers associated with mood, emotion, and motor functioning from a structured clinical conversation in 20 patients admitted to a psychiatric hospital following a suicide risk attempt. Measures were calculated using open-source deep learning algorithms for processing facial expressivity, head movement, and vocal characteristics. Derived digital measures of flattened affect, reduced movement, and slowed speech were compared to suicide risk with the Beck Scale for Suicide Ideation controlling for age and sex, using multiple linear regression. Results: Suicide severity was associated with multiple visual and auditory markers, including speech prevalence (β=−0.68, P=.02, r2=0.40), overall expressivity (β=−0.46, P=.10, r2=0.27), and head movement measured as head pitch variability (β=−1.24, P=.006, r2=0.48) and head yaw variability (β=−0.54, P=.06, r2=0.32). Conclusions: Digital measurements of facial affect, movement, and speech prevalence demonstrated strong effect sizes and linear associations with the severity of suicidal ideation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder.
“Notably, altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder and psychopharmacological treatment has demonstrated restored levels of digital markers (e.g. increased head movement) and a decrease in anger and fear facial expressions [37, 38].”
From Effectiveness of a scalable, remotely delivered stepped-care intervention to reduce symptoms of psychological distress among Polish migrant workers in the Netherlands: study protocol for the RESPOND randomised controlled trial (2023) · cites this paper as [37]
The Results sections report that less speech, facial activity, and head movement were associated with greater suicidal ideation in psychiatric inpatients. The Methods identify participants by admission after a suicide attempt or with suicidal ideation, not by major depressive disorder; the paper therefore does not establish these findings specifically for people with MDD. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPsychopharmacological treatment has demonstrated restored levels of digital markers and a decrease in anger and fear facial expressions.
“Notably, altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder and psychopharmacological treatment has demonstrated restored levels of digital markers (e.g. increased head movement) and a decrease in anger and fear facial expressions [37, 38].”
From Effectiveness of a scalable, remotely delivered stepped-care intervention to reduce symptoms of psychological distress among Polish migrant workers in the Netherlands: study protocol for the RESPOND randomised controlled trial (2023) · cites this paper as [37]
The Methods and Results examine digital markers in relation to suicide severity, not changes after psychopharmacological treatment. The Discussion describes measuring treatment response as a future possibility, while the Facial Activity results report no significant association for fear or anger expressivity. Our copy therefore does not establish treatment-related restoration of markers or decreases in those expressions. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2013) · doi:10.1176/appi.books.9780890425596 · Cited by 116,369 · see citing works
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,975 · see citing works
OpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · see citing works
Nock, M. K., Holmberg, E. B., Photos, V. I., & Michel, B. D. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.309 · Cited by 1,352 · see citing works
A review of depression and suicide risk assessment using speech analysis
Cummins, N., Scherer, S. J., Krajewski, J., Schnieder, S., Epps, J., & Quatieri, T. F. (2015) · Speech Communication · doi:10.1016/j.specom.2015.03.004 · Cited by 1,007 · see citing works
Introducing Parselmouth: A Python interface to Praat
Jadoul, Y., Thompson, B., & de Boer, B. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.07.001 · Cited by 512 · see citing works
Stockmeier, C. A., Shapiro, L. A., Dilley, G. E., Kolli, T. N., Friedman, L., & Rajkowska, G. (1998) · Journal of Neuroscience · doi:10.1523/jneurosci.18-18-07394.1998 · Cited by 429 · see citing works
Psychomotor retardation in depression: Biological underpinnings, measurement, and treatment
Close, J. S., McClintock, S. M., & Croarkin, P. E. (2010) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2010.10.019 · Cited by 401 · see citing works
Suicide risk in schizophrenia: learning from the past to change the future
Pompili, M., AMADOR, X. F., Girardi, P., Harkavy‐Friedman, J., Harrow, M., Kaplan, K. J., Krausz, M. R., Lester, D., Meltzer, H. Y., Modestin, J., et al. (2007) · Annals of General Psychiatry · doi:10.1186/1744-859x-6-10 · Cited by 363 · see citing works
Developing and adopting safe and effective digital biomarkers to improve patient outcomes
Coravos, A., Khozin, S., & Mandl, K. D. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0090-4 · Cited by 361 · see citing works
Stockmeier, C. A. (2003) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(03)00050-5 · Cited by 357 · see citing works
Torous, J. B., Onnela, J., & Keshavan, M. S. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.25 · Cited by 349 · see citing works
Digital phenotyping: a global tool for psychiatry
Insel, T. R. (2018) · World Psychiatry · doi:10.1002/wps.20550 · Cited by 291 · see citing works
Social risk and depression: Evidence from manual and automatic facial expression analysis
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., & Rosenwald, D. P. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553748 · Cited by 197 · see citing works
Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding
Dibeklioğlu, H., Hammal, Z., & Cohn, J. F. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2676878 · Cited by 196 · see citing works
Herrera-Guzmán, I., Gudayol-Ferré, E., Herrera-Guzmán, D., Guàrdia-Olmos, J., Hinojosa-Calvo, E., & Herrera-Abarca, J. E. (2009) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2008.10.015 · Cited by 160 · see citing works
Relevance of Norepinephrine–Dopamine Interactions in the Treatment of Major Depressive Disorder
Mansari, M. E., Guiard, B. P., Chernoloz, O., Ghanbari, R., Katz, N., & Blier, P. (2010) · CNS Neuroscience & Therapeutics · doi:10.1111/j.1755-5949.2010.00146.x · Cited by 146 · see citing works
Kliem, S., Lohmann, A., Mößle, T., & Brähler, E. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1559-9 · Cited by 141 · see citing works
Head Pose and Movement Analysis as an Indicator of Depression
Alghowinem, S., Goecke, R., Wagner, M., Parkerx, G., & Breakspear, M. J. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.53 · Cited by 138 · see citing works
Modulating the serotonin system in the treatment of major depressive disorder
Morrissette, D. A., & Stahl, S. M. (2014) · CNS Spectrums · doi:10.1017/s1092852914000613 · Cited by 65 · see citing works
Galatzer‐Levy, I. R., Abbas, A., Koesmahargyo, V., Yadav, V., Pérez-Rodríguez, M. M., Rosenfield, P., Patil, O., Dockendorf, M. F., Moyer, M., Shipley, L. A., et al. (2021) · JMIR Formative Research · doi:10.2196/26276 · Cited by 53 · see citing works
Self-citationWessely, S. C. (2004) · JAMA · doi:10.1001/jama.292.3.379 · Cited by 49 · see citing works
Abbas, A., Sauder, C., Yadav, V., Koesmahargyo, V., Aghjayan, A., Marecki, S., Evans, M., & Galatzer-Levy, I. R. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.610006 · Cited by 49 · see citing works
Self-citationChu, C., Hom, M. A., Stanley, I. H., Gai, A. R., Nock, M. K., Gutierrez, P. M., & Joiner, T. E. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000262 · Cited by 46 · see citing works
Abbas, A., Yadav, V., Smith, E. C., et al. (2021) · Digital Biomarkers · doi:10.1159/000512383 · Cited by 43 · see citing works
Self-citationA Framework for Advancing Precision Medicine in Clinical Trials for Mental Disorders
Lenze, E. J., Rodebaugh, T. L., & Nicol, G. E. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.0114 · Cited by 41 · see citing works
Torous, J. B., & Walker, R. L. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1231 · Cited by 38 · see citing works
Toward a Distinct Mental Disorder—Suicidal Behavior
Sisti, D. A., Mann, J. J., & Oquendo, M. A. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.0111 · Cited by 36 · see citing works
Prosody Impairment in Depression Measured through Acoustic Analysis
Garcı́a-Toro, M., Talavera, J. A., Sáiz, P. A., & González, A. (2000) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-200012000-00006 · Cited by 35 · see citing works
Coravos, A., Khozin, S., & Mandl, K. D. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0119-8 · Cited by 34 · see citing works
Digital Suicide Prevention: Can Technology Become a Game-changer?
Vahabzadeh, A. B. N., Sahin, N. T., & Kalali, A. H. (2016) · Europe PMC (PubMed Central) · Cited by 33 · see citing works
Depression and posture in patients with Parkinson’s disease
Kim, Y., Cheon, S., Youm, C., et al. (2017) · Gait & Posture · doi:10.1016/j.gaitpost.2017.12.026 · Cited by 27 · see citing works
Digital Measurement of Mental Health: Challenges, Promises, and Future Directions
Abbas, A., Schultebraucks, K., & Galatzer‐Levy, I. R. (2021) · Psychiatric Annals · doi:10.3928/00485713-20201207-01 · Cited by 23 · see citing works
Self-citationHeller, M., & Haynal, V. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.003.0024 · Cited by 20 · see citing works
Kutchins, H., & Kirk, S. A. (1995) · Social Work · doi:10.1093/sw/40.2.286 · Cited by 15 · see citing works
Claassen, C. A. (2013) · Crisis · doi:10.1027/0227-5910/a000208 · Cited by 14 · see citing works
Introduction to Computer Vision and OpenCV
Brahmbhatt, S. (2013) · Apress eBooks · doi:10.1007/978-1-4302-6080-6_1 · Cited by 12 · see citing works
Galatzer‐Levy, I. R., Abbas, A., Yadav, V., et al. (2020) · medRxiv · doi:10.1101/2020.08.24.20178004 · Cited by 4 · see citing works
Self-citationCohen, M. L. (1980) · doi:10.21236/ada096768 · Cited by 2 · see citing works
SSRIs: suicide risk and withdrawal
Lancet, T. (2003) · The Lancet · doi:10.1016/s0140-6736(03)13671-9 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [8]Diefendorf AR Kraepelin E Clinical psychiatry: A textbook for students and physicians, abstracted and adapted from the 7th German edition of Kraepelin's Lehrbuch der Psychiatrie 1907 New York, NY MacMillan Co
- [40]Insel TR Digital phenotyping: a global tool for psychiatry World Psychiatry 2018 10 17 3 276 277 10.1002/wps.20550 10.1002/wps.20550 30192103 30192103 PMC6127813 · DOI
Cited by
Sorted by citations
Roos, R., Witteveen, A. B., Ayuso-Mateos, J. L., Barbui, C., Bryant, R. A., Felez-Nobrega, M., Figueiredo, N., Kalisch, R., Haro, J. M., McDaid, D., et al. (2023) · BMC Psychiatry · doi:10.1186/s12888-023-05288-5 · Cited by 5 · see citing works
“…altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder and psychopharmacological treatment has demonstrated restored levels of digital markers (e.g. increased head movement) and a decrease in anger and fear facial expressions [37, 38].”
Background
- Authors
- Galatzer-Levy, I., Abbas, A., Ries, A., Homan, S., Sels, L., Koesmahargyo, V., Yadav, V., Colla, M., Scheerer, H., Vetter, S., Seifritz, E., Scholz, U., & Kleim, B.
- Affiliations
- AiCure (United States) (US)
- University of Zurich (CH)
- Corresponding author
- Koesmahargyo, Vidya
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-06-03
- Volume
- 23
- Issue
- 6
- Pages
- e25199
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8212625
- OpenAlex
- W3136471953
- Cited by
- 23
- References
- 41
- Citations by year
- 2026: 4 · 2025: 4 · 2023: 9 · 2022: 6
- FWCI
- 2.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
- Ghent University Academic Bibliography (Ghent University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- suicide risk assessment, suicidal ideation, suicide attempts, psychiatric inpatients, facial expressivity, head movement, speech features, video interviews, deep learning, digital markers
- MeSH terms
- Emotions, Humans, Inpatients, Risk Factors, Suicide, Suicide, Attempted, Suicidal Ideation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
