Remote digital measurement of facial and vocal markers of major depressive disorder severity and treatment response: a pilot study
Open accessPublished version of a preprint
Abstract
Abstract Objectives Multiple machine learning-based visual and auditory digital markers have demonstrated associations between Major Depressive Disorder (MDD) status and severity. The current study examines if such measurements can quantify response to antidepressant treatment (ADT) with selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine uptake inhibitors (SNRIs). Methods Visual and auditory markers were acquired through an automated smartphone task that measures facial, vocal, and head movement characteristics across four weeks of treatment (with timepoints at baseline, 2 weeks, and 4 weeks) on ADT (n = 12). The Montgomery-Asberg Depression Rating Scale (MADRS) was collected concordantly through clinical interviews to confirm diagnosis and assess changes in MDD severity. Results Patient responses to ADT demonstrated clinically and statistically significant changes in the MADRS F(2,34) = 51.62, p <.0001. Additionally, patients demonstrated significant increases in multiple digital markers including facial expressivity, head movement, and amount of speech. Finally, patients demonstrated significant decreased frequency of fear and anger facial expressions. Conclusion Digital markers associated with MDD demonstrate validity as measures of treatment response.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder.
“Notably, altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder and psychopharmacological treatment has demonstrated restored levels of digital markers (e.g. increased head movement) and a decrease in anger and fear facial expressions [37, 38].”
Psychopharmacological treatment has demonstrated restored levels of digital markers and a decrease in anger and fear facial expressions.
“Notably, altered speech and vocality, reduced facial expressivity and movement have been found in major depressive disorder and psychopharmacological treatment has demonstrated restored levels of digital markers (e.g. increased head movement) and a decrease in anger and fear facial expressions [37, 38].”
Abbas et al. found that digitally measured facial, vocal, and movement behaviors can measure depression severity and treatment response across 4 weeks of antidepressant treatment.
“Ability of digitally measured facial, vocal, and movement behaviors to measure depression severity and treatment response across 4 weeks of antidepressant treatment”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
Abbas et al. found that monoamine antidepressants such as SSRIs and SNRIs had a significant impact on digital biomarkers, with reduction in symptom severity as assessed by MADRS, indicating improvement in motor functioning and decrease in depression severity.
“The study found that monoamine ADTs, such as SSRIs and SNRIs, had a significant impact on digital biomarker with a reduction in symptom severity as assessed by the MADRS evaluation, indicating an improvement in motor functioning and a decrease in depression severity due to these treatments.”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
McNamara et al. found that low positive emotionality is a strong differentiator of depression.
“Low positive emotionality is a strong differentiator of depression.”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
McNamara et al. found that psychiatric control groups can help distinguish specific factors in the diagnosis of interest.
“Psychiatric control groups can help to distinguish specific factors in the diagnosis of interest.”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
Digital biomarkers encompassing motor function can accurately monitor shifts in depression severity throughout antidepressant therapy.
“Abbas et al. (31) showcased the capacity of digital biomarkers, encompassing motor function, to accurately monitor shifts in depression severity throughout the course of antidepressant therapy.”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
McNamara et al. found that perceived sleep quality and impairment are important predictors of depression.
“Additionally, perceived sleep quality and impairment are also important predictors”
From A narrative review of digital biomarkers in the management of major depressive disorder and treatment-resistant forms (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Praat: Doing Phonetics by Computer
(2011) · Ear and Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 8,004 · see citing works
Toward the future of psychiatric diagnosis: the seven pillars of RDoC
Cuthbert, B. N., & Insel, T. R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-126 · Cited by 3,053 · see citing works
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,975 · see citing works
OpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · see citing works
The NIMH Research Domain Criteria (RDoC) Project: Precision Medicine for Psychiatry
Insel, T. R. (2014) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2014.14020138 · Cited by 1,472 · see citing works
A review of depression and suicide risk assessment using speech analysis
Cummins, N., Scherer, S. J., Krajewski, J., Schnieder, S., Epps, J., & Quatieri, T. F. (2015) · Speech Communication · doi:10.1016/j.specom.2015.03.004 · Cited by 1,007 · see citing works
Digital Phenotyping: Technology for a New Science of Behavior
Insel, T. R. (2017) · JAMA · doi:10.1001/jama.2017.11295 · Cited by 833 · see citing works
Harnessing Smartphone-Based Digital Phenotyping to Enhance Behavioral and Mental Health
Onnela, J.-P., & Rauch, S. L. (2016) · Neuropsychopharmacology · doi:10.1038/npp.2016.7 · Cited by 726 · see citing works
Introducing the Open Affective Standardized Image Set (OASIS)
Kurdi, B., Lozano, S., & Banaji, M. R. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0715-3 · Cited by 669 · see citing works
Introducing Parselmouth: A Python interface to Praat
Jadoul, Y., Thompson, B., & de Boer, B. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.07.001 · Cited by 512 · see citing works
Emotional experience and expression in schizophrenia and depression.
Berenbaum, H., & Oltmanns, T. F. (1992) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.101.1.37 · Cited by 431 · see citing works
Stockmeier, C. A., Shapiro, L. A., Dilley, G. E., Kolli, T. N., Friedman, L., & Rajkowska, G. (1998) · Journal of Neuroscience · doi:10.1523/jneurosci.18-18-07394.1998 · Cited by 429 · see citing works
Psychomotor retardation in depression: Biological underpinnings, measurement, and treatment
Close, J. S., McClintock, S. M., & Croarkin, P. E. (2010) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2010.10.019 · Cited by 401 · see citing works
Psychiatric complications during flooding therapy for posttraumatic stress disorder.
Pitman, R. K., Altman, B., Greenwald, E., Longpre, R. E., Macklin, M. L., Poiré, R. E., & Steketee, G. S. (1991) · PubMed · Cited by 385 · see citing works
Developing and adopting safe and effective digital biomarkers to improve patient outcomes
Coravos, A., Khozin, S., & Mandl, K. D. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0090-4 · Cited by 361 · see citing works
Stockmeier, C. A. (2003) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(03)00050-5 · Cited by 357 · see citing works
Torous, J. B., Onnela, J., & Keshavan, M. S. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.25 · Cited by 349 · see citing works
Labovitz, D. L., Shafner, L., Reyes Gil, M., Virmani, D., & Hanina, A. (2017) · Stroke · doi:10.1161/strokeaha.116.016281 · Cited by 311 · see citing works
The heterogeneity of the depressive syndrome: when numbers get serious
Østergaard, S. D., Jensen, S. O. W., & Bech, P. (2011) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2011.01744.x · Cited by 287 · see citing works
Motor effects of serotonin in the central nervous system
Gerson, S. C., & Baldessarini, R. J. (1980) · Life Sciences · doi:10.1016/0024-3205(80)90368-9 · Cited by 263 · see citing works
The serotonergic system in mood disorders and suicidal behaviour
Mann, J. J. (2013) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0537 · Cited by 235 · see citing works
Coifman, K. G., Bonanno, G. A., Ray, R. D., & Gross, J. J. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.4.745 · Cited by 222 · see citing works
Anger and Hostility in Depression
Riley, W. T., Treiber, F. A., & Woods, M. G. (1989) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-198911000-00002 · Cited by 206 · see citing works
Deconstructing major depression: a validation study of the DSM-IV symptomatic criteria
Lux, V., & Kendler, K. S. (2010) · Psychological Medicine · doi:10.1017/s0033291709992157 · Cited by 204 · see citing works
Social risk and depression: Evidence from manual and automatic facial expression analysis
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., & Rosenwald, D. P. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553748 · Cited by 197 · see citing works
Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding
Dibeklioğlu, H., Hammal, Z., & Cohn, J. F. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2676878 · Cited by 196 · see citing works
Gold, A. L., Morey, R. A., & McCarthy, G. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.03.030 · Cited by 192 · see citing works
Inferior frontal gyrus activity triggers anterior insula response to emotional facial expressions.
Jabbi, M., & Keysers, C. (2008) · Emotion · doi:10.1037/a0014194 · Cited by 190 · see citing works
Digital biomarkers of cognitive function
Dagum, P. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0018-4 · Cited by 190 · see citing works
Differential efficacy of escitalopram and nortriptyline on dimensional measures of depression
Uher, R., Maier, W., Hauser, J., Marušič†, A., Schmael, C., Mors, O. N., Henigsberg, N., Souery, D., Placentino, A., Rietschel, M., et al. (2009) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.108.057554 · Cited by 183 · see citing works
Herrera-Guzmán, I., Gudayol-Ferré, E., Herrera-Guzmán, D., Guàrdia-Olmos, J., Hinojosa-Calvo, E., & Herrera-Abarca, J. E. (2009) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2008.10.015 · Cited by 160 · see citing works
Multiple Testing with the Structure-Adaptive Benjamini–Hochberg Algorithm
Li, A. J., & Barber, R. F. (2018) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12298 · Cited by 160 · see citing works
When distress does not become depression: Emotion context sensitivity and adjustment to bereavement.
Coifman, K. G., & Bonanno, G. A. (2010) · Journal of Abnormal Psychology · doi:10.1037/a0020113 · Cited by 153 · see citing works
Relevance of Norepinephrine–Dopamine Interactions in the Treatment of Major Depressive Disorder
Mansari, M. E., Guiard, B. P., Chernoloz, O., Ghanbari, R., Katz, N., & Blier, P. (2010) · CNS Neuroscience & Therapeutics · doi:10.1111/j.1755-5949.2010.00146.x · Cited by 146 · see citing works
Anisman, H., Du, L., Palkovits, M., Faludi, G., Kovács, G. G., Szontagh-Kisházi, P., Merali, Z., & Poulter, M. O. (2008) · Journal of Psychiatry and Neuroscience · doi:10.1139/jpn.0815 · Cited by 143 · see citing works
Head Pose and Movement Analysis as an Indicator of Depression
Alghowinem, S., Goecke, R., Wagner, M., Parkerx, G., & Breakspear, M. J. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.53 · Cited by 138 · see citing works
EmoMadrid: An emotional pictures database for affect research
Carretié, L., Tapia, M., López‐Martín, S., & Albert, J. (2019) · Motivation and Emotion · doi:10.1007/s11031-019-09780-y · Cited by 104 · see citing works
Modulating the serotonin system in the treatment of major depressive disorder
Morrissette, D. A., & Stahl, S. M. (2014) · CNS Spectrums · doi:10.1017/s1092852914000613 · Cited by 65 · see citing works
Profiles of antidepressant activity with the Montgomery‐Asberg depression rating scale
Montgomery, S., Smeyatsky, N., Ruiter, M. D., & Montgomery, D. (1985) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1985.tb08073.x · Cited by 64 · see citing works
Parkinson’s disease psychosis as a serotonin-dopamine imbalance syndrome
Stahl, S. M. (2016) · CNS Spectrums · doi:10.1017/s1092852916000602 · Cited by 61 · see citing works
McDevitt, R. A., Hiroi, R., Mackenzie, S. M., Robin, N. C., Cohn, A., Kim, J. J., & Neumaier, J. F. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.12.029 · Cited by 60 · see citing works
A Significance Test for Multisample Sphericity
Mendoza, J. L. (1980) · Psychometrika · doi:10.1007/bf02293611 · Cited by 55 · see citing works
Koh, K. B., Kim, D. K., Kim, S. Y., & Park, J. K. (2005) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v66n0411 · Cited by 54 · see citing works
Galatzer‐Levy, I. R., Abbas, A., Koesmahargyo, V., Yadav, V., Pérez-Rodríguez, M. M., Rosenfield, P., Patil, O., Dockendorf, M. F., Moyer, M., Shipley, L. A., et al. (2021) · JMIR Formative Research · doi:10.2196/26276 · Cited by 53 · see citing works
Self-citationHakulinen, C., Jokela, M., Hintsanen, M., Merjonen, P., Pulkki-Råback, L., Seppälä, I., Lyytikäinen, L.-P., Lehtimäki, T., Kähönen, M., Viikari, J., et al. (2013) · Journal of Behavioral Medicine · doi:10.1007/s10865-012-9452-y · Cited by 47 · see citing works
Abbas, A., Yadav, V., Smith, E. C., et al. (2021) · Digital Biomarkers · doi:10.1159/000512383 · Cited by 43 · see citing works
Self-citationA Framework for Advancing Precision Medicine in Clinical Trials for Mental Disorders
Lenze, E. J., Rodebaugh, T. L., & Nicol, G. E. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.0114 · Cited by 41 · see citing works
Prosody Impairment in Depression Measured through Acoustic Analysis
Garcı́a-Toro, M., Talavera, J. A., Sáiz, P. A., & González, A. (2000) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-200012000-00006 · Cited by 35 · see citing works
Depression and posture in patients with Parkinson’s disease
Kim, Y., Cheon, S., Youm, C., et al. (2017) · Gait & Posture · doi:10.1016/j.gaitpost.2017.12.026 · Cited by 27 · see citing works
Jaracz, J., Gattner, K., Moczko, J., & Hauser, J. (2014) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2014.10.005 · Cited by 20 · see citing works
Introduction to Computer Vision and OpenCV
Brahmbhatt, S. (2013) · Apress eBooks · doi:10.1007/978-1-4302-6080-6_1 · Cited by 12 · see citing works
Hanina, A., & Shafner, L. (2019) · Handbook of behavioral neuroscience · doi:10.1016/b978-0-12-803161-2.00031-x · Cited by 3 · see citing works
Emotional Experience and Expression in Schizophrenia and Depression
Berenbaum, H., & Oltmanns, T. F. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.003.0021 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [49]Jacobs BL. Serotonin and behavior: emphasis on motor control. J Clin Psychiatry. (1991) 52(Suppl.): 17– 23. 1752856
Cited by
Sorted by citations
Galatzer-Levy, I., Abbas, A., Ries, A., Homan, S., Sels, L., Koesmahargyo, V., Yadav, V., Colla, M., Scheerer, H., Vetter, S., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/25199 · Cited by 23 · see citing works
Self-citationVignapiano, A., Monaco, F., Pagano, C., Piacente, M., Farina, F., Petrillo, G., Sica, R., Marenna, A., Shin, J. I., Solmi, M., et al. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1321345 · Cited by 14 · see citing works
Roos, R., Witteveen, A. B., Ayuso-Mateos, J. L., Barbui, C., Bryant, R. A., Felez-Nobrega, M., Figueiredo, N., Kalisch, R., Haro, J. M., McDaid, D., et al. (2023) · BMC Psychiatry · doi:10.1186/s12888-023-05288-5 · Cited by 5 · see citing works
“As an exploratory additional analysis, digital markers will be assessed using open-source tools in a Python environment, which is available on GitHub [38, 74].”
BackgroundMethod
- Authors
- Abbas, A., Sauder, C., Yadav, V., Koesmahargyo, V., Aghjayan, A., Marecki, S., Evans, M., & Galatzer-Levy, I. R.
- Affiliations
- AiCure (United States) (US)
- Corresponding author
- Koesmahargyo, Vidya
- Venue
- Frontiers in Digital Health
- Publisher
- Frontiers Media SA
- Published
- 2021-03-31
- Volume
- 3
- Pages
- 610006-610006
- ISSN
- 2673-253X (ISSN-L)
- PMC
- PMC8521884
- OpenAlex
- W3140280866
- Cited by
- 49
- References
- 53
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 11 · 2023: 10 · 2022: 6 · 2021: 3
- FWCI
- 7.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Health Sciences › Medicine › Pharmacology › Treatment of Major Depression
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- major depressive disorder, antidepressant response, digital markers, facial expressivity, head movement, smartphone-based assessment, facial expressions, selective serotonin reuptake inhibitors, serotonin-norepinephrine reuptake inhibitors, depression severity, remote measurement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
