Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions
Open accessAbstract
OBJECTIVE: Major depressive disorder (MDD) is a common psychiatric disorder that leads to persistent changes in mood and interest among other signs and symptoms. We hypothesized that convolutional neural network (CNN) based automated facial expression recognition, pre-trained on an enormous auxiliary public dataset, could provide improve generalizable approach to MDD automatic assessment from videos, and classify remission or response to treatment. METHODS: We evaluated a novel deep neural network framework on 365 video interviews (88 hours) from a cohort of 12 depressed patients before and after deep brain stimulation (DBS) treatment. Seven basic emotions were extracted with a Regional CNN detector and an Imagenet pre-trained CNN, both of which were trained on large-scale public datasets (comprising over a million images). Facial action units were also extracted with the Openface toolbox. Statistics of the temporal evolution of these image features over each recording were extracted and used to classify MDD remission and response to DBS treatment. RESULTS: An Area Under the Curve of 0.72 was achieved using leave-one-subject-out cross-validation for remission classification and 0.75 for response to treatment. CONCLUSION: This work demonstrates the potential for the classification of MDD remission and response to DBS treatment from passively acquired video captured during unstructured, unscripted psychiatric interviews. SIGNIFICANCE: This novel MDD evaluation could be used to augment current psychiatric evaluations and allow automatic, low-cost, frequent use when an expert isn't readily available or the patient is unwilling or unable to engage. Potentially, the framework may also be applied to other psychiatric disorders.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Alternative measures are under study, such as scales for symptom subdimensions or analysis of movements, speech, or facial expression.
“alternative measures are under study for example scales for symptom subdimensions or analysis of movements, speech, or facial expression [82][83][84].”
From Deep brain stimulation for depression. (2022) · cites this paper as [82]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Hamilton, M. (1960) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.23.1.56 · Cited by 32,817 · cited 1× · see citing works
“The severity of MDD and response to treatment are evaluated with the Hamilton Depression Rating Scale (HDRS) [6].”
MethodRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Kessler, R. C., Berglund, P., Demler, O., Jin, R., Koretz, D., Merikangas, K. R., Rush, A. J., Walters, E. E., & Wang, P. S. (2003) · JAMA · doi:10.1001/jama.289.23.3095 · Cited by 7,988 · see citing works
The PHQ-8 as a measure of current depression in the general population
Kroenke, K., Strine, T. W., Spitzer, R. L., Williams, J. B. W., Berry, J. T., & Mokdad, A. H. (2009) · Journal of Affective Disorders · doi:10.1016/j.jad.2008.06.026 · Cited by 5,121 · see citing works
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Deep Brain Stimulation for Treatment-Resistant Depression
Mayberg, H. S., Lozano, A. M., Voon, V., McNeely, H. E., Seminowicz, D., Hamani, C., Schwalb, J. M., & Kennedy, S. H. (2005) · Neuron · doi:10.1016/j.neuron.2005.02.014 · Cited by 4,059 · see citing works
Self-citationIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Presentation and validation of the Radboud Faces Database
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010) · Cognition & Emotion · doi:10.1080/02699930903485076 · Cited by 2,431 · see citing works
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338 · see citing works
Belmaker, R. H., & Agam, G. (2008) · New England Journal of Medicine · doi:10.1056/nejmra073096 · Cited by 2,166 · see citing works
WIDER FACE: A Face Detection Benchmark
Yang, S., Luo, P., Loy, C. C., & Tang, X. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.596 · Cited by 1,907 · see citing works
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861 · see citing works
A meta-analysis of emotional reactivity in major depressive disorder
Bylsma, L. M., Morris, B. H., & Rottenberg, J. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2007.10.001 · Cited by 933 · see citing works
FDDB: A benchmark for face detection in unconstrained settings
Jain, V., & Miller, E. G. (2010) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 879 · see citing works
Face Detection with the Faster R-CNN
Jiang, H., & Miller, E. G. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.82 · Cited by 823 · see citing works
DISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793 · see citing works
Emotion context insensitivity in major depressive disorder
Rottenberg, J., Gross, J. J., & Gotlib, I. H. (2005) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.114.4.627 · Cited by 654 · see citing works
(2014) · doi:10.5555/2615731.2617415 · Cited by 553 · see citing works
Moving faces, looking places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES).
van der Schalk, J., Hawk, S. T., Fischer, A. H., & Doosje, B. (2011) · Emotion · doi:10.1037/a0023853 · Cited by 521 · see citing works
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988258 · Cited by 516 · see citing works
Detecting depression from facial actions and vocal prosody
Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., Zhou, F., & De la Torre, F. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349358 · Cited by 494 · see citing works
Valstar, M., Schuller, B. W., Smith, K., Almaev, T. R., Eyben, F., Krajewski, J., Cowie, R., & Pantić, M. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661807 · Cited by 476 · see citing works
Husain, M. M., Rush, A. J., Fink, M., Knapp, R. G., Petrides, G., Rummans, T. A., Biggs, M. M., O’Connor, K. M., Rasmussen, K. G., Litle, M., et al. (2004) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v65n0406 · Cited by 455 · see citing works
Cross-dataset learning and person-specific normalisation for automatic Action Unit detection
Baltrušaitis, T., Mahmoud, M., & Robinson, P. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7284869 · Cited by 415 · see citing works
Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2013.12.007 · Cited by 255 · see citing works
TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition
Ma, C., Chen, M.-H., Kira, Z., & AlRegib, G. (2018) · Signal Processing Image Communication · doi:10.1016/j.image.2018.09.003 · Cited by 247 · see citing works
Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
Long, X., Gan, C., de Melo, G., Wu, J., Liu, X., & Wen, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00817 · Cited by 233 · see citing works
Automatic behavior descriptors for psychological disorder analysis
Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A., & Morency, L.-P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553789 · Cited by 147 · see citing works
Repetitive Transcranial Magnetic Stimulation for Major Depressive Disorder: A Review
Daskalakis, Z. J., LEVINSON, A. J., & Fitzgerald, P. B. (2008) · The Canadian Journal of Psychiatry · doi:10.1177/070674370805300902 · Cited by 142 · see citing works
(1989) · Cited by 141 · see citing works
Automatically detecting pain using facial actions
Lucey, P., Cohn, J. F., Lucey, S., Matthews, I., Sridharan, S., & Prkachin, K. M. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349321 · Cited by 126 · see citing works
A Compact Embedding for Facial Expression Similarity
Vemulapalli, R., & Agarwala, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00583 · Cited by 121 · see citing works
Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions
Gupta, R., Malandrakis, N., Xiao, B., Guha, T., Segbroeck, M. V., Black, M., Potamianos, A., & Narayanan, S. S. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661810 · Cited by 110 · see citing works
Crowell, A. L., Garlow, S. J., Riva-Posse, P., & Mayberg, H. S. (2015) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2015.00041 · Cited by 105 · see citing works
Self-citationTackling Mental Health by Integrating Unobtrusive Multimodal Sensing
Zhou, D., Luo, J., Silenzio, V., Zhou, Y., Hu, J., Currier, G. W., & Kautz, H. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9381 · Cited by 104 · see citing works
Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender
Stratou, G., Scherer, S. J., Gratch, J. S., & Morency, L. (2014) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0161-4 · Cited by 57 · see citing works
Depression Severity Prediction Based on Biomarkers of Psychomotor Retardation
Syed, Z. S., Sidorov, K. A., & Marshall, D. (2017) · doi:10.1145/3133944.3133947 · Cited by 49 · see citing works
Video-based detection of the clinical depression in adolescents
Maddage, N. C., Senaratne, R., Low, L.-S. A., et al. (2009) · doi:10.1109/iembs.2009.5334815 · Cited by 45 · see citing works
Depression Severity Classification from Speech Emotion
Harati, S., Crowell, A. L., Mayberg, H. S., & Nemati, S. (2018) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2018.8513610 · Cited by 44 · see citing works
Self-citationHarati, S., Crowell, A. L., Huang, Y., Mayberg, H. S., & Nemati, S. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2930604 · Cited by 33 · see citing works
Self-citationPurely Attention Based Local Feature Integration for Video Classification
Long, X., de Melo, G., He, D., Li, F., Chi, Z., Wen, S., & Gan, C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3029554 · Cited by 30 · see citing works
A Multimodal Context-based Approach for Distress Assessment
Ghosh, S., Chatterjee, M., & Morency, L. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663274 · Cited by 27 · see citing works
Gurnani, A., Shah, K., Gajjar, V., Mavani, V., & Khandhediya, Y. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00094 · Cited by 24 · see citing works
McCall, M. V., Riva-Posse, P., Garlow, S. J., Mayberg, H. S., & Crowell, A. L. (2020) · Neurology Psychiatry and Brain Research · doi:10.1016/j.npbr.2020.05.002 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 6
- [8]Ellgring H, Non-verbal communication in depression. Cambridge, U.K: Cambridge University Press, 2007, pp. 4– 17.
- [12]Valstar M, Gratch J, Schuller B, “ AVEC 2016: Depression, mood, and emotion recognition workshop and challenge,” in Proc. ACM Int. Workshop on Audio/Visual Emotion Challenge, 2016, pp. 3– 10.
- [13]Beck AT, Steer RA, Brown GK, Beck depression inventory-II. San Antonio, TX: The Psychological Corporation, 1996, pp. 490– 498.
- [15]Valstar M, Schuller B, Smith K, “ AVEC 2014: 3D dimensional affect and depression recognition challenge,” in Proc. ACM Int. Workshop on Audio/Visual Emotion Challenge, 2014, pp. 3– 10.
- [33]Ren S, He K, Girshick R, “ Faster R-CNN: Towards real-time object detection with region proposal networks,” in Proc. Advances in Neural Information Processing Systems 28, 2015, pp. 91– 99.
- [39]Friesen E and Ekman P, Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press, 1978, vol. 3.
Cited by
Sorted by citations
Deep brain stimulation for depression.
Figee, M., Riva-Posse, P., Choi, K. S., Bederson, L., Mayberg, H. S., & Kopell, B. H. (2022) · Neurotherapeutics · doi:10.1007/s13311-022-01270-3 · Cited by 175 · see citing works
Self-citation
- Authors
- Jiang, Z., Harati, S., Crowell, A., Mayberg, H. S., Nemati, S., & Clifford, G. D.
- Affiliations
- Georgia Institute of Technology (US)
- Stanford University (US)
- Emory University (US)
- Icahn School of Medicine at Mount Sinai (US)
- UC San Diego Health System (US)
- Venue
- IEEE Transactions on Biomedical Engineering
- Published
- 2021-02
- Volume
- 68
- Issue
- 2
- Pages
- 664-672
- ISSN
- 0018-9294 (ISSN-L)
- 1558-2531
- PMC
- PMC7891869
- OpenAlex
- W3044580164
- Cited by
- 60
- References
- 49
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 11 · 2023: 12 · 2022: 11 · 2021: 5
- FWCI
- 6.77
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- IEEE Transactions on Biomedical Engineering (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Cox Foundation
- NSF (1822378)
- NIH (UH3NS103550)
- Hope for Depression Research Foundation
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- major depressive disorder, deep brain stimulation, facial expression recognition, treatment response classification, convolutional neural networks, facial action units, automated mental health assessment
- MeSH terms
- Major Depressive Disorder, Emotions, Facial Expression, Humans, Neural Networks, Computer, Deep Brain Stimulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
