Detecting depression from facial actions and vocal prosody
Abstract
Current methods of assessing psychopathology depend almost entirely on verbal report (clinical interview or questionnaire) of patients, their family, or caregivers. They lack systematic and efficient ways of incorporating behavioral observations that are strong indicators of psychological disorder, much of which may occur outside the awareness of either individual. We compared clinical diagnosis of major depression with automatically measured facial actions and vocal prosody in patients undergoing treatment for depression. Manual FACS coding, active appearance modeling (AAM) and pitch extraction were used to measure facial and vocal expression. Classifiers using leave-one-out validation were SVM for FACS and for AAM and logistic regression for voice. Both face and voice demonstrated moderate concurrent validity with depression. Accuracy in detecting depression was 88% for manual FACS and 79% for AAM. Accuracy for vocal prosody was 79%. These findings suggest the feasibility of automatic detection of depression, raise new issues in automated facial image analysis and machine learning, and have exciting implications for clinical theory and practice.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cohn et al. concluded that specific AUs have higher predictive power for depression than others, suggesting the advantage of using indirect representations for depression prediction.
“They concluded that specific AUs have higher predictive power for depression than others suggesting the advantage of using indirect representations for depression prediction.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [187]
Facial expressions were proven to correlate better than visual activity with personality traits.
“First, FEs were proven to correlate better than visual activity with personality traits [187].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [187]
Cites
Diagnostic and Statistical Manual of Mental Disorders
Williams, J. B. W., & First, M. B. (2013) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199975839.013.104 · Cited by 109,841 · see citing works
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Hamilton, M. (1960) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.23.1.56 · Cited by 32,817 · see citing works
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,124 · see citing works
Diagnostic and Statistical Manual of Mental Disorders
Siegel, L. J., & Fletcher‐Janzen, E. (2008) · Wiley eBooks · doi:10.1002/9780470373699.speced0655 · Cited by 8,468 · see citing works
Praat: Doing Phonetics by Computer
(2011) · Ear and Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 8,004 · see citing works
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019 · see citing works
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409 · see citing works
Structured Clinical Interview for DSM-IV Axis I Disorders
Spitzer, R. L., First, M. B., & Gibbon, M. (1997) · Medical Entomology and Zoology · Cited by 3,582 · see citing works
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,975 · see citing works
Statistical Methods for Rates and Proportions.
Plewis, I., & Fleiss, J. L. (1982) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2982107 · Cited by 1,753 · see citing works
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Self-citationActive Appearance Models Revisited
Matthews, I., & Baker, S. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029666.37597.d3 · Cited by 1,538 · see citing works
Self-citationThe Handbook of emotion elicitation and assessment
(2008) · Choice Reviews Online · doi:10.5860/choice.46-1769 · Cited by 1,127 · see citing works
(2017) · The SAGE Encyclopedia of Communication Research Methods · doi:10.4135/9781483381411.n178 · Cited by 559 · see citing works
Treatment and prevention of depression
Hollon, S. D., Thase, M. E., & Markowitz, J. C. (2002) · Psychological Science in the Public Interest · doi:10.1111/1529-1006.00008 · Cited by 511 · see citing works
Observer-Based Measurement of Facial Expression with the Facial Action Coding System
Cohn, J. F., Ambadar, Z., & Ekman, P. (2007) · doi:10.1093/oso/9780195169157.003.0014 · Cited by 398 · see citing works
Self-citationFacial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
Tong, Y., Liao, W., & Ji, Q. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1094 · Cited by 377 · see citing works
The painful face – Pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., & Solomon, P. E. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.05.007 · Cited by 355 · see citing works
Self-citationAmbadar, Z., Cohn, J. F., & Reed, L. I. (2008) · Journal of Nonverbal Behavior · doi:10.1007/s10919-008-0059-5 · Cited by 319 · see citing works
Self-citationIntroduction to Social Psychology : A European Perspective
Hewstone, M., Stroebe, W., & Stephenson, G. M. (1988) · Cited by 304 · see citing works
Maternal depression and motherese: Temporal and intonational features
Bettes, B. A. (1988) · Child Development · doi:10.2307/1130275 · Cited by 266 · see citing works
Maternal Depression and Motherese: Temporal and Intonational Features
Bettes, B. A. (1988) · Child Development · doi:10.1111/j.1467-8624.1988.tb03261.x · Cited by 264 · see citing works
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248 · see citing works
Schmidt, K. L., Ambadar, Z., Cohn, J. F., & Reed, L. I. (2006) · Journal of Nonverbal Behavior · doi:10.1007/s10919-005-0003-x · Cited by 219 · see citing works
Self-citationAutomatic coding of facial expressions displayed during posed and genuine pain
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.12.010 · Cited by 216 · see citing works
Gibson, R. (2002) · Cited by 195 · see citing works
Moving Smiles: The Role of Dynamic Components for the Perception of the Genuineness of Smiles
Krumhuber, E. G., & Kappas, A. (2005) · Journal of Nonverbal Behavior · doi:10.1007/s10919-004-0887-x · Cited by 173 · see citing works
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168 · see citing works
Self-citationLittlewort, G. C., Bartlett, M. S., & Lee, K. (2007) · International Conference on Multimodal Interfaces · doi:10.1145/1322192.1322198 · Cited by 152 · see citing works
Automated Measurement of Facial Expression in Infant–Mother Interaction: A Pilot Study
Messinger, D. S., Mahoor, M. H., Chow, S., & Cohn, J. F. (2009) · Infancy · doi:10.1080/15250000902839963 · Cited by 151 · see citing works
Self-citation(1989) · Cited by 141 · see citing works
Comparison of Deliberate and Spontaneous Facial Movement in Smiles and Eyebrow Raises
Schmidt, K. L., Bhattacharya, S., & Denlinger, R. (2009) · Journal of Nonverbal Behavior · doi:10.1007/s10919-008-0058-6 · Cited by 117 · see citing works
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K., Solomon, P., & Theobald, B. J. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322197 · Cited by 106 · see citing works
Self-citationAAM Derived Face Representations for Robust Facial Action Recognition
Lucey, S., Matthews, I. A., Hu, C., Ambadar, Z., la Torre, F. D., & Cohn, J. F. (2006) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fgr.2006.17 · Cited by 106 · see citing works
Self-citationZlochower, A. J., & Cohn, J. F. (1996) · Infant Behavior and Development · doi:10.1016/s0163-6383(96)90035-1 · Cited by 102 · see citing works
Self-citationCohn, J. F., & Ekman, P. (2005) · doi:10.1093/oso/9780198529613.003.0002 · Cited by 101 · see citing works
Self-citationImpact of depression on response to comedy: A dynamic facial coding analysis.
Reed, L. I., Sayette, M. A., & Cohn, J. F. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.4.804 · Cited by 100 · see citing works
Self-citationAutomated video-based facial expression analysis of neuropsychiatric disorders
Wang, P., Barrett, F. S., Martin, E. A., Milonova, M., Gur, R. E., Gur, R. C., Kohler, C. G., & Verma, R. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2007.09.030 · Cited by 90 · see citing works
Boker, S. M., Cohn, J. F., Theobald, B.-J., Matthews, I. A., Brick, T. R., & Spies, J. R. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2009.0152 · Cited by 69 · see citing works
Self-citationTemporal Segmentation of Facial Behavior
la Torre, F. D., Campoy, J., Ambadar, Z., & Cohn, J. F. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408961 · Cited by 66 · see citing works
Self-citationBeebe, B. A., & Gerstman, L. J. (1980) · Merrill-palmer Quarterly · Cited by 60 · see citing works
Facial Expression in Affective Disorders
Ekman, P., Matsumoto, D., & Friesen, W. V. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.003.0020 · Cited by 54 · see citing works
The role of structural facial asymmetry in asymmetry of peak facial expressions
Schmidt, K. L., Liu, Y., & Cohn, J. F. (2006) · Laterality Asymmetries of Body Brain and Cognition · doi:10.1080/13576500600832758 · Cited by 30 · see citing works
Self-citationInvestigating the Dynamics of Facial Expression
Reilly, J., Ghent, J. H., & McDonald, J. (2006) · Lecture notes in computer science · doi:10.1007/11919629_35 · Cited by 24 · see citing works
Schmidt, K. L., VanSwearingen, J. M., & Levenstein, R. M. (2005) · Motor Control · doi:10.1123/mcj.9.3.270 · Cited by 24 · see citing works
Automatically detecting action units from faces of pain: Comparing shape and appearance features
Lucey, P., Cohn, J. F., Lucey, S., Sridharan, S., & Prkachin, K. M. (2009) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2009.5204279 · Cited by 22 · see citing works
Self-citationHeller, M., & Haynal, V. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.003.0024 · Cited by 20 · see citing works
Heller, M., & Haynal, V. (1998) · doi:10.1093/oso/9780195104462.003.0020 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Self-citationRingeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409 · see citing works
Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2013.12.007 · Cited by 255 · see citing works
Self-citationMeng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512532 · Cited by 225 · see citing works
Jan, A., Meng, H., Gaus, Y. F. B. A., & Zhang, F. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2017.2721552 · Cited by 219 · see citing works
Automatic behavior descriptors for psychological disorder analysis
Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A., & Morency, L.-P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553789 · Cited by 147 · see citing works
Fan, H., Zhang, X., Xu, Y., Fang, J., Zhang, S., Zhao, X., & Yu, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102161 · Cited by 137 · see citing works
Detecting Dementia Through Interactive Computer Avatars
Tanaka, H., Adachi, H., Ukita, N., Ikeda, M., Kazui, H., Kudo, T., & Nakamura, S. (2017) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2017.2752152 · Cited by 127 · see citing works
Schultebraucks, K., Yadav, V., Shalev, A. Y., Bonanno, G. A., & Galatzer-Levy, I. R. (2022) · Psychological Medicine · doi:10.1017/s0033291720002718 · Cited by 108 · see citing works
Acoustic differences between healthy and depressed people: a cross-situation study
Wang, J., Zhang, L., Liu, T., Pan, W., Hu, B., & Zhu, T. (2019) · BMC Psychiatry · doi:10.1186/s12888-019-2300-7 · Cited by 98 · see citing works
Jiang, Z., Harati, S., Crowell, A., Mayberg, H. S., Nemati, S., & Clifford, G. D. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.3010472 · Cited by 60 · see citing works
Sensor-assisted weighted average ensemble model for detecting major depressive disorder
Mahendran, N., Vincent, D. R., Srinivasan, K., Chang, C.-Y., Garg, A., Gao, L., & Reina, D. G. (2019) · Sensors · doi:10.3390/s19224822 · Cited by 51 · see citing works
Estimating smile intensity: A better way
Girard, J. M., Cohn, J. F., & De la Torre, F. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.10.004 · Cited by 44 · see citing works
Self-citationWang, L., Wang, C., Li, C., Murai, T., Bai, Y., Song, Z., Zhang, S., Zhang, Q., Huang, Y., Bi, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01933-3 · Cited by 33 · see citing works
König, A., Zeghari, R., Guerchouche, R., Duc Tran, M., Bremond, F., Linz, N., Lindsay, H., Langel, K., Ramakers, I., Lemoine, P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047083 · Cited by 17 · see citing works
McCall, M. V., Riva-Posse, P., Garlow, S. J., Mayberg, H. S., & Crowell, A. L. (2020) · Neurology Psychiatry and Brain Research · doi:10.1016/j.npbr.2020.05.002 · Cited by 13 · see citing works
- Authors
- Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., Zhou, F., & De la Torre, F.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Walt Disney (United States) (US)
- Venue
- 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 1-7
- ISSN
- 2156-8103 (ISSN-L)
- OpenAlex
- W2163928333
- Cited by
- 494
- References
- 50
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 32 · 2023: 35 · 2022: 45 · 2021: 29 · 2020: 52 · 2019: 43 · 2018: 45 · 2017: 32 · 2016: 25 · 2015: 22 · 2014: 26 · 2013: 27 · 2012: 11
- FWCI
- 13.28
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Affective Computing and Intelligent Interaction and workshops (publishedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- depression detection, facial action coding system, vocal prosody, facial expressions, voice pitch, active appearance models, major depressive disorder, concurrent validity, machine learning classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
