Brief Personality Measures and Expert Ratings, but Not an Automated Multimodal Stress Index, Relate to Daily Depressive Symptoms in School Employees: An Exploratory Validation Study

Abstract

Background/Objectives: Automatically estimated affective states are increasingly used to monitor mental health, yet such indices are rarely validated against expert clinical judgement of the same recordings. When an automated index fails to match self-report, the source of the failure is ambiguous: the recordings may not contain the relevant signal, or the model may fail to recover a signal that is present. This exploratory study introduced an independent observer standard to separate these possibilities, and examined how expert ratings, brief personality measures and a multimodally estimated stress index each relate to day-to-day self-reported depressive symptoms in school employees without psychiatric diagnosis or treatment contact. Methods: School employees completed daily assessments comprising a fixed three-turn recorded dialogue with a conversational agent followed by the nine-item Patient Health Questionnaire (PHQ-9). Ten participants with baseline trait data and at least four person-days (153 person-days, median 10.5, range 4–44) formed the primary sample; autistic traits (Autism-Spectrum Quotient, AQ) and personality (Japanese Ten Item Personality Inventory, TIPI-J) were measured once at baseline. Three clinicians independently rated 47 recorded sessions from seven participants on Item 1 (Apparent Sadness) of the Montgomery–Åsberg Depression Rating Scale across facial, acoustic, linguistic and overall channels, yielding 41 person-days with a concurrent PHQ-9 and a valid automated index. The index was produced by LightGBM classifiers over speech, facial and linguistic features, trained on binarised PHQ-9/GAD-7 labels from counselling interviews. Associations were tested with permutation-based Spearman correlations, linear mixed models with within-participant permutation tests and cluster bootstrap intervals, and area under the curve (AUC), with false discovery rate (FDR) correction. Results: In the ten-participant trait sample, neuroticism correlated with mean PHQ-9 (ρ = 0.90, q = 0.019) and agreeableness inversely (ρ = −0.84, q = 0.027); both associations persisted when any one participant was omitted. Associations with symptom variability did not survive adjustment for symptom level, to which variability was tightly coupled (ρ = 0.92). In the seven-participant validation subsample, expert ratings of spoken content were associated with same-day PHQ-9 (b = 3.13, cluster bootstrap 95% CI 1.46 to 3.48, permutation p = 0.013) and the association withstood tests for serial dependence, but it was sensitive to which clinicians’ ratings were included, and averaged ratings reached only moderate reliability (ICC(2,3) 0.49–0.56); facial, vocal and overall ratings were not significantly associated. The automated index showed no association with expert ratings after FDR correction (smallest q = 0.22), showed no demonstrable association with concurrent PHQ-9 (withinperson r = 0.08, 95% CI −0.14 to 0.31) and did not discriminate PHQ-9 ≥ 5 above chance (AUC 0.59, 95% CI 0.44–0.77). Only 13.2% of its variance lay between participants, versus 81.3% for PHQ-9. Conclusions: In this small sample, brief personality measures tracked the level of self-reported depressive symptoms, and expert ratings of spoken content were associated with daily symptoms, although the latter association depended on which clinicians’ ratings were included; concurrent validity of the automated index was not demonstrated against either standard. Inspection of the deployed pipeline indicated output saturation arising from a high-sensitivity operating point. Classifier outputs optimised for case detection should not be repurposed as graded severity indices without recalibration. These hypothesis-generating findings require replication in adequately powered samples.