Methodological Gaps in Predicting Mental Health States from Social Media: Triangulating Diagnostic Signals
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
A growing body of research is combining social media data with machine learning to predict mental health states of individuals. An implication of this research lies in informing evidence-based diagnosis and treatment. However, obtaining clinically valid diagnostic information from sensitive patient populations is challenging. Consequently, researchers have operationalized characteristic online behaviors as "proxy diagnostic signals" for building these models. This paper posits a challenge in using these diagnostic signals, purported to support clinical decision-making. Focusing on three commonly used proxy diagnostic signals derived from social media, we find that predictive models built on these data, although offer strong internal validity, suffer from poor external validity when tested on mental health patients. A deeper dive reveals issues of population and sampling bias, as well as of uncertainty in construct validity inherent in these proxies. We discuss the methodological and clinical implications of these gaps and provide remedial guidelines for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Insel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067 · cited 1×
“as the frameworks around mental illnesses evolve, or as the DSM [4], or more recently the National Institute of Mental Health introduced Research Domain Criteria (RDoC) framework [49] offer newer guidelines for mental health diagnostic and treatment.”
Backgroundboyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
The Structured Clinical Interview for DSM-III-R (SCID): I: History, Rationale, and Description
Spitzer, R. L. (1992) · Archives of General Psychiatry · doi:10.1001/archpsyc.1992.01820080032005 · Cited by 4,336 · cited 3×
“Participantswereeligible iftheyhadaprimarypsychoticdisorderlikeschizophrenia, basedonclinicalassessmentscales(e.g.,thePsychiatricDi-agnosticScreeningQuestionnaireorPDSQ[112])aswellasa formalclinicalexaminationfacilitatedbyStructuredClinical InterviewforDSM-5,orSCID[98]—wenotethatallofthese diagnostictoolsarebackedbyso…”
MethodInfluentialHow stigma interferes with mental health care.
Corrigan, P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.7.614 · Cited by 3,648 · cited 1×
“experiences of mental illness [26], some of the proxy diagnostic signals can be leveraged in a respondent-driven sampling framework [44].”
BackgroundLazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
Psychological Aspects of Natural Language Use: Our Words, Our Selves
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145041 · Cited by 2,624 · cited 1×
“A rich body of literature in psycholinguistics has identified the association of linguistic usage to emotion and behavior, including mental health states of individuals [74].”
BackgroundThe Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · cited 3×
“A lack of contextualization in psychiatric practice [57] or theory [7] additionally reduces confidence in their construct validity—an issue recognized in prior critiques of big data approaches [13, 55].”
BackgroundBig Data, new epistemologies and paradigm shifts
Kitchin, R. (2014) · Big Data & Society · doi:10.1177/2053951714528481 · Cited by 2,355 · cited 1×
“However they suffer from significant limitations, which we frame below, drawing upon the critical data literature [13, 51, 55].”
BackgroundImproving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678
Dataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553 · cited 2×
“The other methodological gap we identify in the use of the proxy diagnostic signals for predicting clinical diagnoses relates to dataset shift [99].”
BackgroundDenzin, N. K. (2012) · Journal of Mixed Methods Research · doi:10.1177/1558689812437186 · Cited by 1,444 · cited 2×
“…methods, or data sources compiled via varied mechanisms, to develop a comprehensive understanding of a phenomenon, or to elucidate its complementary aspects [73], Specifically, this approach is used to confirm the results of a research, and provide external validation to existing findings [35].”
ResultFacebook Profiles Reflect Actual Personality, Not Self-Idealization
Back, M. D., Stopfer, J. M., Vazire, S., Gaddis, S., Schmukle, S. C., Egloff, B., & Gosling, S. D. (2010) · Psychological Science · doi:10.1177/0956797609360756 · Cited by 1,169 · cited 1×
“With rising volumes of data and pervasive use, social media has been widely adopted as a lens to provide insights into behaviors [52], mood [42], psychological traits and states [5, 53], andsocialinteractionsofindividuals[56].”
BackgroundDiurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures
Golder, S. A., & Macy, M. W. (2011) · Science · doi:10.1126/science.1202775 · Cited by 1,155
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802 · cited 3×
“In this line of research, on the methodological front, supervised machine learning techniques have gained prominence, providing promising predictive outcomes of mental health states [66].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · cited 3×
“discuss at length surrounding methodological pitfalls of big data [70].”
BackgroundInfluentialSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Quantifying Mental Health Signals in Twitter
Coppersmith, G., Dredze, M., & Harman, C. (2014) · Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w14-3207 · Cited by 722 · cited 6×
“Formental health, a growing body of work, including that in the human computer interaction (HCI) field, is leveraging naturalistic, unobtrusive data from social media to predict mental health states of individuals [21, 25, 28, 29, 31, 34].”
BackgroundInfluentialInstagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519
Natural language processing of social media as screening for suicide risk
Coppersmith, G., Leary, R., Crutchley, P., & Fine, A. (2018) · Biomedical Informatics Insights · doi:10.1177/1178222618792860 · Cited by 506
Social media as a measurement tool of depression in populations
De Choudhury, M., Counts, S., & Horvitz, E. (2013) · Proceedings of the 5th Annual ACM Web Science Conference · doi:10.1145/2464464.2464480 · Cited by 497
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“Research in the HCI field has been instrumental in surfacing the many challenges of transplanting algorithms built with biased data, lacking theoretical and domain-specific validity, in real-world contexts [88, 104].”
BackgroundForecasting the onset and course of mental illness with Twitter data
Reece, A. G., Reagan, A. J., Lix, K. L. M., Dodds, P. S., Danforth, C. M., & Langer, E. J. (2017) · Scientific Reports · doi:10.1038/s41598-017-12961-9 · Cited by 401
Andalibi, N., Haimson, O. L., De Choudhury, M., & Forte, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858096 · Cited by 378 · cited 1×
“Research investigations have ranged from inferring risk tovariousmental illnesses [24, 29, 31, 34]—the largest body of work in this area; comparison of online and offline mental health behaviors [84]; understanding self-disclosure practices and goals [2, 39]; deciphering social support provisions to promote positive m…”
BackgroundMeasuring post traumatic stress disorder in Twitter
Coppersmith, G., Harman, C., & Dredze, M. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14574 · Cited by 316 · cited 2×
“Self-reports: Next, the most popular form of proxy diagnostic signals, this category operationalizes first-hand, public self-disclosures of diagnosis of a mental illness as indicators of a clinical mental illness [8, 15, 21–23, 25, 32, 46, 47, 59– 61, 65, 69, 76, 80, 93, 97, 105–107].”
BackgroundTowards Assessing Changes in Degree of Depression through Facebook
Schwartz, H. A., Eichstaedt, J., Kern, M. L., Park, G., Sap, M., Stillwell, D., Kosinski, M., & Ungar, L. (2014) · Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w14-3214 · Cited by 256 · cited 1×
“External validation: Finally, this category represents humanin-the-loop, collaborative approaches that either seek selfreported information from the individual, or incorporate diagnostic scales and/or expert appraisal for identification of the proxy diagnostic signals [11, 14, 17, 32, 71, 78, 79, 81, 84, 90, 103].”
BackgroundThe Language of Social Support in Social Media and Its Effect on Suicidal Ideation Risk
De Choudhury, M., & Kiciman, E. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14891 · Cited by 202 · cited 1×
“Research investigations have ranged from inferring risk tovariousmental illnesses [24, 29, 31, 34]—the largest body of work in this area; comparison of online and offline mental health behaviors [84]; understanding self-disclosure practices and goals [2, 39]; deciphering social support provisions to promote positive m…”
BackgroundQuantifying the Language of Schizophrenia in Social Media
Mitchell, M., Hollingshead, K., & Coppersmith, G. (2015) · Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w15-1202 · Cited by 172
Birnbaum, M. L., Ernala, S. K., Rizvi, A. F., De Choudhury, M., & Kane, J. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7956 · Cited by 168 · cited 5×
“prior literature [11, 63, 65]: 1) behaviors signaling affiliation to mental health resources, 2) self-reported diagnoses, and 3) clinically appraised self-reports of diagnoses.”
BackgroundMethodInfluentialSelf-citationMental Health Support and its Relationship to Linguistic Accommodation in Online Communities
Sharma, E., & De Choudhury, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174215 · Cited by 156 · cited 1×
“Research investigations have ranged from inferring risk tovariousmental illnesses [24, 29, 31, 34]—the largest body of work in this area; comparison of online and offline mental health behaviors [84]; understanding self-disclosure practices and goals [2, 39]; deciphering social support provisions to promote positive m…”
BackgroundPREDICTING INDIVIDUAL WELL-BEING THROUGH THE LANGUAGE OF SOCIAL MEDIA
SCHWARTZ, H. A., SAP, M., KERN, M. L., EICHSTAEDT, J. C., KAPELNER, A., AGRAWAL, M., BLANCO, E., DZIURZYNSKI, L., PARK, G., STILLWELL, D., et al. (2016) · Biocomputing 2016 · doi:10.1142/9789814749411_0047 · Cited by 148 · cited 1×
“Since social media behaviors are a reliable indicator of people’s personality, psychological states, and even demographic attributes [91], we included the following covariates for the purpose of matching: total number of statuses, chosen language on Twitter, total number of followers and total number of followees.”
BackgroundModeling Stress with Social Media Around Incidents of Gun Violence on College Campuses
Saha, K., & De Choudhury, M. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134727 · Cited by 124 · cited 3×
“[41], Saha and De Choudhury [85], Chancellor et al.”
BackgroundLinguistic Markers Indicating Therapeutic Outcomes of Social Media Disclosures of Schizophrenia
Ernala, S. K., Rizvi, A. F., Birnbaum, M. L., Kane, J. M., & De Choudhury, M. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134678 · Cited by 99 · cited 2×
“Although these tokens reveal symptomatic expression, spirituality and supportseeking behaviors, notable in schizophrenia disclosuresmade on social media [39], the current example demonstrates varied usage of these tokens by healthy controls, in reference to pop-culture or in casual conversations.”
BackgroundSelf-citationInferring Mood Instability on Social Media by Leveraging Ecological Momentary Assessments
Saha, K., Chan, L., De Barbaro, K., Abowd, G. D., & De Choudhury, M. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130960 · Cited by 77
Cited by
Methods in predictive techniques for mental health status on social media: a critical review
Chancellor, S., & De Choudhury, M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0233-7 · Cited by 557
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
“[53] Sindhu Kiranmai Ernala, Michael L.”
From Treatment to Healing:Envisioning a Decolonial Digital Mental Health
Pendse, S. R., Nkemelu, D., Bidwell, N. J., Jadhav, S., Pathare, S., De Choudhury, M., & Kumar, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501982 · Cited by 115
Self-citationGooding, P., & Kariotis, T. (2021) · JMIR Mental Health · doi:10.2196/24668 · Cited by 100
“” Similarly, Ernala et al [73] noted that the field could benefit extensively from cross-disciplinary partnerships and partnerships between “computational and clinical researchers, and patients.”
BackgroundInfluentialA Social Media Study on the Effects of Psychiatric Medication Use
Saha, K., Sugar, B., Torous, J., Abrahao, B., Kıcıman, E., & De Choudhury, M. (2019) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v13i01.3242 · Cited by 96
“We cannot verify if self-reports of medication use corresponded to their actual use (Ernala et al. 2019).”
Background
Metadata
- Authors
- Ernala, S. K., Birnbaum, M. L., Candan, K. A., Rizvi, A. F., Sterling, W. A., Kane, J. M., & De Choudhury, M.
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-16
- OpenAlex
- W2940817972
- Cited by
- 106
- References
- 34
- FWCI
- 25.38
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.