The DOI matches its Crossref record; 31 of the 36 references it cites were found in a registry or an index.
A Collaborative Approach to Identifying Social Media Markers of Schizophrenia by Employing Machine Learning and Clinical Appraisals
Open accessAbstract
BACKGROUND: Linguistic analysis of publicly available Twitter feeds have achieved success in differentiating individuals who self-disclose online as having schizophrenia from healthy controls. To date, limited efforts have included expert input to evaluate the authenticity of diagnostic self-disclosures. OBJECTIVE: This study aims to move from noisy self-reports of schizophrenia on social media to more accurate identification of diagnoses by exploring a human-machine partnered approach, wherein computational linguistic analysis of shared content is combined with clinical appraisals. METHODS: Twitter timeline data, extracted from 671 users with self-disclosed diagnoses of schizophrenia, was appraised for authenticity by expert clinicians. Data from disclosures deemed true were used to build a classifier aiming to distinguish users with schizophrenia from healthy controls. Results from the classifier were compared to expert appraisals on new, unseen Twitter users. RESULTS: Significant linguistic differences were identified in the schizophrenia group including greater use of interpersonal pronouns (P<.001), decreased emphasis on friendship (P<.001), and greater emphasis on biological processes (P<.001). The resulting classifier distinguished users with disclosures of schizophrenia deemed genuine from control users with a mean accuracy of 88% using linguistic data alone. Compared to clinicians on new, unseen users, the classifier's precision, recall, and accuracy measures were 0.27, 0.77, and 0.59, respectively. CONCLUSIONS: These data reinforce the need for ongoing collaborations integrating expertise from multiple fields to strengthen our ability to accurately identify and effectively engage individuals with mental illness online. These collaborations are crucial to overcome some of mental illnesses' biggest challenges by using digital technology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies have reported that individuals living with a range of mental disorders, including depression, psychotic disorders, or other severe mental illnesses, use social media platforms at comparable rates as the general population”
From Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice (2020) · cites this paper as M. L. Birnbaum et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For adolescents and young adults ages 12 to 21 with psychotic disorders and mood disorders, over 97% reported using social media, with average use exceeding 2.5 hours per day (M. L. Birnbaum et al., 2017).”
From Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice (2020) · cites this paper as M. L. Birnbaum et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Several studies have also demonstrated that when compared to a control group, Twitter users with a self-disclosed diagnosis of schizophrenia show unique online communication patterns (Michael L Birnbaum, Ernala, Rizvi, De Choudhury, & Kane, 2017), including more frequent discussion of tobacco use (Hswen et al., 2017), symptoms of depression and anxiety (Hswen, Naslund, Brownstein, & Hawkins, 2018b”
From Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice (2020) · cites this paper as Michael L Birnbaum, Ernala, Rizvi, De Choudhury, & Kane, 2017
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“connecting with mental health providers (M. L. Birnbaum et al., 2017)”
From Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice (2020) · cites this paper as M. L. Birnbaum et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066 · cited 2×
“The classifier was built and evaluated by applying 10-fold cross-validation, an established technique in supervised machine learning [25].”
MethodPersonality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700
Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak
Chew, C., & Eysenbach, G. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014118 · Cited by 1,614
Perkins, D., Gu, H., Boteva, K., & Lieberman, J. (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.10.1785 · Cited by 1,304 · cited 1×
“Results from the Recovery After Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) trial [31] suggest that the median duration of untreated psychosis is 74 weeks [32] and support the established hypothesis that lengthy duration of untreated psychosis (DUP) leads to worse outcomes [31,33].”
BackgroundKane, J. M., Robinson, D. G., Schooler, N. R., Mueser, K. T., Penn, D. L., Rosenheck, R. A., Addington, J., Brunette, M. F., Correll, C. U., Estroff, S. E., et al. (2015) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2015.15050632 · Cited by 896 · cited 1×
“Results from the Recovery After Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) trial [31] suggest that the median duration of untreated psychosis is 74 weeks [32] and support the established hypothesis that lengthy duration of untreated psychosis (DUP) leads to worse outcomes [31,33].”
BackgroundSelf-citationQuantifying Mental Health Signals in Twitter
Coppersmith, G., Dredze, M., & Harman, C. (2014) · Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w14-3207 · Cited by 722 · cited 2×
“Prior work identifying markers of mental illness online used similar filtering techniques based on self-reported diagnoses [22,23].”
BackgroundMethodChristofides, E., Muise, A., & Desmarais, S. (2009) · CyberPsychology & Behavior · doi:10.1089/cpb.2008.0226 · Cited by 696 · cited 1×
“More than 90% of US youth use social media daily [1], placing it ahead of texting, email, and instant messaging, and they disclose considerably more about themselves online than offline [2].”
BackgroundAutomated analysis of free speech predicts psychosis onset in high-risk youths
Bedi, G., Carrillo, F., Cecchi, G. A., Slezak, D. F., Sigman, M., Mota, N. B., Ribeiro, S., Javitt, D. C., Copelli, M., & Corcoran, C. M. (2015) · npj Schizophrenia · doi:10.1038/npjschz.2015.30 · Cited by 673 · cited 2×
“Furthermore, classifiers designed to automatically sort individual cases into diagnostic categories have achieved success in recognizing participants with psychotic disorders from healthy controls based on linguistic differences in writing samples [15] and speech [13,18].”
BackgroundResultSocial media as a measurement tool of depression in populations
De Choudhury, M., Counts, S., & Horvitz, E. (2013) · Proceedings of the 5th Annual ACM Web Science Conference · doi:10.1145/2464464.2464480 · Cited by 497
Broniatowski, D. A., Paul, M., & Dredze, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083672 · Cited by 474
Predicting postpartum changes in emotion and behavior via social media
Choudhury, M. D., Counts, S., & Horvitz, E. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466447 · Cited by 472
Tracking suicide risk factors through Twitter in the US.
Jashinsky, J., Burton, S. H., Hanson, C., West, J. H., Giraud-Carrier, C., Barnes, M. D., & Argyle, T. (2014) · Crisis · doi:10.1027/0227-5910/a000234 · Cited by 363
Measuring post traumatic stress disorder in Twitter
Coppersmith, G., Harman, C., & Dredze, M. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14574 · Cited by 316 · cited 1×
“Main Findings These data contribute to a growing body of literature using language to automatically identify individuals online who may be experiencing mental illness, including depression [16,22,27], postpartum mood disorders [28], suicide [29], posttraumatic stress disorder [30], and bipolar disorder [23].”
BackgroundAffective and Content Analysis of Online Depression Communities
Nguyen, T., Phung, D., Dao, B., Venkatesh, S., & Berk, M. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2315623 · Cited by 231 · cited 1×
“Main Findings These data contribute to a growing body of literature using language to automatically identify individuals online who may be experiencing mental illness, including depression [16,22,27], postpartum mood disorders [28], suicide [29], posttraumatic stress disorder [30], and bipolar disorder [23].”
Background#WhyWeTweetMH: Understanding why people use twitter to discuss mental health problems
Berry, N., Lobban, F., Belousov, M., Emsley, R., Nenadic, G., & Bucci, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6173 · Cited by 228 · cited 2×
“Identified benefits include developing a sense of belonging, establishing and maintaining relationships, accessing support, challenging stigma, raising awareness, and sharing experiences [4,5].”
BackgroundQuantifying the Language of Schizophrenia in Social Media
Mitchell, M., Hollingshead, K., & Coppersmith, G. (2015) · Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w15-1202 · Cited by 172 · cited 1×
“Language-based computational models have achieved more than 80% and 90% accuracy [19,20] in correctly identifying users with self-reported schizophrenia from healthy controls.”
BackgroundKass‐Hout, T., & Alhinnawi, H. (2013) · British Medical Bulletin · doi:10.1093/bmb/ldt028 · Cited by 148 · cited 1×
“There is now clear and convincing evidence that online activity can be used to reliably monitor and predict health-related behaviors [7] ranging from the spread of the influenza virus across the United States to rates of seasonal allergies, HIV infection, cancer, smoking, and obesity [8-10].”
BackgroundOnline social networking in people with psychosis: A systematic review
Highton-Williamson, E., Priebe, S., & Giacco, D. (2015) · International Journal of Social Psychiatry · doi:10.1177/0020764014556392 · Cited by 139 · cited 1×
“Identified benefits include developing a sense of belonging, establishing and maintaining relationships, accessing support, challenging stigma, raising awareness, and sharing experiences [4,5].”
BackgroundBirnbaum, M. L., Rizvi, A. F., Correll, C. U., Kane, J. M., & Confino, J. (2015) · Early Intervention in Psychiatry · doi:10.1111/eip.12237 · Cited by 138 · cited 1×
“Youth with newly diagnosed schizophrenia in particular report frequently utilizing social networking sites throughout the course of illness development and treatment, engaging in social media activity several times daily, and spending several hours per day online [6].”
BackgroundSelf-citationDuration of Untreated Psychosis in Community Treatment Settings in the United States.
Addington, J., Heinssen, R., Robinson, D., Schooler, N., Marcy, P., Brunette, M. F., Correll, C., Estroff, S., Mueser, K., Penn, D., et al. (2015) · Psychiatric Services · doi:10.1176/appi.ps.201400124 · Cited by 121 · cited 1×
“Results from the Recovery After Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) trial [31] suggest that the median duration of untreated psychosis is 74 weeks [32] and support the established hypothesis that lengthy duration of untreated psychosis (DUP) leads to worse outcomes [31,33].”
BackgroundSelf-citationLinguistic dimensions of psychopathology: a quantitative analysis
Junghaenel, D. U., Smyth, J. M., & Santner, L. (2008) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2008.27.1.36 · Cited by 74 · cited 1×
“Comparison With Prior Work Consistent with prior trials [11-15,18,36], first-person pronouns were found to be significantly increased in the psychosis group, suggesting greater interpersonal focus.”
ResultMinor, K. S., Bonfils, K., Luther, L., Firmin, R., Kukla, M., MacLain, V. R., Buck, B., Lysaker, P., & Salyers, M. P. (2015) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2015.02.024 · Cited by 69 · cited 2×
“Prior work in speech and text analysis has identified reliable linguistic markers associated with schizophrenia, including significant differences in word frequency, word categories, and use of self-referential pronouns [11-15].”
BackgroundResultBuck, B., & Penn, D. (2015) · Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000354 · Cited by 64 · cited 2×
“Prior work in speech and text analysis has identified reliable linguistic markers associated with schizophrenia, including significant differences in word frequency, word categories, and use of self-referential pronouns [11-15].”
BackgroundResultBehavioral insights on big data: using social media for predicting biomedical outcomes
Young, S. (2014) · Trends in Microbiology · doi:10.1016/j.tim.2014.08.004 · Cited by 64 · cited 1×
“There is now clear and convincing evidence that online activity can be used to reliably monitor and predict health-related behaviors [7] ranging from the spread of the influenza virus across the United States to rates of seasonal allergies, HIV infection, cancer, smoking, and obesity [8-10].”
BackgroundSelf-reference in psychosis and depression: a language marker of illness
Fineberg, S., Leavitt, J., Deutsch-Link, S., Dealy, S., Landry, C. D., Pirruccio, K., Shea, S., Trent, S., Cecchi, G. A., & Corlett, P. (2016) · Psychological Medicine · doi:10.1017/s0033291716001215 · Cited by 61 · cited 2×
“Additionally, these data replicate findings that biological processes, including words such as “body” and “health,” are more frequently used in psychosis [17], suggesting a greater awareness or focus on health status.”
BackgroundFacebook Displays as Predictors of Binge Drinking
D’Angelo, J. D., Kerr, B. R., & Moreno, M. A. (2014) · Bulletin of Science Technology & Society · doi:10.1177/0270467615584044 · Cited by 57 · cited 1×
“At the same time, there is compelling evidence to suggest that linguistic and behavioral changes manifest on the pages of social media before they are clinically detected, providing the prospect for earlier intervention [22,28,34].”
BackgroundHong, K., Nenkova, A., March, M. E., Parker, A. P., Verma, R., & Kohler, C. G. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.10.002 · Cited by 57 · cited 3×
“Prior work in speech and text analysis has identified reliable linguistic markers associated with schizophrenia, including significant differences in word frequency, word categories, and use of self-referential pronouns [11-15].”
BackgroundResultAutomated Characterization and Identification of Schizophrenia in Writing
Strous, R. D., Koppel, M., Fine, J. B., Nachliel, S., Shaked, G., & Zivotofsky, A. Z. (2009) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0b013e3181b09068 · Cited by 53 · cited 3×
“Prior work in speech and text analysis has identified reliable linguistic markers associated with schizophrenia, including significant differences in word frequency, word categories, and use of self-referential pronouns [11-15].”
BackgroundResultF120. USING DIGITAL MEDIA ADVERTISING IN EARLY PSYCHOSIS INTERVENTION
Birnbaum, M., Garrett, C., Baumel, A., Scovel, M., Rizvi, A., Muscat, W., & Kane, J. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600571 · Cited by 38 · cited 1×
“Recently, Birnbaum et al [35] used Google AdWords to explore aspects of digital advertising most effective at engaging individuals online.”
MethodSelf-citationBirnbaum, M., Candan, K. A., Libby, I., Pascucci, O., & Kane, J. (2016) · Early Intervention in Psychiatry · doi:10.1111/eip.12179 · Cited by 35 · cited 1×
“Furthermore, prior work has demonstrated that words that might have been automatically identified as self-disclosure such as “psychosis,” schizophrenia,” and “delusion” are often used inappropriately online [21] and may represent a major limitation to these computational models.”
BackgroundSelf-citationBuck, B., Minor, K. S., & Lysaker, P. H. (2015) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2014.12.015 · Cited by 30 · cited 2×
“Prior work in speech and text analysis has identified reliable linguistic markers associated with schizophrenia, including significant differences in word frequency, word categories, and use of self-referential pronouns [11-15].”
BackgroundResult
Research works printed but not linked: 3
- McManus, K, 2015, AMIA Jt Summits Transl Sci Proc, 2015, 122
- Bishop, CM, 2006, Pattern Recognition and Machine Learning
- CoppersmithGHarmanCDredzeMMeasuring post traumatic stress disorder in TwitterProceedings of the International AAAI Conference on Web and Social Media Eighth International AAAI Conference on Weblogs and Social Media2014Eighth International AAAI Conference on Weblogs and Social MediaJun 1-4, 2014Ann Arbor, MI16
Cited by
Sorted by citations
Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice
Naslund, J. A., Bondre, A., Torous, J., & Aschbrenner, K. A. (2020) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-020-00134-x · Cited by 746
Methods in predictive techniques for mental health status on social media: a critical review
Chancellor, S., & De Choudhury, M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0233-7 · Cited by 557
AI in Health: State of the Art, Challenges, and Future Directions
Wang, F., & Preininger, A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677908 · Cited by 398
“Researchers have examined social media analytics and mental health, and they identified markers in social media activity associated with worsening psychotic symptoms [57], schizophrenia [58], risk of suicidal ideation [59], and depression [60].”
BackgroundLeveraging Large Language Models for Mental Health Prediction via Online Text DataPreprint
Xu, X., Yao, B., Dong, Y.-Z., Yu, H., Hendler, J. A., Dey, A., & Wang, D.-K. (2023) · arXiv.org · doi:10.48550/arxiv.2307.14385 · Cited by 291
“Since then, researchers have proposed a wide range of feature engineering methods and built machine-learning models for the prediction [13, 67, 69, 88, 103].”
BackgroundMental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data
Xu, X., Yao, B., Dong, Y., Gabriel, S., Yu, H., Hendler, J., Ghassemi, M., Dey, A. K., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3643540 · Cited by 278
“Since then, researchers have proposed a wide range of feature engineering methods and built machine-learning models for the prediction [13, 67, 69, 88, 103].”
Background
Metadata
- Authors
- Birnbaum, M. L., Ernala, S. K., Rizvi, A. F., De Choudhury, M., & Kane, J. M.
- Affiliations
- Northwell Health (US)
- Georgia Institute of Technology (US)
- Corresponding author
- Birnbaum, Michael L
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2017-08-14
- Volume
- 19
- Issue
- 8
- Pages
- e289
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5575421
- OpenAlex
- W2748635631
- Cited by
- 168
- References
- 36
- Citations by year
- 2026: 6 · 2025: 20 · 2024: 20 · 2023: 17 · 2022: 22 · 2021: 30 · 2020: 23 · 2019: 21 · 2018: 8 · 2017: 1
- FWCI
- 20.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of General Medical Sciences (R01GM112697)
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- schizophrenia, social media, Twitter, linguistic analysis, self-disclosure, authenticity assessment, clinical evaluation, human-machine collaboration, mental disorder detection, friendship, biological processes, online engagement
- MeSH terms
- Machine Learning, Humans, Schizophrenia, Social Behavior, Social Support, Internet, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
