CoChat

Natural Language Processing Reveals Vulnerable Mental Health Support Groups and Heightened Health Anxiety on Reddit During COVID-19: Observational Study

Open access
Low, Daniel M, Rumker, Laurie, Talkar, Tanya, Torous, John, Cecchi, Guillermo, Ghosh, Satrajit S · 2020 · Journal of Medical Internet Research

Published version of a preprint

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Background: The COVID-19 pandemic is impacting mental health, but it is not clear how people with different types of mental health problems were differentially impacted as the initial wave of cases hit. Objective: The aim of this study is to leverage natural language processing (NLP) with the goal of characterizing changes in 15 of the world’s largest mental health support groups (eg, r/schizophrenia, r/SuicideWatch, r/Depression) found on the website Reddit, along with 11 non–mental health groups (eg, r/PersonalFinance, r/conspiracy) during the initial stage of the pandemic. Methods: We created and released the Reddit Mental Health Dataset including posts from 826,961 unique users from 2018 to 2020. Using regression, we analyzed trends from 90 text-derived features such as sentiment analysis, personal pronouns, and semantic categories. Using supervised machine learning, we classified posts into their respective support groups and interpreted important features to understand how different problems manifest in language. We applied unsupervised methods such as topic modeling and unsupervised clustering to uncover concerns throughout Reddit before and during the pandemic. Results: We found that the r/HealthAnxiety forum showed spikes in posts about COVID-19 early on in January, approximately 2 months before other support groups started posting about the pandemic. There were many features that significantly increased during COVID-19 for specific groups including the categories “economic stress,” “isolation,” and “home,” while others such as “motion” significantly decreased. We found that support groups related to attention-deficit/hyperactivity disorder, eating disorders, and anxiety showed the most negative semantic change during the pandemic out of all mental health groups. Health anxiety emerged as a general theme across Reddit through independent supervised and unsupervised machine learning analyses. For instance, we provide evidence that the concerns of a diverse set of individuals are converging in this unique moment of history; we discovered that the more users posted about COVID-19, the more linguistically similar (less distant) the mental health support groups became to r/HealthAnxiety (ρ=–0.96, P<.001). Using unsupervised clustering, we found the suicidality and loneliness clusters more than doubled in the number of posts during the pandemic. Specifically, the support groups for borderline personality disorder and posttraumatic stress disorder became significantly associated with the suicidality cluster. Furthermore, clusters surrounding self-harm and entertainment emerged. Conclusions: By using a broad set of NLP techniques and analyzing a baseline of prepandemic posts, we uncovered patterns of how specific mental health problems manifest in language, identified at-risk users, and revealed the distribution of concerns across Reddit, which could help provide better resources to its millions of users. We then demonstrated that textual analysis is sensitive to uncover mental health complaints as they appear in real time, identifying vulnerable groups and alarming themes during COVID-19, and thus may have utility during the ongoing pandemic and other world-changing events such as elections and protests.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/22635

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Mention of COVID-19–related words across mental health support groups. Timeline landmarks were chosen from NBC News timeline given that US users are the most prevalent across Reddit. Global, China, and US confirmed COVID-19 cases are displayed. The overall acute rise in COVID-19–related words occurs on March 11, 2020. The correlation between the mean proportion of COVID-19–related posts and global COVID-19 cases is ρ=0.83 (P<.001). The health anxiety subreddit has a large increase in COVID-19–related posts almost 2 months before the general increase. r/alcoholism has the most amount of posts related to COVID-19 on March 27. adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDAnonymous: Eating Disorders Anonymous; ptsd: posttraumatic stress disorder; WHO: World Health Organization.
Figure 1 Mention of COVID-19–related words across mental health support groups. Timeline landmarks were chosen from NBC News timeline given that US users are the most prevalent across Reddit. Global, China, and US confirmed COVID-19 cases are displayed. The overall acute rise in COVID-19–related words occurs on March 11, 2020. The correlation between the mean proportion of COVID-19–related posts and global COVID-19 cases is ρ=0.83 (P<.001). The health anxiety subreddit has a large increase in COVID-19–related posts almost 2 months before the general increase. r/alcoholism has the most amount of posts related to COVID-19 on March 27. adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDAnonymous: Eating Disorders Anonymous; ptsd: posttraumatic stress disorder; WHO: World Health Organization.CC BY · PubMed Central
Figure 2 Trend analysis of linguistic features over time. A) Significant change in average feature values from January to April 2020 and 2019 across subreddits. COVID19 support subreddit was created in 2020 and, therefore, does not appear in 2019. Features important for classifying a subreddit are added to the y-axis. Change is defined by slope x R^2 (ie, increases have a positive slope and tend toward red, decreases have a negative slope and tend toward blue). Significant trends after multiple comparison correction on full results are displayed. There is significantly more absolute change in 2020 than in 2019 (P<.001) and 2018 (P<.001). B) Rank of subreddits by the amount of negative semantic change throughout COVID-19 (January 1, 2020, to April 20, 2020) across significant full results. In bold are the mental health subreddits with the most negative semantic change using the following features with emotional valence: negative sentiment; the lexicons about economic stress, isolation, substance use, guns, domestic stress, and suicidality; LIWC measures of anger, anxiety, death, negations, negative emotion, and sadness; and three positive features inversely weighed, compound sentiment, positive sentiment, and positive emotion. r/ptsd and r/conspiracy decreased in negative semantic features. adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDanonymous: Eating Disorders Anonymous; LIWC: Linguistic Inquiry and Word Count; ptsd: posttraumatic stress disorder.
Figure 2 Trend analysis of linguistic features over time. A) Significant change in average feature values from January to April 2020 and 2019 across subreddits. COVID19 support subreddit was created in 2020 and, therefore, does not appear in 2019. Features important for classifying a subreddit are added to the y-axis. Change is defined by slope x R^2 (ie, increases have a positive slope and tend toward red, decreases have a negative slope and tend toward blue). Significant trends after multiple comparison correction on full results are displayed. There is significantly more absolute change in 2020 than in 2019 (P<.001) and 2018 (P<.001). B) Rank of subreddits by the amount of negative semantic change throughout COVID-19 (January 1, 2020, to April 20, 2020) across significant full results. In bold are the mental health subreddits with the most negative semantic change using the following features with emotional valence: negative sentiment; the lexicons about economic stress, isolation, substance use, guns, domestic stress, and suicidality; LIWC measures of anger, anxiety, death, negations, negative emotion, and sadness; and three positive features inversely weighed, compound sentiment, positive sentiment, and positive emotion. r/ptsd and r/conspiracy decreased in negative semantic features. adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDanonymous: Eating Disorders Anonymous; LIWC: Linguistic Inquiry and Word Count; ptsd: posttraumatic stress disorder.CC BY · PubMed Central
Figure 3 Unsupervised clustering reveals post groupings with representation across mental health subreddits. A) Unsupervised clustering of pre-pandemic (year 2019) posts from 15 mental health subreddits, presented in 2D UMAP space. Posts in two thematically-related, adjacent clusters were collapsed into a single “Resources” cluster. Three clusters could not be assigned identifiable themes. Two of these—annotated as ”Unspecified”—were the largest clusters in the dataset, containing 6329 and 4272 total posts, respectively, while the next largest cluster contained 1620 posts. The other cluster without an identifiable theme was characterized by very long posts. This ”Long Posts” cluster had an average post length of 886 words, while the cluster with the next most lengthy posts had an average post length of 554 words. As a result, this “Long Posts” cluster had an overwhelming number of cluster-characteristic text features, which made any core linguistic theme poorly discernible. The identified clusters were not an approximation of post subreddit of origin, as demonstrated by several metrics quantifying the lack of correspondence between cluster labels and post subreddit of origin: Homogeneity (0.20), Completeness (0.22), V-measure (0.21), and Adjusted Rand-Index (0.08). B) Unsupervised clustering of mid-pandemic posts using the same process resulted primarily in replication of cluster annotations observed in the pre-pandemic data, with a few clusters (e.g., Seeking Advice) detected only in the pre-pandemic clustering and a few (e.g., Entertainment) detected only in the mid-pandemic clustering. Two clusters increased notably in size in the mid-pandemic clustering: Suicidality (204% increase in number of posts) and Loneliness (233% increase in number of posts). C) Enrichment of clusters on mental health subreddits during the pre-pandemic period and the mid-pandemic period, using clusters detected during each time period, respectively. Associations were assessed with hypergeometric tests, and those displayed here passed strict Bonferroni correction for multiple hypothesis testing. Associations present only for the pre-pandemic or only for the mid-pandemic time period are shown in bold.
Figure 3 Unsupervised clustering reveals post groupings with representation across mental health subreddits. A) Unsupervised clustering of pre-pandemic (year 2019) posts from 15 mental health subreddits, presented in 2D UMAP space. Posts in two thematically-related, adjacent clusters were collapsed into a single “Resources” cluster. Three clusters could not be assigned identifiable themes. Two of these—annotated as ”Unspecified”—were the largest clusters in the dataset, containing 6329 and 4272 total posts, respectively, while the next largest cluster contained 1620 posts. The other cluster without an identifiable theme was characterized by very long posts. This ”Long Posts” cluster had an average post length of 886 words, while the cluster with the next most lengthy posts had an average post length of 554 words. As a result, this “Long Posts” cluster had an overwhelming number of cluster-characteristic text features, which made any core linguistic theme poorly discernible. The identified clusters were not an approximation of post subreddit of origin, as demonstrated by several metrics quantifying the lack of correspondence between cluster labels and post subreddit of origin: Homogeneity (0.20), Completeness (0.22), V-measure (0.21), and Adjusted Rand-Index (0.08). B) Unsupervised clustering of mid-pandemic posts using the same process resulted primarily in replication of cluster annotations observed in the pre-pandemic data, with a few clusters (e.g., Seeking Advice) detected only in the pre-pandemic clustering and a few (e.g., Entertainment) detected only in the mid-pandemic clustering. Two clusters increased notably in size in the mid-pandemic clustering: Suicidality (204% increase in number of posts) and Loneliness (233% increase in number of posts). C) Enrichment of clusters on mental health subreddits during the pre-pandemic period and the mid-pandemic period, using clusters detected during each time period, respectively. Associations were assessed with hypergeometric tests, and those displayed here passed strict Bonferroni correction for multiple hypothesis testing. Associations present only for the pre-pandemic or only for the mid-pandemic time period are shown in bold.CC BY · PubMed Central
Figure 4 Latent dirichlet allocation (LDA) reveals prominent topics in mental health subreddits. A) Distribution of midpandemic posts from 15 mental health subreddits across 10 topics extracted using LDA on prepandemic mental health subreddit posts. Topic distribution was assessed for midpandemic posts between March 16, 2020, and April 20, 2020, to capture the phase of the pandemic right after stay-at-home orders had been announced or enacted for many areas in the United States. Inspection of the topic distribution indicated that there was minimal shift in most topics for all subreddits between the pre- and midpandemic time frames. We tested changes in topic distributions across all 27 subreddits using a Wilcoxon signed rank test (COVID19_support was not available during 2019). B) Manually labelled topics and the top 10 terms associated with each topic derived from an LDA model created on midpandemic subreddit posts. ADHD: attention-deficit/hyperactivity disorder; PTSD: posttraumatic stress disorder.
Figure 4 Latent dirichlet allocation (LDA) reveals prominent topics in mental health subreddits. A) Distribution of midpandemic posts from 15 mental health subreddits across 10 topics extracted using LDA on prepandemic mental health subreddit posts. Topic distribution was assessed for midpandemic posts between March 16, 2020, and April 20, 2020, to capture the phase of the pandemic right after stay-at-home orders had been announced or enacted for many areas in the United States. Inspection of the topic distribution indicated that there was minimal shift in most topics for all subreddits between the pre- and midpandemic time frames. We tested changes in topic distributions across all 27 subreddits using a Wilcoxon signed rank test (COVID19_support was not available during 2019). B) Manually labelled topics and the top 10 terms associated with each topic derived from an LDA model created on midpandemic subreddit posts. ADHD: attention-deficit/hyperactivity disorder; PTSD: posttraumatic stress disorder.CC BY · PubMed Central
Figure 5 Characterization of r/COVID19_support through supervised and unsupervised methods. A) Proportion of r/COVID19_support posts (March 11 to April 20, 2020) that each binary classifier trained on prepandemic data detects. B) Distribution of prepandemic model topics (left) and midpandemic model topics (right) for posts in r/COVID19_support, highlighting prominent topics in the posts, such as health anxiety and issues in school, work, and home scenarios. The distribution of topics indicate common themes of pain points, which could help guide the medium and content of mental health resources. C) Distribution of unsupervised cluster representation among posts from r/COVID19_support. Although many posts were assigned to unspecified clusters, the substantial portion of posts assigned to the suicidality cluster is notable. ADHD: attention-deficit/hyperactivity disorder; EDanonymous: Eating Disorders Anonymous; PTSD: posttraumatic stress disorder.
Figure 5 Characterization of r/COVID19_support through supervised and unsupervised methods. A) Proportion of r/COVID19_support posts (March 11 to April 20, 2020) that each binary classifier trained on prepandemic data detects. B) Distribution of prepandemic model topics (left) and midpandemic model topics (right) for posts in r/COVID19_support, highlighting prominent topics in the posts, such as health anxiety and issues in school, work, and home scenarios. The distribution of topics indicate common themes of pain points, which could help guide the medium and content of mental health resources. C) Distribution of unsupervised cluster representation among posts from r/COVID19_support. Although many posts were assigned to unspecified clusters, the substantial portion of posts assigned to the suicidality cluster is notable. ADHD: attention-deficit/hyperactivity disorder; EDanonymous: Eating Disorders Anonymous; PTSD: posttraumatic stress disorder.CC BY · PubMed Central
Figure 6 Supervised dimensionality reduction to measure how certain subreddits are becoming more or less similar over time. A) Supervised dimensionality reduction of posts within 15-day time windows with starting day displayed (r/healthanxiety becomes more similar to other subreddits). B) Median pairwise distance with r/healthanxiety for each time window over 50 bootstrapping samples displaying only extreme values with regard to normal 2019 fluctuations (top and bottom 5th percentiles), which indicates they are less likely to be part of normal fluctuations in distance. C) The median distance across all subreddits (last row in B) shows subreddits becoming more similar to r/healthanxiety during the increase in COVID-19–related posts (Figure 2 mean values were split into 7 time windows to match subreddit trends, and the mean was taken for each window). r/healthanxiety is the only trend that significantly correlates with COVID-19 posts after Benjamini-Hochberg multiple comparison correction (ρ=–0.96, P<.001). adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDanonymous: Eating Disorders Anonymous; ptsd: posttraumatic stress disorder; UMAP: Uniform Manifold Approximation and Projection.
Figure 6 Supervised dimensionality reduction to measure how certain subreddits are becoming more or less similar over time. A) Supervised dimensionality reduction of posts within 15-day time windows with starting day displayed (r/healthanxiety becomes more similar to other subreddits). B) Median pairwise distance with r/healthanxiety for each time window over 50 bootstrapping samples displaying only extreme values with regard to normal 2019 fluctuations (top and bottom 5th percentiles), which indicates they are less likely to be part of normal fluctuations in distance. C) The median distance across all subreddits (last row in B) shows subreddits becoming more similar to r/healthanxiety during the increase in COVID-19–related posts (Figure 2 mean values were split into 7 time windows to match subreddit trends, and the mean was taken for each window). r/healthanxiety is the only trend that significantly correlates with COVID-19 posts after Benjamini-Hochberg multiple comparison correction (ρ=–0.96, P<.001). adhd: attention-deficit/hyperactivity disorder; bpd: borderline personality disorder; EDanonymous: Eating Disorders Anonymous; ptsd: posttraumatic stress disorder; UMAP: Uniform Manifold Approximation and Projection.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 5

  1. 10.18653/v1/w19-3003 · doi:10.18653/v1/w19-3003
  2. Pennebaker, JW, 2015, LIWC 2015 Operator’s Manual
  3. De ChoudhuryMGamonMCountsSHorvitzEPredicting depression via social media2013Seventh International AAAI Conference on Weblogs and Social Media AAAI Publications2013Cambridge, Massachusetts
  4. Park, A, 2017, AMIA Annu Symp Proc, 2017, 1362
  5. McInnes, L, 2018, arXiv, 1

Cited by

Items 301–339 of 339

Metadata

Authors
Low, D. M., Rumker, L., Talkar, T., Torous, J., Cecchi, G., & Ghosh, S. S.
Affiliations
  • Harvard University (US)
  • Broad Institute (US)
  • Beth Israel Deaconess Medical Center (US)
  • IBM Research - Thomas J. Watson Research Center (US)
Corresponding author
Low, Daniel M
Venue
Journal of Medical Internet Research
Published
2020-10-12
Volume
22
Issue
10
Pages
e22635
ISSN
1438-8871 (ISSN-L)
OpenAlex
W3085890295
Cited by
340
References
17
Citations by year
2026: 34 · 2025: 46 · 2024: 58 · 2023: 78 · 2022: 72 · 2021: 49 · 2020: 3
FWCI
55.20
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Massachusetts Institute of Technology
  • National Institutes of Health
  • National Human Genome Research Institute (T32HG002295)
  • National Institute on Deafness and Other Communication Disorders (T32DC000038)
Topics
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Social Sciences › Psychology › Clinical Psychology › COVID-19 and Mental Health
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
COVID-19 pandemic, mental health, online support groups, Reddit, health anxiety, suicide, loneliness, social isolation, financial stress, attention deficit hyperactivity disorder, eating disorders, borderline personality disorder, post-traumatic stress disorder, self-harm, text mining, sentiment analysis, topic modeling, unsupervised clustering, online communities
MeSH terms
Betacoronavirus, COVID-19, SARS-CoV-2, Adolescent, Adult, Anxiety, Borderline Personality Disorder, Female, Humans, Male, Mental Health, Middle Aged, Natural Language Processing, Pneumonia, Viral, Self-Help Groups, Stress Disorders, Post-Traumatic, Global Health, Coronavirus Infections, Young Adult, Pandemics, Suicidal Ideation, Social Media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.