CoChat

Quantifying and contextualizing the impact of bioRxiv preprints through automated social media audience segmentation

Open access
Carlson, Jedidiah, Harris, Kelley · 2020 · PLoS Biology

Published version of a preprint

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

Abstract

Engagement with scientific manuscripts is frequently facilitated by Twitter and other social media platforms. As such, the demographics of a paper's social media audience provide a wealth of information about how scholarly research is transmitted, consumed, and interpreted by online communities. By paying attention to public perceptions of their publications, scientists can learn whether their research is stimulating positive scholarly and public thought. They can also become aware of potentially negative patterns of interest from groups that misinterpret their work in harmful ways, either willfully or unintentionally, and devise strategies for altering their messaging to mitigate these impacts. In this study, we collected 331,696 Twitter posts referencing 1,800 highly tweeted bioRxiv preprints and leveraged topic modeling to infer the characteristics of various communities engaging with each preprint on Twitter. We agnostically learned the characteristics of these audience sectors from keywords each user's followers provide in their Twitter biographies. We estimate that 96% of the preprints analyzed are dominated by academic audiences on Twitter, suggesting that social media attention does not always correspond to greater public exposure. We further demonstrate how our audience segmentation method can quantify the level of interest from nonspecialist audience sectors such as mental health advocates, dog lovers, video game developers, vegans, bitcoin investors, conspiracy theorists, journalists, religious groups, and political constituencies. Surprisingly, we also found that 10% of the preprints analyzed have sizable (>5%) audience sectors that are associated with right-wing white nationalist communities. Although none of these preprints appear to intentionally espouse any right-wing extremist messages, cases exist in which extremist appropriation comprises more than 50% of the tweets referencing a given preprint. These results present unique opportunities for improving and contextualizing the public discourse surrounding scientific research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.3000860

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Topic modeling output for 4 selected preprints. Each panel shows the LDA topic modeling results for one of 4 selected preprints: (A) “DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning,” by Graving and colleagues. (B) “Engineering Brain Parasites for Intracellular Delivery of Therapeutic Proteins,” by Graving and colleagues. (C) “Sex differences in gene expression in the human fetal brain,” by O’Brien and colleagues. (D) “How face perception unfolds over time,” by Dobs and colleagues. In each panel, every account that tweeted about that paper is represented by a horizontal stack of 12 colored segments that represent the account’s estimated membership probability in each of the 12 inferred audience sectors. The top 15 keywords, hashtags, or emoji associated with each topic are shown in the corresponding legend to the right of each panel. Topics inferred to correspond to academic audience sectors are indicated with a emoji at the beginning of the list of associated keywords. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and interactive versions of this figure for each of the 1,800 preprints analyzed can be accessed at https://carjed.github.io/audiences. LDA, latent Dirichlet allocation.
Fig 1 Topic modeling output for 4 selected preprints. Each panel shows the LDA topic modeling results for one of 4 selected preprints: (A) “DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning,” by Graving and colleagues. (B) “Engineering Brain Parasites for Intracellular Delivery of Therapeutic Proteins,” by Graving and colleagues. (C) “Sex differences in gene expression in the human fetal brain,” by O’Brien and colleagues. (D) “How face perception unfolds over time,” by Dobs and colleagues. In each panel, every account that tweeted about that paper is represented by a horizontal stack of 12 colored segments that represent the account’s estimated membership probability in each of the 12 inferred audience sectors. The top 15 keywords, hashtags, or emoji associated with each topic are shown in the corresponding legend to the right of each panel. Topics inferred to correspond to academic audience sectors are indicated with a emoji at the beginning of the list of associated keywords. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and interactive versions of this figure for each of the 1,800 preprints analyzed can be accessed at https://carjed.github.io/audiences. LDA, latent Dirichlet allocation.CC BY · PubMed Central
Fig 2 Comparison of academic audience fractions estimated by Altmetric versus our estimates. Each point represents an individual preprint, with Altmetric’s estimated academic audience fraction shown on the x-axis and the academic audience fractions estimated through our topic modeling approach shown on the y-axis. The loess-smoothed curve fit to the data indicates a nonlinear relationship between these 2 sets of estimates. The size of each point indicates the total number of tweets referencing that preprint. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and an interactive version of this figure can be accessed at https://carjed.github.io/audiences.
Fig 2 Comparison of academic audience fractions estimated by Altmetric versus our estimates. Each point represents an individual preprint, with Altmetric’s estimated academic audience fraction shown on the x-axis and the academic audience fractions estimated through our topic modeling approach shown on the y-axis. The loess-smoothed curve fit to the data indicates a nonlinear relationship between these 2 sets of estimates. The size of each point indicates the total number of tweets referencing that preprint. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and an interactive version of this figure can be accessed at https://carjed.github.io/audiences.CC BY · PubMed Central
Fig 3 Political skew of nonacademic audience sectors by bioRxiv category. The x-axis shows the ratio between the estimated sizes of left-wing audience sectors (associated with the hashtag “#resist”) and right-wing audience sectors (associated with the hashtag “#maga”) among all tweets referencing preprints in a given bioRxiv category. Data shown are for the 14 bioRxiv categories where this ratio exceeds 2:1 or 1:2. The y-axis shows the -log10 p-value of a chi-square test for whether the sizes of these audience sectors match an underlying null distribution, assuming 62.5% of users lean left and 37.5% of users lean right, based on a recent poll of US Twitter users’ political ideologies. Preprint categories with statistically significant differences (after Bonferroni multiple testing correction) are annotated above the dashed line. The size of each point indicates the total number of users affiliated with political audience sectors for that category. Data for the information depicted in this figure are available at https://github.com/carjed/audiences.
Fig 3 Political skew of nonacademic audience sectors by bioRxiv category. The x-axis shows the ratio between the estimated sizes of left-wing audience sectors (associated with the hashtag “#resist”) and right-wing audience sectors (associated with the hashtag “#maga”) among all tweets referencing preprints in a given bioRxiv category. Data shown are for the 14 bioRxiv categories where this ratio exceeds 2:1 or 1:2. The y-axis shows the -log10 p-value of a chi-square test for whether the sizes of these audience sectors match an underlying null distribution, assuming 62.5% of users lean left and 37.5% of users lean right, based on a recent poll of US Twitter users’ political ideologies. Preprint categories with statistically significant differences (after Bonferroni multiple testing correction) are annotated above the dashed line. The size of each point indicates the total number of users affiliated with political audience sectors for that category. Data for the information depicted in this figure are available at https://github.com/carjed/audiences.CC BY · PubMed Central
Fig 4 Distributions of white nationalist homophily by category. Each point represents a single preprint, and the position on the y-axis indicates the proportion of users who tweeted about that preprint whose follower network homophily with the white nationalist reference panel is greater than h = 2%. Boxplots summarizing the distributions of these proportions per bioRxiv category are shown beneath each set of points. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and an interactive version of this figure can be accessed at https://carjed.github.io/audiences.
Fig 4 Distributions of white nationalist homophily by category. Each point represents a single preprint, and the position on the y-axis indicates the proportion of users who tweeted about that preprint whose follower network homophily with the white nationalist reference panel is greater than h = 2%. Boxplots summarizing the distributions of these proportions per bioRxiv category are shown beneath each set of points. Data for the information depicted in this figure are available at https://github.com/carjed/audiences, and an interactive version of this figure can be accessed at https://carjed.github.io/audiences.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 60 references

Research works printed but not linked: 5

  1. To tweet or not to tweet? In: Science | AAAS [Internet]. 20 Nov 2014. [cited 2019 Dec 10]. Available from: https://www.sciencemag.org/careers/2014/10/tweet-or-not-tweet
  2. Kang J-H, Lerman K. Using Lists to Measure Homophily on Twitter. Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. 2012. [cited 2019 Nov 5]. Available from: https://www.aaai.org/ocs/index.php/WS/AAAIW12/paper/viewPaper/5425/5628
  3. Birney E. Race, genetics and pseudoscience: an explainer. In: Ewan’s Blog: Bioinformatician at large [Internet]. 24 10 2019 [cited 2019 Oct 31]. Available from: http://ewanbirney.com/2019/10/race-genetics-and-pseudoscience-an-explainer.html
  4. Sommer W. Why a Red “X” Is the New Symbol of Conservative Twitter. In: The Daily Beast [Internet]. 10 8 2018 [cited 2019 Aug 30]. Available from: https://www.thedailybeast.com/why-a-red-x-is-the-new-symbol-of-conservative-twitter
  5. “It”s a toxic place.’ How the online world of white nationalists distorts population genetics. In: Science | AAAS [Internet]. 22 May 2018. [cited 2020 Jan 16]. Available from: https://www.sciencemag.org/news/2018/05/it-s-toxic-place-how-online-world-white-nationalists-distorts-population-genetics

Cited by

See all 66

Metadata

Authors
Carlson, J., & Harris, K.
Affiliations
University of Washington (US)
Corresponding author
Carlson, Jedidiah
Venue
PLoS Biology
Published
2020-09-22
Volume
18
Issue
9
Pages
e3000860
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
OpenAlex
W3088297402
Cited by
61
References
55
Citations by year
2026: 6 · 2025: 4 · 2024: 11 · 2023: 9 · 2022: 17 · 2021: 10 · 2020: 4
FWCI
9.91
Type
journal article
Language
en
Open access
Locations
  • PLoS Biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institute of General Medical Sciences (1R35GM133428-01)
  • Burroughs Wellcome Fund
  • Pew Charitable Trusts
  • Searle Scholars Program
  • Alfred P. Sloan Foundation
  • Fred Hutchinson Cancer Research Center
Topics
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Health › Social Media in Health Education
Keywords
Twitter engagement, audience segmentation, social media audiences, public perception of science, topic modeling, right-wing extremism, conspiracy theorists, science communication
MeSH terms
Academies and Institutes, Political Activism, Preprints as Topic, Electronic Data Processing, Humans, Publications, Science, Social Change, Databases as Topic, Internet, Access to Information, Information Literacy, Social Media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Quantifying and contextualizing the impact of bioRxiv preprints through automated social media audience segmentation. https://app.cochat.ai/papers/10.1371/journal.pbio.3000860
The paper
Carlson, J., & Harris, K. (2020). Quantifying and contextualizing the impact of bioRxiv preprints through automated social media audience segmentation. PLoS Biology. https://doi.org/10.1371/journal.pbio.3000860

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.