Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 33 sources its claims cite were checked for support.

Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review

Open access

Abstract

BACKGROUND: The pathway from evidence generation to consumption contains many steps which can lead to overstatement or misinformation. The proliferation of internet-based health news may encourage selection of media and academic research articles that overstate strength of causal inference. We investigated the state of causal inference in health research as it appears at the end of the pathway, at the point of social media consumption. METHODS: We screened the NewsWhip Insights database for the most shared media articles on Facebook and Twitter reporting about peer-reviewed academic studies associating an exposure with a health outcome in 2015, extracting the 50 most-shared academic articles and media articles covering them. We designed and utilized a review tool to systematically assess and summarize studies' strength of causal inference, including generalizability, potential confounders, and methods used. These were then compared with the strength of causal language used to describe results in both academic and media articles. Two randomly assigned independent reviewers and one arbitrating reviewer from a pool of 21 reviewers assessed each article. RESULTS: We accepted the most shared 64 media articles pertaining to 50 academic articles for review, representing 68% of Facebook and 45% of Twitter shares in 2015. Thirty-four percent of academic studies and 48% of media articles used language that reviewers considered too strong for their strength of causal inference. Seventy percent of academic studies were considered low or very low strength of inference, with only 6% considered high or very high strength of causal inference. The most severe issues with academic studies' causal inference were reported to be omitted confounding variables and generalizability. Fifty-eight percent of media articles were found to have inaccurately reported the question, results, intervention, or population of the academic study. CONCLUSIONS: We find a large disparity between the strength of language as presented to the research consumer and the underlying strength of causal inference among the studies most widely shared on social media. However, because this sample was designed to be representative of the articles selected and shared on social media, it is unlikely to be representative of all academic and media work. More research is needed to determine how academic institutions, media organizations, and social network sharing patterns impact causal inference and language as received by the research consumer.

Overview

CoChat Summary

TL;DR

In a systematic review of the 50 most-shared academic health studies and 64 corresponding media articles on Facebook/Twitter in 2015, most academic studies had low or very low strength of causal inference, and both academic and media language often overstated that inference. Media articles also frequently misreported key study characteristics.

Key findings

  • 70% of academic studies were rated low or very low strength of causal inference (54% low, 16% very low), and only 6% were rated high or very high.
  • 34% of academic studies and 48% of media articles used causal language reviewers considered too strong for the assessed causal inference; 58% of media articles inaccurately reported the question, results, intervention, or population of the academic study.
  • The most severe academic issues were covariates/confounding (54% high or very high severity), generalizability (52%), and statistical methods (32%).
  • Media language was stronger than academic language in 55% of cases, matched in 39%, and weaker in 6%; 83% of media articles linked to low/very low inference studies used moderate or strong causal language.

Methods

  • Systematic review using NewsWhip Insights to identify the most-shared Facebook and Twitter media articles in 2015 reporting on peer-reviewed academic studies associating an exposure with a health outcome; extracted 50 academic articles and 64 associated media articles, representing 68% of Facebook shares and 45% of Twitter shares in 2015.
  • Two randomly assigned independent reviewers plus one arbitrating reviewer from a pool of 21 assessed each article using a custom review tool; protocol followed PRISMA and was registered with PROSPERO (CRD42016045197).
  • The tool rated strength of causal inference on a 5-point scale, causal language on a 3-point scale, and media accuracy; analysis used descriptive statistics in R and two-tailed t-tests.

Limitations

  • Paper-stated: the sample is representative of articles selected and shared on social media, not all academic or media work; the study cannot isolate the roles of academic institutions, media, social media platforms, or the public; only published articles were reviewed, without replicating calculations, data, or code or assessing all error sources such as cherry-picking; only 50 article sets were reviewed; the review tool is subjective, resource-intensive, and exact reviewer agreement was imperfect; these limitations likely biased results toward finding higher strength of causal inference.
  • Text-evident: the descriptive cross-sectional design cannot establish causal direction in the research-to-consumption pathway, and the media sample is limited to 64 articles linked to the 50 academic articles.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0196346

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 Results of summary measures for strength of causal inference and language. Language for categories of strength of causal inference has been lightly edited for this publication to better reflect the instructions given to the reviewers and for consistency with the rest of the manuscript. Reviewers were instructed to consider only the causal inference aspect of the study for these measures. The original language referred to the “study” and “results,” which has been edited in the figure to “causal inference” where appropriate for clarity. The original language and instructions are available in the attached review tool and on metacausal.com.
Fig 1 Results of summary measures for strength of causal inference and language. Language for categories of strength of causal inference has been lightly edited for this publication to better reflect the instructions given to the reviewers and for consistency with the rest of the manuscript. Reviewers were instructed to consider only the causal inference aspect of the study for these measures. The original language referred to the “study” and “results,” which has been edited in the figure to “causal inference” where appropriate for clarity. The original language and instructions are available in the attached review tool and on metacausal.com.CC0 · PubMed Central
Fig 2 Summary of severity of issues in causal estimate in academic article by category.
Fig 2 Summary of severity of issues in causal estimate in academic article by category.CC0 · PubMed Central
Fig 3 Strength of causal inference vs. strength of causal language in academic and media articles. The upper charts represent an assumed set of theoretically preferred regions, where we would prefer studies of the form exposure vs. outcome which reach the public to have language matching the strength of causal inference, a slight preference towards stronger causal inference, media language matching the academic language, and a preference for understated vs. overstated strength of causal language. The lower charts represent the empirical results of the study, where darker regions have more articles. The raw number of articles in each box is available in supplemental S4 Table.
Fig 3 Strength of causal inference vs. strength of causal language in academic and media articles. The upper charts represent an assumed set of theoretically preferred regions, where we would prefer studies of the form exposure vs. outcome which reach the public to have language matching the strength of causal inference, a slight preference towards stronger causal inference, media language matching the academic language, and a preference for understated vs. overstated strength of causal language. The lower charts represent the empirical results of the study, where darker regions have more articles. The raw number of articles in each box is available in supplemental S4 Table.CC0 · PubMed Central

Verification

Claims the paper makes

13 Valid · 1 Not valid · 5 Not checked · 64 Own results and interpretations

49 more claims · 1 with their cited sources checked

Create a free account to see all 83 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 24
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 45 references

Research works printed but not linked: 2

  1. 2011, The Oxford 2011 Levels of Evidence
  2. Clarivate Analytics. Journal Citation Reports. 2015.

Cited by

See all 97

Metadata

Authors
Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., McClure, E. S., Suarez, E. A., & on behalf of the CLAIMS research team
Affiliations
  • University of North Carolina at Chapel Hill (US)
  • Boston Children's Hospital (US)
  • Harvard University (US)
  • Tufts University (US)
  • Boston University (US)
  • Johns Hopkins University (US)
Corresponding author
Haber, Noah
Venue
PLoS ONE
Published
2018-05-30
Volume
13
Issue
5
Pages
e0196346
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2805549342
Cited by
97
References
45
Citations by year
2026: 5 · 2025: 7 · 2024: 7 · 2023: 19 · 2022: 11 · 2021: 20 · 2020: 11 · 2019: 16 · 2018: 1
FWCI
4.12
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, public-domain)
  • Europe PMC (PubMed Central) (publishedVersion, public-domain)
  • PubMed Central (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (acceptedVersion)
  • OpenBU (Boston University) (acceptedVersion, cc-by)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
Eunice Kennedy Shriver National Institute of Child Health and Human Development (P2CHD050924)
Topics
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
Keywords
causal inference, quality of evidence, causal language, health misinformation, media coverage of research, social media sharing, Facebook, Twitter, research generalizability, confounding, health news
MeSH terms
Communication, Communications Media, Humans, Language, Causality, Internet, Biomedical Research, Social Media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. https://app.cochat.ai/papers/10.1371/journal.pone.0196346
The paper
Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., McClure, E. S., Suarez, E. A., & on behalf of the CLAIMS research team. (2018). Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. PLoS ONE. https://doi.org/10.1371/journal.pone.0196346

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.