The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 33 sources its claims cite were checked for support.
Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review
Open accessAbstract
BACKGROUND: The pathway from evidence generation to consumption contains many steps which can lead to overstatement or misinformation. The proliferation of internet-based health news may encourage selection of media and academic research articles that overstate strength of causal inference. We investigated the state of causal inference in health research as it appears at the end of the pathway, at the point of social media consumption. METHODS: We screened the NewsWhip Insights database for the most shared media articles on Facebook and Twitter reporting about peer-reviewed academic studies associating an exposure with a health outcome in 2015, extracting the 50 most-shared academic articles and media articles covering them. We designed and utilized a review tool to systematically assess and summarize studies' strength of causal inference, including generalizability, potential confounders, and methods used. These were then compared with the strength of causal language used to describe results in both academic and media articles. Two randomly assigned independent reviewers and one arbitrating reviewer from a pool of 21 reviewers assessed each article. RESULTS: We accepted the most shared 64 media articles pertaining to 50 academic articles for review, representing 68% of Facebook and 45% of Twitter shares in 2015. Thirty-four percent of academic studies and 48% of media articles used language that reviewers considered too strong for their strength of causal inference. Seventy percent of academic studies were considered low or very low strength of inference, with only 6% considered high or very high strength of causal inference. The most severe issues with academic studies' causal inference were reported to be omitted confounding variables and generalizability. Fifty-eight percent of media articles were found to have inaccurately reported the question, results, intervention, or population of the academic study. CONCLUSIONS: We find a large disparity between the strength of language as presented to the research consumer and the underlying strength of causal inference among the studies most widely shared on social media. However, because this sample was designed to be representative of the articles selected and shared on social media, it is unlikely to be representative of all academic and media work. More research is needed to determine how academic institutions, media organizations, and social network sharing patterns impact causal inference and language as received by the research consumer.
Overview
CoChat Summary
TL;DR
In a systematic review of the 50 most-shared academic health studies and 64 corresponding media articles on Facebook/Twitter in 2015, most academic studies had low or very low strength of causal inference, and both academic and media language often overstated that inference. Media articles also frequently misreported key study characteristics.
Key findings
- 70% of academic studies were rated low or very low strength of causal inference (54% low, 16% very low), and only 6% were rated high or very high.
- 34% of academic studies and 48% of media articles used causal language reviewers considered too strong for the assessed causal inference; 58% of media articles inaccurately reported the question, results, intervention, or population of the academic study.
- The most severe academic issues were covariates/confounding (54% high or very high severity), generalizability (52%), and statistical methods (32%).
- Media language was stronger than academic language in 55% of cases, matched in 39%, and weaker in 6%; 83% of media articles linked to low/very low inference studies used moderate or strong causal language.
Methods
- Systematic review using NewsWhip Insights to identify the most-shared Facebook and Twitter media articles in 2015 reporting on peer-reviewed academic studies associating an exposure with a health outcome; extracted 50 academic articles and 64 associated media articles, representing 68% of Facebook shares and 45% of Twitter shares in 2015.
- Two randomly assigned independent reviewers plus one arbitrating reviewer from a pool of 21 assessed each article using a custom review tool; protocol followed PRISMA and was registered with PROSPERO (CRD42016045197).
- The tool rated strength of causal inference on a 5-point scale, causal language on a 3-point scale, and media accuracy; analysis used descriptive statistics in R and two-tailed t-tests.
Limitations
- Paper-stated: the sample is representative of articles selected and shared on social media, not all academic or media work; the study cannot isolate the roles of academic institutions, media, social media platforms, or the public; only published articles were reviewed, without replicating calculations, data, or code or assessing all error sources such as cherry-picking; only 50 article sets were reviewed; the review tool is subjective, resource-intensive, and exact reviewer agreement was imperfect; these limitations likely biased results toward finding higher strength of causal inference.
- Text-evident: the descriptive cross-sectional design cannot establish causal direction in the research-to-consumption pathway, and the media sample is limited to 64 articles linked to the 50 academic articles.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
13 Valid · 1 Not valid · 5 Not checked · 64 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Media outlets may spin [26–28] and encourage dissemination of eye-catching, potentially overstated, inaccurate [29], and/or misleading headlines in order to gain larger audience size [28, 30, 31].”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [26] I Boutron, 2010, Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes, JAMA source
- Partly supportedcites [27] S Lockyer, 2013, "Spin" in wound care research: the reporting and interpretation of randomized controlled trials with statistically non-significant primary outcome results or unspecified primary outcomes, Trials sourceExperimental — accuracy not yet measured
The Background describes spin as language and emphasis in trial reports that can mislead readers, and the Key findings report spin in wound-trial publications. The paper does not establish that media outlets spin headlines or do so to attract a larger audience; those are different actors and a motive the cited passages do not address. No page numbers are available in our copy. (LLM, conf=0.76)
- Partly supportedcites [28] R Haneef, 2015, Interpretation of Results of Studies Evaluating an Intervention Highlighted in Google Health News: A Cross-Sectional Study of News, PloS One sourceExperimental — accuracy not yet measured
The Results section (‘Prevalence of spin in news headlines’; page not supplied) reports spin in 58 of 130 headlines, including hype and misleading extrapolation. The paper does not establish that media outlets encouraged such headlines to gain a larger audience; the Discussion says it did not determine whether the spin originated in scientific articles or was added by journalists. (LLM, conf=0.89)
- Partly supportedcites [29] G Schwitzer, 2008, How do US journalists cover treatments, tests, products, and procedures? An evaluation of 500 stories, PLoS Med sourceExperimental — accuracy not yet measured
The Results and Impact section reports widespread shortcomings in health-news stories, and Next Steps says short health headlines can omit important context and mislead readers. Next Steps also discusses sensationalism as a way for journalists to attract attention. It does not establish that media outlets encourage dissemination of such headlines specifically to increase audience size. (LLM, conf=0.82)
- Partly supportedcites [30] R Moynihan, 2000, Coverage by the news media of the benefits and risks of medications, N Engl J Med sourceExperimental — accuracy not yet measured
The Conclusions (page unavailable) describe prior evidence of inaccurate coverage, overstatement, and sensationalism; the Results (page unavailable) show that medication stories often reported relative benefits without absolute benefits. These passages support concern about slanted coverage, but the paper does not establish that outlets disseminated misleading headlines to gain a larger audience. (LLM, conf=0.84)
- Partly supportedcites [31] A Yavchitz, 2012, Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study, PLoS Med sourceExperimental — accuracy not yet measured
The Abstract reports spin in 21 of 41 news items, and the Discussion says spin in press releases could increase journalists’ interest. The paper does not establish that media outlets promote misleading headlines to gain a larger audience; its proposed motive concerns interest from journalists, not audience size. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent evidence suggests that exposure to health information in traditional and social media may impact health behaviors [2].”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Introduction
What the sources say
- Partly supportedcites [2] R Grilli, 2002, Mass media interventions: effects on health services utilisation, Cochrane Database Syst Rev sourceExperimental — accuracy not yet measured
The Background says mass media may influence individual health behaviours and gives examples involving dietary choices and contraceptive use. The review covers traditional mass-media channels, but it does not establish an effect of social media on health behaviours; it identifies the Internet as a channel for future research. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The authors of both publications later made statements retracting their conclusions, noting that they had overstated the strength of causal inference in light of the studies’ lack of methodological rigor [3].”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Introduction
What the sources say
- Could not verifycites [3] A Matthews, 2016, Impact of statin related media coverage on use of statins: interrupted time series analysis with UK primary care data, BMJ sourceExperimental — accuracy not yet measured
The Strengths and limitations section says the interrupted time series cannot confirm causality, but it does not report that the authors of two publications later retracted their conclusions or acknowledged a lack of methodological rigor. Searches for retractions, later statements, and overstated causal inference found no passage addressing those assertions. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“An estimated 200,000 patients temporarily ceased taking statins–drugs with strong evidence for efficacy and safety for treating hypertension [4, 5]–after press outlets reported on two studies suggesting that the medication was associated with high rates of adverse events.”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Introduction
What the sources say
- Supportedcites [3] A Matthews, 2016, Impact of statin related media coverage on use of statins: interrupted time series analysis with UK primary care data, BMJ source
The Impact on public health section estimates 218,971 excess UK patients stopped taking statins in the six months after media coverage, consistent with the rounded figure of 200,000. The Introduction links the coverage to two BMJ articles and discussion of purportedly high side-effect rates; Key findings says the increase in cessation was temporary. The paper does not establish that every individual who stopped later restarted. (LLM, conf=0.87)
hedgedstates a causen = 200,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Publication bias [20, 21], lack of replicability and replication [22], implication of a causal relationship even after stating that the evidence is insufficient to reach causal conclusions [23–25], and related phenomena may also introduce error into bodies of published literature.”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Introduction
What the sources say
- Supportedcites [23] SS Cofield, 2010, Use of causal language in observational studies of obesity and nutrition, Obes Facts source
The Introduction says causal language in observational studies can mislead readers. The Descriptive and Inferential Statistics section reports that 31 papers used causal language while also including a statement disavowing causation. Together, these passages support the cited concern that such reporting may introduce error into the literature. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [24] V Prasad, 2013, Observational studies often make clinical practice recommendations: an empirical evaluation of authors' attitudes, J Clin Epidemiol source
- Partly supportedcites [25] K Chiu, 2017, 'Spin' in published biomedical literature: A methodological systematic review, PLoS Biology sourceExperimental — accuracy not yet measured
The Results (Practices used to spin results) identify causal language in observational studies as spin because those designs do not permit causal inference. The Discussion likewise describes implying cause and effect as spin. Neither passage establishes the more specific sequence claimed here: authors first stating that the evidence is insufficient for a causal conclusion and then implying one. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
49 more claims · 1 with their cited sources checked
Create a free account to see all 83 claims and to verify the 2 references still open.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“in another study, roughly the same fraction of health-related papers widely shared on social media used inappropriately strong causal language (38).”
From Misinformation in and about science (2021) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Among the 35 academic articles rated as “Low” or “Very low” strength of causal inference, 45% (n = 16) were written with strong or moderate causal language.”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Results
Reported 45 · recomputed 45.714286
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Abstract > Results
“We accepted the most shared 64 media articles pertaining to 50 academic articles for review, representing 68% of Facebook and 45% of Twitter shares in 2015.”
§ Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review > Results
“The cumulative number of shares among the 231 screened articles screened represents 68% (938,596/1,375,152) of Facebook shares and 45% (189,777/423,996) of Twitter shares (S1 Fig)”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · cited 1×
“The final protocol for this project was developed and reported using PRISMA guidelines for systematic review [35], and registered with PROSPERO [36] as protocol CRD42016045197 on August 5, 2016.”
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 2×
“While similar tools and frameworks have recently been proposed, most notably the ROBINS-I [44] and related Risk of Bias 2.0 [45] tools, the CLAIMS tool differs in several key respects in order to be most useful for research consumers and reviewers.”
MethodInfluentialWang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“A host of factors can render studies inaccurate [15] or not useful in practice [16].”
BackgroundSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“Publication bias [20, 21], lack of replicability and replication [22], implication of a causal relationship even after stating that the evidence is insufficient to reach causal conclusions [23–25], and related phenomena may also introduce error into bodies of published literature.”
BackgroundThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
Research works printed but not linked: 2
- 2011, The Oxford 2011 Levels of Evidence
- Clarivate Analytics. Journal Citation Reports. 2015.
Cited by
Public health and online misinformation: challenges and recommendations
Swire-Thompson, B., & Lazer, D. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094127 · Cited by 1,170
“In response to a rapid decline in human papillomavirus (HPV) vaccine uptake, Danish public health officials created c r i These technologies can be particularly useful for rural communities.”
BackgroundMisinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
“One analysis found that nearly a third of 525 papers in top obesity or nutrition journals make inappropriate causal claims in their abstracts or titles (37); in another study, roughly the same fraction of health-related papers widely shared on social media used inappropriately strong causal language (38).”
BackgroundYeung, A. W. K., Tosevska, A., Klager, E., Eibensteiner, F., Tsagkaris, C., Parvanov, E. D., Nawaz, F. A., Völkl-Kernstock, S., Schaden, E., Kletecka-Pulker, M., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/28152 · Cited by 203
“Often, media articles may report research findings with misinterpretation and overstatement that can lead to confusion, misinformation, and mistrust in scientific reporting [6].”
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 233
“This is indicated by recent reports of poor interpretability of systematic review results by readers [184–186] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [50, 187].”
Prime, H., Andrews, K., Markwell, A., González, A., Janus, M., Tricco, A. C., Bennett, T. A., & Atkinson, L. R. (2023) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-022-00423-2 · Cited by 157
Self-citation
Metadata
- Authors
- Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., McClure, E. S., Suarez, E. A., & on behalf of the CLAIMS research team
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Boston Children's Hospital (US)
- Harvard University (US)
- Tufts University (US)
- Boston University (US)
- Johns Hopkins University (US)
- Corresponding author
- Haber, Noah
- ORCID
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-05-30
- Volume
- 13
- Issue
- 5
- Pages
- e0196346
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5976147
- OpenAlex
- W2805549342
- Cited by
- 97
- References
- 45
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 7 · 2023: 19 · 2022: 11 · 2021: 20 · 2020: 11 · 2019: 16 · 2018: 1
- FWCI
- 4.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (acceptedVersion)
- OpenBU (Boston University) (acceptedVersion, cc-by)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (P2CHD050924)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Keywords
- causal inference, quality of evidence, causal language, health misinformation, media coverage of research, social media sharing, Facebook, Twitter, research generalizability, confounding, health news
- MeSH terms
- Communication, Communications Media, Humans, Language, Causality, Internet, Biomedical Research, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. https://app.cochat.ai/papers/10.1371/journal.pone.0196346
- The paper
- Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., McClure, E. S., Suarez, E. A., & on behalf of the CLAIMS research team. (2018). Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. PLoS ONE. https://doi.org/10.1371/journal.pone.0196346
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.