Poor replication validity of biomedical association studies reported by newspapers
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To investigate the replication validity of biomedical association studies covered by newspapers. METHODS: We used a database of 4723 primary studies included in 306 meta-analysis articles. These studies associated a risk factor with a disease in three biomedical domains, psychiatry, neurology and four somatic diseases. They were classified into a lifestyle category (e.g. smoking) and a non-lifestyle category (e.g. genetic risk). Using the database Dow Jones Factiva, we investigated the newspaper coverage of each study. Their replication validity was assessed using a comparison with their corresponding meta-analyses. RESULTS: Among the 5029 articles of our database, 156 primary studies (of which 63 were lifestyle studies) and 5 meta-analysis articles were reported in 1561 newspaper articles. The percentage of covered studies and the number of newspaper articles per study strongly increased with the impact factor of the journal that published each scientific study. Newspapers almost equally covered initial (5/39 12.8%) and subsequent (58/600 9.7%) lifestyle studies. In contrast, initial non-lifestyle studies were covered more often (48/366 13.1%) than subsequent ones (45/3718 1.2%). Newspapers never covered initial studies reporting null findings and rarely reported subsequent null observations. Only 48.7% of the 156 studies reported by newspapers were confirmed by the corresponding meta-analyses. Initial non-lifestyle studies were less often confirmed (16/48) than subsequent ones (29/45) and than lifestyle studies (31/63). Psychiatric studies covered by newspapers were less often confirmed (10/38) than the neurological (26/41) or somatic (40/77) ones. This is correlated to an even larger coverage of initial studies in psychiatry. Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim. CONCLUSION: Journalists preferentially cover initial findings although they are often contradicted by meta-analyses and rarely inform the public when they are disconfirmed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a recent study finding that the results of over half of studies on an association between exposure and outcome covered in the news are refuted by subsequent meta-analyses [33]”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“journalists seldom report when the studies covered previously fail to pan out (67).”
From Misinformation in and about science (2021) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Influence of life stress on depression: moderation by a polymorphism in the 5-HTT gene
Caspi, A., Sugden, K., Moffitt, T. E., Taylor, A., Craig, I. W., Harrington, H., McClay, J., Mill, J., Martin, J., Braithwaite, A., et al. (2003) · Science · doi:10.1126/science.1083968 · Cited by 8,120 · cited 2×
“In the second case, the initial study was published in 2003 in Science and claimed that a genetic factor was associated with depression when subjects were exposed to stressful life events [28].”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“These empirical studies, as well as theoretical considerations [1], have highlighted that initial findings on a novel scientific question are especially vulnerable to refutation.”
Background1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 1×
“We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23].”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8– 11], and iii) clinical trials [12–16].”
ResultReplication validity of genetic association studies
Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., & Contopoulos-Ioannidis, D. G. (2001) · Nature Genetics · doi:10.1038/ng749 · Cited by 1,832
Risch, N., Herrell, R., Lehner, T., Liang, K.-Y., Eaves, L., Hoh, J., Griem, A., Kovacs, M., Ott, J., & Merikangas, K. R. (2009) · JAMA · doi:10.1001/jama.2009.878 · Cited by 1,655 · cited 2×
“Finally, the meta-analysis published in 2009 by Risch and coworkers in JAMA (IF = 30) and contradicting "positive" findings [29], was covered by only four newspaper articles and all four mentioned that the initial finding "has not held up to scientific scrutiny".”
BackgroundThe economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · cited 1×
“Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8– 11], and iii) clinical trials [12–16].”
Result
Cited by
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Modernizing the Bradford Hill criteria for assessing causal relationships in observational data
Cox, L. A. (2018) · Critical Reviews in Toxicology · doi:10.1080/10408444.2018.1518404 · Cited by 115
“…meta-analyses, although such negative findings are seldom reported: “Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim” (Dumas-Mallet et al. 2017).”
BackgroundResultHaber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196346 · Cited by 97
“The interaction between academia and media highlights some of these complexities, such as a recent study finding that the results of over half of studies on an association between exposure and outcome covered in the news are refuted by subsequent meta-analyses [33], and another showing that those articles publicly cri…”
BackgroundHeterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 103
The drug delivery field at the inflection point: Time to fight its way out of the egg
Park, K. (2017) · Journal of Controlled Release · doi:10.1016/j.jconrel.2017.07.030 · Cited by 74
Metadata
- Authors
- Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Dumas-Mallet, Estelle
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-02-21
- Volume
- 12
- Issue
- 2
- Pages
- e0172650
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5319681
- OpenAlex
- W2589877982
- Cited by
- 76
- References
- 39
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 7 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 18 · 2019: 8 · 2018: 12 · 2017: 8
- FWCI
- 11.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Oskar-Bordeaux (Universite de Bordeaux) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- newspaper coverage of science, meta-analysis, negative results, lifestyle risk factors, genetic risk factors, psychiatric research, brain research, somatic diseases, science journalism
- MeSH terms
- Humans, Internal Medicine, Life Style, Neurology, Newspapers as Topic, Periodicals as Topic, Psychiatry, Risk Factors, Meta-Analysis as Topic, Reproducibility of Results, Health Behavior, Causality, Databases, Factual, Journalism, Information Dissemination, Biomedical Research, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Poor replication validity of biomedical association studies reported by newspapers. https://app.cochat.ai/papers/10.1371/journal.pone.0172650
- The paper
- Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F. (2017). Poor replication validity of biomedical association studies reported by newspapers. PLOS ONE. https://doi.org/10.1371/journal.pone.0172650
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.