CoChat

Poor replication validity of biomedical association studies reported by newspapers

Open access

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Abstract

OBJECTIVE: To investigate the replication validity of biomedical association studies covered by newspapers. METHODS: We used a database of 4723 primary studies included in 306 meta-analysis articles. These studies associated a risk factor with a disease in three biomedical domains, psychiatry, neurology and four somatic diseases. They were classified into a lifestyle category (e.g. smoking) and a non-lifestyle category (e.g. genetic risk). Using the database Dow Jones Factiva, we investigated the newspaper coverage of each study. Their replication validity was assessed using a comparison with their corresponding meta-analyses. RESULTS: Among the 5029 articles of our database, 156 primary studies (of which 63 were lifestyle studies) and 5 meta-analysis articles were reported in 1561 newspaper articles. The percentage of covered studies and the number of newspaper articles per study strongly increased with the impact factor of the journal that published each scientific study. Newspapers almost equally covered initial (5/39 12.8%) and subsequent (58/600 9.7%) lifestyle studies. In contrast, initial non-lifestyle studies were covered more often (48/366 13.1%) than subsequent ones (45/3718 1.2%). Newspapers never covered initial studies reporting null findings and rarely reported subsequent null observations. Only 48.7% of the 156 studies reported by newspapers were confirmed by the corresponding meta-analyses. Initial non-lifestyle studies were less often confirmed (16/48) than subsequent ones (29/45) and than lifestyle studies (31/63). Psychiatric studies covered by newspapers were less often confirmed (10/38) than the neurological (26/41) or somatic (40/77) ones. This is correlated to an even larger coverage of initial studies in psychiatry. Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim. CONCLUSION: Journalists preferentially cover initial findings although they are often contradicted by meta-analyses and rarely inform the public when they are disconfirmed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0172650

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Preferential coverage of initial findings and influence of the impact factor (IF). The figure shows the percentage of primary studies that are covered by newspapers depending on the study type (lifestyle versus non-lifestyle). Studies of the lifestyle category described associations linking a pathology to a risk factor on which each subject can act. Regarding non-lifestyle articles, the figure also contrasts initial articles with subsequent ones. Differences in the media coverage between initial studies and subsequent ones were statistically significant (see text) except for studies published in prestigious journals (IF ≥ 30). Raw data are given in Supporting Information (S2 Text).
Fig 1 Preferential coverage of initial findings and influence of the impact factor (IF). The figure shows the percentage of primary studies that are covered by newspapers depending on the study type (lifestyle versus non-lifestyle). Studies of the lifestyle category described associations linking a pathology to a risk factor on which each subject can act. Regarding non-lifestyle articles, the figure also contrasts initial articles with subsequent ones. Differences in the media coverage between initial studies and subsequent ones were statistically significant (see text) except for studies published in prestigious journals (IF ≥ 30). Raw data are given in Supporting Information (S2 Text).CC BY · PubMed Central
Fig 2 Replication validity of primary articles reported by newspapers. The figure shows the percentage of primary articles echoed by newspapers whose main finding was consistent with the corresponding meta-analysis. We considered here the same three categories as in Fig 1: primary articles of the lifestyle category and initial or subsequent non-lifestyle studies. Raw data are given in Supporting Information (S2 Text).
Fig 2 Replication validity of primary articles reported by newspapers. The figure shows the percentage of primary articles echoed by newspapers whose main finding was consistent with the corresponding meta-analysis. We considered here the same three categories as in Fig 1: primary articles of the lifestyle category and initial or subsequent non-lifestyle studies. Raw data are given in Supporting Information (S2 Text).CC BY · PubMed Central
Fig 3 Replication validity of primary studies reported by newspapers in three biomedical domains. The blue bars show the percentage of primary studies covered by newspapers whose main finding was consistent with the corresponding meta-analysis. The red bars show the percentage of initial findings among primary studies echoed by newspapers and related to four psychiatric disorders (PSY), four neurological diseases (NEURO) and four somatic diseases (SOMA). Raw data are given in Supporting Information (S2 Text).
Fig 3 Replication validity of primary studies reported by newspapers in three biomedical domains. The blue bars show the percentage of primary studies covered by newspapers whose main finding was consistent with the corresponding meta-analysis. The red bars show the percentage of initial findings among primary studies echoed by newspapers and related to four psychiatric disorders (PSY), four neurological diseases (NEURO) and four somatic diseases (SOMA). Raw data are given in Supporting Information (S2 Text).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 39 references

Cited by

See all 84

Metadata

Authors
Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F.
Affiliations
Centre National de la Recherche Scientifique (FR)
Corresponding author
Dumas-Mallet, Estelle
Venue
PLOS ONE
Published
2017-02-21
Volume
12
Issue
2
Pages
e0172650
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2589877982
Cited by
76
References
39
Citations by year
2026: 4 · 2025: 2 · 2024: 7 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 18 · 2019: 8 · 2018: 12 · 2017: 8
FWCI
11.25
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Oskar-Bordeaux (Universite de Bordeaux) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
newspaper coverage of science, meta-analysis, negative results, lifestyle risk factors, genetic risk factors, psychiatric research, brain research, somatic diseases, science journalism
MeSH terms
Humans, Internal Medicine, Life Style, Neurology, Newspapers as Topic, Periodicals as Topic, Psychiatry, Risk Factors, Meta-Analysis as Topic, Reproducibility of Results, Health Behavior, Causality, Databases, Factual, Journalism, Information Dissemination, Biomedical Research, Journal Impact Factor

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Poor replication validity of biomedical association studies reported by newspapers. https://app.cochat.ai/papers/10.1371/journal.pone.0172650
The paper
Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F. (2017). Poor replication validity of biomedical association studies reported by newspapers. PLOS ONE. https://doi.org/10.1371/journal.pone.0172650

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.