Poor replication validity of biomedical association studies reported by newspapers
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To investigate the replication validity of biomedical association studies covered by newspapers. METHODS: We used a database of 4723 primary studies included in 306 meta-analysis articles. These studies associated a risk factor with a disease in three biomedical domains, psychiatry, neurology and four somatic diseases. They were classified into a lifestyle category (e.g. smoking) and a non-lifestyle category (e.g. genetic risk). Using the database Dow Jones Factiva, we investigated the newspaper coverage of each study. Their replication validity was assessed using a comparison with their corresponding meta-analyses. RESULTS: Among the 5029 articles of our database, 156 primary studies (of which 63 were lifestyle studies) and 5 meta-analysis articles were reported in 1561 newspaper articles. The percentage of covered studies and the number of newspaper articles per study strongly increased with the impact factor of the journal that published each scientific study. Newspapers almost equally covered initial (5/39 12.8%) and subsequent (58/600 9.7%) lifestyle studies. In contrast, initial non-lifestyle studies were covered more often (48/366 13.1%) than subsequent ones (45/3718 1.2%). Newspapers never covered initial studies reporting null findings and rarely reported subsequent null observations. Only 48.7% of the 156 studies reported by newspapers were confirmed by the corresponding meta-analyses. Initial non-lifestyle studies were less often confirmed (16/48) than subsequent ones (29/45) and than lifestyle studies (31/63). Psychiatric studies covered by newspapers were less often confirmed (10/38) than the neurological (26/41) or somatic (40/77) ones. This is correlated to an even larger coverage of initial studies in psychiatry. Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim. CONCLUSION: Journalists preferentially cover initial findings although they are often contradicted by meta-analyses and rarely inform the public when they are disconfirmed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a recent study finding that the results of over half of studies on an association between exposure and outcome covered in the news are refuted by subsequent meta-analyses [33]”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“journalists seldom report when the studies covered previously fail to pan out (67).”
From Misinformation in and about science (2021) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Influence of life stress on depression: moderation by a polymorphism in the 5-HTT gene
Caspi, A., Sugden, K., Moffitt, T. E., Taylor, A., Craig, I. W., Harrington, H., McClay, J., Mill, J., Martin, J., Braithwaite, A., et al. (2003) · Science · doi:10.1126/science.1083968 · Cited by 8,120 · cited 2×
“In the second case, the initial study was published in 2003 in Science and claimed that a genetic factor was associated with depression when subjects were exposed to stressful life events [28].”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“These empirical studies, as well as theoretical considerations [1], have highlighted that initial findings on a novel scientific question are especially vulnerable to refutation.”
Background1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 1×
“We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23].”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8– 11], and iii) clinical trials [12–16].”
ResultReplication validity of genetic association studies
Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., & Contopoulos-Ioannidis, D. G. (2001) · Nature Genetics · doi:10.1038/ng749 · Cited by 1,832
Risch, N., Herrell, R., Lehner, T., Liang, K.-Y., Eaves, L., Hoh, J., Griem, A., Kovacs, M., Ott, J., & Merikangas, K. R. (2009) · JAMA · doi:10.1001/jama.2009.878 · Cited by 1,655 · cited 2×
“Finally, the meta-analysis published in 2009 by Risch and coworkers in JAMA (IF = 30) and contradicting "positive" findings [29], was covered by only four newspaper articles and all four mentioned that the initial finding "has not held up to scientific scrutiny".”
BackgroundThe economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · cited 1×
“Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8– 11], and iii) clinical trials [12–16].”
ResultReproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Evaluating Dopamine Reward Pathway in ADHD: Clinical Implications
Volkow, N. D., Wang, G.-J., Kollins, S. H., Wigal, T. L., Newcorn, J. H., Telang, F., Fowler, J. S., Zhu, W., Logan, J., Ma, Y., et al. (2009) · JAMA · doi:10.1001/jama.2009.1308 · Cited by 728 · cited 1×
“In contrast, according to a subsequent study published in 2009 by JAMA (IF = 30), the dopamine transporter level was significantly decreased (-19%) in the left striatum of ADHD patients, but not in the right [27].”
ResultDopamine transporter density in patients with attention deficit hyperactivity disorder
Dougherty, D. D., Bonab, A. A., Spencer, T. J., Rauch, S. L., Madras, B. K., & Fischman, A. J. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04030-1 · Cited by 684 · cited 1×
“In the first case, the initial study published in 1999 by The Lancet (IF = 39) reported that the dopamine transporter level was increased (+70%) in both striatum of ADHD patients [26].”
BackgroundCoverage by the news media of the benefits and risks of medications
Moynihan, R., Bero, L., Ross-Degnan, D., Henry, D., Lee, K., Watkins, J., Mah, C., & Soumerai, S. B. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006013422206 · Cited by 500 · cited 1×
“We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23].”
BackgroundHeres, S., Davis, J., Maino, K., Jetzinger, E., Kissling, W., & Leucht, S. (2006) · American Journal of Psychiatry · doi:10.1176/appi.ajp.163.2.185 · Cited by 450 · cited 1×
“We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23].”
BackgroundYavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001308 · Cited by 293 · cited 1×
“We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23].”
BackgroundEmpirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Ioannidis, J. P. A. (2011) · JAMA · doi:10.1001/jama.2011.713 · Cited by 252
Bartlett, C., Sterne, J., & Egger, M. (2002) · BMJ · doi:10.1136/bmj.325.7355.81 · Cited by 222 · cited 3×
“Previous studies [24, 25] and a preliminary step of the present study led us to subdivide all association studies into two categories: a lifestyle category corresponding to any association between a pathology and a risk factor on which each subject can individually decide to act (e.”
BackgroundResultEvolution of treatment effects over time: Empirical insight from recursive cumulative metaanalyses
Ioannidis, J. P. A. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.021529998 · Cited by 192
Kitzinger, J., & Reilly, J. (1997) · European Journal of Communication · doi:10.1177/0267323197012003002 · Cited by 187
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time
Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., & Ioannidis, J. P. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.018 · Cited by 177
Breaking Free of Sample Size Dogma to Perform Innovative Translational Research
Bacchetti, P., Deeks, S. G., & McCune, J. M. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3001628 · Cited by 159 · cited 1×
“In other words, the low replication validity of many initial studies does not hinder the advancement of knowledge as long as science remains self-corrective [18].”
BackgroundMedical messages in the media – barriers and solutions to improving medical journalism
Larsson, A., Oxman, A. D., Carling, C. L. L., & Herrin, J. S. (2003) · Health Expectations · doi:10.1046/j.1369-7625.2003.00228.x · Cited by 136
Stryker, J. E. (2002) · Preventive Medicine · doi:10.1006/pmed.2002.1102 · Cited by 121 · cited 3×
“Previous studies [24, 25] and a preliminary step of the present study led us to subdivide all association studies into two categories: a lifestyle category corresponding to any association between a pathology and a risk factor on which each subject can individually decide to act (e.”
BackgroundResultJensen, J. D., & Hurley, R. J. (2012) · Public Understanding of Science · doi:10.1177/0963662510387759 · Cited by 112 · cited 1×
“Indeed, contrary to common beliefs, reception studies show that scientists are viewed by the public as more trustworthy when news coverage of health research acknowledges its uncertainty, especially if this is attributed to scientific authors [37, 38].”
BackgroundGan, H. K., You, B., Pond, G. R., & Chen, E. X. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs141 · Cited by 107
Establishment of genetic associations for complex diseases is independent of early study findings
Trikalinos, T. A., Ntzani, E., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2004) · European Journal of Human Genetics · doi:10.1038/sj.ejhg.5201227 · Cited by 102
The Quality of Media Reports on Discoveries Related to Human Genetic Diseases
Holtzman, N. A., Bernhardt, B. A., Mountcastle-Shah, E., Rodgers, J. E., Tambor, E. S., & Geller, G. (2005) · Public Health Genomics · doi:10.1159/000086756 · Cited by 84 · cited 1×
“Importantly, such biased newspaper coverage can have important social consequences [32, 33].”
BackgroundBeware the creeping cracks of bias
Sarewitz, D. (2012) · Nature · doi:10.1038/485149a · Cited by 80
Gonon, F., Konsman, J.-P., Cohen, D., & Boraud, T. (2012) · PLoS ONE · doi:10.1371/journal.pone.0044275 · Cited by 77 · cited 6×
“Newspaper coverage of scientific studies As previously described [19], we used the Dow Jones Factiva database to find newspaper articles echoing the scientific studies of our database.”
BackgroundMethodResultSelf-citationReplication and contradiction of highly cited research papers in psychiatry: 10-year follow-up
Tajika, A., Ogawa, Y., Takeshima, N., Hayasaka, Y., & Furukawa, T. A. (2015) · British Journal of Psychiatry · doi:10.1192/bjp.bp.113.143701 · Cited by 61 · cited 1×
“Likewise, highly cited early studies investigating the effectiveness of psychiatric treatments are often contradicted by subsequent studies [16].”
ResultBias in genetic association studies and impact factor
Munafò, M. R., Stothart, G., & Flint, J. (2009) · Molecular Psychiatry · doi:10.1038/mp.2008.77 · Cited by 60
Against conventional wisdom: when the public, the media, and medical practice collide
Jensen, J. D., Krakow, M., John, K. K., & Liu, M. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-s3-s4 · Cited by 37 · cited 1×
“Indeed, contrary to common beliefs, reception studies show that scientists are viewed by the public as more trustworthy when news coverage of health research acknowledges its uncertainty, especially if this is attributed to scientific authors [37, 38].”
BackgroundBourdaa, M., Konsman, J. P., Sécail, C., Venturini, T., Veyrat-Masson, I., & Gonon, F. (2013) · Public Understanding of Science · doi:10.1177/0963662513484842 · Cited by 32 · cited 1×
“Indeed, television programs rarely explicitly report on specific scientific publications [31].”
BackgroundSelf-citationDumas-Mallet, E., Button, K. S., Boraud, T., Munafo, M. R., & Gonon, F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158064 · Cited by 31 · cited 5×
“However, another factor might contribute to the low replication validity of psychiatric studies covered by newspapers: we previously showed that initial association studies linking biomarkers or risk factors to psychiatric disorders were less often confirmed by corresponding meta-analyses than those related to neurolo…”
BackgroundMethodResultSelf-citationHorwitz, A. V. (2005) · The Journals of Gerontology Series B · doi:10.1093/geronb/60.special_issue_2.s48 · Cited by 29
Society needs more than wonder to respect science
Watts, S. (2014) · Nature · doi:10.1038/508151a · Cited by 12 · cited 2×
“Therefore, with Susan Watts [35], we advocate that society "needs science journalism to weigh up the values and the vices of new science" (p. 151).”
BackgroundWhat should be said to the lay public regarding ADHD etiology
Gonon, F., Bézard, E., & Boraud, T. (2011) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.31236 · Cited by 8 · cited 1×
“Importantly, such biased newspaper coverage can have important social consequences [32, 33].”
BackgroundSelf-citation
Cited by
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Modernizing the Bradford Hill criteria for assessing causal relationships in observational data
Cox, L. A. (2018) · Critical Reviews in Toxicology · doi:10.1080/10408444.2018.1518404 · Cited by 115
“…meta-analyses, although such negative findings are seldom reported: “Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim” (Dumas-Mallet et al. 2017).”
BackgroundResultHaber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196346 · Cited by 97
“The interaction between academia and media highlights some of these complexities, such as a recent study finding that the results of over half of studies on an association between exposure and outcome covered in the news are refuted by subsequent meta-analyses [33], and another showing that those articles publicly cri…”
BackgroundHeterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 103
The drug delivery field at the inflection point: Time to fight its way out of the egg
Park, K. (2017) · Journal of Controlled Release · doi:10.1016/j.jconrel.2017.07.030 · Cited by 74
Metadata
- Authors
- Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Dumas-Mallet, Estelle
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-02-21
- Volume
- 12
- Issue
- 2
- Pages
- e0172650
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5319681
- OpenAlex
- W2589877982
- Cited by
- 76
- References
- 39
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 7 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 18 · 2019: 8 · 2018: 12 · 2017: 8
- FWCI
- 11.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Oskar-Bordeaux (Universite de Bordeaux) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- newspaper coverage of science, meta-analysis, negative results, lifestyle risk factors, genetic risk factors, psychiatric research, brain research, somatic diseases, science journalism
- MeSH terms
- Humans, Internal Medicine, Life Style, Neurology, Newspapers as Topic, Periodicals as Topic, Psychiatry, Risk Factors, Meta-Analysis as Topic, Reproducibility of Results, Health Behavior, Causality, Databases, Factual, Journalism, Information Dissemination, Biomedical Research, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.