CoChat

Explosion of formulaic research articles, including inappropriate study designs and false discoveries, based on the NHANES US national health database

Open access
Suchak, Tulsi, Aliu, Anietie E., Harrison, Charlie, Zwiggelaar, Reyer, Geifman, Nophar, Spick, Matt · 2025 · PLOS Biology

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

With the growth of artificial intelligence (AI)-ready datasets such as the National Health and Nutrition Examination Survey (NHANES), new opportunities for data-driven research are being created, but also generating risks of data exploitation by paper mills. In this work, we focus on two areas of potential concern for AI-supported research efforts. First, we describe the production of large numbers of formulaic single-factor analyses, relating single predictors to specific health conditions, where multifactorial approaches would be more appropriate. Employing AI-supported single-factor approaches removes context from research, fails to capture interactions, avoids false discovery correction, and is an approach that can easily be adopted by paper mills. Second, we identify risks of selective data usage, such as analyzing limited date ranges or cohort subsets without clear justification, suggestive of data dredging, and post-hoc hypothesis formation. Using a systematic literature search for single-factor analyses, we identified 341 NHANES-derived research papers published over the past decade, each proposing an association between a predictor and a health condition from the wide range contained within NHANES. We found evidence that research failed to take account of multifactorial relationships, that manuscripts did not account for the risks of false discoveries, and that researchers selectively extracted data from NHANES rather than utilizing the full range of data available. Given the explosion of AI-assisted productivity in published manuscripts (the systematic search strategy used here identified an average of 4 papers per annum from 2014 to 2021, but 190 in 2024–9 October alone), we highlight a set of best practices to address these concerns, aimed at researchers, data controllers, publishers, and peer reviewers, to encourage improved statistical practices and mitigate the risks of paper mills using AI-assisted workflows to introduce low-quality manuscripts to the scientific literature.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.3003152

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 PRISMA four-phase flow diagram of the article identification process.
Fig 1 PRISMA four-phase flow diagram of the article identification process.CC BY · PubMed Central
Fig 2 Summary of distribution of journals: Tree map of 341 manuscripts by journal, grouped by publisher family. BioMed Central is part of the Springer Nature publishing group but is shown as the BMC family of journals here. Complete data included in S1 Data Table A. Colors are used to assist visual separation of blocks and do not represent data types or categories.
Fig 2 Summary of distribution of journals: Tree map of 341 manuscripts by journal, grouped by publisher family. BioMed Central is part of the Springer Nature publishing group but is shown as the BMC family of journals here. Complete data included in S1 Data Table A. Colors are used to assist visual separation of blocks and do not represent data types or categories.CC BY · PubMed Central
Fig 3 Number of publications by year: (A) single-factor NHANES analyses identified in this review (B) total publications by year identified by a PubMed search for “biobank”. Both data series show publication counts for 2024 to 9 October only. The data underlying these figures are additionally reported in S1 Data Table D.
Fig 3 Number of publications by year: (A) single-factor NHANES analyses identified in this review (B) total publications by year identified by a PubMed search for “biobank”. Both data series show publication counts for 2024 to 9 October only. The data underlying these figures are additionally reported in S1 Data Table D.CC BY · PubMed Central
Fig 4 Network analysis of health conditions and their associations with predictor variables from NHANES published research focusing on single-factor relationships. Outer shell nodes are predictor variables, inner shell nodes are health conditions with four or more associated predictor variables and edges represent a paper in this systematic review describing a statistically significant single-factor relationship. Diameters of nodes are proportional to the number of papers. Only nodes with more than four edges are labeled. The data underlying these figures are taken from the pivot tables presented in S1 Data Tables B and C.
Fig 4 Network analysis of health conditions and their associations with predictor variables from NHANES published research focusing on single-factor relationships. Outer shell nodes are predictor variables, inner shell nodes are health conditions with four or more associated predictor variables and edges represent a paper in this systematic review describing a statistically significant single-factor relationship. Diameters of nodes are proportional to the number of papers. Only nodes with more than four edges are labeled. The data underlying these figures are taken from the pivot tables presented in S1 Data Tables B and C.CC BY · PubMed Central
Fig 5 UpSet plot illustrating selections of annual data taken from NHANES that were employed for the 341 publications analyzed in this work. The data are taken from S1 Data Table A, using the start dates to the end dates of subsets of NHANES data analyzed by each manuscript.
Fig 5 UpSet plot illustrating selections of annual data taken from NHANES that were employed for the 341 publications analyzed in this work. The data are taken from S1 Data Table A, using the start dates to the end dates of subsets of NHANES data analyzed by each manuscript.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 31 references

Research works printed but not linked: 1

  1. About COPE.

Cited by

See all 113

Metadata

Authors
Suchak, T., Aliu, A. E., Harrison, C., Zwiggelaar, R., Geifman, N., & Spick, M.
Affiliations
  • University of Surrey (GB)
  • Aberystwyth University (GB)
Corresponding author
Spick, Matt
Venue
PLOS Biology
Published
2025-05-08
Volume
23
Issue
5
Pages
e3003152
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
OpenAlex
W4410206242
Cited by
95
References
30
Citations by year
2026: 54 · 2025: 41
FWCI
98.30
Type
journal article
Language
en
Open access
Locations
  • PLoS Biology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
Biotechnology and Biological Sciences Research Council (BB/Y006933/1)
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
paper mills, single factor analysis, NHANES, artificial intelligence in research, p-hacking, false discovery, multifactorial analysis, research quality, statistical practice, peer review
MeSH terms
Artificial Intelligence, Humans, Nutrition Surveys, Research Design, United States, Databases, Factual

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.