CoChat

Hasty generalizations and generics in medical research: A systematic review

Open access
Peters, Uwe, Sherling, Henrik Røed, Chin-Yee, Benjamin · 2024 · PLOS ONE

The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index; 14 of the 67 sources its claims cite were checked for support.

Abstract

It is unknown to what extent medical researchers generalize study findings beyond their samples when their sample size, sample diversity, or knowledge of conditions that support external validity do not warrant it. It is also unknown to what extent medical researchers describe their results with precise quantifications or unquantified generalizations, i.e., generics, that can obscure variations between individuals. We therefore systematically reviewed all prospective studies (n = 533) published in the top four highest ranking medical journals, Lancet, New England Journal of Medicine (NEJM), Journal of the American Medical Association (JAMA), and the British Medical Journal (BMJ), from January 2022 to May 2023. We additionally reviewed all NEJM Journal Watch clinical research summaries (n = 143) published during the same time. Of all research articles reporting prospective studies, 52.5% included generalizations beyond specific national study populations, with the numbers of articles with generics varying significantly between journals (JAMA = 12%; Lancet = 77%) (p < 0.001, V = 0.48). There was no evidence that articles containing broader generalizations or generics were correlated with larger or more nationally diverse samples. Moreover, only 10.2% of articles with generalizations beyond specific national populations reported external validity strengthening factors that could potentially support such extrapolations. There was no evidence that original research articles and NEJM Journal Watch summaries intended for practitioners differed in their use of broad generalizations, including generics. Finally, from the journal with the highest citation impact, articles containing broader conclusions were correlated with more citations. Since there was no evidence that studies with generalizations beyond specific national study populations or with generics were associated with larger, more nationally diverse samples, or with reports of population similarity that may permit extensions of conclusions, our findings suggest that the generalizations in many articles were insufficiently supported. Caution against overly broad generalizations in medical research is warranted.

Overview

CoChat Summary

TL;DR

This systematic review of 533 prospective studies published in the four highest-impact medical journals (2022–2023) found that 52.5% contained result claims or recommendations generalizing beyond the specific national population sampled, with no evidence that such articles had larger or more nationally diverse samples. Generics appeared in 46.2% of all articles and varied sharply by journal (Lancet ~77% vs. JAMA ~12%), and the authors conclude that many of these generalizations were insufficiently supported.

Key findings

  • Of 533 articles, 285 (53.5%) were coded as generalized; 246 contained generics (86.3% of generalized articles, 46.2% of all 533). Only 5 of the 285 generalized articles mentioned sample country or region in their generalized claims, so 52.5% (n = 280) of all articles contained claims not limited to a specific national sample.
  • Journal differences were large and significant: generalized articles ranged from Lancet 83.7% to JAMA 19.1% (χ²(3) = 119.99, p < 0.001, V = 0.47); articles with generics ranged from Lancet 76.6% to JAMA 11.8% (χ²(3) = 120.87, p < 0.001, V = 0.48). Lancet vs. JAMA odds ratio for generalized scope was 21.74 (95% CI 11.32–41.75).
  • No evidence that generalized articles or articles with generics had larger samples (r = −0.09, p = 0.08; r = −0.06, p = 0.22) or higher country/region counts (r = −0.05, p = 0.29; r = −0.03, p = 0.52). Of 285 generalized articles, 201 (70.5%) had only Western or only non-Western samples, and in 194 of these (68.1%) conclusions were not limited accordingly.
  • Only 78 (27.4%) of generalized articles reported external validity strengthening factors, and only 29 (10.2%) concerned factors supporting generalization across nationalities; thus in 251 (88.1%) of generalized articles (47.1% of all 533) no reasons were offered for national similarity. Citation impact: generalized articles were only marginally higher overall (p = 0.049, r = −0.10), but for NEJM generalized articles had significantly higher impact (p = 0.005, r = −0.22) and more generalized claims correlated with higher impact (rs = 0.18, p = 0.005). NEJM Journal Watch summaries did not differ significantly from primary articles in generalized claims (χ²(1) = 0.14, p = 0.71) or generics (χ²(1) = 0.20, p = 0.66).

Methods

  • Systematic review reported per PRISMA and preregistered on OSF; included all prospective studies published January 1, 2022–May 22, 2023 in Lancet, NEJM, JAMA, and BMJ (533 articles), plus all NEJM Journal Watch summaries of prospective studies from those journals (143 summaries across 35 issues). Retrospective studies, meta-analyses, systematic reviews, and guidelines were excluded.
  • Three coders extracted title, journal, Google Scholar citation count, sample size, and sample country/region (categorized Western/non-Western), and identified all result claims across the entire article (title, highlights, abstract, introduction, discussion, conclusion), classifying each as restricted or generalized (unrestricted, hedged, or practice-related), with generics as a subset.
  • Inter-rater agreement was substantial to high (Cohen's κ = 0.80–0.97; project-naïve coders on 25% of data: κ = 0.88–0.89). Non-parametric tests (χ², Kruskal-Wallis H, Mann-Whitney U, rank-biserial correlation, Spearman correlation, ϕ) were used; impact was analyzed via Relative Citation Rate; α = 0.05, two-tailed, in JASP and SPSS 29.0.

Limitations

  • Stated: the analysis only captured whether generalizability strengthening factors were *reported*, not whether authors held relevant implicit background knowledge; only nationality was coded, not gender or age; only generalized claims (not all result claims) were counted, so restricted-vs-generalized claim proportions cannot be presented; only four top journals were included, so results may not generalize to the broader medical literature (though the authors argue estimates are likely conservative); country/region as a proxy for national diversity ignores expatriates, mixed demographics, and other diversity dimensions, and some Western countries are themselves highly diverse.
  • Evident from the text: the citation-impact analysis relies on Google Scholar counts normalized as RCR, and the overall generalized-vs-restricted impact difference was only marginally significant (p = 0.049), with the association driven by NEJM alone; the study is observational and correlational, so no causal claims about editorial policy effects on generalization use can be established (the authors themselves call for further research on this).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0306749

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 PRISMA flowcharts of the systematic reviews.
Fig 1 PRISMA flowcharts of the systematic reviews.CC BY · PubMed Central
Fig 2 The proportion of articles with generalized, generic, unrestricted, hedged, or practice-related result claims (derived by dividing the number of articles with these claims with the total number of articles of each journal). Error bars indicate standard error for the variability in proportion estimates.
Fig 2 The proportion of articles with generalized, generic, unrestricted, hedged, or practice-related result claims (derived by dividing the number of articles with these claims with the total number of articles of each journal). Error bars indicate standard error for the variability in proportion estimates.CC BY · PubMed Central
Fig 3 Number of generalized claims in total shown by claim type and journal.
Fig 3 Number of generalized claims in total shown by claim type and journal.CC BY · PubMed Central
Fig 4 Interval plot showing the mean overall generics count of each journal’s total number of articles.
Fig 4 Interval plot showing the mean overall generics count of each journal’s total number of articles.CC BY · PubMed Central
Fig 5 Comparisons of the proportions of clinical research articles and NEJM Journal Watch research summaries with generalized claims. Error bars indicate standard error for the variability in proportion estimates.
Fig 5 Comparisons of the proportions of clinical research articles and NEJM Journal Watch research summaries with generalized claims. Error bars indicate standard error for the variability in proportion estimates.CC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 0 Not valid · 32 Not checked · 103 Own results and interpretations

82 more claims

Create a free account to see all 144 claims and to verify the 53 references still open.

See all claims — free account
  • Probably not a substantial claim. 33
  • It may state two separate claims as one. 32

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentχ²(3) = 119.99reported p < .001 · recomputed p < .001

    “χ2 (3) = 119.99, p < .001”

  • Consistentχ²(3) = 120.87reported p < .001 · recomputed p < .001

    “χ2 (3) = 120.87, p < .001”

  • Consistentz = -1.97reported p = .049 · recomputed p = .049

    “z =-1.97, p = .049”

  • Consistentz = -0.311reported p = .75 · recomputed p = .756

    “z =-0.311, p = .75”

    The paper states an exact rank test

  • Consistentz = -0.13reported p = .9 · recomputed p = .897

    “z =-0.13, p = .90”

5 of 14 · the rest aren’t loaded

9 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Research works printed but not linked: 4

  1. M Krifka, 1995, The Generic Book
  2. K. Vela, 2023, NEJM Journal Watch, J Electron Resour Medical Libr, 20, 33
  3. JAMA Network. Instructions for Authors. JAMA. 2024; https://jamanetwork.com/journals/jama/pages/instructions-for-authors. Accessed April 5, 2024.
  4. The Lancet. Randomised Controlled Trials Guidelines. 2024; https://www.thelancet.com/pb/assets/raw/Lancet/authors/RCTguidelines-1668613849943.pdf. Accessed April 5, 2024.

Cited by

See all 11

Metadata

Authors
Peters, U., Sherling, H. R., & Chin-Yee, B.
Affiliations
  • Utrecht University (NL)
  • University of Cambridge (GB)
  • Western University (CA)
Corresponding author
  • Peters, Uwe
  • Sherling, Henrik Røed
  • Chin-Yee, Benjamin
Venue
PLOS ONE
Published
2024-07-05
Volume
19
Issue
7
Pages
e0306749
ISSN
1932-6203 (ISSN-L)
OpenAlex
W4400350189
Cited by
11
References
42
Citations by year
2026: 8 · 2025: 3
FWCI
1.38
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
Keywords
overgeneralization, generalizability, sample size, sample diversity, population similarity, prospective study, medical journal articles, citation impact, reporting quality
MeSH terms
Humans, Periodicals as Topic, Prospective Studies, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.