The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
There is widespread evidence of the extensive use of statistical methods in medical research. Just the same, standards are generally low and a growing body of literature points to statistical errors in most medical journals. However, there is no comprehensive study contrasting the top medical journals of basic and clinical science for recent practice in their use of statistics.All original research articles in Volume 10, Numbers 1–6 of Nature Medicine (Nat Med) and Volume 350, Numbers 1–26 of The New England Journal of Medicine (NEJM) were screened for their statistical content. Types, frequencies, and complexity of applied statistical methods were systematically recorded. A 46-item checklist was used to evaluate statistical quality for a subgroup of papers.94.5 percent (95% CI 87.6–98.2) of NEJM articles and 82.4 percent (95% CI 65.5–93.2) of Nat Med articles contained inferential statistics. NEJM papers were significantly more likely to use advanced statistical methods (p < 0.0001). Statistical errors were identified in a considerable proportion of articles, although not always serious in nature. Documentation of applied statistical methods was generally poor and insufficient, particularly in Nat Med.Compared to 1983, a vast increase in usage and complexity of statistical methods could be observed for NEJM papers. This does not necessarily hold true for Nat Med papers, as the results of the study indicate that basic science sticks with basic analysis. As statistical errors seem to remain common in medical literature, closer attention to statistical methodology should be seriously considered to raise standards.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
Cited by
A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003
Austin, P. C. (2008) · Statistics in Medicine · doi:10.1002/sim.3150 · Cited by 1,249
Austin, P. C. (2007) · The Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2007.07.021 · Cited by 589
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
To test or not to test: Preliminary assessment of normality when comparing two independent samples
Rochon, J., Gondan, M., & Kieser, M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-81 · Cited by 296
“[7] criticized that contributors to medical journals often failed to examine and report that assumptions had been met when conducting Student’s t test.”
BackgroundMethodCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Metadata
- Authors
- Strasak, A. M., Zaman, Q., Marinell, G., Pfeiffer, K. P., & Ulmer, H.
- Venue
- The American Statistician
- Publisher
- Informa UK Limited
- Published
- 2007-02
- Volume
- 61
- Issue
- 1
- Pages
- 47-55
- ISSN
- 0003-1305 (ISSN-L)
- 1537-2731
- OpenAlex
- W1575656708
- Cited by
- 103
- References
- 2
- FWCI
- 2.70
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine. https://app.cochat.ai/papers/10.1198/000313007x170242
- The paper
- Strasak, A. M., Zaman, Q., Marinell, G., Pfeiffer, K. P., & Ulmer, H. (2007). The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine. The American Statistician. https://doi.org/10.1198/000313007x170242
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.