CoChat
Sign Up
Cited by 182Benford’s Law and Fraud DetectionMeta-analysis and systematic reviewsStatistical Methods in Epidemiology

Incongruence between test statistics and P values in medical papers

Open access
García-Berthou, Emili, Alcaraz, Carles·2004·BMC Medical Research Methodology
DOI 10.1186/1471-2288-4-13PMC PMC443510
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Given an observed test statistic and its degrees of freedom, one may compute the observed P value with most statistical packages. It is unknown to what extent test statistics and P values are congruent in published medical papers. METHODS: We checked the congruence of statistical results reported in all the papers of volumes 409-412 of Nature (2001) and a random sample of 63 results from volumes 322-323 of BMJ (2001). We also tested whether the frequencies of the last digit of a sample of 610 test statistics deviated from a uniform distribution (i.e., equally probable digits). RESULTS: 11.6% (21 of 181) and 11.1% (7 of 63) of the statistical results published in Nature and BMJ respectively during 2001 were incongruent, probably mostly due to rounding, transcription, or type-setting errors. At least one such error appeared in 38% and 25% of the papers of Nature and BMJ, respectively. In 12% of the cases, the significance level might change one or more orders of magnitude. The frequencies of the last digit of statistics deviated from the uniform distribution and suggested digit preference in rounding and reporting. CONCLUSIONS: This incongruence of test statistics and P values is another example that statistical practice is generally poor, even in the most renowned scientific journals, and that quality of papers should be more controlled and valued.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2288-4-13

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two studies evaluated p-values in the medical field (Garcia-Berthou & Alcaraz, 2004) and in psychiatry (Berle & Starcevic, 2007) instead of in psychology.

    “two studies evaluated p-values in the medical field (Garcia-Berthou & Alcaraz, 2004) and in psychiatry (Berle & Starcevic, 2007) instead of in psychology.”

    From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Garcia-Berthou & Alcaraz, 2004

  • The percentage of gross inconsistencies ranged from .4% (Garcia-Berthou & Alcaraz, 2004) to 2.3% (Caperos & Pardo, 2013), with a median of 1.1%, .3 percentage points lower than the 1.4% found in the current study.

    “The percentage of gross inconsistencies ranged from .4 % (Garcia-Berthou & Alcaraz, 2004) to 2.3 % (Caperos & Pardo, 2013), with a median of 1.1 % (.3 percentage points lower than the 1.4 % found in the current study).”

    From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Garcia-Berthou & Alcaraz, 2004

  • Twenty-five to 38% of medical articles, and up to 50% of psychology papers, contain p-values that do not match the reported test statistic and degrees of freedom.

    “Twenty-five to 38% of medical articles (74), and up to 50% in psychology papers (47), contain p-values that don't match the reported test-statistic and degrees of freedom.”

    From The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials (2021) · cites this paper as [74]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
García-Berthou, E., & Alcaraz, C.
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2004-05-28
Volume
4
Issue
1
Pages
13-13
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/1471-2288-4-13
PMC
PMC443510
OpenAlex
W2162929498
Cited by
181
References
31
Citations by year
2026: 4 · 2025: 2 · 2024: 6 · 2023: 5 · 2022: 5 · 2021: 5 · 2020: 10 · 2019: 11 · 2018: 6 · 2017: 12 · 2016: 17 · 2015: 9 · 2014: 4 · 2013: 10 · 2012: 8
FWCI
12.49
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (unspecified)
Locations
  • BMC Medical Research Methodology (publishedVersion)
  • PubMed Central (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Universitat de Girona Digital Repository (Universitat de Girona) (submittedVersion, cc-by)
  • Repositori UJI (Universitat Jaume I) (submittedVersion, other-oa)
  • RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
test statistics, p-values, statistical reporting errors, rounding errors, digit preference, medical research papers, nature, British Medical Journal
MeSH terms
Data Interpretation, Statistical, Humans, Periodicals as Topic, Quality Control, Confidence Intervals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review