The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Incongruence between test statistics and P values in medical papers
Open accessAbstract
BACKGROUND: Given an observed test statistic and its degrees of freedom, one may compute the observed P value with most statistical packages. It is unknown to what extent test statistics and P values are congruent in published medical papers. METHODS: We checked the congruence of statistical results reported in all the papers of volumes 409-412 of Nature (2001) and a random sample of 63 results from volumes 322-323 of BMJ (2001). We also tested whether the frequencies of the last digit of a sample of 610 test statistics deviated from a uniform distribution (i.e., equally probable digits). RESULTS: 11.6% (21 of 181) and 11.1% (7 of 63) of the statistical results published in Nature and BMJ respectively during 2001 were incongruent, probably mostly due to rounding, transcription, or type-setting errors. At least one such error appeared in 38% and 25% of the papers of Nature and BMJ, respectively. In 12% of the cases, the significance level might change one or more orders of magnitude. The frequencies of the last digit of statistics deviated from the uniform distribution and suggested digit preference in rounding and reporting. CONCLUSIONS: This incongruence of test statistics and P values is another example that statistical practice is generally poor, even in the most renowned scientific journals, and that quality of papers should be more controlled and valued.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“two studies evaluated p-values in the medical field (Garcia-Berthou & Alcaraz, 2004) and in psychiatry (Berle & Starcevic, 2007) instead of in psychology.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Garcia-Berthou & Alcaraz, 2004
The cited source says what the claim attributes to it.
In the citing paper
“The percentage of gross inconsistencies ranged from .4 % (Garcia-Berthou & Alcaraz, 2004) to 2.3 % (Caperos & Pardo, 2013), with a median of 1.1 % (.3 percentage points lower than the 1.4 % found in the current study).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Garcia-Berthou & Alcaraz, 2004
In the citing paper
“Twenty-five to 38% of medical articles (74), and up to 50% in psychology papers (47), contain p-values that don't match the reported test-statistic and degrees of freedom.”
From The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials (2021) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897
Statistical guidelines for contributors to medical journals.
Altman, D. G., Gore, S. M., Gardner, M. J., & Pocock, S. (1983) · BMJ · doi:10.1136/bmj.286.6376.1489 · Cited by 662
Comparison of pregnancy dating by last menstrual period, ultrasound scanning, and their combination
Savitz, D. A., Terry, J. W., Dole, N., Thorp, J. M., Siega-Riz, A. M., & Herring, A. H. (2002) · American Journal of Obstetrics and Gynecology · doi:10.1067/mob.2002.127601 · Cited by 437
Guidelines for Statistical Reporting in Articles for Medical Journals
Bailar, J. C., & MOSTELLER, F. (1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-2-266 · Cited by 387
On the accuracy of statistical procedures in Microsoft Excel 2003
McCullough, B., & Wilson, B. K. (1999) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(99)00004-3 · Cited by 299
Statistics in medical journals: developments in the 1980s.
Altman, D. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101206 · Cited by 267
On the accuracy of statistical procedures in Microsoft Excel 2010
Mélard, G. (2014) · Computational Statistics · doi:10.1007/s00180-014-0482-5 · Cited by 238
On the accuracy of statistical procedures in Microsoft Excel 2007
McCullough, B. D., & Heiser, D. A. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.03.004 · Cited by 238
Statistics in medical journals
Altman, D. G. (1982) · Statistics in Medicine · doi:10.1002/sim.4780010109 · Cited by 227
Improving Doctors' Understanding of Statistics
Altman, D. G., & Bland, J. M. (1991) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983040 · Cited by 193
Statistical reviewing for medical journals
Altman, D. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981215)17:23 · Cited by 157
Ehrenberg, A. S. C. (1977) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344922 · Cited by 153
Assessing the Reliability of Statistical Software: Part I
McCullough, B. D. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480597 · Cited by 144
Sharp, S. J., & Thompson, S. G. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001215)19:23 · Cited by 99
The Reporting of Computation-Based Results in Statistics
Hoaglin, D. C., & Andrews, D. F. (1975) · The American Statistician · doi:10.1080/00031305.1975.10477393 · Cited by 90
Ehrenberg, A. S. C. (1981) · The American Statistician · doi:10.1080/00031305.1981.10479310 · Cited by 75
Diaconis, P. W., & Freedman, D. A. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10482518 · Cited by 68
Statistics Notes: Presentation of numerical data
Altman, D. G., & Bland, J. M. (1996) · BMJ · doi:10.1136/bmj.312.7030.572 · Cited by 67
Observer error and birthweight: digit preference in recording
Edouard, L., & Senthilselvan, A. (1997) · Public Health · doi:10.1016/s0033-3506(97)90004-4 · Cited by 65
Assessing the Reliability of Statistical Software: Part II
McCullough, B. D. (1999) · The American Statistician · doi:10.2307/2685736 · Cited by 51
Should There Be Statistical Guidelines for Medical Research Papers?
O'Fallon, J. R., Dubey, S. D., Salsburg, D. S., Edmonson, J. H., Soffer, A., & Colton, T. (1978) · Biometrics · doi:10.2307/2530390 · Cited by 50
Assessing the Reliability of Statistical Software: Part II
McCullough, B. D. (1999) · The American Statistician · doi:10.1080/00031305.1999.10474450 · Cited by 44
Increasing the accessibility of data
Smith, G. D. (1994) · BMJ · doi:10.1136/bmj.308.6943.1519 · Cited by 43
Publishing raw data and real time statistical analysis on e-journals
Hutchon, D. J. R. (2001) · BMJ · doi:10.1136/bmj.322.7285.530 · Cited by 37
Poor-Quality Medical Research: What Can Journals Do?
Altman, D. G. (2002) · JAMA · doi:10.1001/jama.287.21.2765 · Cited by 34
Delamothe, T. (1996) · BMJ · doi:10.1136/bmj.312.7041.1241 · Cited by 31
Quality assessment of data on blood pressure in the WHO MONICA Project
Kuulasmaa, K., Hense, H., & Tolonen, H. K. (1998) · STM:n Hallinnonalan avoin julkaisuarkisto (Julkari) · Cited by 15
Hawkins, B. A. (1999) · Nature · doi:10.1038/22862 · Cited by 9
Editorial Code for Presentation of Statistical Analyses
Finney, D. J., & Harper, J. L. (1994) · Experimental Agriculture · doi:10.1017/s0014479700024637 · Cited by 7
Preprocessing histograms of age at menopause using the fast Fourier transform
Greer, W. (2003) · Maturitas · doi:10.1016/s0378-5122(03)00039-2 · Cited by 7
Clemons, T. E., & Pagano, M. (1999) · The American Statistician · doi:10.1080/00031305.1999.10474480 · Cited by 3
Eliminating Bias Due to Rounding
Green, D. (1990) · Teaching Statistics · doi:10.1111/j.1467-9639.1990.tb00119.x · Cited by 2
Research works printed but not linked: 1
- Benford F: The Law of Anomalous Numbers. Proc Amer Phil Soc. 1938, 78: 551-572.
Cited by
Sorted by citations
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Serdar, C. C., Cihan, M., Yücel, D., & Serdar, M. A. (2020) · Biochemia Medica · doi:10.11613/bm.2021.010502 · Cited by 1,868
A dirty dozen: twelve p-value misconceptions.
Goodman, S. (2008) · Seminars in hematology (Print) · doi:10.1053/j.seminhematol.2008.04.003 · Cited by 694
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Metadata
- Authors
- García-Berthou, E., & Alcaraz, C.
- Venue
- BMC Medical Research Methodology
- Published
- 2004-05-28
- Volume
- 4
- Issue
- 1
- Pages
- 13-13
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC443510
- OpenAlex
- W2162929498
- Cited by
- 181
- References
- 31
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 6 · 2023: 5 · 2022: 5 · 2021: 5 · 2020: 10 · 2019: 11 · 2018: 6 · 2017: 12 · 2016: 17 · 2015: 9 · 2014: 4 · 2013: 10 · 2012: 8
- FWCI
- 12.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion)
- PubMed Central (publishedVersion, other-oa)
- PubMed (publishedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Universitat de Girona Digital Repository (Universitat de Girona) (submittedVersion, cc-by)
- Repositori UJI (Universitat Jaume I) (submittedVersion, other-oa)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- test statistics, p-values, statistical reporting errors, rounding errors, digit preference, medical research papers, nature, British Medical Journal
- MeSH terms
- Data Interpretation, Statistical, Humans, Periodicals as Topic, Quality Control, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
