Peer review of statistics in medical research: the other problem
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Peer review has long been criticised for failing to identify flaws in research. Here Peter Bacchetti argues that it is also guilty of the opposite: finding flaws that are not there The process of peer review before publication has long been criticised for failing to prevent the publication of statistics that are wrong, unclear, or suboptimal. 1 2 My concern here, however, is not with failing to find flaws, but with the complementary problem of finding flaws that are not really there. My impression as a collaborating and consulting statistician is that spurious criticism of sound statistics is increasingly common, mainly from subject matter reviewers with limited statistical knowledge. Of the subject matter manuscript reviews I see that raise statistical issues, perhaps half include a mistaken criticism. In grant reviews unhelpful statistical comments seem to be a near certainty, mainly due to unrealistic expectations concerning sample size planning. While funding or publication of bad research is clearly undesirable, so is preventing the funding or publication of good research. Responding to misguided comments requires considerable time and effort, and poor reviews are demoralising—a subtler but possibly more serious cost. This paper discusses the problem, its causes, and what might improve the situation. Although the main focus is on statistics, many of the causes and potential improvements apply to peer review generally. #### Summary points Peer reviewers often make unfounded statistical criticisms, particularly in difficult areas such as sample size and multiple comparisons These spurious statistical comments waste time and sap morale Reasons include overvaluation of criticism for its own sake, inappropriate statistical dogmatism, time pressure, and lack of rewards for good peer reviewing Changes in the culture of peer review could improve things, particularly honouring good performance Mistaken criticism is a general problem, but may be especially acute for statistics. The examples below illustrate …
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333
Does Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278
What Makes a Good Reviewer and a Good Review for a General Medical Journal?
Black, N. (1998) · JAMA · doi:10.1001/jama.280.3.231 · Cited by 266
Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Callaham, M. L. (1998) · JAMA · doi:10.1001/jama.280.3.229 · Cited by 122
Cited by
Likert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850
“As Bacchetti (2002) has pointed out, many of these comments are unfounded or wrong, and appear to result from a review culture that encourages ‘‘overvaluation of criticism for its own sake, inappropriate statistical dogmatism’’, and is subject to ‘‘time pressure, and lack of rewards for good peer…”
Association of Amyloid and Tau With Cognition in Preclinical Alzheimer Disease: A Longitudinal Study
Hanseeuw, B. J., Betensky, R. A., Jacobs, H. I. L., Schultz, A. P., Sepulcre, J., Becker, J. A., Cosio, D. M. O., Farrell, M., Quiroz, Y. T., Mormino, E. C., et al. (2019) · JAMA Neurology · doi:10.1001/jamaneurol.2019.1424 · Cited by 865
Supporting thinking on sample sizes for thematic analyses: a quantitative tool
Fugard, A. J. B., & Potts, H. W. W. (2015) · International Journal of Social Research Methodology · doi:10.1080/13645579.2015.1005453 · Cited by 749
Local Standards for Sample Size at CHI
Caine, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858498 · Cited by 657
CD4+ T Cells Expressing PD-1, TIGIT and LAG-3 Contribute to HIV Persistence during ART
Fromentin, R., Bakeman, W., Lawani, M. B., Khoury, G., Hartogensis, W., DaFonseca, S., Killian, M., Epling, L., Hoh, R., Sinclair, E., et al. (2016) · PLOS Pathogens · doi:10.1371/journal.ppat.1005761 · Cited by 422
Self-citationMuiesan, M. L., Salvetti, M., Monteduro, C., Bonzi, B., Paini, A., Viola, S., Poisa, P., Rizzoni, D., Castellano, M., & Agabiti-Rosei, E. (2004) · Hypertension · doi:10.1161/01.hyp.0000121223.44837.de · Cited by 330
Editorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub3 · Cited by 312
“Characteristics of excluded studies [ordered by study ID] Abby 1994 Non-comparative study Bacchetti 2002 Non-comparative study Baxt 1998 A study of association of reviewer characteristics and performance 32Editorial peer review for improving the quality of reports of biomedical studies (Review)…”
BackgroundReporting of sample size calculation in randomised controlled trials: review
Charles, P., Giraudeau, B., Dechartres, A., Baron, G., & Ravaud, P. (2009) · BMJ · doi:10.1136/bmj.b1732 · Cited by 274
Repeatable evaluation of search services in dynamic environments
Jensen, E. C., Beitzel, S. M., Chowdhury, A., & Frieder, O. (2007) · ACM Transactions on Information Systems · doi:10.1145/1292591.1292592 · Cited by 269
Mansournia, M. A., Collins, G. S., Nielsen, R. O., Nazemipour, M., Jewell, N. P., Altman, D. G., & Campbell, M. J. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-103652 · Cited by 263
Morphine Does Not Provide Adequate Analgesia for Acute Procedural Pain Among Preterm Neonates
Carbajal, R., Lenclen, R., Jugie, M., Paupe, A., Barton, B. A., & Anand, K. J. S. (2005) · Pediatrics · doi:10.1542/peds.2004-1425 · Cited by 255
Current sample size conventions: Flaws, harms, and alternatives
Bacchetti, P. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-17 · Cited by 232
Self-citationA Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
Breaking Free of Sample Size Dogma to Perform Innovative Translational Research
Bacchetti, P., Deeks, S. G., & McCune, J. M. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3001628 · Cited by 159
“We therefore recommend that investigators pursue all issues of potential interest regardless of whether they were prespeci ed, avoid formal adjustments to P values for “multiple comparisons” (31, 32), and adjust methods and designs in response to early results when necessary.”
MethodSelf-citationIn vivo evidence of glutamate toxicity in multiple sclerosis
Azevedo, C. J., Kornak, J., Chu, P., Sampat, M., Okuda, D. T., Cree, B. A., Nelson, S. J., Hauser, S. L., & Pelletier, D. (2014) · Annals of Neurology · doi:10.1002/ana.24202 · Cited by 100
CLINICAL RESEARCH AND STATISTICAL METHODS IN THE UROLOGY LITERATURE
SCALES, C. D., NORRIS, R. D., PETERSON, B. L., PREMINGER, G. M., & DAHM, P. (2005) · Journal of Urology · doi:10.1097/01.ju.0000173640.91654.b5 · Cited by 87
Fernandes-Taylor, S., Hyun, J. K., Reeder, R. N., & Harris, A. H. (2011) · BMC Research Notes · doi:10.1186/1756-0500-4-304 · Cited by 75
“Rather, statistical experts are often consulted only during the analysis phase, if at all, and even then may not be credited with authorship [35].”
BackgroundShafer, A. T., Beason-Held, L., An, Y., Williams, O. A., Huo, Y., Landman, B. A., Caffo, B. S., & Resnick, S. M. (2021) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2021.03.013 · Cited by 16
Hooker, A. R., Sagui-Henson, S. J., Daubenmier, J., Moran, P. J., Hartogensis, W., Acree, M., Kristeller, J., Epel, E. S., Mason, A. E., & Hecht, F. M. (2022) · Mindfulness · doi:10.1007/s12671-022-01951-2 · Cited by 6
“The overall pattern of results with multiple outcomes is important to assess, as has been advocated by many methodological experts (Bacchetti, 2002).”
Background
Metadata
- Authors
- Bacchetti, P.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2002-05-25
- Volume
- 324
- Issue
- 7348
- Pages
- 1271.1-1273
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2017627972
- Cited by
- 156
- References
- 6
- Type
- peer review
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.