The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies
Open accessPublished version of a preprint
Abstract
We investigated whether statistical-reporting inconsistencies could be avoided if journals implement the tool statcheck in the peer-review process. In a preregistered pretest–posttest quasi-experiment covering more than 7,000 articles and more than 147,000 extracted statistics, we compared the prevalence of reported p values that were inconsistent with their degrees of freedom and test statistics in two journals that implemented statcheck in their peer-review process ( Psychological Science and Journal of Experimental and Social Psychology ) and two matched control journals ( Journal of Experimental Psychology: General and Journal of Personality and Social Psychology ) before and after statcheck was implemented. Preregistered multilevel logistic regression analyses showed that the decrease in both inconsistencies and decision inconsistencies around p = .05 is considerably steeper in statcheck journals than in control journals, offering preliminary support for the notion that statcheck can be a useful tool for journals to avoid statistical-reporting inconsistencies in published articles. We discuss limitations and implications of these findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Publication Manual of the American Psychological Association (4th ed.).
Gullickson, T. (1996) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/004700 · Cited by 1,240
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationThe prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 7×
“We estimate that statcheck detects roughly 60% of all reported NHST results (Nuijten et al., 2016).”
InfluentialSelf-citationReplication in Psychological Science
Lindsay, D. S. (2015) · Psychological Science · doi:10.1177/0956797615616374 · Cited by 280 · cited 2×
“Indeed, around the same time that statcheck was introduced, both PS and JESP published editorials emphasizing the importance of open practices and replication (Giner-Sorolla, 2016; Lindsay, 2015).”
InfluentialStatistical Reform in Psychology: Is Anything Changing?
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01881.x · Cited by 122 · cited 1×
“Many conclusions in psychological research are based on the results of null hypothesis significance tests (NHSTs; Cumming et al., 2007; Nuijten et al., 2016).”
Association for Psychological Science
Kraut, A., & Brookhart, S. (2010) · The Corsini Encyclopedia of Psychology · doi:10.1002/9780470479216.corpsy0087 · Cited by 111
Approximated adjusted fractional Bayes factors: A general method for testing informative hypotheses
Gu, X., Mulder, J., & Hoijtink, H. (2017) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12110 · Cited by 100 · cited 1×
“We computed approximated adjusted fractional Bayes factors (Gu et al., 2018) using the default implementation in the R package BFpack (Version 1.0.0; Mulder et al., 2021).”
InfluentialSociety for Research in Child Development
(2012) · SpringerReference · doi:10.1007/springerreference_180608 · Cited by 99
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79 · cited 6×
“Whether this is problematic is debatable, but we previously argued that such corrections can and should be reported in a way that does not render the full result inconsistent (Nuijten et al., 2017) to allow a reproducible report of applied corrections.”
InfluentialSelf-citation
Research works printed but not linked: 3
- JESP piloting the use of statcheck. (2017 August 16). https://www.sciencedirect.com/journal/journal-of-experimental-social-psychology/about/news#jesp-piloting-the-use-of-statcheck
- The Journal of Consumer Research. (2021). JCR’s data policy in practice. https://web.archive.org/web/20230606191958/https://consumerresearcher.com/jcrs-data-policy-in-practice
- Journal of Experimental Social Psychology. (2022). Guide for authors. https://web.archive.org/web/20221205222546/https://www.elsevier.com/journals/journal-of-experimental-social-psychology/0022-1031/guide-for-authors
Cited by
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28
“…pointed us to a study that compared the levels of statistical inconsistencies between journals that implemented StatCheck and those that did not [132], and we retrieved through EPPI Reviewer and OpenAlex a preprint concluding that providing article templates does not improve completeness of…”
On the Dearth of Retractions in Social Work: A Cross-Sectional Study of Ten Leading Journals
Dunleavy, D. J. (2025) · Metrics · doi:10.3390/metrics2030016 · Cited by 4
“…available as a web app (https://michelenuijten.shinyapps.io/ statcheck-web/; accessed on 20 August 2024) and R package, extracts the results of significance tests from manuscripts, recomputes reported p -values based on the associated test statistics and degrees of freedom (where reported) [117].”
ReproduceMe: Lessons from a pilot project on computational reproducibility
Baker, D. H., Berg, M., Hansford, K. J., Quinn, B. P. A., Segala, F. G., & Warden-English, E. L. (2024) · Meta-Psychology · doi:10.15626/mp.2023.4021 · Cited by 4
Over half of bilingual advantage articles have a statistical reporting error
Schroeder, S. R. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00147-5 · Cited by 1
ReproduceMe: lessons from a pilot project on computational reproducibilityPreprint
Baker, D. H., Berg, M. G. V. D., Hansford, K. J., et al. (2023) · doi:10.31234/osf.io/k8d4u · Cited by 1
Metadata
- Authors
- Nuijten, M. B., & Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- Corresponding author
- Nuijten, Michèle B.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2024-04
- Volume
- 7
- Issue
- 2
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W4400067224
- Cited by
- 6
- References
- 17
- Citations by year
- 2025: 4 · 2024: 2
- FWCI
- 2.45
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (unspecified)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion, cc-by)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- ERC Consolidator (grant number 726361)
- NWO Veni (grant number 11507)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- statcheck, statistical reporting errors, peer review, p-values, degrees of freedom, test statistics, editorial practice, reporting consistency, preregistration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies. https://app.cochat.ai/papers/10.1177/25152459241258945
- The paper
- Nuijten, M. B., & Wicherts, J. M. (2024). Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/25152459241258945
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.