Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies

Open access
Nuijten, Michèle B., Wicherts, Jelte M. · 2024 · Advances in Methods and Practices in Psychological Science

Published version of a preprint

Abstract

We investigated whether statistical-reporting inconsistencies could be avoided if journals implement the tool statcheck in the peer-review process. In a preregistered pretest–posttest quasi-experiment covering more than 7,000 articles and more than 147,000 extracted statistics, we compared the prevalence of reported p values that were inconsistent with their degrees of freedom and test statistics in two journals that implemented statcheck in their peer-review process ( Psychological Science and Journal of Experimental and Social Psychology ) and two matched control journals ( Journal of Experimental Psychology: General and Journal of Personality and Social Psychology ) before and after statcheck was implemented. Preregistered multilevel logistic regression analyses showed that the decrease in both inconsistencies and decision inconsistencies around p = .05 is considerably steeper in statcheck journals than in control journals, offering preliminary support for the notion that statcheck can be a useful tool for journals to avoid statistical-reporting inconsistencies in published articles. We discuss limitations and implications of these findings.

Overview

Integrity signals

Registry

Crossref · 10.1177/25152459241258945

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 17 references

Research works printed but not linked: 3

  1. JESP piloting the use of statcheck. (2017 August 16). https://www.sciencedirect.com/journal/journal-of-experimental-social-psychology/about/news#jesp-piloting-the-use-of-statcheck
  2. The Journal of Consumer Research. (2021). JCR’s data policy in practice. https://web.archive.org/web/20230606191958/https://consumerresearcher.com/jcrs-data-policy-in-practice
  3. Journal of Experimental Social Psychology. (2022). Guide for authors. https://web.archive.org/web/20221205222546/https://www.elsevier.com/journals/journal-of-experimental-social-psychology/0022-1031/guide-for-authors

Cited by

See all 7

Metadata

Authors
Nuijten, M. B., & Wicherts, J. M.
Affiliations
Tilburg University (NL)
Corresponding author
Nuijten, Michèle B.
Venue
Advances in Methods and Practices in Psychological Science
Published
2024-04
Volume
7
Issue
2
ISSN
  • 2515-2459 (ISSN-L)
  • 2515-2467
OpenAlex
W4400067224
Cited by
6
References
17
Citations by year
2025: 4 · 2024: 2
FWCI
2.45
Type
journal article
Language
en
Open access
Licence
Locations
  • Advances in Methods and Practices in Psychological Science (publishedVersion, cc-by)
  • Research portal (Tilburg University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Funders
  • ERC Consolidator (grant number 726361)
  • NWO Veni (grant number 11507)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
statcheck, statistical reporting errors, peer review, p-values, degrees of freedom, test statistics, editorial practice, reporting consistency, preregistration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies. https://app.cochat.ai/papers/10.1177/25152459241258945
The paper
Nuijten, M. B., & Wicherts, J. M. (2024). Implementing Statcheck During Peer Review Is Related to a Steep Decline in Statistical-Reporting Inconsistencies. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/25152459241258945

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.