The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 33 of the 54 sources its claims cite were checked for support.
The prevalence of statistical reporting errors in psychology (1985–2013)
Open accessAbstract
This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.
Overview
CoChat Summary
TL;DR
Using the R package statcheck, this study analyzed 258,105 NHST results from 30,717 articles in eight psychology journals (1985–2013) and found that 49.6% of articles with NHST results contained at least one inconsistent p-value and 12.9% contained at least one gross inconsistency. Contrary to concerns that reporting errors are increasing, the prevalence of inconsistencies was stable or declined over time, though gross inconsistencies were more common in p-values reported as significant than nonsignificant.
Key findings
- Across all journals and years, 49.6% (8,273/16,695) of articles with NHST results contained at least one inconsistency, and 12.9% (2,150) contained at least one gross inconsistency. At the p-value level, 9.7% (24,961/258,105) were inconsistent and 1.4% (3,581) were grossly inconsistent; the mean per-article inconsistency rate was 10.6% and the gross inconsistency rate was 1.6%.
- Gross inconsistencies were more prevalent in p-values reported as significant (1.56%) than in p-values reported as nonsignificant (.97%). Reported p-values were significant 76.6% of the time, whereas computed p-values were significant 74.4% of the time, overestimating significant findings by 2.2 percentage points.
- The prevalence of inconsistencies did not generally increase from 1985 to 2013; inconsistencies decreased or remained stable across APA journals, and articles with gross inconsistencies decreased in all journals except PS and PLOS. JPSP had the highest percentage of articles with at least one inconsistency (57.6%) but one of the lowest p-value inconsistency rates (9.0%); PLoS had the highest p-value inconsistency rate (14.0%).
- NHST use increased: statcheck detected NHST results in 54.4% of articles overall, and the number of NHST results per article increased across APA journals (b = .25, R² = .68).
Methods
- Automated statcheck (version 1.0.1) extracted APA-formatted t, F, r, χ², and Z statistics with degrees of freedom and p-values from HTML articles, recalculated p-values, and flagged inconsistencies; a gross inconsistency was one that changed the statistical conclusion at α = .05. One-tailed test detection was used.
- The sample included all HTML articles from eight journals (PLOS, JPSP, JCCP, DP, JAP, PS, FP, JEPG) over years determined by HTML availability (1985–2013 for APA journals; later starts for PS, FP, and PLOS): 30,717 articles downloaded, 16,695 with detectable NHST results, and 258,105 NHST results. Analyses were descriptive only because the selected journals’ APA-reported NHST results were treated as the population.
- A validity check compared statcheck with Wicherts et al. (2011) manual coding of 49 articles: statcheck read 67.5% of manually extracted results; with one-tailed detection it found 56/775 (7.2%) inconsistencies and 8/775 (1.0%) gross inconsistencies, versus manual findings of 49/1,148 (4.3%) and 10/1,148 (.9%). Inter-rater reliability was .76 for inconsistencies and .89 for gross inconsistencies when excluding p = .000.
Limitations
- The paper states that statcheck cannot read results not reported exactly in APA format or results in tables, and cannot account for multiple-testing corrections; it may therefore slightly overestimate inconsistencies. The sample is limited to eight flagship journals and may not represent all psychology; without raw data, it is impossible to determine whether the p-value, test statistic, or degrees of freedom is erroneous; and statcheck cannot distinguish core-hypothesis p-values from manipulation checks.
- Evident from the text: the HTML-only restriction and automated extraction may miss PDF-only or non-APA results, and because no inferential statistics were used, conclusions apply only to APA-reported NHST results in the eight selected journals.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
25 Valid · 2 Not valid · 16 Not checked · 172 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Abstract
What the sources say
- Supportedcites Cumming et al., 2007 source
The ‘Statistical Practices in Leading Journals’ section reports NHST use in almost all sampled articles (96.9–97.8%). The ‘Change, but Little Reform Yet’ section says NHST dominates overwhelmingly in those journals and is deeply ingrained in psychologists’ thinking, supporting the broad paraphrase. The supplied passages do not provide page numbers. (LLM, conf=0.88)
- Not matchedcites Hubbard & Ryan, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Sterling, 1959 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, there is evidence that many reported p-values do not match their accompanying test statistic and degrees of freedom (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp, Nuijten, Dominguez-Alvarez, Van Assen, & Wicherts, 2014; Wicherts, Bakker, & Molenaar, 2011).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Abstract
What the sources say
- Supportedcites Bakker & Wicherts, 2011 source
Study 1’s Method describes recalculating reported p-values from the accompanying test statistic and degrees of freedom to identify mismatches. Study 2’s Results report errors in 17.1% of exactly reported results in Study 1 and at least one error in 35% of the articles examined in Study 2. The copy does not provide page numbers for these passages. (LLM, conf=0.93)
- Supportedcites Bakker & Wicherts, 2014 source
The Method section (Calculation of variables; page unavailable) defines reporting errors as p-values inconsistent with values recalculated from the reported test statistic and degrees of freedom. The Results section (Errors in the reporting of statistical results; page unavailable) reports that 45% of articles had at least one such error. This supports the claim without implying that 45% of individual p-values were incorrect. (LLM, conf=0.91)
- Supportedcites Berle & Starcevic, 2007 source
The Discussion reports that p-value inconsistencies were common in the two psychiatry journals surveyed and that many papers contained at least one. The Method explains that the authors checked reported p-values against those calculated from the accompanying test statistic and degrees of freedom. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck’s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Method > “statcheck”
What the sources say
- Partly supportedcites Wicherts, Bakker, and Molenaar (2011) sourceExperimental — accuracy not yet measured
Methods > Data Retrieval describes extracting statistical results from the papers, and Results > Errors in the Reporting of Statistical Results reports findings from 49 papers. Methods > Detecting Reporting Errors says the authors independently verified all 49 inconsistencies, but says the inconsistencies were initially detected automatically in Excel, not by a manual search for errors. No page numbers are available in our copy. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“we conclude that there does not seem to be a difference in the prevalence of reporting inconsistencies between results in APA format and results that are not exactly in APA format (see Appendix A).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Discussion
What the sources say
- Could not verifycites Bakker & Wicherts, 2011 sourceExperimental — accuracy not yet measured
The General discussion reports different inconsistency rates for exactly and inexactly reported statistical results (17–19% versus 7–8%). Those categories concern how p values were reported, not whether results followed APA format. Our copy does not compare inconsistency rates for APA-formatted and non-APA-formatted results, so it cannot verify the claimed lack of a difference. (LLM, conf=0.75)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Bakker and Wicherts (2011) included only t, F, and χ 2 values”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Results > Prevalence of inconsistencies as found by other studies
What the sources say
- Supportedcites Bakker and Wicherts (2011) source
The Study 1 Method says the authors checked only χ², t, and F tests. The Study 2 Results likewise describe selecting articles with those tests and counting results from those three types. The restriction concerns the tests included in their analysis, not every value reported in the articles. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
113 more claims · 6 with their cited sources checked
Create a free account to see all 215 claims and every source verdict.
- It may state two separate claims as one. 61
- Probably not a substantial claim. 41
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“involve questionable research practices, or even outright misconduct (e.g. Horbach and Halffman 2017b; Nuijten et al. 2016)”
From The ability of different peer review procedures to flag problematic publications (2019) · cites this paper as Nuijten et al. 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From this perspective, it is not surprising that analysis pipelines are peppered with errors, ranging from the mundane to the serious [53–55].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, as S&B combines the measurement of text similarity [9] and fact-checking of reagent identities, it can either be applied independently or used in parallel with other tools, such as those that detect duplicated images [10], and/or incorrect statistical results [12, 13].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“nor has the rate of statistical errors made in mainstream psychological journals (29, 30).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“Errata and corrections have also not increased”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [30]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedF(2, 56) = 1.203reported p < .001 · recomputed p = .308
“F(2,56) = 1.203, p < .001”
- Consistentt(37) = -4.93reported p < .001 · recomputed p < .001
“t(37) = −4.93, p < .001”
- Consistentt(28) = 2reported p < .05 · recomputed p = .055
“t(28) = 2.0, p < .05”
- ConsistentF(2, 56) = 12.03reported p < .001 · recomputed p < .001
“F(2,56) = 12.03, p < .001”
- ConsistentF(1, 76) = 23.95reported p < .001 · recomputed p < .001
“F(l, 76) = 23.95, p < .001”
1 more recomputed test
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Method > “statcheck”
“If we do not include these 11 inconsistencies due to p = .000, statcheck finds an inconsistency percentage of 5.8 % (45 of 775 results)”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Author note > Appendix A: Results validity check statcheck
“Secondly, seven of the results that statcheck classified as an error, but Wicherts et al. did not, were results in which the p-value was reported to be zero (p = .000).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| No. of gross inconsistencies | 10 (0.9 %) | 17 (2.3 %) | 8 (1.0 %) |”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Author note > Appendix A: Results validity check statcheck
Reported 2.3 · recomputed 2.193548
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · cited 1×
“Considering that the ubiquitous significance level of .05 is arbitrary, and that there is a vast amount of critique on NHST in general (see e.g., Cohen, 1994; Fidler & Cumming, 2005; Krueger, 2001; Rozeboom, 1960; Wagenmakers, 2007), it should be clear that it is more important that p -values are…”
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 2×
“…of .05 is arbitrary, and that there is a vast amount of critique on NHST in general (see e.g., Cohen, 1994; Fidler & Cumming, 2005; Krueger, 2001; Rozeboom, 1960; Wagenmakers, 2007), it should be clear that it is more important that p -values are accurately reported than that they are below .05.”
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 3×
“This is in line with the finding of John et al. (2012) that social psychologists admit to more QRPs, find them more applicable to their field, and find them more defensible as compared to other subgroups in psychology (but see also Fiedler & Schwarz, 2015, on this issue).”
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 2×
“Checking articles with statcheck can also prevent such inconsistencies from distorting meta-analyses or analyses of p -value distributions (Simonsohn et al., 2014; Van Assen et al., 2014).”
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 2×
“Even though this can be a legitimate concern for some studies with particularly sensitive data, these are exceptions; the data of most psychology studies could be published without risks (Nosek et al., 2012).”
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Cited by
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Self-citationReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationSzucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“22Similar analyses and results are reported by Berle & Starcevic (2007), Garcı́a-Berthou & Alcaraz (2004), and Nuijten et al. (2016).”
MethodOpen science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“Another useful checking tool is statcheck, an R toolbox that scans documents for inconsistencies in reported statistical values [64].”
Method
Metadata
- Authors
- Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Nuijten, Michèle B.
- Venue
- Behavior Research Methods
- Published
- 2015-10-23
- Volume
- 48
- Issue
- 4
- Pages
- 1205-1226
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC5101263
- OpenAlex
- W1910875068
- Cited by
- 510
- References
- 45
- Citations by year
- 2026: 19 · 2025: 36 · 2024: 25 · 2023: 85 · 2022: 43 · 2021: 59 · 2020: 35 · 2019: 60 · 2018: 57 · 2017: 55 · 2016: 33 · 2015: 2
- FWCI
- 44.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Research portal (Tilburg University) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (acceptedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- statistical reporting errors, p-values, null hypothesis significance testing, statcheck, psychology journals, data sharing, peer review
- MeSH terms
- Humans, Bias, Prevalence, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The prevalence of statistical reporting errors in psychology (1985–2013). https://app.cochat.ai/papers/10.3758/s13428-015-0664-2
- The paper
- Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. https://doi.org/10.3758/s13428-015-0664-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.