Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 33 of the 54 sources its claims cite were checked for support.

The prevalence of statistical reporting errors in psychology (1985–2013)

Open access
Nuijten, Michèle B., Hartgerink, Chris H. J., van Assen, Marcel A. L. M., Epskamp, Sacha, Wicherts, Jelte M. · 2016 · Behavior Research Methods

Abstract

This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.

Overview

CoChat Summary

TL;DR

Using the R package statcheck, this study analyzed 258,105 NHST results from 30,717 articles in eight psychology journals (1985–2013) and found that 49.6% of articles with NHST results contained at least one inconsistent p-value and 12.9% contained at least one gross inconsistency. Contrary to concerns that reporting errors are increasing, the prevalence of inconsistencies was stable or declined over time, though gross inconsistencies were more common in p-values reported as significant than nonsignificant.

Key findings

  • Across all journals and years, 49.6% (8,273/16,695) of articles with NHST results contained at least one inconsistency, and 12.9% (2,150) contained at least one gross inconsistency. At the p-value level, 9.7% (24,961/258,105) were inconsistent and 1.4% (3,581) were grossly inconsistent; the mean per-article inconsistency rate was 10.6% and the gross inconsistency rate was 1.6%.
  • Gross inconsistencies were more prevalent in p-values reported as significant (1.56%) than in p-values reported as nonsignificant (.97%). Reported p-values were significant 76.6% of the time, whereas computed p-values were significant 74.4% of the time, overestimating significant findings by 2.2 percentage points.
  • The prevalence of inconsistencies did not generally increase from 1985 to 2013; inconsistencies decreased or remained stable across APA journals, and articles with gross inconsistencies decreased in all journals except PS and PLOS. JPSP had the highest percentage of articles with at least one inconsistency (57.6%) but one of the lowest p-value inconsistency rates (9.0%); PLoS had the highest p-value inconsistency rate (14.0%).
  • NHST use increased: statcheck detected NHST results in 54.4% of articles overall, and the number of NHST results per article increased across APA journals (b = .25, R² = .68).

Methods

  • Automated statcheck (version 1.0.1) extracted APA-formatted t, F, r, χ², and Z statistics with degrees of freedom and p-values from HTML articles, recalculated p-values, and flagged inconsistencies; a gross inconsistency was one that changed the statistical conclusion at α = .05. One-tailed test detection was used.
  • The sample included all HTML articles from eight journals (PLOS, JPSP, JCCP, DP, JAP, PS, FP, JEPG) over years determined by HTML availability (1985–2013 for APA journals; later starts for PS, FP, and PLOS): 30,717 articles downloaded, 16,695 with detectable NHST results, and 258,105 NHST results. Analyses were descriptive only because the selected journals’ APA-reported NHST results were treated as the population.
  • A validity check compared statcheck with Wicherts et al. (2011) manual coding of 49 articles: statcheck read 67.5% of manually extracted results; with one-tailed detection it found 56/775 (7.2%) inconsistencies and 8/775 (1.0%) gross inconsistencies, versus manual findings of 49/1,148 (4.3%) and 10/1,148 (.9%). Inter-rater reliability was .76 for inconsistencies and .89 for gross inconsistencies when excluding p = .000.

Limitations

  • The paper states that statcheck cannot read results not reported exactly in APA format or results in tables, and cannot account for multiple-testing corrections; it may therefore slightly overestimate inconsistencies. The sample is limited to eight flagship journals and may not represent all psychology; without raw data, it is impossible to determine whether the p-value, test statistic, or degrees of freedom is erroneous; and statcheck cannot distinguish core-hypothesis p-values from manipulation checks.
  • Evident from the text: the HTML-only restriction and automated extraction may miss PDF-only or non-APA results, and because no inferential statistics were used, conclusions apply only to APA-reported NHST results in the eight selected journals.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3758/s13428-015-0664-2

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The percentage of articles with American Psychological Association (APA)-reported null-hypothesis significance testing (NHST) results over the years, averaged over all APA journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal – light gray panels for the APA journals and white panels for the non-APA journals (Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). For each trend we report the unstandardized linear regression coefficient (b) and the coefficient of determination (R2) of the linear trend
Fig. 1 The percentage of articles with American Psychological Association (APA)-reported null-hypothesis significance testing (NHST) results over the years, averaged over all APA journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal – light gray panels for the APA journals and white panels for the non-APA journals (Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). For each trend we report the unstandardized linear regression coefficient (b) and the coefficient of determination (R2) of the linear trendCC BY · PubMed Central
Fig. 2 The average number of American Psychological Association (APA)-reported null-hypothesis significance testing (NHST) results per article that contains NHST results over the years, averaged over all APA journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal (light gray panels for the APA journals and white panels for the non-APA journals – Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). For each trend we report the unstandardized linear regression coefficient (b) and the coefficient of determination (R2) of the linear trend
Fig. 2 The average number of American Psychological Association (APA)-reported null-hypothesis significance testing (NHST) results per article that contains NHST results over the years, averaged over all APA journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal (light gray panels for the APA journals and white panels for the non-APA journals – Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). For each trend we report the unstandardized linear regression coefficient (b) and the coefficient of determination (R2) of the linear trendCC BY · PubMed Central
Fig. 3 The average percentage of articles within a journal with at least one (gross) inconsistency and the average percentage of (grossly) inconsistent p-values per article, split up by journal. Inconsistencies are depicted in white and gross inconsistencies in grey. For the journals Journal of Personality and Social Psychology (JPSP), Journal of Experimental Psychology: General (JEPG), Developmental Psychology (DP), Frontiers in Psychology (FP), Public Library of Science (PLoS), Journal of Consulting and Clinical Psychology (JCCP), Psychological Science (PS), and Journal of Applied Psychology (JAP), respectively, the number of articles with null-hypothesis significance testing (NHST) results is 4,346, 821, 2,607, 702, 2,487, 2,413, 1,681, and 1,638, and the average number of NHST results in an article is 23.4, 23.0, 14.4, 14.5, 12.7, 11.4, 9.3, and 9.2
Fig. 3 The average percentage of articles within a journal with at least one (gross) inconsistency and the average percentage of (grossly) inconsistent p-values per article, split up by journal. Inconsistencies are depicted in white and gross inconsistencies in grey. For the journals Journal of Personality and Social Psychology (JPSP), Journal of Experimental Psychology: General (JEPG), Developmental Psychology (DP), Frontiers in Psychology (FP), Public Library of Science (PLoS), Journal of Consulting and Clinical Psychology (JCCP), Psychological Science (PS), and Journal of Applied Psychology (JAP), respectively, the number of articles with null-hypothesis significance testing (NHST) results is 4,346, 821, 2,607, 702, 2,487, 2,413, 1,681, and 1,638, and the average number of NHST results in an article is 23.4, 23.0, 14.4, 14.5, 12.7, 11.4, 9.3, and 9.2CC BY · PubMed Central
Fig. 4 Average percentage of inconsistencies (open circles) and gross inconsistencies (solid circles) in an article over the years averaged over all American Psychological Association (APA) journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel) and split up per journal (light gray panels for the APA journals and white panels for non-APA journals – Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). The unstandardized regression coefficient b and the coefficient of determination R2 of the linear trend are shown per journal for both inconsistencies (incons) and gross inconsistencies (gross) over the years
Fig. 4 Average percentage of inconsistencies (open circles) and gross inconsistencies (solid circles) in an article over the years averaged over all American Psychological Association (APA) journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel) and split up per journal (light gray panels for the APA journals and white panels for non-APA journals – Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). The unstandardized regression coefficient b and the coefficient of determination R2 of the linear trend are shown per journal for both inconsistencies (incons) and gross inconsistencies (gross) over the yearsCC BY · PubMed Central
Fig. 5 Percentage of articles with at least one inconsistency (open circles) or at least one gross inconsistency (solid circles), split up by journal. The unstandardized regression coefficient b and the coefficient of determination R2 of the linear trend are shown per journal for both inconsistencies (incons) as gross inconsistencies (gross) over the years. APA American Psychological Assocation, DP Developmental Psychology, JCCP Journal of Consulting and Clinical Psychology, JEPG Journal of Experimental Psychology: General , JPSP Journal of Personality and Social Psychology, JAP Journal of Applied Psychology, PS Psychological Science, FP Frontiers in Psychology, PLoS Public Library of Science
Fig. 5 Percentage of articles with at least one inconsistency (open circles) or at least one gross inconsistency (solid circles), split up by journal. The unstandardized regression coefficient b and the coefficient of determination R2 of the linear trend are shown per journal for both inconsistencies (incons) as gross inconsistencies (gross) over the years. APA American Psychological Assocation, DP Developmental Psychology, JCCP Journal of Consulting and Clinical Psychology, JEPG Journal of Experimental Psychology: General , JPSP Journal of Personality and Social Psychology, JAP Journal of Applied Psychology, PS Psychological Science, FP Frontiers in Psychology, PLoS Public Library of ScienceCC BY · PubMed Central
Fig. 6 The percentage of gross inconsistencies in p-values reported as significant (white bars) and nonsignificant (gray bars), split up by journal. For the journals Journal of Applied Psychology (JAP), Journal of Consulting and Clinical Psychology (JCCP), Developmental Psychology (DP), Public Library of Science (PLoS), Psychological Science (PS), Frontiers in Psychology (FP), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: General (JEPG), respectively, the total number of significant p-values was 11,654, 21,120, 29,962, 22,071, 12,482, 7,377, 78,889, and 14,084, and the total number of nonsignificant p-values was 3,119, 5,558, 6,698, 9,134, 2,936, 2,712, 17,868, and 4,407
Fig. 6 The percentage of gross inconsistencies in p-values reported as significant (white bars) and nonsignificant (gray bars), split up by journal. For the journals Journal of Applied Psychology (JAP), Journal of Consulting and Clinical Psychology (JCCP), Developmental Psychology (DP), Public Library of Science (PLoS), Psychological Science (PS), Frontiers in Psychology (FP), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: General (JEPG), respectively, the total number of significant p-values was 11,654, 21,120, 29,962, 22,071, 12,482, 7,377, 78,889, and 14,084, and the total number of nonsignificant p-values was 3,119, 5,558, 6,698, 9,134, 2,936, 2,712, 17,868, and 4,407CC BY · PubMed Central
Fig. 7 The percentage of gross inconsistencies in p-values reported as significant (solid line) and nonsignificant (dotted line), over the years, averaged over journals. The size of the open and solid circles represents the number of significant and nonsignificant p-values in that year, respectively
Fig. 7 The percentage of gross inconsistencies in p-values reported as significant (solid line) and nonsignificant (dotted line), over the years, averaged over journals. The size of the open and solid circles represents the number of significant and nonsignificant p-values in that year, respectivelyCC BY · PubMed Central
Fig. 8 The total number of downloaded articles and the number of published articles that contain NHST results over the years, averaged over all American Psychological Association (APA) journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal (light gray panels for the APA journals and white panels for the non-APA journals –Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). Note that the y-axes in the plot for All APA Journals, FP, and PLOS are different from the others and continue until 1,000, 1,050, and 3,750, respectively. The unstandardized regression coefficient ‘b’ and the coefficient of determination ‘R2’ of the linear trend are shown per journal for both the downloaded articles (down) as articles with null-hypothesis significance testing results (NHST) over the years
Fig. 8 The total number of downloaded articles and the number of published articles that contain NHST results over the years, averaged over all American Psychological Association (APA) journals (Developmental Psychology (DP), Journal of Consulting and Clinical Psychology (JCCP), Journal of Experimental Psychology: General (JEPG), Journal of Personality and Social Psychology (JPSP), and Journal of Applied Psychology (JAP); dark gray panel), and split up per journal (light gray panels for the APA journals and white panels for the non-APA journals –Psychological Science (PS), Frontiers in Psychology (FP), and Public Library of Science (PLoS)). Note that the y-axes in the plot for All APA Journals, FP, and PLOS are different from the others and continue until 1,000, 1,050, and 3,750, respectively. The unstandardized regression coefficient ‘b’ and the coefficient of determination ‘R2’ of the linear trend are shown per journal for both the downloaded articles (down) as articles with null-hypothesis significance testing results (NHST) over the yearsCC BY · PubMed Central
Fig. 9 The average number of exact and inexact null-hypothesis significance testing (NHST) results per article over the years, averaged over all journals (grey panel), and split up by journal (white panels). The unstandardized regression coefficient ‘b’ and the coefficient of determination ‘R2’ of the linear trend are shown per journal for both exact (ex) as inexact (inex) p-values over the years. APA American Psychological Assocation, DP Developmental Psychology, JCCP Journal of Consulting and Clinical Psychology, JEPG Journal of Experimental Psychology: General , JPSP Journal of Personality and Social Psychology, JAP Journal of Applied Psychology, PS Psychological Science, FP Frontiers in Psychology, PLoS Public Library of Science
Fig. 9 The average number of exact and inexact null-hypothesis significance testing (NHST) results per article over the years, averaged over all journals (grey panel), and split up by journal (white panels). The unstandardized regression coefficient ‘b’ and the coefficient of determination ‘R2’ of the linear trend are shown per journal for both exact (ex) as inexact (inex) p-values over the years. APA American Psychological Assocation, DP Developmental Psychology, JCCP Journal of Consulting and Clinical Psychology, JEPG Journal of Experimental Psychology: General , JPSP Journal of Personality and Social Psychology, JAP Journal of Applied Psychology, PS Psychological Science, FP Frontiers in Psychology, PLoS Public Library of ScienceCC BY · PubMed Central

Verification

Claims the paper makes

25 Valid · 2 Not valid · 16 Not checked · 172 Own results and interpretations

113 more claims · 6 with their cited sources checked

Create a free account to see all 215 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 61
  • Probably not a substantial claim. 41

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

7 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Not confirmedF(2, 56) = 1.203reported p < .001 · recomputed p = .308

    “F(2,56) = 1.203, p < .001”

  • Consistentt(37) = -4.93reported p < .001 · recomputed p < .001

    “t(37) = −4.93, p < .001”

  • Consistentt(28) = 2reported p < .05 · recomputed p = .055

    “t(28) = 2.0, p < .05”

  • ConsistentF(2, 56) = 12.03reported p < .001 · recomputed p < .001

    “F(2,56) = 12.03, p < .001”

  • ConsistentF(1, 76) = 23.95reported p < .001 · recomputed p < .001

    “F(l, 76) = 23.95, p < .001”

5 of 6 · the rest aren’t loaded

1 more recomputed test

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 45 references

Cited by

See all 533

Metadata

Authors
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M.
Affiliations
  • Tilburg University (NL)
  • University of Amsterdam (NL)
Corresponding author
Nuijten, Michèle B.
Venue
Behavior Research Methods
Published
2015-10-23
Volume
48
Issue
4
Pages
1205-1226
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W1910875068
Cited by
510
References
45
Citations by year
2026: 19 · 2025: 36 · 2024: 25 · 2023: 85 · 2022: 43 · 2021: 59 · 2020: 35 · 2019: 60 · 2018: 57 · 2017: 55 · 2016: 33 · 2015: 2
FWCI
44.63
Type
journal article
Language
en
Open access
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • Research portal (Tilburg University) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (acceptedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
statistical reporting errors, p-values, null hypothesis significance testing, statcheck, psychology journals, data sharing, peer review
MeSH terms
Humans, Bias, Prevalence, Behavioral Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The prevalence of statistical reporting errors in psychology (1985–2013). https://app.cochat.ai/papers/10.3758/s13428-015-0664-2
The paper
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. https://doi.org/10.3758/s13428-015-0664-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.