The prevalence of statistical reporting errors in psychology (1985–2013)
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 33 of the 54 sources its claims cite were checked for support.
Abstract
This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.
Overview
CoChat Summary
TL;DR
Using the R package statcheck, this study analyzed 258,105 NHST results from 30,717 articles in eight psychology journals (1985–2013) and found that 49.6% of articles with NHST results contained at least one inconsistent p-value and 12.9% contained at least one gross inconsistency. Contrary to concerns that reporting errors are increasing, the prevalence of inconsistencies was stable or declined over time, though gross inconsistencies were more common in p-values reported as significant than nonsignificant.
Key findings
- Across all journals and years, 49.6% (8,273/16,695) of articles with NHST results contained at least one inconsistency, and 12.9% (2,150) contained at least one gross inconsistency. At the p-value level, 9.7% (24,961/258,105) were inconsistent and 1.4% (3,581) were grossly inconsistent; the mean per-article inconsistency rate was 10.6% and the gross inconsistency rate was 1.6%.
- Gross inconsistencies were more prevalent in p-values reported as significant (1.56%) than in p-values reported as nonsignificant (.97%). Reported p-values were significant 76.6% of the time, whereas computed p-values were significant 74.4% of the time, overestimating significant findings by 2.2 percentage points.
- The prevalence of inconsistencies did not generally increase from 1985 to 2013; inconsistencies decreased or remained stable across APA journals, and articles with gross inconsistencies decreased in all journals except PS and PLOS. JPSP had the highest percentage of articles with at least one inconsistency (57.6%) but one of the lowest p-value inconsistency rates (9.0%); PLoS had the highest p-value inconsistency rate (14.0%).
- NHST use increased: statcheck detected NHST results in 54.4% of articles overall, and the number of NHST results per article increased across APA journals (b = .25, R² = .68).
Methods
- Automated statcheck (version 1.0.1) extracted APA-formatted t, F, r, χ², and Z statistics with degrees of freedom and p-values from HTML articles, recalculated p-values, and flagged inconsistencies; a gross inconsistency was one that changed the statistical conclusion at α = .05. One-tailed test detection was used.
- The sample included all HTML articles from eight journals (PLOS, JPSP, JCCP, DP, JAP, PS, FP, JEPG) over years determined by HTML availability (1985–2013 for APA journals; later starts for PS, FP, and PLOS): 30,717 articles downloaded, 16,695 with detectable NHST results, and 258,105 NHST results. Analyses were descriptive only because the selected journals’ APA-reported NHST results were treated as the population.
- A validity check compared statcheck with Wicherts et al. (2011) manual coding of 49 articles: statcheck read 67.5% of manually extracted results; with one-tailed detection it found 56/775 (7.2%) inconsistencies and 8/775 (1.0%) gross inconsistencies, versus manual findings of 49/1,148 (4.3%) and 10/1,148 (.9%). Inter-rater reliability was .76 for inconsistencies and .89 for gross inconsistencies when excluding p = .000.
Limitations
- The paper states that statcheck cannot read results not reported exactly in APA format or results in tables, and cannot account for multiple-testing corrections; it may therefore slightly overestimate inconsistencies. The sample is limited to eight flagship journals and may not represent all psychology; without raw data, it is impossible to determine whether the p-value, test statistic, or degrees of freedom is erroneous; and statcheck cannot distinguish core-hypothesis p-values from manipulation checks.
- Evident from the text: the HTML-only restriction and automated extraction may miss PDF-only or non-APA results, and because no inferential statistics were used, conclusions apply only to APA-reported NHST results in the eight selected journals.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
25 Valid · 2 Not valid · 16 Not checked · 172 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Abstract
What the sources say
- Supportedcites Cumming et al., 2007 matched to “Statistical Reform in Psychology: Is Anything Changing?” (2007) source
The ‘Statistical Practices in Leading Journals’ section reports NHST use in almost all sampled articles (96.9–97.8%). The ‘Change, but Little Reform Yet’ section says NHST dominates overwhelmingly in those journals and is deeply ingrained in psychologists’ thinking, supporting the broad paraphrase. The supplied passages do not provide page numbers. (LLM, conf=0.88)
- Not matchedcites Hubbard & Ryan, 2000
More than one paper fits this citation.
- Not checked — no full text of the sourcecites Sterling, 1959 matched to “Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa” (1959) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, there is evidence that many reported p-values do not match their accompanying test statistic and degrees of freedom (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp, Nuijten, Dominguez-Alvarez, Van Assen, & Wicherts, 2014; Wicherts, Bakker, & Molenaar, 2011).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Abstract
What the sources say
- Supportedcites Bakker & Wicherts, 2011 matched to “The (mis)reporting of statistical results in psychology journals” (2011) source
Study 1’s Method describes recalculating reported p-values from the accompanying test statistic and degrees of freedom to identify mismatches. Study 2’s Results report errors in 17.1% of exactly reported results in Study 1 and at least one error in 35% of the articles examined in Study 2. The copy does not provide page numbers for these passages. (LLM, conf=0.93)
- Supportedcites Bakker & Wicherts, 2014 matched to “Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research” (2014) source
The Method section (Calculation of variables; page unavailable) defines reporting errors as p-values inconsistent with values recalculated from the reported test statistic and degrees of freedom. The Results section (Errors in the reporting of statistical results; page unavailable) reports that 45% of articles had at least one such error. This supports the claim without implying that 45% of individual p-values were incorrect. (LLM, conf=0.91)
- Supportedcites Berle & Starcevic, 2007 matched to “Inconsistencies between reported test statistics and p‐values in two psychiatry journals” (2007) source
The Discussion reports that p-value inconsistencies were common in the two psychiatry journals surveyed and that many papers contained at least one. The Method explains that the authors checked reported p-values against those calculated from the accompanying test statistic and degrees of freedom. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck’s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Method > “statcheck”
What the sources say
- Partly supportedcites Wicherts, Bakker, and Molenaar (2011) matched to “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results” (2011) sourceExperimental — accuracy not yet measured
Methods > Data Retrieval describes extracting statistical results from the papers, and Results > Errors in the Reporting of Statistical Results reports findings from 49 papers. Methods > Detecting Reporting Errors says the authors independently verified all 49 inconsistencies, but says the inconsistencies were initially detected automatically in Excel, not by a manual search for errors. No page numbers are available in our copy. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“we conclude that there does not seem to be a difference in the prevalence of reporting inconsistencies between results in APA format and results that are not exactly in APA format (see Appendix A).”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Discussion
What the sources say
- Could not verifycites Bakker & Wicherts, 2011 matched to “The (mis)reporting of statistical results in psychology journals” (2011) sourceExperimental — accuracy not yet measured
The General discussion reports different inconsistency rates for exactly and inexactly reported statistical results (17–19% versus 7–8%). Those categories concern how p values were reported, not whether results followed APA format. Our copy does not compare inconsistency rates for APA-formatted and non-APA-formatted results, so it cannot verify the claimed lack of a difference. (LLM, conf=0.75)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Bakker and Wicherts (2011) included only t, F, and χ 2 values”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Results > Prevalence of inconsistencies as found by other studies
What the sources say
- Supportedcites Bakker and Wicherts (2011) matched to “The (mis)reporting of statistical results in psychology journals” (2011) source
The Study 1 Method says the authors checked only χ², t, and F tests. The Study 2 Results likewise describe selecting articles with those tests and counting results from those three types. The restriction concerns the tests included in their analysis, not every value reported in the articles. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
113 more claims · 6 with their cited sources checked
Create a free account to see all 215 claims and every source verdict.
- It may state two separate claims as one. 61
- Probably not a substantial claim. 41
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“involve questionable research practices, or even outright misconduct (e.g. Horbach and Halffman 2017b; Nuijten et al. 2016)”
From The ability of different peer review procedures to flag problematic publications (2019) · cites this paper as Nuijten et al. 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From this perspective, it is not surprising that analysis pipelines are peppered with errors, ranging from the mundane to the serious [53–55].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, as S&B combines the measurement of text similarity [9] and fact-checking of reagent identities, it can either be applied independently or used in parallel with other tools, such as those that detect duplicated images [10], and/or incorrect statistical results [12, 13].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“nor has the rate of statistical errors made in mainstream psychological journals (29, 30).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“Errata and corrections have also not increased”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [30]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedF(2, 56) = 1.203reported p < .001 · recomputed p = .308
“F(2,56) = 1.203, p < .001”
- Consistentt(37) = -4.93reported p < .001 · recomputed p < .001
“t(37) = −4.93, p < .001”
- Consistentt(28) = 2reported p < .05 · recomputed p = .055
“t(28) = 2.0, p < .05”
- ConsistentF(2, 56) = 12.03reported p < .001 · recomputed p < .001
“F(2,56) = 12.03, p < .001”
- ConsistentF(1, 76) = 23.95reported p < .001 · recomputed p < .001
“F(l, 76) = 23.95, p < .001”
1 more recomputed test
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Method > “statcheck”
“If we do not include these 11 inconsistencies due to p = .000, statcheck finds an inconsistency percentage of 5.8 % (45 of 775 results)”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Author note > Appendix A: Results validity check statcheck
“Secondly, seven of the results that statcheck classified as an error, but Wicherts et al. did not, were results in which the p-value was reported to be zero (p = .000).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| No. of gross inconsistencies | 10 (0.9 %) | 17 (2.3 %) | 8 (1.0 %) |”
§ The prevalence of statistical reporting errors in psychology (1985–2013) > Author note > Appendix A: Results validity check statcheck
Reported 2.3 · recomputed 2.193548
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · cited 1×
“Considering that the ubiquitous significance level of .05 is arbitrary, and that there is a vast amount of critique on NHST in general (see e.g., Cohen, 1994; Fidler & Cumming, 2005; Krueger, 2001; Rozeboom, 1960; Wagenmakers, 2007), it should be clear that it is more important that p -values are…”
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 2×
“…of .05 is arbitrary, and that there is a vast amount of critique on NHST in general (see e.g., Cohen, 1994; Fidler & Cumming, 2005; Krueger, 2001; Rozeboom, 1960; Wagenmakers, 2007), it should be clear that it is more important that p -values are accurately reported than that they are below .05.”
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 3×
“This is in line with the finding of John et al. (2012) that social psychologists admit to more QRPs, find them more applicable to their field, and find them more defensible as compared to other subgroups in psychology (but see also Fiedler & Schwarz, 2015, on this issue).”
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 2×
“Checking articles with statcheck can also prevent such inconsistencies from distorting meta-analyses or analyses of p -value distributions (Simonsohn et al., 2014; Van Assen et al., 2014).”
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 2×
“Even though this can be a legitimate concern for some studies with particularly sensitive data, these are exceptions; the data of most psychology studies could be published without risks (Nosek et al., 2012).”
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Publication Manual of the American Psychological Association (4th ed.).
Gullickson, T. (1996) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/004700 · Cited by 1,240
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998 · cited 1×
“…for a meta-analysis, this effect size will be incorrect as well, which could affect the outcome of the meta-analysis (Bakker & Wicherts, 2011; in fact, the misreporting of all kinds of statistics is a problem for meta-analyses; Gotzsche, Hrobjartsson, Maric, & Tendal, 2007; Levine & Hullett, 2002).”
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 729
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · cited 1×
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 9×
“To avoid bias in the prevalence of reporting errors, we performed a validity study of statcheck, in which we compared statcheck ’ s results with the results of Wicherts, Bakker, and Molenaar (2011), who performed a manual search for and verification of reporting errors in a sample of 49 articles.”
InfluentialSelf-citationPublic Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 10×
“For instance, it has been found that journals with lower impact factors have a higher prevalence of reporting inconsistencies than high impact journals (Bakker & Wicherts, 2011).”
InfluentialSelf-citationNull hypothesis significance testing: On the survival of a flawed method.
Krueger, J. (2001) · American Psychologist · doi:10.1037//0003-066x.56.1.16 · Cited by 282 · cited 1×
“…level of .05 is arbitrary, and that there is a vast amount of critique on NHST in general (see e.g., Cohen, 1994; Fidler & Cumming, 2005; Krueger, 2001; Rozeboom, 1960; Wagenmakers, 2007), it should be clear that it is more important that p -values are accurately reported than that they…”
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279 · cited 1×
“This is in line with the finding of John et al. (2012) that social psychologists admit to more QRPs, find them more applicable to their field, and find them more defensible as compared to other subgroups in psychology (but see also Fiedler & Schwarz, 2015, on this issue).”
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · cited 1×
“Whatever the cause of this over-representation might be, there seems to be too much focus on getting B perfect, ^ significant results (see also Giner-Sorolla, 2012).”
Why Growing Retractions Are (Mostly) a Good Sign
Fanelli, D. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001563 · Cited by 272 · cited 4×
“Furthermore, we manually downloaded all articles in HTML from two journals in general psychology: Psychological Science (PS; 2003 – 2013) and Frontiers in Psychology (FP; 2010 – 2013).”
InfluentialMeta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Self-citationData Extraction Errors in Meta-analyses That Use Standardized Mean Differences
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007) · JAMA · doi:10.1001/jama.298.4.430 · Cited by 238
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202 · cited 1×
“One objection that has been raised is that due to privacy concerns data cannot be made publicly available (see e.g., Finkel, Eastwick, & Reis, 2015).”
Educational and psychological measurement
(1982) · Serials Review · doi:10.1016/0098-7913(82)90053-3 · Cited by 186
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
Sterling, T. D., Rosenbaum, W. L., & Jj, W. (1995) · The American Statistician · doi:10.2307/2684823 · Cited by 158
Statistical Reform in Psychology: Is Anything Changing?
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01881.x · Cited by 122
The Historical Growth of Statistical Significance Testing in Psychology–and Its Future Prospects
Hubbard, R., & Ryan, P. A. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970808 · Cited by 118 · cited 1×
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research
Bakker, M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103360 · Cited by 101 · cited 2×
“…there is evidence that many reported p -values do not match their accompanying test statistic and degrees of freedom (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp, Nuijten, Dominguez-Alvarez, Van…”
Self-citationPsychology must learn a lesson from fraud case
Wicherts, J. M. (2011) · Nature · doi:10.1038/480007a · Cited by 86 · cited 9×
“For instance, it has been found that journals with lower impact factors have a higher prevalence of reporting inconsistencies than high impact journals (Bakker & Wicherts, 2011).”
InfluentialSelf-citationPublish (your data) or (let the data) perish! Why not publish your data too?
Wicherts, J. M., & Bakker, M. (2012) · Intelligence · doi:10.1016/j.intell.2012.01.004 · Cited by 85 · cited 1×
“Sharing data for instance facilitates aggregating data for better effect size estimates, enable reanalyzing published articles, and increase credibility of scientific findings (see also Nosek, Spies, & Motyl, 2012; Sakaluk et al., 2014; Wicherts, 2013; Wicherts & Bakker, 2012).”
MethodSelf-citationStatistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69 · cited 4×
“Finally, the lowest percentage of articles with at least one gross inconsistency is 2.6 % (Berle & Starcevic, 2007) and the highest is 20.5 % (Veldkamp et al., 2014), with a median of 14.3 % (1.4 percentage points higher than the 12.9 % found in the current study).”
InfluentialSelf-citationThe Life of p : “Just Significant” Results are on the Rise
Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.863371 · Cited by 64 · cited 5×
“If gross inconsistencies are indicative of QRPs and QRPs have increased over the years, we would expect an increase of gross inconsistencies over the years (see also Leggett et al., 2013).”
InfluentialHubbard, R., & Ryan, P. A. (2000) · Educational and Psychological Measurement · doi:10.1177/0013164400605001 · Cited by 64 · cited 1×
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
(2014) · Data Archiving and Networked Services (DANS) · Cited by 46
Self-citationToo Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
Self-citationInconsistencies between reported test statistics and p‐values in two psychiatry journals
Berle, D., & Starcevic, V. (2007) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.225 · Cited by 38 · cited 5×
“Our study can be considered a large replication of several previous studies (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp et al., 2014; Wicherts et al., 2011).”
InfluentialTeaching Confidence Intervals: Problems and Potential Solutions
Fidler, F. M., & Cumming, G. (2005) · Cited by 32
Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Sakaluk, J., Williams, A., & Biernat, M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614549257 · Cited by 32 · cited 4×
“This solution would be in line with the notion of Analytic Review (Sakaluk, Williams, & Biernat, 2014), in which a reviewer receives the data file and syntax of a manuscript to check if the reported analyses were actually conducted and reported correctly.”
InfluentialScience revolves around the data
(2013) · Journal of Open Psychology Data · doi:10.5334/jopd.e1 · Cited by 29 · cited 1×
“Sharing data for instance facilitates aggregating data for better effect size estimates, enable reanalyzing published articles, and increase credibility of scientific findings (see also Nosek, Spies, & Motyl, 2012; Sakaluk et al., 2014; Wicherts, 2013; Wicherts & Bakker, 2012).”
MethodSelf-citationPublication Manual of the American Psychological Association (3rd. ed.)
(1984) · Bulletin of the Psychonomic Society · doi:10.3758/bf03333766 · Cited by 28
Consistency errors in p-values reported in Spanish psychology journals
Caperos, J., & Pardo, A. (2013) · Psicothema · doi:10.7334/psicothema2012.207 · Cited by 23 · cited 3×
“The percentage of gross inconsistencies ranged from .4 % (Garcia-Berthou & Alcaraz, 2004) to 2.3 % (Caperos & Pardo, 2013), with a median of 1.1 % (.3 percentage points lower than the 1.4 % found in the current study).”
InfluentialRise in retractions is a signal of integrity
Fanelli, D. (2014) · Nature · doi:10.1038/509033a · Cited by 18 · cited 1×
“On the other hand, it has been found that the number of published corrections to the literature did not change over time, suggesting no change in QRPs over time (Fanelli, 2013, 2014).”
Cited by
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Self-citationReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationSzucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“Another useful checking tool is statcheck, an R toolbox that scans documents for inconsistencies in reported statistical values [64].”
MethodThe Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
Self-citation
Metadata
- Authors
- Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- University of Amsterdam (NL)
- Corresponding author
- Nuijten, Michèle B.
- Venue
- Behavior Research Methods
- Published
- 2015-10-23
- Volume
- 48
- Issue
- 4
- Pages
- 1205-1226
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC5101263
- OpenAlex
- W1910875068
- Cited by
- 510
- References
- 45
- Citations by year
- 2026: 19 · 2025: 36 · 2024: 25 · 2023: 85 · 2022: 43 · 2021: 59 · 2020: 35 · 2019: 60 · 2018: 57 · 2017: 55 · 2016: 33 · 2015: 2
- FWCI
- 44.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Research portal (Tilburg University) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (acceptedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- statistical reporting errors, p-values, null hypothesis significance testing, statcheck, psychology journals, data sharing, peer review
- MeSH terms
- Humans, Bias, Prevalence, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.