Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science

Open access
Veldkamp, Coosje L. S., Nuijten, Michèle B., Dominguez-Alvarez, Linda, van Assen, Marcel A. L. M., Wicherts, Jelte M. · 2014 · PLoS ONE

Abstract

Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this 'co-piloting' currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0114876

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flow chart for composition of sample.
Figure 1 Flow chart for composition of sample.CC BY · PubMed Central
Figure 2 The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval). Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).
Figure 2 The probability per journal that an article contains at least one p-value comprising an error or gross error (with 95% confidence interval). Note. JAP = Journal of Applied Psychology (n = 42); JCCP = Journal of Consulting and Clinical Psychology (n = 67); JCN = Journal of Cognitive Neuroscience (n = 107); JCPP = Journal of Child Psychology and Psychiatry (n = 39); JPSP = Journal of Personality and Social Psychology (n = 133); PP = Psychophysiology (n = 42); TOTAL = all articles together (N = 430).CC BY · PubMed Central
Figure 3 The probability per journal that a given p-value comprises an error or a gross error (with 95% confidence interval). Note. JAP = Journal of Applied Psychology (n = 340); JCCP = Journal of Consulting and Clinical Psychology (n = 833); JCN = Journal of Cognitive Neuroscience (n = 1,721); JCPP = Journal of Child Psychology and Psychiatry (n = 444); JPSP = Journal of Personality and Social Psychology (n = 4,018); PP = Psychophysiology (n = 749); TOTAL = all p-values together (N = 8,105).
Figure 3 The probability per journal that a given p-value comprises an error or a gross error (with 95% confidence interval). Note. JAP = Journal of Applied Psychology (n = 340); JCCP = Journal of Consulting and Clinical Psychology (n = 833); JCN = Journal of Cognitive Neuroscience (n = 1,721); JCPP = Journal of Child Psychology and Psychiatry (n = 444); JPSP = Journal of Personality and Social Psychology (n = 4,018); PP = Psychophysiology (n = 749); TOTAL = all p-values together (N = 8,105).CC BY · PubMed Central
Figure 4 The percentage of articles in which co-piloting occurred for various processes (with 95% confidence intervals). Note. statistical analyses = conducting the statistical analyses (N = 335); write up sample details = writing the sample details in the manuscript (N = 330); write up results = writing up the results in the manuscript (N = 334); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 326); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 333); data sharing now = having access to the data at the moment the survey was being filled in (N = 332).
Figure 4 The percentage of articles in which co-piloting occurred for various processes (with 95% confidence intervals). Note. statistical analyses = conducting the statistical analyses (N = 335); write up sample details = writing the sample details in the manuscript (N = 330); write up results = writing up the results in the manuscript (N = 334); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 326); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 333); data sharing now = having access to the data at the moment the survey was being filled in (N = 332).CC BY · PubMed Central
Figure 5 The probability that a p-value in the first (or only) study reported comprises an error: co-piloted studies versus non-co-piloted studies. Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).
Figure 5 The probability that a p-value in the first (or only) study reported comprises an error: co-piloted studies versus non-co-piloted studies. Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).CC BY · PubMed Central
Figure 6 The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies. Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).
Figure 6 The probability that a p-value in the first (or only) study reported comprises a gross error: co-piloted studies versus non-co-piloted studies. Note. statistical analyses = conducting the statistical analyses (N = 2,247); write up sample details = writing the sample details in the manuscript (N = 2,215); write up results = writing up the results in the manuscript (N = 2,231); check results in manuscript = checking of the results in the manuscript by someone other than the person who wrote up the results in the manuscript (N = 2,185); data sharing at submission = having access to the data at the moment the manuscript was submitted (N = 2,228); data sharing now = having access to the data at the moment the survey was being filled in (N = 2,226).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 1

  1. American Psychological Association (2010) Publication Manual of the American Psychological Association. Sixth Edition. Washington, DC: American Psychological Association.

Cited by

See all 69

Metadata

Authors
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M.
Affiliations
Tilburg University (NL)
Corresponding author
Veldkamp, Coosje L. S.
Venue
PLoS ONE
Published
2014-12-10
Volume
9
Issue
12
Pages
e114876
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2011295155
Cited by
69
References
40
Citations by year
2025: 1 · 2024: 1 · 2023: 9 · 2022: 7 · 2021: 5 · 2020: 6 · 2019: 8 · 2018: 9 · 2017: 8 · 2016: 13 · 2015: 2
FWCI
10.64
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research portal (Tilburg University) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
Keywords
statistical reporting errors, data sharing, degrees of freedom, statistical analysis, research integrity, peer review
MeSH terms
Data Accuracy, Cooperative Behavior, Data Interpretation, Statistical, Humans, Periodicals as Topic, Psychology, Research Design, Statistics as Topic, Information Dissemination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science. https://app.cochat.ai/papers/10.1371/journal.pone.0114876
The paper
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014). Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science. PLoS ONE. https://doi.org/10.1371/journal.pone.0114876

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.