CoChat

Detection of data fabrication using statistical tools

Open access

The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.

Abstract

Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Study 1, we tested the validity of statistical methods to detect fabricated data at the study level using summary statistics. Using (arguably) genuine data from the Many Labs 1 project on the anchoring effect (k=36) and fabricated data for the same effect by our participants (k=39), we tested the validity of our newly proposed 'reversed Fisher method', variance analyses, and extreme effect sizes, and a combination of these three indicators using the original Fisher method. Results indicate that the variance analyses perform fairly well when the homogeneity of population variances is accounted for and that extreme effect sizes perform similarly well in distinguishing genuine from fabricated data. The performance of the 'reversed Fisher method' was poor and depended on the types of tests included. In Study 2, we tested the validity of statistical methods to detect fabricated data using raw data. Using (arguably) genuine data from the Many Labs 3 project on the classic Stroop task (k=21) and fabricated data for the same effect by our participants (k=28), we investigated the performance of digit analyses, variance analyses, multivariate associations, and extreme effect sizes, and a combination of these four methods using the original Fisher method. Results indicate that variance analyses, extreme effect sizes, and multivariate associations perform fairly well to excellent in detecting fabricated data using raw data, while digit analyses perform at chance levels. The two studies provide mixed results on how the use of random number generators affects the detection of data fabrication. Ultimately, we consider the variance analyses, effect sizes, and multivariate associations valuable tools to detect potential data anomalies in empirical (summary or raw) data. However, we argue against widespread (possible automatic) application of these tools, because some fabricated data may be irregular in one aspect but not in another. Considering how violations of the assumptions of fabrication detection methods may yield high false positive or false negative probabilities, we recommend comparing potentially fabricated data to genuine data on the same topic.

Overview

Integrity signals

Registry

Crossref · 10.31234/osf.io/jkws4

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

1 retracted

Cited by

See all 17

Metadata

Authors
Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M.
Affiliations
Tilburg University (NL)
Published
2019-08-19
OpenAlex
W3162976510
Cited by
16
References
82
Citations by year
2025: 2 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 4 · 2020: 1
Type
posted content
Language
en
Open access
Locations
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
research misconduct, analysis of variance, multivariate association, random number generators, anchoring effect, Stroop task, diagnostic performance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.