The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Detection of data fabrication using statistical tools
Open accessAbstract
Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Study 1, we tested the validity of statistical methods to detect fabricated data at the study level using summary statistics. Using (arguably) genuine data from the Many Labs 1 project on the anchoring effect (k=36) and fabricated data for the same effect by our participants (k=39), we tested the validity of our newly proposed 'reversed Fisher method', variance analyses, and extreme effect sizes, and a combination of these three indicators using the original Fisher method. Results indicate that the variance analyses perform fairly well when the homogeneity of population variances is accounted for and that extreme effect sizes perform similarly well in distinguishing genuine from fabricated data. The performance of the 'reversed Fisher method' was poor and depended on the types of tests included. In Study 2, we tested the validity of statistical methods to detect fabricated data using raw data. Using (arguably) genuine data from the Many Labs 3 project on the classic Stroop task (k=21) and fabricated data for the same effect by our participants (k=28), we investigated the performance of digit analyses, variance analyses, multivariate associations, and extreme effect sizes, and a combination of these four methods using the original Fisher method. Results indicate that variance analyses, extreme effect sizes, and multivariate associations perform fairly well to excellent in detecting fabricated data using raw data, while digit analyses perform at chance levels. The two studies provide mixed results on how the use of random number generators affects the detection of data fabrication. Ultimately, we consider the variance analyses, effect sizes, and multivariate associations valuable tools to detect potential data anomalies in empirical (summary or raw) data. However, we argue against widespread (possible automatic) application of these tools, because some fabricated data may be irregular in one aspect but not in another. Considering how violations of the assumptions of fabrication detection methods may yield high false positive or false negative probabilities, we recommend comparing potentially fabricated data to genuine data on the same topic.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
1 retracted
When contact changes minds: An experiment on transmission of support for gay equality
LaCour, M. J., & GREEN, D. P. (2014) · Science · doi:10.1126/science.1256151 · Cited by 136
RetractedTversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Social Influence: Compliance and Conformity
Cialdini, R. B., & Goldstein, N. J. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142015 · Cited by 6,077
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Cited by
Methods to assess research misconduct in health-related research: A scoping review
Bordewijk, E. M., Li, W., van Eekelen, R., Wang, R., Showell, M., Mol, B. W., & van Wely, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.012 · Cited by 68
International multi‐stakeholder consensus statement on clinical trial integrity
Khan, K. S. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17451 · Cited by 30
The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials
Hunter, K. E., Aberoumand, M., Libesman, S., Sotiropoulos, J. X., Williams, J. G., Aagerup, J., Wang, R., Mol, B. W., Li, W., Barba, A., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1738 · Cited by 28
“47”
Maximal positive controls: A method for estimating the largest plausible effect size
Hilgard, J. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104082 · Cited by 25
Khan, K. S., Fawzy, M., & Chien, P. F. W. (2023) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.14837 · Cited by 20
“56,57 This conclusion reinforces the need for research to establish the pre-test probabilities when evaluating the measurement properties of RCT integrity tests to gauge the level of their performance.”
Background
Metadata
- Authors
- Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M.
- Affiliations
- Tilburg University (NL)
- Publisher
- Center for Open Science
- Published
- 2019-08-19
- OpenAlex
- W3162976510
- Semantic Scholar
- 24fe958de5a6b21ec71fa120c67a43a369fec1d1
- Cited by
- 16
- References
- 82
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 4 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- research misconduct, analysis of variance, multivariate association, random number generators, anchoring effect, Stroop task, diagnostic performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Detection of data fabrication using statistical tools. https://app.cochat.ai/papers/10.31234/osf.io/jkws4
- The paper
- Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M. (2019). Detection of data fabrication using statistical tools. https://doi.org/10.31234/osf.io/jkws4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.