Detection of data fabrication using statistical tools
Open accessThe DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Abstract
Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Study 1, we tested the validity of statistical methods to detect fabricated data at the study level using summary statistics. Using (arguably) genuine data from the Many Labs 1 project on the anchoring effect (k=36) and fabricated data for the same effect by our participants (k=39), we tested the validity of our newly proposed 'reversed Fisher method', variance analyses, and extreme effect sizes, and a combination of these three indicators using the original Fisher method. Results indicate that the variance analyses perform fairly well when the homogeneity of population variances is accounted for and that extreme effect sizes perform similarly well in distinguishing genuine from fabricated data. The performance of the 'reversed Fisher method' was poor and depended on the types of tests included. In Study 2, we tested the validity of statistical methods to detect fabricated data using raw data. Using (arguably) genuine data from the Many Labs 3 project on the classic Stroop task (k=21) and fabricated data for the same effect by our participants (k=28), we investigated the performance of digit analyses, variance analyses, multivariate associations, and extreme effect sizes, and a combination of these four methods using the original Fisher method. Results indicate that variance analyses, extreme effect sizes, and multivariate associations perform fairly well to excellent in detecting fabricated data using raw data, while digit analyses perform at chance levels. The two studies provide mixed results on how the use of random number generators affects the detection of data fabrication. Ultimately, we consider the variance analyses, effect sizes, and multivariate associations valuable tools to detect potential data anomalies in empirical (summary or raw) data. However, we argue against widespread (possible automatic) application of these tools, because some fabricated data may be irregular in one aspect but not in another. Considering how violations of the assumptions of fabrication detection methods may yield high false positive or false negative probabilities, we recommend comparing potentially fabricated data to genuine data on the same topic.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
1 retracted
When contact changes minds: An experiment on transmission of support for gay equality
LaCour, M. J., & Green, D. P. (2014) · Science · doi:10.1126/science.1256151 · Cited by 113
RetractedTversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Social Influence: Compliance and Conformity
Cialdini, R. B., & Goldstein, N. J. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142015 · Cited by 6,077
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Statistical Methods for Research Workers.
I., L., & Fisher, R. A. (1926) · Journal of the Royal Statistical Society · doi:10.2307/2341488 · Cited by 2,128
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Note on the Frequency of Use of the Different Digits in Natural Numbers
Newcomb, S. (1881) · American Journal of Mathematics · doi:10.2307/2369148 · Cited by 974
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Self-citationA Dirty Dozen: Twelve P-Value Misconceptions
Goodman, S. (2008) · Seminars in Hematology · doi:10.1053/j.seminhematol.2008.04.003 · Cited by 680
A Statistical Derivation of the Significant-Digit Law
Hill, T. P. (1995) · Statistical Science · doi:10.1214/ss/1177009869 · Cited by 670
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
Self-citationGeneration of random sequences by human subjects: A critical survey of literature.
Wagenaar, W. A. (1972) · Psychological Bulletin · doi:10.1037/h0032060 · Cited by 489
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468
The effective use of Benford's Law to assist in detecting fraud in accounting data
Durtschi, C., Hillison, W., & Pacini, C. (2004) · Digital Commons - University of South Florida (University of South Florida) · Cited by 466
Sobel, M. E., & Agresti, A. (1993) · Contemporary Sociology · doi:10.2307/2074579 · Cited by 407
Youngstrom, E. A. (2014) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jst062 · Cited by 387
Probability and Statistical Inference
Borowiak, D. (1989) · Technometrics · doi:10.1080/00401706.1989.10488575 · Cited by 291
The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications
Bik, E. M., Casadevall, A., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.00809-16 · Cited by 287
The analysis of 168 randomised controlled trials to test data integrity
Carlisle, J. B. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07128.x · Cited by 255
Carlisle, J. B. (2017) · Anaesthesia · doi:10.1111/anae.13938 · Cited by 229
Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone
Simonsohn, U. (2013) · Psychological Science · doi:10.1177/0956797613480366 · Cited by 216
Computer algorithms for plagiarism detection
Parker, A., & Hamblen, J. O. (1989) · IEEE Transactions on Education · doi:10.1109/13.28038 · Cited by 216
Flawed science: The fraudulent research practices of social psychologist Diederik Stapel
Levelt, W. J. M., Drenth, J. P. H., & Noort, E. (2012) · Cited by 210
Not the First Digit! Using Benford's Law to Detect Fraudulent Scientif ic Data
Diekmann, A. (2007) · Journal of Applied Statistics · doi:10.1080/02664760601004940 · Cited by 209
Kranke, P., Apfel, C. C., & Roewer, N. (2000) · Anesthesia & Analgesia · doi:10.1213/00000539-200004000-00053 · Cited by 156
A Simple Explanation of Benford's Law
Fewster, R. M. (2009) · The American Statistician · doi:10.1198/tast.2009.0005 · Cited by 155
A basic theory of Benford’s Law
Berger, A., & Hill, T. P. (2011) · Probability Surveys · doi:10.1214/11-ps175 · Cited by 141
Brown, N. J. L., & Heathers, J. A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550616673876 · Cited by 127
Registered Replication Report on Mazar, Amir, and Ariely (2008)
Verschuere, B., Meijer, E. H., Jim, A., Hoogesteyn, K., Orthey, R., McCarthy, R. J., Skowronski, J. J., Acar, O. A., Aczel, B., Bakos, B. E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918781032 · Cited by 117
Distributions of p -values smaller than .05 in psychology: what is going on?
Hartgerink, C. H. J., van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · PeerJ · doi:10.7717/peerj.1935 · Cited by 89
Self-citationCarlisle, J. B., Dexter, F., Pandit, J. J., Shafer, S. L., & Yentis, S. M. (2015) · Anaesthesia · doi:10.1111/anae.13126 · Cited by 87
Ulrich, R., & Miller, J. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000086 · Cited by 85
Registered Replication Report: Study 1 From Finkel, Rusbult, Kumashiro, & Hannon (2002)
Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutoğlu, B., Bahník, Š., Bowen, J. D., Bredow, C. A., Bromberg, C., Caprariello, P. A., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616664694 · Cited by 82
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 69
Self-citationFanelli, D., Costas, R., Fang, F. C., Casadevall, A., & Bik, E. M. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-018-0023-7 · Cited by 65
Evidence for non‐random sampling in randomised, controlled trials by Yuhji Saitoh
Carlisle, J. B., & Loadsman, J. A. (2017) · Anaesthesia · doi:10.1111/anae.13650 · Cited by 64
Data fabrication: Can people generate random digits?
Mosimann, J. E., Wiseman, C. V., & Edelman, R. E. (1995) · Accountability in Research · doi:10.1080/08989629508573866 · Cited by 62
Comment: What p-hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2015) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56
Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018) · PeerJ Preprints · doi:10.7287/peerj.preprints.26968v1 · Cited by 50
Stricker, J., & Günther, A. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000356 · Cited by 48
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
Self-citationIrregularities in LaCour (2014)Preprint
Broockman, D., Kalla, J., & Aronow, P. M. (2020) · doi:10.31222/osf.io/qy2se · Cited by 43
Detecting fabrication of data in a multicenter collaborative animal study.
Bailey, K. R. (1991) · Controlled Clinical Trials · doi:10.1016/0197-2456(91)90037-m · Cited by 35
The GRIMMER test: A method for testing the validity of reported measures of variability
Anaya, J. (2016) · PeerJ Preprints · doi:10.7287/peerj.preprints.2400v1 · Cited by 31
Bolland, M. J., Gamble, G. D., Avenell, A., & Grey, A. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.001 · Cited by 31
An appraisal of the Carlisle-Stouffer-Fisher method for assessing study data integrity and fraud
Mascha, E. J., Vetter, T. R., & Pittet, J.-F. (2017) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002415 · Cited by 30
The Baltimore Case: A Trial of Politics, Science, and Character
Chalk, R. A., & Kevles, D. J. (2000) · Academe · doi:10.2307/40252375 · Cited by 28
Koppers, L., Wormer, H., & Ickstadt, K. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-016-9841-7 · Cited by 26
Hüllemann, S., Schüpfer, G., & Mauch, J. (2017) · Der Anaesthesist · doi:10.1007/s00101-017-0333-1 · Cited by 25
Improving the Conduct and Reporting of Statistical Analysis in Psychology
Sijtsma, K., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Psychometrika · doi:10.1007/s11336-015-9444-2 · Cited by 24
Self-citationWidening the search for suspect data – is the flood of retractions about to become a tsunami?
Loadsman, J. A., & McCulloch, T. J. (2017) · Anaesthesia · doi:10.1111/anae.13962 · Cited by 21
Hartgerink, C. H. J., & van Zelst, M. (2018) · Publications · doi:10.3390/publications6020021 · Cited by 21
Self-citationRegularity of digits and significant digits of random variables
Hill, T. P., & Schürger, K. (2005) · Stochastic Processes and their Applications · doi:10.1016/j.spa.2005.05.003 · Cited by 20
Proceedings of the 4th World Conference on Research Integrity
O’Brien, S. P., Chan, D. T.-M., Leung, F., Ko, E. J., Kwak, J. S., Gwon, T., Lee, J. M., Lee, M., Nolte, H., Gommel, M., et al. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0012-9 · Cited by 17
Self-citationSensitivity to statistical regularities: People (largely) follow Benford’s law
Burns, B. D. (2009) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 16
Errors and Integrity in Seeking and Reporting Apparent Research Misconduct
Kharasch, E. D., & Houle, T. T. (2017) · Anesthesiology · doi:10.1097/aln.0000000000001875 · Cited by 15
Seeking and reporting apparent research misconduct: errors and integrity
Kharasch, E. D., & Houle, T. T. (2018) · Anaesthesia · doi:10.1111/anae.14147 · Cited by 15
How does correlation structure differ between real and fabricated data-sets?
Akhtar‐Danesh, N., & Dehghan-Kooshkghazi, M. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-18 · Cited by 14
The value of statistical tools to detect data fabrication
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Research Ideas and Outcomes · doi:10.3897/rio.2.e8860 · Cited by 14
Self-citationMiller, D. R. (2015) · Anaesthesia · doi:10.1111/anae.13165 · Cited by 12
Collateral damage: How one misconduct case brought a biology institute to its knees
Cyranoski, D. (2015) · Nature · doi:10.1038/520600a · Cited by 12
Uniform occurrence of digits for folded and mixture distributions on finite intervals
Mosimann, J. E., & Ratnaparkhi, M. V. (1996) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919608813325 · Cited by 11
688, 112 Statistical Results: Content Mining Psychology Articles for Statistical Test Results
Hartgerink, C. (2016) · International Conference on Data Technologies and Applications · doi:10.3390/data1030014 · Cited by 10
Self-citationPutting the record straight: granisetron′s efficacy as an antiemetic ‘post‐Fujii’
Kranke, P. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07318.x · Cited by 9
Piraino, S. W. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/179135 · Cited by 6
Seeking and reporting apparent research misconduct: errors and integrity – a reply
Carlisle, J. B. (2017) · Anaesthesia · doi:10.1111/anae.14148 · Cited by 6
Image doctoring must be halted
(2017) · Nature · doi:10.1038/546575a · Cited by 5
Editorial expression of concern
Mcnutt, M. K. (2015) · Science · doi:10.1126/science.aac6184 · Cited by 5
Difficulties Detecting Fraud? The Use of Benford's Law on Regression Tables
Bauer, J., & Gross, J. (2011) · De Gruyter eBooks · doi:10.1515/9783110508420-010 · Cited by 5
Joint Editors-in-Chief request for determination regarding papers published by Dr. Yoshitaka Fujii
(2012) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2012.10.001 · Cited by 4
Chapter Eight. Detecting Fraud and Errors Using Benfordʼs Law
Nigrini, M. J. (2015) · Princeton University Press eBooks · doi:10.1515/9781400866595-011 · Cited by 4
Errors in published papers are multifactorial
Moppett, I. K. (2017) · Anaesthesia · doi:10.1111/anae.14048 · Cited by 3
Ethisch klimaat op het werk : Op zoek naar het nieuwe normaal
Ellemers, N. (2017) · De Psycholoog · Cited by 2
Transcripts Of 28 Interviews With Researchers Who Fabricated Data For An Experiment
Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.832490 · Cited by 2
Self-citation
Cited by
Methods to assess research misconduct in health-related research: A scoping review
Bordewijk, E. M., Li, W., van Eekelen, R., Wang, R., Showell, M., Mol, B. W., & van Wely, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.012 · Cited by 68
International multi‐stakeholder consensus statement on clinical trial integrity
Khan, K. S. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17451 · Cited by 30
The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials
Hunter, K. E., Aberoumand, M., Libesman, S., Sotiropoulos, J. X., Williams, J. G., Aagerup, J., Wang, R., Mol, B. W., Li, W., Barba, A., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1738 · Cited by 28
“47”
Maximal positive controls: A method for estimating the largest plausible effect size
Hilgard, J. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104082 · Cited by 27
Khan, K. S., Fawzy, M., & Chien, P. F. W. (2023) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.14837 · Cited by 20
“56,57 This conclusion reinforces the need for research to establish the pre-test probabilities when evaluating the measurement properties of RCT integrity tests to gauge the level of their performance.”
Background
Metadata
- Authors
- Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M.
- Affiliations
- Tilburg University (NL)
- Publisher
- Center for Open Science
- Published
- 2019-08-19
- OpenAlex
- W3162976510
- Semantic Scholar
- 24fe958de5a6b21ec71fa120c67a43a369fec1d1
- Cited by
- 16
- References
- 82
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 4 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- research misconduct, analysis of variance, multivariate association, random number generators, anchoring effect, Stroop task, diagnostic performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.