Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone
The DOI matches its Crossref record; 52 of the 53 references it cites were found in a registry or an index.
Abstract
I argue that requiring authors to post the raw data supporting their published results has the benefit, among many others, of making fraud much less likely to go undetected. I illustrate this point by describing two cases of suspected fraud I identified exclusively through statistical analysis of reported means and standard deviations. Analyses of the raw data behind these published results provided invaluable confirmation of the initial suspicions, ruling out benign explanations (e.g., reporting errors, unusual distributions), identifying additional signs of fabrication, and also ruling out one of the suspected fraud's explanations for his anomalous results. If journals, granting agencies, universities, or other entities overseeing research promoted or required data posting, it seems inevitable that fraud would be reduced.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Researchers who fabricated randomised trials have been discovered because their baseline tables were not realistic. 3 -5”
From Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials. (2022) · cites this paper as [3]
In the citing paper
“Statistical approaches have been used to detect fraud in trials using individual patient data. 3,6,48”
From Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials. (2022) · cites this paper as [3]
Citations
Cites
5 retracted
Mixed-motive conflict in social dilemmas: Mood as input to competitive and cooperative goals.
Sanna, L. J., Parks, C. D., & Chang, E. C. (2003) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.7.1.26 · Cited by 39
RetractedSanna, L. J., Chang, E. C., Miceli, P. M., & Lundberg, K. B. (2010) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2010.12.013 · Cited by 29
RetractedSauna, L. J., Chang, E. C., Parks, C. D., & Kennedy, L. A. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02458.x · Cited by 28
RetractedSmeesters, D., & Liu, J. (. (2011) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2011.02.010 · Cited by 24
RetractedRETRACTED: Effects of Messiness on Preferences for Simplicity
Liu, J., Smeesters, D., & Trampe, D. (2011) · Journal of Consumer Research · doi:10.1086/662139 · Cited by 10
RetractedAn Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Proceedings of the American philosophical society
(1838) · Journal of the Franklin Institute · doi:10.1016/s0016-0032(38)92229-x · Cited by 5,524
Ego depletion: Is the active self a limited resource?
Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.74.5.1252 · Cited by 5,312
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 1×
“One of the most well-known judgment biases is the belief in the law of small numbers (Tversky & Kahneman, 1971)—the belief that even small samples closely resemble underlying populations.”
BackgroundAn Introduction to the Bootstrap.
Buckland, S. T., Efron, B., & Tibshirani, R. J. (1994) · Biometrics · doi:10.2307/2532810 · Cited by 3,103
The automated will: nonconscious activation and pursuit of behavioral goals.
Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K., & Trötschel, R. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.81.6.1014 · Cited by 1,865
Emotional distress regulation takes precedence over impulse control: If you feel bad, do it!
Tice, D. M., Bratslavsky, E., & Baumeister, R. F. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.80.1.53 · Cited by 1,361 · cited 1×
“…(2011) d = 0.37, n = 27.5 Vohs & Heatherton (2000) d = 0.84, n = 15.5 Webb & Sheeran (2003) d = 0.89, n = 17 Bargh et al. (2001) d = 0.71, n = 15 Tice et al. (2001) d = 1.04, n = 11.75 De Cremer & van Dijk (2002) d = 1.67, n = 34.25 Kerr et al. Study 1 (2009) d = 0.36, n = 44.5 Kerr et al.…”
InfluentialSelf-Regulatory Failure: A Resource-Depletion Approach
Vohs, K. D., & Heatherton, T. F. (2000) · Psychological Science · doi:10.1111/1467-9280.00250 · Cited by 974
When ostracism leads to aggression: The moderating effects of control deprivation
Warburton, W. A., Williams, K. D., & Cairns, D. R. (2006) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2005.03.005 · Cited by 573
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Gender Differences in Cooperation and Competition: The Male-Warrior Hypothesis
Vugt, M. V., Cremer, D. D., & Janssen, D. P. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01842.x · Cited by 541
The two-sample trimmed t for unequal population variances
YUEN, K. K. (1974) · Biometrika · doi:10.1093/biomet/61.1.165 · Cited by 486
McGregor, H. A., Lieberman, J. D., Greenberg, J., Solomon, S., Arndt, J., Simon, L., & Pyszczynski, T. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.3.590 · Cited by 403
Can implementation intentions help to overcome ego-depletion?
Webb, T. L., & Sheeran, P. (2003) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(02)00527-9 · Cited by 382
Has Mendel's work been rediscovered?
Fisher, R. A. (1936) · Annals of Science · doi:10.1080/00033793600200111 · Cited by 344
Anchoring Effects on Consumers’ Willingness‐to‐Pay and Willingness‐to‐Accept
Simonson, I., & Drolet, A. (2004) · Journal of Consumer Research · doi:10.1086/425103 · Cited by 279
The analysis of 168 randomised controlled trials to test data integrity
Carlisle, J. B. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07128.x · Cited by 255 · cited 1×
“…were too similar across conditions to have originated in random samples, an approach to identifying problematic data that has been employed before (Carlisle, 2012; Fisher, 1936; Gaffan & Gaffan, 1992; Kalai, McKay, & Bar-Hillel, 1998; Roberts, 1987; Sternberg & Roberts, 2006).1 These preliminary…”
MethodAn Introduction to the Theory of Statistics.
Allen, R. G. D., Yule, G. U., & Kendall, M. G. (1937) · The Economic Journal · doi:10.2307/2225339 · Cited by 254
Power and Choice: Their Dynamic Interplay in Quenching the Thirst for Personal Control
Inesi, M. E., Botti, S., Dubois, D., Rucker, D. D., & Galinsky, A. D. (2011) · Psychological Science · doi:10.1177/0956797611413936 · Cited by 231
Ayduk, Ö., Gyurak, A., & Luerssen, A. (2008) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2007.07.004 · Cited by 223 · cited 1×
“…% 12 % 17 % 0%50 % 10 0% 15 0% 20 0% 25 0% Study 2: Time d = 1.64, n = 20 Study 3: Hot Sauce d = 1.85, n = 15 Study 4: Fish d = 1.29, n = 15 Ayduk et al. (2008) d = 0.35, n = 32.25 Klinesmith et al. (2006) d = 1.71, n = 15 McGregor et al. (1998) [1] d = 0.95, n = 18.5 McGregor et al.…”
InfluentialCackowski, J. M., & Nasar, J. L. (2003) · Environment and Behavior · doi:10.1177/0013916503256267 · Cited by 191
Introduction to the Bootstrap World
Boos, D. D. (2003) · Statistical Science · doi:10.1214/ss/1063994971 · Cited by 152
Cremer, D. D., & van Dijk, E. (2002) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(02)00009-4 · Cited by 148
Direct Risk Aversion: Evidence From Risky Prospects Valued Below Their Worst Outcome
Simonsohn, U. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02349.x · Cited by 136
Self-citationKerr, N. L., Rumble, A. C., Park, E. S., Ouwerkerk, J. W., Parks, C. D., Gallucci, M., & Lange, P. A. M. V. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.02.017 · Cited by 126
The New and Improved Two-Sample t Test
Keselman, H. J., Othman, A. R., Wilcox, R. R., & Fradette, K. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01501008.x · Cited by 124
Guns, Testosterone, and Aggression: An Experimental Test of a Mediational Hypothesis
Klinesmith, J., Kasser, T., & McAndrew, F. T. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01745.x · Cited by 109
de Kwaadsteniet, E. W., van Dijk, E., Wit, A., & Cremer, D. D. (2005) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2005.06.004 · Cited by 107
Bootstrap Methods for Testing Homogeneity of Variances
Boos, D. D., & Brownie, C. (1989) · Technometrics · doi:10.1080/00401706.1989.10488477 · Cited by 105
Overestimating Others’ Willingness to Pay
Frederick, S. (2012) · Journal of Consumer Research · doi:10.1086/662060 · Cited by 97
On the Robustness of Anchoring Effects in WTP and WTA Experiments
Drew Fudenberg, B., Levine, D. K., & Maniadis, Z. (2012) · American Economic Journal Microeconomics · doi:10.1257/mic.4.2.131 · Cited by 92
Psychology must learn a lesson from fraud case
Wicherts, J. M. (2011) · Nature · doi:10.1038/480007a · Cited by 86
Publish (your data) or (let the data) perish! Why not publish your data too?
Wicherts, J. M., & Bakker, M. (2012) · Intelligence · doi:10.1016/j.intell.2012.01.004 · Cited by 85
An Introduction to the Theory of Statistics.
Brillinger, D. R., & Plackett, R. L. (1972) · Journal of the American Statistical Association · doi:10.2307/2284679 · Cited by 73
The role of interpersonal perceptions in the prime-to-behavior pathway.
Smeesters, D., Wheeler, S. C., & Kay, A. C. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0012959 · Cited by 51
Keren, G. B., & Willemsen, M. C. (2008) · Journal of Behavioral Decision Making · doi:10.1002/bdm.628 · Cited by 35
Why are lotteries valued less? Multiple tests of a direct risk-aversion mechanism
Newman, G. E., & Daniel Mochon, B. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001807 · Cited by 31
Gaffan, E. A., & Gaffan, D. (1992) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.18.3.298 · Cited by 19
Less-than-expected variability in evidence for three stages in memory formation.
Roberts, S. (1987) · Behavioral Neuroscience · doi:10.1037/0735-7044.101.1.120 · Cited by 8
Nutritional supplements and infection in the elderly: why do the findings conflict?
Sternberg, S., & Roberts, S. (2006) · Nutrition Journal · doi:10.1186/1475-2891-5-30 · Cited by 7
Less-than-expected variability in evidence for three stages in memory formation.
Roberts, S. (1987) · Behavioral Neuroscience · doi:10.1037//0735-7044.101.1.120 · Cited by 6
Rubin, D. B., & Stigler, S. M. (1979) · Science · doi:10.1126/science.205.4412.1204-a · Cited by 6
Retraction of Johnson, Smeesters, and Wheeler (2012).
Johnson, C. S., Smeesters, D., & Wheeler, S. C. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0030145 · Cited by 5
Sanna, L. J., Chang, E. C., Miceli, P. M., & Lundberg, K. B. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.11.015 · Cited by 4
Smeesters, D., & Liu, J. (. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.11.016 · Cited by 4
OBLAD, A. G. (1979) · Science · doi:10.1126/science.472736 · Cited by 3
Research works printed but not linked: 2
- Kalai G., 1998, The two famous rabbis experiments: How similar is too similar (Discussion Paper 182)
- Zwaan R. A., Groenen P. J. F., van der Heijden A. J., te Lindert R. (2012). Report by the Committee for Inquiry into Scientific Integrity. Retrieved from http://www.eur.nl/fileadmin/ASSETS/press/2012/Juli/report_Committee_for_inquiry_prof._Smeesters.publicversion.28_6_2012.pdf
Cited by
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Self-citation
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2013-08-27
- Volume
- 24
- Issue
- 10
- Pages
- 1875-1888
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2142075785
- Cited by
- 216
- References
- 53
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 9 · 2023: 8 · 2022: 11 · 2021: 14 · 2020: 24 · 2019: 18 · 2018: 41 · 2017: 11 · 2016: 25 · 2015: 19 · 2014: 11 · 2013: 13
- FWCI
- 59.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- University of Michigan
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- data sharing, scientific fraud, fraud detection, fabrication, research integrity
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Publishing, Research, Universities, Scientific Misconduct, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.