Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : an observational study
Open accessPublished version of a preprint
Abstract
For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies have documented severe shortcomings in the state of computational reproducibility across fields including psychology, economics, medicine, political science, life sciences, geoscience, and computer science.
“recent studies have documented severe shortcomings in the state of computational reproducibility across fields including psychology (Hardwicke et al., 2021; Obels et al., 2020; Hardwicke et al., 2018), economics (Gertler et al., 2018; McCullough et al., 2006), medicine (Naudet et al., 2018), political science (Stockemer et al., 2018), life sciences (Andrew et al., 2015; Gilbert et al., 2012; Ioann”
From CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark (2024) · cites this paper as Hardwicke et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Schwarzer, G., Carpenter, J. R., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0 · Cited by 1,230 · see citing works
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · see citing works
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · see citing works
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
Self-citationIs Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'
Chang, A. C., & Li, P. (2015) · doi:10.2139/ssrn.2669564 · Cited by 217 · see citing works
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79 · see citing works
Self-citationLessons from a Decade of Replications at the <i>Quarterly Journal of Political Science</i>
Eubank, N. (2016) · Political Science and Politics · doi:10.1017/s1049096516000196 · Cited by 35 · see citing works
Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Sakaluk, J., Williams, A., & Biernat, M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614549257 · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- [2]LeBel EP, McCarthy RJ, Earp BD, Elson M, Vanpaemel W 2018 A unified framework to quantify the credibility of scientific findings. Adv. Methods Pract. Psychol. Sci. 1, 389– 402. (10.1177/2515245918787489) · DOI
- [4]Obels P, Lakens D, Coles N, Gottfried J, Green SA 2019 Analysis of open data and computational reproducibility in registered reports in psychology. PsyArXiv. (10.31234/osf.io/fk8vh) · DOI
- [6]Minocher R, Atmaca S, Bavero C, McElreath R, Beheim B 2020 Reproducibility of social learning research declines exponentially over 63 years of publication. PsyArXiv (10.31234/osf.io/4nzc7) PMC8441137 34540248 · DOI
- [7]Nuijten MB, Bakker M, Maassen E, Wicherts JM 2018 Verify original results through reanalysis before replicating. Behav. Brain Sci. 41, e143 (10.1017/S0140525X18000791) 31064583 · DOI
- [10]Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, Ioannidis JPA 2018 Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ 360, k400 (10.1136/bmj.k400) 29440066 PMC5809812 · DOI · Paper page
- [11]Hardwicke TE, Serghiou S, Janiaud P, Danchev V, Crüwell S, Goodman SN, Ioannidis JP 2020 Calibrating the scientific ecosystem through meta-research. Annu. Rev. Stat. Appl. 7, 11– 37. (10.1146/annurev-statistics-031219-041104) · DOI · Paper page
- [12]Hardwicke TE, Thibault RT, Kosie J, Wallach JD, Kidwell MC, Ioannidis JPA 2020 Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). MetaArXiv (10.31222/osf.io/9sz2y) PMC8785283 33682488 · DOI
- [13]Hardwicke TE, Ioannidis JPA 2018 Populating the Data Ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS ONE 13, e0201856 (10.1371/journal.pone.0201856) 30071110 PMC6072126 · DOI · Paper page
- [15]Kidwell MC et al. 2016 Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol. 14, e1002456 (10.1371/journal.pbio.1002456) 27171007 PMC4865119 · DOI · Paper page
- [17]Vines TH et al. 2014 The availability of research data declines rapidly with article age. Curr. Biol. 24, 94– 97. (10.1016/j.cub.2013.11.014) 24361065 · DOI · Paper page
- [18]Eich E 2014 Business not as usual. Psychol. Sci. 25, 3– 6. (10.1177/0956797613512465) 24285431 · DOI
- [24]Vuorre M, Crump MJC 2020 Sharing and organizing research products as R packages. Behav. Res. Methods. (10.3758/s13428-020-01436-x) PMC8062383 32875401 · DOI
- [25]Borghi J, Van Gulick A. 2020 Data management and sharing: practices and perceptions of psychology researchers. PsyArXiv. (doi:10.31234/osf.io/7g3ae) (10.31234/osf.io/7g3ae) · DOI
- [26]Klein O et al. 2018 A practical guide for transparency in psychological science. Collabra: Psychol. 4, 1– 15. (10.1525/collabra.158)30637365 · DOI
- [27]Rouder JN, Haaf JM, Snyder HK 2019 Minimizing mistakes in psychological science. Adv. Methods Pract. Psychol. Sci. 2, 3– 11. (10.1177/2515245918801915) · DOI
- [28]Xie Y 2015 Dynamic documents with R and knitr, 2nd edn Boca Raton, FL: CRC Press.
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Self-citationHardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187 · see citing works
Self-citationCrüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., et al. (2023) · Psychological Science · doi:10.1177/09567976221140828 · Cited by 54 · see citing works
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28 · see citing works
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4 · see citing works
“Yet, recent studies have documented severe shortcomings in the state of computational reproducibility across fields including psychology (Hardwicke et al., 2021; Obels et al., 2020; Hardwicke et al., 2018), economics (Gertler et al., 2018; McCullough et al., 2006), medicine (Naudet et al., 2018),…”
- Authors
- Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C.
- Affiliations
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Max Planck Institute for Evolutionary Anthropology (DE)
- University of California, Los Angeles (US)
- Stanford University (US)
- Tilburg University (NL)
- Corresponding author
- Hardwicke, Tom E.
- ORCID
- Venue
- Royal Society Open Science
- Publisher
- The Royal Society
- Published
- 2021-01
- Volume
- 8
- Issue
- 1
- Pages
- 201494-201494
- ISSN
- 2054-5703 (ISSN-L)
- PMC
- PMC7890505
- OpenAlex
- W2782009241
- Cited by
- 108
- References
- 9
- Citations by year
- 2026: 16 · 2025: 14 · 2024: 30 · 2023: 21 · 2022: 19 · 2021: 6 · 2020: 1 · 2017: 1
- FWCI
- 17.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Royal Society Open Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Research portal (Tilburg University) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Stiftung Charité
- Einstein Stiftung Berlin
- Laura and John Arnold Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- open data, reproducibility, author response
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
