Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)
Open accessPublished version of a preprint
The DOI matches its Crossref record; 79 of the 80 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In disciplines beyond clinical trials, preregistration has yet to become standard practice.”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“In psychology, recent estimates for the prevalence of preregistration are lacking, but it likely remains around or below 10% [39,40].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Home visits: a strategy to improve newborn survival - Authors' reply
(2008) · Americanae (AECID Library) · doi:10.1016/s0140 · Cited by 22,524
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Serious concerns about the credibility and utility of some scientific research (Ioannidis, 2005; Ioannidis, Greenland, et al., 2014) have prompted calls for increasingly adopting research practices that enhance reproducibility and transparency (Miguel et al., 2014; Munafò et al., 2017; Nosek…”
BackgroundSelf-citation1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationPromoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 1×
“Preregistration is relatively new to psychology (Nosek et al., 2018, 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012), in which they have become the expected norm (Zarin et al., 2017).”
BackgroundScope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816 · cited 2×
“In addition, potential sources of bias, such as conflicts of interest and funding sources, may not be disclosed (Bekelman et al., 2003; Cristea & Ioannidis, 2018).”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 1×
“For example, in biomedicine at least, there is a geometric growth in the number of meta-analyses, and in many fields multiple meta-analyses are often conducted once several studies appear on the same research question (Ioannidis, 2016).”
BackgroundSelf-citationIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 2×
“Although the current findings suggest that routine replication and evidence synthesis is relatively rare in psychology, many high-profile replication attempts have been conducted in recent years (Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012).”
BackgroundResultThe state of OA: a large-scale analysis of the prevalence and impact of Open Access articles
Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West, J., & Haustein, S. (2018) · PeerJ · doi:10.7717/peerj.4375 · Cited by 1,223
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Reproducibility crisis: Blame it on the antibodies
Baker, M. (2015) · Nature · doi:10.1038/521274a · Cited by 830
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761 · cited 1×
“…have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results (Ioannidis, Munafò, et al., 2014; John et al., 2012; Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012; Simmons et al., 2011; Szucs & Ioannidis, 2017).”
ResultSelf-citationReplications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · cited 1×
“Journal policies that use badges to encourage data sharing (Kidwell et al., 2016) or mandate data sharing (Hardwicke et al., 2018; Nuijten et al., 2017) have been associated with marked increases in data availability in the journals that adopted them.”
BackgroundSelf-citationThe Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · cited 3×
“transparency, reproducibility, meta-research, psychology, open science self-correction (Ioannidis, 2012), and enhance credibility (Vazire, 2017).”
BackgroundSelf-citationNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
WaveLab and Reproducible Research
Buckheit, J. B., & Donoho, D. L. (1995) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-2544-7_5 · Cited by 537
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Self-citationAssessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
Self-citationBarreto, M. L., Teixeira, M. G., Bastos, F. I., Ximenes, R. A., Barata, R. B., & Rodrigues, L. C. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60202-x · Cited by 396
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · cited 1×
“This is consistent with accumulating evidence that suggests that the data underlying scientific claims are rarely immediately available (Alsheikh-Ali et al., 2011; Iqbal et al., 2016), although some modest improvement has been observed in recent years in biomedicine (Wallach et al.”
BackgroundSelf-citationTierney, J. F., Vale, C., Riley, R., Smith, C. T., Stewart, L., Clarke, M., & Rovers, M. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001855 · Cited by 361
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 359
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Self-citationPreregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 319
Self-citationReproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · cited 7×
“Using previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the social sciences (Hardwicke, Wallach, et al., 2020) as a guide, we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related…”
BackgroundMethodResultInfluentialSelf-citationThe Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review
Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., et al. (2016) · Royal Society Open Science · doi:10.1098/rsos.150547 · Cited by 284
Data Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 283
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269 · cited 5×
“%) than social-science articles in general (31% and 15%, respectively; Hardwicke, Wallach, et al., 2020) but less likely than biomedical articles (69% and 65%, respectively; Wallach et al., 2018).”
BackgroundResultInfluentialSelf-citationUpdate on Trial Registration 11 Years after the ICMJE Policy Was Established
Zarin, D. A., Tse, T., Williams, R. J., & Rajakannan, T. (2017) · New England Journal of Medicine · doi:10.1056/nejmsr1601330 · Cited by 239
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · cited 1×
“, 2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al., 2014; Wagenmakers et al., 2012) and enabling the detection and mitigation of questionable research practices such as selectiveoutcome reporting (Franco et al.”
BackgroundSimultaneous Confidence Intervals and Sample Size Determination for Multinomial Proportions
Sison, C. P., & Glaz, J. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476521 · Cited by 236
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra: Psychology · doi:10.1525/collabra.158 · Cited by 233
Self-citationNaudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227
Self-citationPractical Tips for Ethical Data Sharing
Meyer, M. N. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747656 · Cited by 225
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
Self-citationA Unified Framework to Quantify the Credibility of Scientific Findings
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918787489 · Cited by 214
Claesen, A., Gomes, S. L. B. T., tuerlinckx, F., & vanpaemel, W. (2019) · doi:10.31234/osf.io/d8wex · Cited by 200
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
Extending the Mertonian Norms: Scientists' Subscription to Norms of Research
Anderson, M. S., Ronning, E. A., Vries, R. D., & Martinson, B. C. (2010) · The Journal of Higher Education · doi:10.1080/00221546.2010.11779057 · Cited by 184
Ten simple rules to consider regarding preprint submission
Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005473 · Cited by 162
Seven Easy Steps to Open Science: An Annotated Reading List
Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A., Parsons, S., & Schulte-Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 153
Are We Wasting a Good Crisis? The Availability of Psychological Research Data after the Storm
Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015) · Collabra · doi:10.1525/collabra.13 · Cited by 152
Ensuring the quality and specificity of preregistrations
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000937 · Cited by 138
Mapping the universe of registered reports
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0444-y · Cited by 138
Self-citationThe Evolution of Trial Registries and Their Use to Assess the Clinical Trial Enterprise
Dickersin, K. (2012) · JAMA · doi:10.1001/jama.2012.4230 · Cited by 127 · cited 1×
“Preregistration is relatively new to psychology (Nosek et al., 2018, 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012), in which they have become the expected norm (Zarin et al., 2017).”
BackgroundUnderreporting in Psychology Experiments: Evidence From a Study Registry
Franco, A., Malhotra, N., & Simonovits, G. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 117
Statistical Methods for Extreme Event Attribution in Climate Science
Naveau, P., Hannart, A., & Ribes, A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041314 · Cited by 117
Extending the Mertonian Norms: Scientists' Subscription to Norms of Research
Anderson, M. S., Ronning, E. A., Vries, R. G. D., et al. (2010) · The Journal of Higher Education · doi:10.1353/jhe.0.0095 · Cited by 109
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
Self-citationQuality Uncertainty Erodes Trust in Science
Vazire, S. (2017) · Collabra: Psychology · doi:10.1525/collabra.74 · Cited by 106 · cited 3×
“However, there is a growing (or reemerging; David, 2008) appreciation that the scientific community needs to be able to access all of this information to comprehensively evaluate, interpret, and independently verify scientific claims (Munafò et al., 2017; Vazire, 2017).”
BackgroundCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Self-citationEvangelou, E., Trikalinos, T. A., & Ioannidis, J. P. A. (2005) · The FASEB Journal · doi:10.1096/fj.05-4784lsf · Cited by 92
Self-citationHas open data arrived at the British Medical Journal (BMJ) ? An observational study
Rowhani-Farid, A., & Barnett, A. G. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011784 · Cited by 84
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78
Self-citationDavid, P. A. (2008) · Capitalism and Society · doi:10.2202/1932-0213.1040 · Cited by 67
Psychologists Are Open to Change, yet Wary of Rules
Fuchs, H. M., Jenny, M., & Fiedler, S. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459521 · Cited by 50
Does psychology have a conflict-of-interest problem?
Chivers, T. (2019) · Nature · doi:10.1038/d41586-019-02041-5 · Cited by 40
Open science practices in clinical psychology journals: An audit study.
Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000414 · Cited by 34
Improving Disclosure of Financial Conflicts of Interest for Research on Psychosocial Interventions
Cristea, I. A., & Ioannidis, J. P. A. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.0382 · Cited by 31
Self-citationBadges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.1 · Cited by 31
Badges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 26
Research, regulatory, and clinical decision-making: the importance of scientific integrity
Wallach, J. D., Gonsalves, G., & Ross, J. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.021 · Cited by 25
Self-citationThe Virtuous Cycle of a Data Ecosystem
Voytek, B. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005037 · Cited by 25
Computational Reproducibility via Containers in Social PsychologyPreprint
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mf82t · Cited by 21
Good Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationBest Laid Plans: A Guide to Reporting Preregistration Deviations
Willroth, E. C., & Atherton, O. E. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231213802 · Cited by 220
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Self-citationSurvey of open science practices and attitudes in the social sciences
Ferguson, J., Littman, R., Christensen, G., Paluck, E. L., Swanson, N., Wang, Z., Miguel, E. A., Birke, D. J., & Pezzuto, J.-H. (2023) · Nature Communications · doi:10.1038/s41467-023-41111-1 · Cited by 135
Reproducibility in Neuroimaging Analysis: Challenges and Solutions
Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 118
Metadata
- Authors
- Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A.
- Affiliations
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- University of Bristol (GB)
- Princeton University (US)
- Yale University (US)
- University of Utah (US)
- Corresponding author
- Hardwicke, Tom E.
- ORCID
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2021-03-08
- Volume
- 17
- Issue
- 1
- Pages
- 239-251
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- PMC
- PMC8785283
- OpenAlex
- W3134007699
- Cited by
- 187
- References
- 76
- Citations by year
- 2026: 20 · 2025: 29 · 2024: 28 · 2023: 51 · 2022: 34 · 2021: 22
- FWCI
- 30.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Perspectives on Psychological Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- MetArXiv (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- MetArXiv (OSF Preprints) (submittedVersion)
- Funders
- fonds de recherche du québec - santé
- Einstein Stiftung Berlin
- stiftung charité
- Laura and John Arnold Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- research transparency, reproducibility, research practices, psychology literature, preregistration, data sharing, funding disclosure, conflicts of interest, replication study, systematic reviews, meta-analysis, research credibility, open science
- MeSH terms
- Systematic Reviews as Topic, Humans, Publications, Research Design, Reproducibility of Results, Prevalence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.