Ensuring the quality and specificity of preregistrations
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called “OSF Preregistration,” http://osf.io/prereg/). The Prereg Challenge format was a “structured” workflow with detailed instructions and an independent review to confirm completeness; the “Standard” format was “unstructured” with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the “structured” format restricted the opportunistic use of researcher degrees of freedom better (Cliff’s Delta = 0.49) than the “unstructured” format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“One study found that 2 researchers could only agree on the number of hypotheses in 14% of the preregistrations they assessed [45].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
(2011) · Cited by 14,317
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 5×
“Clearly, distinguishing these 2 modes of research is vitally important for maintaining the validity of statistical inferences in confirmatory analysis and for avoiding mistaking the generation of a hypothesis in exploratory analysis as the testing of a hypothesis in confirmatory analysis [1,2].”
BackgroundInfluentialSelf-citation
Research works printed but not linked: 2
- N Cliff, 1996, Ordinal Methods for Behavior Data Analysis
- A Claesen, 2019, Preregistration: Comparing Dream to Reality
Cited by
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 319
Self-citationWhere to prospectively register a systematic review
Pieper, D., & Rombey, T. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01877-1 · Cited by 192
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
Seven Easy Steps to Open Science: An Annotated Reading List
Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A., Parsons, S., & Schulte-Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 153
Preregistration of secondary data analysis: A template and tutorial
Van den Akker, O. R., Weston, S., Campbell, L., Chopik, B., Damian, R., Davis-Kean, P., Hall, A., Kosie, J., Kruse, E., Olsen, J., et al. (2021) · Meta-Psychology · doi:10.15626/mp.2020.2625 · Cited by 122
Self-citationConsortium for Transparency in Exercise Science (COTES) Collaborators, Caldwell, A. R., Vigotsky, A. D., Tenan, M. S., Radel, R., Mellor, D. T., Kreutzer, A., Lahart, I. M., Mills, J. P., & Boisgontier, M. P. (2020) · Sports Medicine · doi:10.1007/s40279-019-01227-1 · Cited by 114
Self-citationBishop, D. V. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021819886519 · Cited by 108
Why most psychological research findings are not even wrong
Scheel, A. M. (2022) · Infant and Child Development · doi:10.1002/icd.2295 · Cited by 125
The benefits of preregistration and Registered Reports
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2376046 · Cited by 104
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72
“One study found that 2 researchers could only agree on the number of hypotheses in 14% of the preregistrations they assessed [45].”
BackgroundA reproducible systematic map of research on the illusory truth effect
Henderson, E. L., Westwood, S. J., & Simons, D. J. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-021-01995-w · Cited by 51
Making ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 71
The Invisible Workload of Open Research
Hostler, T. (2023) · Journal of Trial and Error · doi:10.36850/mr5 · Cited by 72
Transparency Is Now the Default at Psychological Science
Hardwicke, T. E., & Vazire, S. (2023) · Psychological Science · doi:10.1177/09567976231221573 · Cited by 57
Preregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A.-M. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 48
van den Akker, O. R., Bakker, M., van Assen, M. A. L. M., Pennington, C. R., Verweij, L., Elsherif, M. M., Claesen, A., Gaillard, S. D. M., Yeung, S. K., Frankenberger, J.-L., et al. (2024) · Psychological Methods · doi:10.1037/met0000687 · Cited by 52
Self-citationThe academic impact of Open Science: a scoping review
Klebel, T., Traag, V., Grypari, I., Stoy, L., & Ross‐Hellauer, T. (2025) · Royal Society Open Science · doi:10.1098/rsos.241248 · Cited by 51
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F. F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231187988 · Cited by 51
Self-citationProcess and Bureaucracy: Scientific Reform as Civilisation
Penders, B. (2022) · Bulletin of Science Technology & Society · doi:10.1177/02704676221126388 · Cited by 42
Heckelei, T., Hüttel, S., Rommel, J., & Odening, M. (2023) · German Journal of Agricultural Economics · doi:10.30430/gjae.2023.0231 · Cited by 49
Heirene, R. M., LaPlante, D. A., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. (2024) · Meta-Psychology · doi:10.15626/mp.2021.2909 · Cited by 44
Self-citationMeta-analyzing the multiverse: A peek under the hood of selective reporting.
Olsson-Collentine, A., van Aert, R. C. M., Bakker, M., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000559 · Cited by 52
Self-citationReproducible Research in R: A Tutorial on How to Do the Same Thing More Than Once
Peikert, A., Lissa, C. J. V., & Brandmaier, A. M. (2021) · Psych · doi:10.3390/psych3040053 · Cited by 40
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 45
Self-citationEvans, T. R., Branney, P., Clements, A. J., & Hatton, E. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1969233 · Cited by 39
Keyvan, E., Adesemoye, E. T., Champomier‐Vergès, M., Chanséaume, E., Mardon, J., Nedelkoska, D. N., Palamutoğlu, R., Russo, P., Sarand, I., Songré-Ouattara, L. T., et al. (2025) · Frontiers in Nutrition · doi:10.3389/fnut.2025.1653666 · Cited by 33
A Practical Guide to Registered Reports for Economists
Arpinon, T., & Espinosa, R. (2022) · Journal of the Economic Science Association · doi:10.1007/s40881-022-00123-1 · Cited by 39
Center for Open Science: Strategic PlanPreprint
Nosek, B. A., Shaw, L. C., Errington, T. M., Pfeiffer, N., & Zaringhalam, M. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/x2w9h · Cited by 30
Self-citationHardwicke, T. E., Thibault, R. T., Clarke, B., Moodie, N., Crüwell, S., Schiavone, S. R., Handcock, S. A., Nghiem, K. A., Mody, F., Eerola, T., et al. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241283477 · Cited by 37
Predicting treatment response using machine learning: A registered report
Jankowsky, K., Krakau, L., Schroeders, U., Zwerenz, R., & Beutel, M. E. (2023) · British Journal of Clinical Psychology · doi:10.1111/bjc.12452 · Cited by 32
Crüwell, S., & Evans, N. J. (2021) · Royal Society Open Science · doi:10.1098/rsos.210155 · Cited by 30
Registration of health and medical research
Cashin, A. G., Richards, G. C., DeVito, N., Mellor, D. T., & Lee, H. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111836 · Cited by 31
Self-citationEnding publication bias: A values-based approach to surface null and negative results
Curry, S., Mercado-Lara, E., Arechavala‐Gomeza, V., Begley, C. G., Bernard, C., Bernard, R., Bertuzzi, S., Bhalla, N., Bowers, D., Brod, S., et al. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003368 · Cited by 26
de Ridder, J. (2022) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2022.02.003 · Cited by 26
Improving Social Science: Lessons from the Open Science Movement
Engzell, P., & Rohrer, J. M. (2020) · PS Political Science & Politics · doi:10.1017/s1049096520000967 · Cited by 26
Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102327 · Cited by 24
Reflections on Preregistration: Core Criteria, Badges, Complementary Workflows
Thibault, R. T., Pennington, C. R., & Munafo, M. R. (2023) · Journal of Trial and Error · doi:10.36850/mr6 · Cited by 26
Syed, M. (2025) · Journal of Trial and Error · doi:10.36850/e5ce-4cc5 · Cited by 20
Heinl, C., Scholman-Végh, A. M. D., Mellor, D. T., Schönfelder, G., Strech, D., Chamuleau, S. A. J., & Bert, B. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac016 · Cited by 18
Self-citationMeier, M., Lonsdorf, T. B., LUPIEN, S. J., Stalder, T., Laufer, S., Sicorello, M., Linz, R., & Puhlmann, L. M. C. (2022) · Comprehensive Psychoneuroendocrinology · doi:10.1016/j.cpnec.2022.100144 · Cited by 20
Evaluating the Pedagogical Effectiveness of Study Preregistration in the Undergraduate Dissertation
Pownall, M., Pennington, C. R., Norris, E., Juanchich, M., Smailes, D., Russell, P. S., Gooch, D., Evans, T. R., Persson, S., Mak, M. H., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231202724 · Cited by 19
Wudarczyk, O. A., Kırtay, M., Kuhlen, A. K., Rahman, R. A., Haynes, J., Hafner, V. V., & Pischedda, D. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.630789 · Cited by 16
Decisions, Decisions, Decisions: An Ethnographic Study of Researcher Discretion in Practice
van Drimmelen, T., Slagboom, M. N., Reis, R., Bouter, L. M., & van der Steen, J. T. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00481-5 · Cited by 16
Osborne, C., & Norris, E. (2022) · Cogent Psychology · doi:10.1080/23311908.2022.2066304 · Cited by 15
Assessing Open Science practices in physical activity behaviour change intervention evaluations
Norris, E., Sulevani, I., Finnerty, A. N., & Castro, O. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2021-001282 · Cited by 14
Preregistration of Psychology Meta-Analyses: A Cross-Sectional Study of Prevalence and Practice
Sandoval‐Lentisco, A., Tortajada, M., López‐Nicolás, R., López-López, J. A., Wagenmakers, E., Sánchez‐Meca, J., & Hardwicke, T. E. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241300113 · Cited by 16
Ten simple rules for designing and conducting undergraduate replication projects
Moreau, D., & Wiebels, K. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010957 · Cited by 14
Hanging the Anchor Off a New Ship: A Meta-Analysis of Anchoring EffectsPreprint
Röseler, L., & Schütz, A. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/wf2tn · Cited by 14
Humblot, C., Alvanoudi, P., Alves, E., Assunção, R., Belović, M. M., Tüccar, T. B., Chassard, C., Derrien, M., Karagöz, M. F., Karakaya, S., et al. (2025) · Frontiers in Nutrition · doi:10.3389/fnut.2025.1650633 · Cited by 13
On the Use and Misuses of Preregistration: A Reply to Klonsky (2024)
Vize, C. E., Phillips, N. L., Miller, J. D., & Lynam, D. R. (2024) · Assessment · doi:10.1177/10731911241275256 · Cited by 15
Girard, C., Longo, L., Chesnokova, H., Epinat‐Duclos, J., & Prado, J. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102328 · Cited by 12
Review of Four Preregistration Registries for Special Education Researchers
Fleming, J. I., McLucas, A. S., & Cook, B. G. (2023) · Remedial and Special Education · doi:10.1177/07419325231160293 · Cited by 12
Pennington, C. R., & Heim, D. (2021) · Addiction Research & Theory · doi:10.1080/16066359.2021.1931142 · Cited by 12
Chen, Y., Sun, Y., Li, X., Lin, Y., He, P., & Wang, Q. (2025) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-025-09017-9 · Cited by 10
Transparency in Cognitive Training Meta-analyses: A Meta-review
Sandoval‐Lentisco, A., López‐Nicolás, R., Tortajada, M., López-López, J. A., & Sánchez‐Meca, J. (2024) · Neuropsychology Review · doi:10.1007/s11065-024-09638-2 · Cited by 10
Open Research Reforms and the Capitalist University: Areas of Opposition and Alignment
Hostler, T. (2024) · Collabra Psychology · doi:10.1525/collabra.121383 · Cited by 11
Why does preregistration increase the persuasiveness of evidence? A Bayesian rationalizationPreprint
Peikert, A., Ernst, M. S., & Brandmaier, A. M. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cs8wb · Cited by 13
Planning falsifiable confirmatory research.
Kennedy, J. E. (2024) · Psychological Methods · doi:10.1037/met0000639 · Cited by 10
Birkenmaier, L., Wagner, C., & Lechner, C. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02863 · Cited by 12
Binney, R. J., Smith, L., Rossit, S., Demeyere, N., Learmonth, G., Olgiati, E., Halai, A. D., Rounis, E., Evans, J. J., Edelstyn, N. M. J., et al. (2025) · Brain Communications · doi:10.1093/braincomms/fcaf162 · Cited by 10
Not all pre-registrations are equal
Waldron, S., & Allen, C. P. G. (2022) · Neuropsychopharmacology · doi:10.1038/s41386-022-01418-x · Cited by 8
Open science as a better gatekeeper for science and society: a perspective from neurolaw
Chuan-Peng, H., Jiang, X., Jeffrey, R., & Zuo, X. (2018) · 中国科学通报:英文版 · doi:10.1016/j.scib.2018.11.015 · Cited by 9
Grimes, D. R. (2024) · eLife · doi:10.7554/elife.93050 · Cited by 7
A systematic review of pre-registration in autism research journals
Poole, D., Linden, A. H., Sedgewick, F., Allchin, O., & Hobson, H. (2024) · Autism · doi:10.1177/13623613241308312 · Cited by 8
Bertin, P., & Nera, K. (2025) · International Review of Social Psychology · doi:10.5334/irsp.979 · Cited by 7
Barreiro‐Hurlé, J. (2021) · Spanish Journal of Agricultural Research · doi:10.5424/sjar/2021194-18956 · Cited by 7
Novel prediction and the problem of low-quality accommodation
Syrjänen, P. (2023) · Synthese · doi:10.1007/s11229-023-04400-2 · Cited by 7
Holst, M., Faust, A., & Strech, D. (2022) · Health Research Policy and Systems · doi:10.1186/s12961-022-00841-2 · Cited by 9
Mak, M. H. (2024) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2024.100111 · Cited by 6
Ruytenbeek, N., & Holtgraves, T. (2025) · Journal of Pragmatics · doi:10.1016/j.pragma.2025.01.010 · Cited by 9
A Checklist for Incentivizing and Facilitating Good Theory Building
Lange, J., Freyer, N., Musfeld, P., Schönbrodt, F. D., & Leising, D. (2025) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000604 · Cited by 7
Thibault, R. T., Kovács, M., Hardwicke, T. E., Sarafoglou, A., Ioannidis, J. P. A., & Munafo, M. R. (2023) · Royal Society Open Science · doi:10.1098/rsos.230568 · Cited by 6
Challenges in Identifying Individualized Brain Biomarkers of Late Life Depression
Hannon, K., & Bijsterbosch, J. D. (2023) · Advances in Geriatric Medicine and Research · doi:10.20900/agmr20230010 · Cited by 8
Les pratiques de recherche ouvertes en psychologie
Lantian, A. (2020) · Psychologie Française · doi:10.1016/j.psfr.2020.09.001 · Cited by 6
Preregistration of Preregistration evaluation 2016
van Assen, M. A. L. M., Bakker, M., Crompvoets, E. A. V., Mellor, D. T., Wicherts, J. M., Veldkamp, C. L. S., Nosek, B. A., Ong, H. H., & Soderberg, C. K. (2016) · Open Science Framework · doi:10.17605/osf.io/jsznk · Cited by 6
Self-citationWhy I declare a conflict of interest and you should not
Acem, E., Aczél, B., Albayrak‐Aydemir, N., Brown, N. J. L., Dudda, L. A., Elsherif, M. M., Gjoneska, B., Kowal, M., Krishna, A., Miłkoś, S., et al. (2025) · Theory and Society · doi:10.1007/s11186-025-09641-3 · Cited by 5
Eerola, T. (2024) · Musicae Scientiae · doi:10.1177/10298649241300885 · Cited by 7
Is Social Work Research in Crisis?
Dunleavy, D. J., & Lacasse, J. R. (2023) · Research on Social Work Practice · doi:10.1177/10497315231188684 · Cited by 7
A Two-Sided Discussion of Preregistration of NLP Research
Søgaard, A., Hershcovich, D., & de Lhoneux, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.6 · Cited by 9
Accessibility of clinical study reports supporting medicine approvals: a cross-sectional evaluation
Hopkins, A. M., Modi, N. D., Rockhold, F. W., Hoffmann, T., Menz, B. D., Veroniki, A. A., McKinnon, R. A., Rowland, A., Swain, S. M., Ross, J. S., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111263 · Cited by 6
Open science practices in the false memory literature
Wiechert, S., Leistra, P., Ben‐Shakhar, G., Pertzov, Y., & Verschuère, B. (2024) · Memory · doi:10.1080/09658211.2024.2387108 · Cited by 5
Determinants of Citation in Epidemiological Studies on Phthalates: A Citation Analysis
Urlings, M. J. E., Duyx, B., Swaen, G. M. H., Bouter, L. M., & Zeegers, M. P. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00260-y · Cited by 5
Practicing Good Laboratory Hygiene, Even in a Pandemic
Kravitz, D. J., Mitroff, S. R., & Bauer, P. J. (2020) · Psychological Science · doi:10.1177/0956797620920547 · Cited by 5
Clinical Trials to authors: Please pre-register your studies!
Kimmelman, J. (2021) · Clinical Trials · doi:10.1177/17407745211057186 · Cited by 5
RegCheck: A tool for structured comparisons between study registrations and papersPreprint
Cummins, J., Clarke, B., Hussey, I., & Elson, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.13330 · Cited by 5
The Minimally Meaningful Effect Size: A Vital Component of Pre-RegistrationsPreprint
Camilleri, C., Beribisky, N., & Cribbie, R. A. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/jbgtm · Cited by 5
Towards Transparency and Open Science
Peikert, A. (2023) · doi:10.18452/27056 · Cited by 4
Open Science Practices in Psychiatric Genetics: A Primer
Kępińska, A. P., Johnson, J., & Huckins, L. M. (2023) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2023.08.007 · Cited by 3
Sustainable neuroscience through open science
Puhlmann, L. M. C., Koppold, A., Feld, G. B., Lonsdorf, T. B., Hilger, K., Vogel, S., Gürsoy, Ç. N., Kastrinogiannis, A., Kulke, L., Lischke, A., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02426-3 · Cited by 3
“Estimating Software Project Effort Using Analogies”: Reflections After 28 Years
Shepperd, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3534032 · Cited by 3
How do psychology researchers interpret the results of multiple replication studies?
van den Akker, O. R., Wicherts, J. M., Alvarez, L. D., Bakker, M., & van Assen, M. A. L. M. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02235-5 · Cited by 4
Self-citationPownall, M., Pennington, C. R., Norris, E., Juanchich, M., Smailes, D., Russell, P. S., Gooch, D., Evans, T. R., Persson, S., Mak, M. H., et al. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xg2ah · Cited by 3
The Open Empirical Cycle for Hypothesis Evaluation in PsychologyPreprint
Hoijtink, H., Bruin, J. D., Duken, S. B., Flores-Dourojeanni, J. P., Frankenhuis, W. E., & Lissa, C. J. V. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/wsxbh · Cited by 3
Rubin, M. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3t7pc · Cited by 3
Spitzer, L., Bošnjak, M., & Mueller, S. (2024) · Meta-Psychology · doi:10.15626/mp.2023.4039 · Cited by 2
Gould, E., Jones, C. S., Yen, J. D. L., Fraser, H. S., Wootton, H. F., Good, M., Duncan, D. H., Hauser, C. E., Wintle, B. C., & Rumpff, L. (2026) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70311 · Cited by 2
Hahn, L., Glöckner, A., Gollwitzer, M., Hellmann, J. H., Lange, J., Schindler, S., & Sassenberg, K. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357568 · Cited by 2
Metadata
- Authors
- Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- Utrecht University (NL)
- Center for Open Science (US)
- Corresponding author
- Bakker, Marjan
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-12-09
- Volume
- 18
- Issue
- 12
- Pages
- e3000937
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC7725296
- OpenAlex
- W2906405410
- Cited by
- 138
- References
- 65
- Citations by year
- 2026: 8 · 2025: 25 · 2024: 23 · 2023: 30 · 2022: 19 · 2021: 18 · 2020: 7
- FWCI
- 18.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- European Research Council (726361)
- Laura and John Arnold Foundation
- Templeton World Charity Foundation
- John Templeton Foundation
- Templeton Religion Trust
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- preregistration, researcher degrees of freedom, hypothesis specification, false positives, Open Science Framework, inter-rater reliability, research quality
- MeSH terms
- Data Collection, Humans, Quality Control, Registries, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.