Ensuring the quality and specificity of preregistrations
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called “OSF Preregistration,” http://osf.io/prereg/). The Prereg Challenge format was a “structured” workflow with detailed instructions and an independent review to confirm completeness; the “Standard” format was “unstructured” with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the “structured” format restricted the opportunistic use of researcher degrees of freedom better (Cliff’s Delta = 0.49) than the “unstructured” format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“One study found that 2 researchers could only agree on the number of hypotheses in 14% of the preregistrations they assessed [45].”
From Open Science 2.0: Towards a truly collaborative research ecosystem (2023) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
(2011) · Cited by 14,317
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 5×
“Clearly, distinguishing these 2 modes of research is vitally important for maintaining the validity of statistical inferences in confirmatory analysis and for avoiding mistaking the generation of a hypothesis in exploratory analysis as the testing of a hypothesis in confirmatory analysis [1,2].”
BackgroundInfluentialSelf-citationFanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Dominance statistics: Ordinal analyses to answer ordinal questions.
Cliff, N. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.494 · Cited by 1,485
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 3×
“Clearly, distinguishing these 2 modes of research is vitally important for maintaining the validity of statistical inferences in confirmatory analysis and for avoiding mistaking the generation of a hypothesis in exploratory analysis as the testing of a hypothesis in confirmatory analysis [1,2].”
BackgroundThe Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Self-citationRegistered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 3×
“A more substantive review could have added benefits like identifying lack of clarity on degrees of freedom that authors miss, such as through the Registered Reports publishing format ([26,56, 64]; http://cos.”
BackgroundMethodInfluentialSelf-citationThe Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Registered Reports: A new publishing initiative at Cortex
Chambers, C. D. (2013) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 570
Characteristics of Clinical Trials Registered in ClinicalTrials.gov, 2007-2010
Califf, R. M. (2012) · JAMA · doi:10.1001/jama.2012.3424 · Cited by 516
Pre-registration in social psychology—A discussion and suggested template
van 't Veer, A. E., & Giner-Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 439
Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 319
Self-citationPreregistering qualitative research
L. Haven, T., & Van Grootel, D. L. (2019) · Accountability in Research · doi:10.1080/08989621.2019.1580147 · Cited by 311
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
Update on Trial Registration 11 Years after the ICMJE Policy Was Established
Zarin, D. A., Tse, T., Williams, R. J., & Rajakannan, T. (2017) · New England Journal of Medicine · doi:10.1056/nejmsr1601330 · Cited by 239
COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019) · Trials · doi:10.1186/s13063-019-3173-2 · Cited by 233
Bakker, M., & Wicherts, J. M. (2014) · Psychological Methods · doi:10.1037/met0000014 · Cited by 209
Self-citationChan, A.-W., Hrobjartsson, A., Jorgensen, K. J., Gotzsche, P. C., & Altman, D. G. (2008) · BMJ · doi:10.1136/bmj.a2299 · Cited by 202
Claesen, A., Gomes, S. L. B. T., tuerlinckx, F., & vanpaemel, W. (2019) · doi:10.31234/osf.io/d8wex · Cited by 200
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
de Groot, A. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 188
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172792 · Cited by 162
Self-citationWhat’s next for Registered Reports?
Chambers, C. (2019) · Nature · doi:10.1038/d41586-019-02674-6 · Cited by 154
Open Science Practices are on the Rise: The State of Social Science (3S) SurveyPreprint
Christensen, G., Wang, Z., Paluck, E. L., Swanson, N., Birke, D. J., Miguel, E. A., & Littman, R. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5rksu · Cited by 152
Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results
Chan, A.-W. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050230 · Cited by 144
Underreporting in Psychology Experiments: Evidence From a Study Registry
Franco, A., Malhotra, N., & Simonovits, G. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 117
Gelman, A., & Loken, E. (2014) · American Scientist · Cited by 111
Motyl, M., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B., Prims, J. P., Sun, J., Washburn, A. N., Wong, K. M., et al. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000084 · Cited by 110
Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., Doshi, P., Ehmsen, J., Gresham, G., Guo, N., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.014 · Cited by 108
PsychDisclosure.org: Grassroots Support for Reforming Reporting Standards in Psychology
LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613491437 · Cited by 103
Preregistration Becoming the Norm in Psychological Science
Lindsay, B. A. N. A. D. S. (2018) · APS observer · Cited by 100
Make journals report clinical trials properly
Goldacre, B. M. (2016) · Nature · doi:10.1038/530007a · Cited by 94
Outcome reporting bias in clinical trials: why monitoring matters
Ioannidis, J. P., Caplan, A. L., & Dal-Ré, R. (2017) · BMJ · doi:10.1136/bmj.j408 · Cited by 74
Study preregistration: An evaluation of a method for transparent reporting
Toth, A. A., Banks, G. C., Mellor, D., O’Boyle, E. H., Dickson, A., Davis, D. J., DeHaven, A., Bochantin, J., & Borns, J. (2021) · Journal of Business and Psychology · doi:10.1007/s10869-020-09695-3 · Cited by 73 · cited 2×
“Recently, participants who had completed a preregistration were asked about their experiences with preregistration in a survey [66].”
BackgroundSelf-citationJones, P. M., Chow, J. T. Y., Arango, M. F., Fridfinnson, J. A., Gai, N., Lam, K., & Turkstra, T. P. (2017) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002272 · Cited by 64
Seven Selfish Reasons for Preregistration
Wagenmakers, E. (2016) · APS observer · Cited by 62
Evidence of selective reporting bias in hematology journals: A systematic review
Wayant, C., Scheckel, C., Hicks, C., Nissen, T., Leduc, L., Som, M., & Vassar, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0178379 · Cited by 53
Opportunistic biases: Their origins, effects, and an integrated solution.
DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015) · American Psychologist · doi:10.1037/a0039191 · Cited by 51
Outcome reporting bias in randomized-controlled trials investigating antipsychotic drugs
Lancee, M., Lemmens, C. M. C., Kahn, R. S., Vinkers, C. H., & Luykx, J. J. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.203 · Cited by 51
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 45
Self-citationToo Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
Self-citationTen reasons why journals must review manuscripts before results are known
Chambers, C. (2014) · Addiction · doi:10.1111/add.12728 · Cited by 37
Selective outcome reporting in obesity clinical trials: a cross‐sectional review
Rankin, J., Ross, A., Baker, J., O'Brien, M., Scheckel, C., & Vassar, M. (2017) · Clinical Obesity · doi:10.1111/cob.12199 · Cited by 32
van der Steen, J. T., van den Bogert, C. A., van Soest‐Poortvliet, M. C., FARSANI, S. F. F., Otten, R. H. J., ter Riet, G., & Bouter, L. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0188247 · Cited by 32
Crüwell, S., & Evans, N. J. (2021) · Royal Society Open Science · doi:10.1098/rsos.210155 · Cited by 30
Improving the Conduct and Reporting of Statistical Analysis in Psychology
Sijtsma, K., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Psychometrika · doi:10.1007/s11336-015-9444-2 · Cited by 24
Self-citationCurtailing the Use of Preregistration : A Misused Term
Rice, D. B., & Moher, D. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619858427 · Cited by 22
fMRI Preregistration TemplatePreprint
Flannery, J. (2018) · OSF Preprints (OSF Preprints) · Cited by 5
Research works printed but not linked: 2
- N Cliff, 1996, Ordinal Methods for Behavior Data Analysis
- A Claesen, 2019, Preregistration: Comparing Dream to Reality
Cited by
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 319
Self-citationWhere to prospectively register a systematic review
Pieper, D., & Rombey, T. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01877-1 · Cited by 192
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
Metadata
- Authors
- Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- Utrecht University (NL)
- Center for Open Science (US)
- Corresponding author
- Bakker, Marjan
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-12-09
- Volume
- 18
- Issue
- 12
- Pages
- e3000937
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC7725296
- OpenAlex
- W2906405410
- Cited by
- 138
- References
- 65
- Citations by year
- 2026: 8 · 2025: 25 · 2024: 23 · 2023: 30 · 2022: 19 · 2021: 18 · 2020: 7
- FWCI
- 18.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- European Research Council (726361)
- Laura and John Arnold Foundation
- Templeton World Charity Foundation
- John Templeton Foundation
- Templeton Religion Trust
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- preregistration, researcher degrees of freedom, hypothesis specification, false positives, Open Science Framework, inter-rater reliability, research quality
- MeSH terms
- Data Collection, Humans, Quality Control, Registries, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.