Comparing Dream to Reality: An Assessment of Adherence of the First Generation of Preregistered Studies
Open accessPreprint — published as 10.1098/rsos.211037 · Paper page
Abstract
Doing research inevitably involves making numerous decisions that can influence research outcomes in such a way that it leads to overconfidence in statistical conclusions. One proposed method to increase the interpretability of a research finding is preregistration, which involves documenting analytic choices on a public, third-party repository prior to any influence by data. To investigate whether, in psychology, preregistration lives up to that potential, we focused on all articles published in Psychological Science with a preregistered badge between February 2015 and November 2017, and assessed the adherence to their corresponding preregistration plans. We observed deviations from the plan in all studies, and, more importantly, in all but one study, at least one of these deviations was not fully disclosed. We discuss examples and possible explanations, and highlight good practices for preregistering research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“Preregistration is increasingly common in medical and psychological science, and even though it is not perfect (Claesen et al., 2019) at least it has made research more transparent and controllable, which has a positive impact on the possibilities to replicate earlier findings.”
BackgroundEnsuring the quality and specificity of preregistrations
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000937 · Cited by 138 · see citing works
TARG Meta-Research Group & Collaborators (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076264 · Cited by 33 · see citing works
- Authors
- Claesen, A., Gomes, S. L. B. T., tuerlinckx, F., & vanpaemel, W.
- Affiliations
- KU Leuven (BE)
- Corresponding author
- Claesen, Aline
- Venue
- Royal Society Open Science
- Publisher
- Center for Open Science
- Published
- 2019-05-09
- Volume
- 8
- Issue
- 10
- Pages
- 211037-211037
- ISSN
- 2054-5703 (ISSN-L)
- PMC
- PMC8548785
- OpenAlex
- W3211113958
- Cited by
- 200
- References
- 0
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 32 · 2023: 33 · 2022: 58 · 2021: 23 · 2020: 19 · 2019: 6
- FWCI
- 21.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Royal Society Open Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- KU Leuven (C14/19/054)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- preregistration, research transparency, reporting guidelines, sample size reporting, exclusion criteria, statistical analysis plan
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
