The effect of preregistration on trust in empirical research findings: results of a registered report
Open accessAbstract
The crisis of confidence has undermined the trust that researchers place in the findings of their peers. In order to increase trust in research, initiatives such as preregistration have been suggested, which aim to prevent various questionable research practices. As it stands, however, no empirical evidence exists that preregistration does increase perceptions of trust. The picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. This registered report presents an empirical assessment of the extent to which preregistration increases the trust of 209 active academics in the reported outcomes, and how familiarity with another researcher influences that trust. Contrary to our expectations, we report ambiguous Bayes factors and conclude that we do not have strong evidence towards answering our research questions. Our findings are presented along with evidence that our manipulations were ineffective for many participants, leading to the exclusion of 68% of complete datasets, and an underpowered design as a consequence. We discuss other limitations and confounds which may explain why the findings of the study deviate from a previously conducted pilot study. We reflect on the benefits of using the registered report submission format in light of our results. The OSF page for this registered report and its pilot can be found here: http://dx.doi.org/10.17605/OSF.IO/B3K75 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Preregistration can help researchers to carefully plan their hypotheses and hypothesis tests.
“Preregistration can help researchers to carefully plan their hypotheses and hypothesis tests (Field et al., 2020).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Field et al., 2020
There is currently no clear evidence that preregistration increases the perceived credibility of research.
“However, I would note that there is currently no clear evidence that preregistration increases the perceived credibility of research (Field et al., 2020).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Field et al., 2020
Preregistration has also been proposed as a method of identifying when researchers fool themselves into believing their own conclusions due to confirmation and hindsight biases.
“Preregistration has also been proposed as a method of identifying when researchers fool themselves into believing their own conclusions (Feynman, 1974) due to confirmation and hindsight biases”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Field et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Self-citationRegistered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
How scientists fool themselves – and how they can stop
Nuzzo, R. (2015) · Nature · doi:10.1038/526182a · Cited by 296 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Flagged first, then by citations
Research works printed but not linked: 18
- [3]Simmons JP, Nelson LD, Simonsohn U 2011 False–positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359– 1366. (10.1177/0956797611417632) 22006061 · DOI · Paper page
- [5]Spellman BA, Gilbert EA, Corker KS 2018 Open science. In Stevens' handbook of experimental psychology and cognitive neuroscience (4th edn), Volume 5: Methodology (eds J Wixted, EJ Wagenmakers), pp. 729–776. New York, NY: Wiley.
- [7]Munafò MR et al. 2017 A manifesto for reproducible science. Nat. Hum. Behav. 1, 1– 9. (10.1038/s41562-016-0001) PMC7610724 33954258 · DOI
- [8]Pain E 2015 Register your study as a new publication option. Science 350, 15 See https://www.sciencemag.org/careers/2015/12/register-your-study-new-publication-option.
- [12]van't Veer AE, Giner-Sorolla R 2016 Pre-registration in social psychology—a discussion and suggested template. J. Exp. Social Psychol. 67, 2– 12. (10.1016/j.jesp.2016.03.004) · DOI
- [16]John LK, Loewenstein G, Prelec D 2012 Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 24, 524– 532. (10.1177/0956797611430953) 22508865 · DOI · Paper page
- [17]Rosenthal R 1979 The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638– 641. (10.1037/0033-2909.86.3.638) · DOI · Paper page
- [20]Chambers CD 2013 Registered reports: a new publishing initiative at Cortex. Cortex 49, 609– 610. (10.1016/j.cortex.2012.12.016) 23347556 · DOI
- [21]Leonelli S 2014 Data interpretation in the digital age. Perspect. Sci. 22, 397– 417. (10.1162/POSC_a_00140) 25729262 PMC4340525 · DOI
- [22]Metzger MJ, Flanagin AJ 2013 Credibility and trust of information in online environments: the use of cognitive heuristics. J. Pragmatics 59, 210– 220. (10.1016/j.pragma.2013.07.012) · DOI · Paper page
- [23]Gigerenzer G, Goldstein DG 1999 Betting on one good reason: the Take The Best heuristic. In Simple heuristics that make us smart (eds G Gigerenzer, PM Todd, the ABC Research Group), pp. 75–96. New York, NY: Oxford University Press.
- [24]Rouder JN, Morey RD, Speckman PL, Province JM 2012 Default Bayes factors for ANOVA designs. J. Math. Psychol. 56, 356– 374. (10.1016/j.jmp.2012.08.001) · DOI · Paper page
- [27]Jeffreys H 1961 Theory of probability. Oxford, UK: Oxford University Press.
- [28]Houtkoop BL, Chambers C, Macleod M, Bishop DV, Nichols TE, Wagenmakers EJ 2018 Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70– 85. (10.1177/2515245917751886) · DOI
- [29]Bayarri MJ, Berger JO, Forte A, Garcıia-Donato G 2012 Criteria for Bayesian model choice with application to variable selection. Ann. Stat. 40, 1550– 1577. (10.1214/12-AOS1013) · DOI
- [34]@nichoisa. (14 June 2019). Watching a conference panel now in which a PI asserted that preregistered reports are not possible if a PhD student has to finish in 3 years (prereg generally ok). I'm sceptical of her assertion but maybe those with experience differ on this? [Twitter post]. See https://twitter.com/nichoisa/status/1139569547810353153.
- [35]Claesen A et al. 2019 Preregistration: comparing dream to reality See https://psyarxiv.com/d8wex/.
- [36]Button KS, Chambers CD, Lawrence N, Munafò MR 2019 Grassroots training for reproducible science: a consortium-based approach to the empirical dissertation. Psychol. Learn. Teach. 19, 77– 90. (10.1177/1475725719857659) · DOI
Cited by
Sorted by citations
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“…and exploratory data analyses, with the assumption being that the results of exploratory hypothesis tests are less credible than those of confirmatory hypothesis tests (e.g., Centre for Open Science, n. d; ; Field et al., 2020; Nosek & Lindsay, 2018; Nosek et al., 2018; Wagenmakers et al., 2012).”
BackgroundMethodInfluential
- Authors
- Field, S. M., Wagenmakers, E.-J., Kiers, H. A. L., Hoekstra, R., Ernst, A. F., & van Ravenzwaaij, D.
- Affiliations
- University of Groningen (NL)
- University of Amsterdam (NL)
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
- Corresponding author
- Field, Sarahanne M.
- Venue
- Royal Society Open Science
- Publisher
- The Royal Society
- Published
- 2020-04
- Volume
- 7
- Issue
- 4
- Pages
- 181351
- ISSN
- 2054-5703 (ISSN-L)
- PMC
- PMC7211853
- OpenAlex
- W3011113154
- Cited by
- 43
- References
- 7
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 6 · 2023: 8 · 2022: 10 · 2021: 7 · 2020: 4 · 2019: 1
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Royal Society Open Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- preregistration, trust crisis, questionable research practices, peer trust, Bayes factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
