Evaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Open accessAbstract
Registered Reports (RRs) is a publishing model in which initial peer review is conducted prior to knowing the outcomes of the research. In-principle acceptance of papers at this review stage combats publication bias, and provides a clear distinction between confirmatory and exploratory research. Some editors raise a practical concern about adopting RRs. By reducing publication bias, RRs may produce more negative or mixed results and, if such results are not valued by the research community, receive less citations as a consequence. If so, by adopting RRs, a journal’s impact factor may decline. Despite known flaws with impact factor, it is still used as a heuristic for judging journal prestige and quality. Whatever the merits of considering impact factor as a decision-rule for adopting RRs, it is worthwhile to know whether RRs are cited less than other articles. We will conduct a naturalistic comparison of citation and altmetric impact between published RRs and comparable empirical articles from the same journals.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Initial evidence suggests that Registered Reports are cited as often or even more than other articles in the same journals.
“and (2) cited as often or even more than other articles in the same journals 28 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [28]
There is initial evidence that the ultimate acceptance rate for RRs is higher than traditional papers.
“There is initial evidence that the ultimate acceptance rate for RRs is higher than traditional papers 28”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [28]
Initial evidence against the just-world hypothesis is that an initial investigation of altmetric and citation impact suggests no difference or slightly better performance for RRs compared to similar articles published at the same time in the same journals.
“Initial evidence against the just-world hypothesis is that an initial investigation of altmetric and citation impact suggests no difference or slightly better performance for RRs compared to similar articles published at the same time in the same journals 28”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [28]
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Self-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Self-citationInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Self-citationAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · see citing works
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · see citing works
Citation Advantage of Open Access Articles
Eysenbach, G. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040157 · Cited by 829 · see citing works
How to Detect and Handle Outliers
Crosby, T. A., Iglewicz, B., & Hoaglin, D. C. (1994) · Technometrics · doi:10.2307/1269377 · Cited by 764 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
Self-citationRegistered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 574 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468 · see citing works
Self-citationDetecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · see citing works
Nosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242 · see citing works
Self-citationA simple proposal for the publication of journal citation distributionsPreprint
Larivière, V., Kiermer, V., MacCallum, C. J., Mcnutt, M. K., Patterson, M., Pulverer, B., Swaminathan, S., Taylor, S., & Curry, S. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/062109 · Cited by 216 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Preregistration Becoming the Norm in Psychological Science
Lindsay, B. A. N. A. D. S. (2018) · APS observer · Cited by 124 · see citing works
Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1996) · Scientometrics · doi:10.1007/bf02093487 · Cited by 71 · see citing works
Pulverer, B. (2015) · The EMBO Journal · doi:10.15252/embj.201570010 · Cited by 22 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499 · see citing works
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 323 · see citing works
Self-citationSoderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 289 · see citing works
Self-citationKohl, S. H., Mehler, D. M. A., Lührs, M., Thibault, R. T., Konrad, K., & Sorger, B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00594 · Cited by 160 · see citing works
What’s next for Registered Reports?
Chambers, C. (2019) · Nature · doi:10.1038/d41586-019-02674-6 · Cited by 154 · see citing works
Bradley, S. H., DeVito, N., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C., & Gill, P. J. (2020) · Journal of the Royal Society of Medicine · doi:10.1177/0141076820956799 · Cited by 90 · see citing works
Büttner, F., Toomey, E. C., McClean, S., Roe, M., & Delahunt, E. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101863 · Cited by 51 · see citing works
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 36 · see citing works
Briker, R., & Gerpott, F. H. (2023) · Organizational Research Methods · doi:10.1177/10944281231210309 · Cited by 27 · see citing works
Opening the Door to Registered Reports: Census of Journals Publishing Registered Reports (2013–2020)
Montoya, A. K., Krenzer, W. L. D., & Fossum, J. L. (2021) · Collabra Psychology · doi:10.1525/collabra.24404 · Cited by 22 · see citing works
A guide for social science journal editors on easing into open science
Silverstein, P., Elman, C., Montoya, A. K., McGillivray, B. C., Pennington, C. R., Harrison, C. H., Steltenpohl, C. N., Röer, J. P., Corker, K. S., Charron, L., et al. (2024) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00141-5 · Cited by 24 · see citing works
Protocol transparency is vital for registered reports
Chambers, C. D., & Mellor, D. T. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0449-6 · Cited by 20 · see citing works
Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102327 · Cited by 18 · see citing works
Open Science: A Review of Its Effectiveness and Implications for Service Research
Vaerenbergh, Y. V., Hazée, S., & Zwienenberg, T. J. (2025) · Journal of Service Research · doi:10.1177/10946705251338461 · Cited by 16 · see citing works
What senior academics can do to support reproducible and open research: a short, three-step guide
Kowalczyk, O. S., Lautarescu, A., Blok, E., Dall’Aglio, L., & Westwood, S. J. (2022) · BMC Research Notes · doi:10.1186/s13104-022-05999-0 · Cited by 16 · see citing works
The Chinese Open Science Network (COSN): Building an Open Science Community From Scratch
Jin, H., Wang, Q., Yang, Y., Zhang, H., Gao, M. M., Jin, S., Chen, Y., Xu, T., Zheng, Y.-R., Ji, C., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221144986 · Cited by 16 · see citing works
A qualitative analysis of stakeholder experiences with Registered Reports Funding Partnerships
Drax, K., Clark, R. W. A., Chambers, C., Munafo, M. R., & Thompson, J. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.17029.1 · Cited by 13 · see citing works
Pennington, C. R., & Heim, D. (2021) · Addiction Research & Theory · doi:10.1080/16066359.2021.1931142 · Cited by 9 · see citing works
Shaw, S. D., & Nave, G. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114129 · Cited by 8 · see citing works
Baldwin, J. R. (2023) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13898 · Cited by 7 · see citing works
Anthony, N., Tisseaux, A., & Naudet, F. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.05.016 · Cited by 6 · see citing works
Accelerating addiction research via Open Science and Team Science
Dora, J., Piccirillo, M. L., Foster, K. T., & King, K. M. (2023) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/bs.plm.2023.06.004 · Cited by 3 · see citing works
NSF Science of Science Proposal: Randomized Trial of Registered ReportsPreprint
Nosek, B. A., Mellor, D. T., Errington, T. M., Soderberg, C. K., Krall, T., Grasty, F. L., Brooks, R. E., Davis, R., Kousta, S. T., Swaminathan, S., et al. (2020) · doi:10.31222/osf.io/jvxab · Cited by 3 · see citing works
Self-citation(FULL GUIDE) A Guide for Social Science Journal Editors on Easing into Open SciencePreprint
Silverstein, P., Elman, C., Montoya, A. K., McGillivray, B. C., Pennington, C. R., Harrison, C. H., Steltenpohl, C. N., Röer, J. P., Corker, K. S., Charron, L., et al. (2023) · doi:10.31219/osf.io/hstcx · Cited by 3 · see citing works
Liu, M., Sang, Y., Hiver, P. V., & Al‐Hoorie, A. H. (2025) · Journal of Second Language Studies · doi:10.1075/jsls.00045.liu · Cited by 2 · see citing works
Bradley, S. H., DeVito, N., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C., & Gill, P. J. (2020) · doi:10.31222/osf.io/jy37f · Cited by 2 · see citing works
Psychology and Law, Meet Open Science
McAuliff, B. D., Fessinger, M. B., Perillo, A. D., & Perillo, J. T. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197649138.013.5 · Cited by 1 · see citing works
Meier, M., Lonsdorf, T. B., LUPIEN, S. J., Tobias Stalder, A., Laufer, S., Sicorello, M., Linz, R., & Puhlmann, L. M. C. (2022) · doi:10.31234/osf.io/d2zvc
A primer on open science for cognitive psychologists
Pennington, C. R. (2025) · The Cognitive Psychology Bulletin · doi:10.53841/bpscog.2025.1.10.36
- Authors
- Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M.
- Affiliations
- Center for Open Science (US)
- Publisher
- Center for Open Science
- Published
- 2017-12-04
- OpenAlex
- W2777107367
- Cited by
- 29
- References
- 32
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 7 · 2022: 3 · 2021: 7 · 2020: 5 · 2019: 1 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion, public-domain)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- impact factor, citation impact, altmetrics, publication bias, pre-registered study, journal prestige, negative results, exploratory research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
