Initial evidence of research quality of registered reports compared with the standard publishing model
Open accessPublished version of a preprint
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 18 of the 54 sources its claims cite were checked for support.
Abstract
In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs such as reducing novelty. Here, 353 researchers peer reviewed a pair of papers from 29 published RRs from psychology and neuroscience and 57 non-RR comparison papers. RRs numerically outperformed comparison papers on all 19 criteria (mean difference 0.46, scale range -4 to +4) with effects ranging from RRs being statistically indistinguishable from comparison papers in novelty (0.13, 95% credible interval [-0.24, 0.49]) and creativity (0.22, [-0.14, 0.58]) to sizeable improvements in rigour of methodology (0.99, [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]) and overall paper quality (0.66, [0.30, 1.02]). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature.
Overview
CoChat Summary
TL;DR
In a naturalistic comparison, 353 researchers peer reviewed paired Registered Reports (RRs) and standard-model articles, rating RRs higher on all 19 evaluated criteria. RRs showed the largest advantages in methodological and analysis rigor, with little evidence of costs to novelty or creativity.
Key findings
- Across 29 RRs and 57 matched comparison articles, RRs outperformed comparison papers on all 19 criteria, with an average difference of 0.46 on a −4 to +4 scale and a range of 0.13 to 0.96; 12 of 19 criteria had 95% credible intervals excluding zero.
- Largest advantages were for rigor of methodology (0.99, 95% CrI [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]), and overall paper quality (0.66, [0.30, 1.02]); smallest differences were novelty (0.13, [−0.24, 0.49]) and creativity (0.22, [−0.14, 0.58]).
- Objective coding showed more preregistration (59% of RRs vs 7% author-based and 0% journal-based comparisons), open data (86% vs 18% and 14%), open materials (59% vs 11% and 7%), and larger median sample sizes (220 vs 118 and 78).
- Between-subjects analysis using only the first reviewed article showed a similar pattern, with an average effect size of 0.43 and range 0.04 to 0.93.
Methods
- Observational quasi-experimental study using 29 published RRs in psychology and neuroscience reporting novel research (replication RRs excluded) and 57 comparison articles matched by journal/date or lead author/date, totaling 86 articles.
- 353 recruited researchers, matched to articles by self-reported expertise, each peer reviewed one RR and one comparison article in randomized order; they rated 19 criteria on bipolar 9-point scales from −4 to +4.
- Preregistered Bayesian multilevel models estimated within-subjects and between-subjects effects; article coding assessed sample size, sample justification, difficult-to-reach populations, study design, open practices, and statcheck-detected reporting errors.
Limitations
- The paper states that the naturalistic design lacked random assignment of research to RR versus standard publication, and authors/editors self-select into RRs; matching on topic, journal, and lead author was intended to reduce but not eliminate confounding.
- The paper also states that RRs are a new format, so early-adoption behavior may not generalize; quality was assessed via peer-review ratings rather than replicability or generalizability, and reviewer representativeness is unknown.
- Evident from the text: reviewers were not fully blinded to article type, with 54% correctly identifying the RR and 54% correctly identifying the comparison paper as not an RR; exploratory analyses showed larger perceived RR advantages among reviewers more familiar with or more favorable toward RRs, though every subgroup estimate remained directionally favorable.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
8 Valid · 1 Not valid · 12 Not checked · 165 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Positive results are much more likely to be published than negative results 5-7”
What the sources say
- Supportedcites [5] Smith, R. Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99, 178–182 (2006). source
In the Bias section (page not provided), Smith says editorial peer review has been strongly biased against negative studies and that authors often do not write them up. This supports the claim that positive results are much more likely to be published. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [6] Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012). source
- Supportedcites [7] Fanelli, D. ‘Positive’ results increase down the hierarchy of the sciences. PLoS ONE 5, e10068 (2010). source
The Discussion section, “2C—Non-publication of negative and/or statistically non-significant results” (page unavailable), describes a publication bias against negative results. It says such results can remain unpublished because researchers may not submit them or reviewers and editors may reject them. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
What the sources say
- Not checked — no full text of the sourcecites [6] Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012). source
- Supportedcites [7] Fanelli, D. ‘Positive’ results increase down the hierarchy of the sciences. PLoS ONE 5, e10068 (2010). source
The Discussion section on non-publication of negative results says such results may remain unpublished because researchers do not submit them or editors and reviewers reject them. It also explicitly describes a publication bias against negative results. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [8] Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505 (2014). source
- Not checked — no full text of the sourcecites [9] Dickersin, K. The existence of publication bias and risk factors for its occurrence. JAMA 263, 1385–1389 (1990). source
- Not checked — no full text of the sourcecites [10] Mahoney, M. J. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cogn. Ther. Res. 1, 161–175 (1977). source
- Not checked — no full text of the sourcecites [11] Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975). source
- Not checked — no full text of the sourcecites [12] Sterling, T. D. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J. Am. Stat. Assoc. 54, 30–34 (1959). source
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
What the sources say
- Supportedcites [18] Begley, C. G. & Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012). source
In BUILDING A STRONGER SYSTEM, the authors say that pressure for a clear, complete ‘perfect’ story can encourage selected data sets or massaged results. Improving the reliability of preclinical cancer studies calls for more publication of negative findings, while CONFIRMING RESEARCH FINDINGS links selective presentation of experiments to irreproducible results. Together, these passages support the citing sentence’s hedged account of how publication preferences may undermine the literature’s credibility. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [19] Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712–712 (2011). source
- Partly supportedcites [20] Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). sourceExperimental — accuracy not yet measured
The Main Text (page not provided) says original effect sizes tended to be inflated and that this could stem from publication bias; it also discusses concerns about false positives and reproducibility. It does not say that attention to positive, novel, or clean findings causes publication bias, selective reporting, or questionable research practices. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [21] Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018). source
- Partly supportedcites [22] Klein, R. A. et al. Many Labs 2: investigating variation in replicability across samples and settings. Adv. Methods Pract. Psychol. Sci. 1, 443–490 (2018). sourceExperimental — accuracy not yet measured
The Implications section says positive results are perceived as more publishable and describes analytic and post hoc practices that can inflate false positives and diminish credibility. The Discussion also identifies publication bias and p-hacking as possible explanations for replication failures, but says the study cannot determine whether they caused those failures. Our copy does not connect attention to novel or clean findings specifically to these practices. (LLM, conf=0.84)
- Partly supportedcites [23] Klein, R. A. et al. Investigating variation in replicability: a ‘many labs’ replication project. Soc. Psychol. 45, 142–152 (2014). sourceExperimental — accuracy not yet measured
The introduction (p. 4) says infrequent direct replication can leave spurious findings in the published literature, supporting the concern about credibility. It does not link that problem to attention to positive, novel, or clean findings, or discuss publication bias, selective reporting, or questionable research practices as the claimed pathway. (LLM, conf=0.84)
- Partly supportedcites [24] Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). sourceExperimental — accuracy not yet measured
Insights on Reproducibility (p. 13) links a preference for positive results to publication and reporting bias and says these biases may inflate published effect sizes. Implications and Limitations (p. 15) discusses incentives favoring innovation over verification. The paper does not establish the claimed link to questionable research practices or specifically discuss attention to “clean” findings. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [25] Ebersole, C. R. et al. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67, 68–82 (2016). source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reality, 99.43% of the reviewers provided data for both articles.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“better alignment between the research question and methodology (estimate = 0.37, [0.01, 0.73], standardized ES = 0.19, [0.01, 0.37])”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
46 more claims
Create a free account to see all 186 claims and every source verdict.
- Probably not a substantial claim. 116
- It may state two separate claims as one. 42
- The quote only partly states this claim. 10
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Of the 29 RR and control article matches, each had an average of 12.5 reviewers with a low of 3 and a high of 24 ( SD =5.59).”
Reported 12.5 · recomputed 12.172414
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Results
“Finally, a majority of reviewers correctly identified the RR as a RR (54%; 190 correct, 38 incorrect, 125 unsure) and the comparison paper as not a RR (54%; 192 correct, 30 incorrect, 131 unsure).”
§ Results
“In the within-subjects model, 19 of 19 outcome criteria remained directionally consistent with an RR performance advantage for reviewers that guessed one paper correctly ( n =168), and 14 of 19 outcome criteria remained directionally consistent for reviewers that guessed neither paper correctly ( n =87).”
Citations
Cites
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationThe Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationJusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Podobnik, B., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 643
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
Self-citation
Metadata
- Authors
- Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A.
- Affiliations
- Center for Open Science (US)
- University of California, Davis (US)
- The University of Melbourne (AU)
- University of California, Riverside (US)
- Corresponding author
- Nosek, Brian A.
- ORCID
- Venue
- Nature Human Behaviour
- Published
- 2021-06-24
- Volume
- 5
- Issue
- 8
- Pages
- 990-997
- ISSN
- 2397-3374 (ISSN-L)
- OpenAlex
- W3174993952
- Semantic Scholar
- b870928f45680afa8e394c85bfaddd487cd6c4d2
- Cited by
- 295
- References
- 41
- Citations by year
- 2026: 27 · 2025: 47 · 2024: 43 · 2023: 61 · 2022: 85 · 2021: 28
- FWCI
- 39.83
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- public-domain
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Human Behaviour (publishedVersion)
- MetArXiv (OSF Preprints) (submittedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (acceptedVersion)
- MetArXiv (OSF Preprints) (acceptedVersion)
- Funders
- James S. McDonnell Foundation (220020498)
- Arnold Ventures, no reference number
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- research quality, publication bias, research credibility, novelty, creativity, methodological quality, open science
- MeSH terms
- Data Analysis, Humans, Neurosciences, Psychology, Registries, Research, Research Design, Peer Review, Research, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Initial evidence of research quality of registered reports compared with the standard publishing model. https://app.cochat.ai/papers/10.1038/s41562-021-01142-4
- The paper
- Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01142-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.