The DOI matches its Crossref record.
An excess of positive results: Comparing the standard Psychology literature with Registered Reports
Open accessAbstract
Selectively publishing results that support the tested hypotheses ('positive' results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports, peer review and the decision to publish take place before results are known. We compared the results in published Registered Reports (N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature (N = 152) in Psychology. Analysing the first hypothesis of each paper, we found 96% positive results in standard reports, but only 44% positive results in Registered Reports. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type-1 error inflation in the Registered-Reports literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Preregistration, and especially registered reports, helps to increase the reporting of null findings”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Scheel, Schijen, & Lakens, 2020
In the citing paper
“Making acceptance decisions prior to knowing the outcomes should and does mitigate publication bias favoring positive results 26,27 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]
In the citing paper
“Initial evidence suggests that RRs are (1) effective at mitigating publication bias with a sharp increase in publishing negative results compared to the standard model 26,27 ,”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]
In the citing paper
“This hypothesis was supported in two prior investigations 26,27”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]
In the citing paper
“Incentive focused innovations such as Registered Reports can shift reward away from achieving exciting results and toward asking important questions and designing rigorous studies to investigate them”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Scheel et al., 2020
Citations
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Kohl, S. H., Mehler, D. M. A., Lührs, M., Thibault, R. T., Konrad, K., & Sorger, B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00594 · Cited by 171
Robust Bayesian meta-analysis: Addressing publication bias with model-averaging.
Maier, M., Bartos̆, F., & Wagenmakers, E. (2022) · Psychological Methods · doi:10.1037/met0000405 · Cited by 153
Metadata
- Authors
- Scheel, A. M., Schijen, M., & Lakens, D.
- Publisher
- Center for Open Science
- Published
- 2020-02-05
- OpenAlex
- W4252093822
- Cited by
- 136
- References
- 0
- Citations by year
- 2025: 4 · 2024: 13 · 2023: 9 · 2022: 55 · 2021: 31 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- TU/e Research Portal (submittedVersion, other-oa)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, selective reporting, positive results, hypothesis testing, pre-registered study, type I error inflation, psychology literature, peer review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
