CoChat
Sign Up
PreprintCited by 136Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic Publishing and Open Access

An excess of positive results: Comparing the standard Psychology literature with Registered Reports

Open access
Scheel, Anne M., Schijen, Mitchell, Lakens, Daniel·2020
DOI 10.31234/osf.io/p6e9c
Open this paper in CoChat — free account

Abstract

Selectively publishing results that support the tested hypotheses ('positive' results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports, peer review and the decision to publish take place before results are known. We compared the results in published Registered Reports (N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature (N = 152) in Psychology. Analysing the first hypothesis of each paper, we found 96% positive results in standard reports, but only 44% positive results in Registered Reports. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type-1 error inflation in the Registered-Reports literature.

Integrity signals

Registry

Crossref · 10.31234/osf.io/p6e9c

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Preregistration, and especially registered reports, helps to increase the reporting of null findings.

    “Preregistration, and especially registered reports, helps to increase the reporting of null findings”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Scheel, Schijen, & Lakens, 2020

  • Making acceptance decisions prior to knowing the outcomes should and does mitigate publication bias favoring positive results.

    “Making acceptance decisions prior to knowing the outcomes should and does mitigate publication bias favoring positive results 26,27 .”

    From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]

  • Initial evidence suggests that Registered Reports are effective at mitigating publication bias with a sharp increase in publishing negative results compared to the standard model.

    “Initial evidence suggests that RRs are (1) effective at mitigating publication bias with a sharp increase in publishing negative results compared to the standard model 26,27 ,”

    From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]

  • The hypothesis that RRs receive greater scrutiny and ultimate rejection for negative than positive results was supported in two prior investigations.

    “This hypothesis was supported in two prior investigations 26,27”

    From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [27]

  • Incentive focused innovations such as Registered Reports can shift reward away from achieving exciting results and toward asking important questions and designing rigorous studies.

    “Incentive focused innovations such as Registered Reports can shift reward away from achieving exciting results and toward asking important questions and designing rigorous studies to investigate them”

    From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Scheel et al., 2020

Cited by

Sorted by citations

Related

Authors
Scheel, A. M., Schijen, M., & Lakens, D.
ORCID
Publisher
Center for Open Science
Published
2020-02-05
DOI
10.31234/osf.io/p6e9c
OpenAlex
W4252093822
Cited by
136
References
0
Citations by year
2025: 4 · 2024: 13 · 2023: 9 · 2022: 55 · 2021: 31 · 2020: 23 · 2019: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • TU/e Research Portal (submittedVersion, other-oa)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
publication bias, selective reporting, positive results, hypothesis testing, pre-registered study, type I error inflation, psychology literature, peer review

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review