CoChat
Sign Up
Cited by 455Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods and Bayesian Inference

The ironic effect of significant results on the credibility of multiple-study articles.

Schimmack, Ulrich·2012·Psychological Methods
DOI 10.1037/a0029487
Open this paper in CoChat — free account

Abstract

Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power decreases as a function of the number of statistical tests that are being conducted (Maxwell, 2004). The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959). The problem of low power in multiple-study articles is illustrated using Bem's (2011) article on extrasensory perception and Gailliot et al.'s (2007) article on glucose and self-regulation. I conclude with several recommendations that can increase the credibility of scientific evidence in psychological journals. One major recommendation is to pay more attention to the power of studies to produce positive results without the help of questionable research practices and to request that authors justify sample sizes with a priori predictions of effect sizes. It is also important to publish replication studies with nonsignificant results if these studies have high power to replicate a published finding.

Integrity signals

Registry

Crossref · 10.1037/a0029487

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Clean papers in which the evidence consistently supports the proposed conclusion are more likely to be published than papers with equivocal evidence, exceptions, or other uncertainties.

    “And, clean papers in which the evidence consistently supports the proposed conclusion are more likely to be published than papers with equivocal evidence, exceptions, or other uncertainties 16,17 .”

    From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [16]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Schimmack, U.
Affiliations
University of Toronto (CA)
Corresponding author
Schimmack, Ulrich
ORCID
Schimmack, Ulrich
Venue
Psychological Methods
Publisher
American Psychological Association (APA)
Published
2012
Volume
17
Issue
4
Pages
551-566
ISSN
  • 1082-989X (ISSN-L)
  • 1939-1463
DOI
10.1037/a0029487
OpenAlex
W2164150544
Cited by
455
References
76
Citations by year
2026: 8 · 2025: 10 · 2024: 14 · 2023: 15 · 2022: 24 · 2021: 55 · 2020: 27 · 2019: 56 · 2018: 36 · 2017: 47 · 2016: 42 · 2015: 48 · 2014: 38 · 2013: 29 · 2012: 5
FWCI
43.66
Type
journal article
Language
en
Open access
closed
Locations
  • Psychological Methods (publishedVersion)
  • PubMed (publishedVersion)
Funders
Social Sciences and Humanities Research Council of Canada
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords
statistical power, research credibility, questionable research practices, non-significant results, sample size justification, replication study, extrasensory perception
MeSH terms
Humans, Psychology, Research Design, Statistics as Topic, Publication Bias, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review