The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
The ironic effect of significant results on the credibility of multiple-study articles.
Abstract
Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power decreases as a function of the number of statistical tests that are being conducted (Maxwell, 2004). The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959). The problem of low power in multiple-study articles is illustrated using Bem's (2011) article on extrasensory perception and Gailliot et al.'s (2007) article on glucose and self-regulation. I conclude with several recommendations that can increase the credibility of scientific evidence in psychological journals. One major recommendation is to pay more attention to the power of studies to produce positive results without the help of questionable research practices and to request that authors justify sample sizes with a priori predictions of effect sizes. It is also important to publish replication studies with nonsignificant results if these studies have high power to replicate a published finding.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“And, clean papers in which the evidence consistently supports the proposed conclusion are more likely to be published than papers with equivocal evidence, exceptions, or other uncertainties 16,17 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [16]
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
GPOWER: A general power analysis program
Erdfelder, E., Faul, F., & Buchner, A. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203630 · Cited by 4,864
Required sample size to detect the mediated effect
Fritz, M. S., & MacKinnon, D. P. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01882.x · Cited by 4,810
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 5×
“Achieve 80% Power to Produce Significant Results in All”
BackgroundResultInfluential
Cited by
Sorted by citations
An opportunity cost model of subjective effort and task performance
Kurzban, R., Duckworth, A., Kable, J. W., & Myers, J. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12003196 · Cited by 1,381
Why self-control seems (but may not be) limited
Inzlicht, M., Schmeichel, B. J., & Macrae, C. N. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.12.009 · Cited by 1,026
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Metadata
- Authors
- Schimmack, U.
- Affiliations
- University of Toronto (CA)
- Corresponding author
- Schimmack, Ulrich
- ORCID
- Schimmack, Ulrich
- Venue
- Psychological Methods
- Published
- 2012
- Volume
- 17
- Issue
- 4
- Pages
- 551-566
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2164150544
- Cited by
- 455
- References
- 76
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 14 · 2023: 15 · 2022: 24 · 2021: 55 · 2020: 27 · 2019: 56 · 2018: 36 · 2017: 47 · 2016: 42 · 2015: 48 · 2014: 38 · 2013: 29 · 2012: 5
- FWCI
- 43.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Social Sciences and Humanities Research Council of Canada
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- statistical power, research credibility, questionable research practices, non-significant results, sample size justification, replication study, extrasensory perception
- MeSH terms
- Humans, Psychology, Research Design, Statistics as Topic, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
