Cited by 455Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods and Bayesian Inference

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.

The ironic effect of significant results on the credibility of multiple-study articles.

Schimmack, Ulrich·2012·Psychological Methods

DOI 10.1037/a0029487

Open this paper in CoChat — free account

Abstract

Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power decreases as a function of the number of statistical tests that are being conducted (Maxwell, 2004). The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959). The problem of low power in multiple-study articles is illustrated using Bem's (2011) article on extrasensory perception and Gailliot et al.'s (2007) article on glucose and self-regulation. I conclude with several recommendations that can increase the credibility of scientific evidence in psychological journals. One major recommendation is to pay more attention to the power of studies to produce positive results without the help of questionable research practices and to request that authors justify sample sizes with a priori predictions of effect sizes. It is also important to publish replication studies with nonsignificant results if these studies have high power to replicate a published finding.

Integrity signals

Registry

Crossref · 10.1037/a0029487

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
A power primer.
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
GPOWER: A general power analysis program
Erdfelder, E., Faul, F., & Buchner, A. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203630 · Cited by 4,864
Required sample size to detect the mediated effect
Fritz, M. S., & MacKinnon, D. P. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01882.x · Cited by 4,810
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 5×
“Achieve 80% Power to Produce Significant Results in All”
BackgroundResultInfluential
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Initial Severity and Antidepressant Benefits: A Meta-Analysis of Data Submitted to the Food and Drug Administration
Kirsch, I., Deacon, B. J., Huedo-Medina, T. B., Scoboria, A., Moore, T. J., & Johnson, B. T. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050045 · Cited by 2,442
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis
Cipriani, A., Furukawa, T. A., Salanti, G., Geddes, J. R., Higgins, J. P., Churchill, R., Watanabe, N., Nakagawa, A., Omori, I. M., McGuire, H., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60046-5 · Cited by 1,654
Psychology as the science of self-reports and finger movements: whatever happened to actual behavior?
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00051.x · Cited by 1,629
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1×
“As a result, they have to rely on fortune (Wegner, 1992) or questionable research methods (Simmons et al., 2011; Vul et al., 2009) to report inflated observed effect sizes that reach statistical significance in small samples.”
Background
The New Yorker
Gottlieb, R. (1987) · The New Yorker · Cited by 1,311
Self-control relies on glucose as a limited energy source: Willpower is more than a metaphor.
Gailliot, M. T., Baumeister, R. F., DeWall, C. N., Maner, J. K., Plant, E. A., Tice, D. M., Brewer, L. E., & Schmeichel, B. J. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.2.325 · Cited by 1,297
On the malleability of automatic attitudes: Combating automatic prejudice with images of admired and disliked individuals.
Dasgupta, N., & Greenwald, A. G. (2001) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.81.5.800 · Cited by 1,102
Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 4×
“…number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959).”
BackgroundResultInfluential
Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
On the malleability of automatic attitudes: Combating automatic prejudice with images of admired and disliked individuals.
Dasgupta, N., & Greenwald, A. G. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.81.5.800 · Cited by 809
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
The Use of Predicted Confidence Intervals When Planning Experiments and the Misuse of Power When Interpreting Results
Goodman, S. N., & Berlin, J. A. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-3-199408010-00008 · Cited by 658
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2007) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 601
Things I have learned (so far).
Cohen, J. (1992) · American Psychological Association eBooks · doi:10.1037/10109-028 · Cited by 553
Big correlations in little studies: Inflated fMRI correlations reflect low statistical power – Commentary on Vul et al. (2009)
Yarkoni, T. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01127.x · Cited by 549
A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance--Or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 474
Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Replication studies: Bad copy
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 406
Retractions in the scientific literature: is the incidence of research fraud increasing?
Steen, R. G. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.040923 · Cited by 405
Revisiting Truth or Triviality
Mitchell, G. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432343 · Cited by 343
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.5.646 · Cited by 340
Unpublished results hide the decline effect
Schooler, J. W. (2011) · Nature · doi:10.1038/470437a · Cited by 335
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323
Wishful Seeing: More Desired Objects Are Seen as Closer
Balcetis, E., & Dunning, D. (2010) · Psychological Science · doi:10.1177/0956797609356283 · Cited by 320
Does the Brain Consume Additional Glucose during Self-Control Tasks?
Kurzban, R. (2010) · Evolutionary Psychology · doi:10.1177/147470491000800208 · Cited by 250
Voodoo Correlations Are Everywhere—Not Only in Neuroscience
Fiedler, K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611400237 · Cited by 248
Additivity of nonconscious affect: Combined effects of priming and exposure.
Murphy, S. T., Monahan, J. L., & Zajonc, R. B. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.4.589 · Cited by 245
The Effect of Sample Size on the Meaning of Significance Tests
Royall, R. M. (1986) · The American Statistician · doi:10.1080/00031305.1986.10475424 · Cited by 234
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 221
Retractions in the scientific literature: do authors deliberately commit research fraud?
Steen, R. G. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.038125 · Cited by 217
Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033423 · Cited by 198
Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests
Mudge, J. F., Baker, L. F., Edge, C. B., & Houlahan, J. E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032734 · Cited by 192
What makes an article influential? Predicting impact in social and personality psychology
Haslam, N., Ban, L., Kaufmann, L. M., Loughnan, S., Peters, K., Whelan, J., & Wilson, S. G. (2008) · Scientometrics · doi:10.1007/s11192-007-1892-8 · Cited by 182
On the Post Hoc Power in Testing Mean Differences
Yuan, K.-H., & Maxwell, S. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030002141 · Cited by 168
Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa
Sterling, T. D., Rosenbaum, W. L., & Jj, W. (1995) · The American Statistician · doi:10.2307/2684823 · Cited by 168
Moderation of Resource Depletion in the Self-Control Strength Model: Differing Effects of Two Modes of Self-Control
Dvorak, R., & Simons, J. (2009) · Personality and Social Psychology Bulletin · doi:10.1177/0146167208330855 · Cited by 164
Detecting and Correcting the Lies That Data Tell
Schmidt, F. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691610369339 · Cited by 163
Ten frequently asked questions about implicit measures and their frequently supposed, but not entirely correct answers.
Gawronski, B. (2009) · Canadian Psychology/Psychologie canadienne · doi:10.1037/a0013848 · Cited by 144
Meta-analytic interval estimation for standardized and unstandardized mean differences.
Bonett, D. G. (2009) · Psychological Methods · doi:10.1037/a0016619 · Cited by 133
The surprisingly limited malleability of implicit racial evaluations
Joy-Gaba, J. A., & Nosek, B. A. (2010) · Social Psychology · doi:10.1027/1864-9335/a000020 · Cited by 123
Fighting Publication Bias: Introducing the Negative Results Section
Dirnagl, U., & Lauritzen, M. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.51 · Cited by 111
Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science
Ledgerwood, A., & Sherman, J. W. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611427304 · Cited by 72
The Effect of Sample Size on the Meaning of Significance Tests
Royall, R. M. (1986) · The American Statistician · doi:10.2307/2684616 · Cited by 70
Introduction to the Special Section: Data, Data, Everywhere . . . Especially in My File Drawer
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432124 · Cited by 58
Writing an Empirical Article
Bem, D. J. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511807862.002 · Cited by 47
SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing
Strube, M. (2006) · Behavior Research Methods · doi:10.3758/bf03192746 · Cited by 41
The Same Old New Look: Publication Bias in a Study of Wishful Seeing
Francis, G. (2012) · i-Perception · doi:10.1068/i0519ic · Cited by 36
Replication, replication, replication
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · University of Hertfordshire Research Archive (University of Hertfordshire) · Cited by 33
Editorial.
Smith, E. R. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0026676 · Cited by 33
The Premature Demise of the Solo Experiment
Wegner, D. M. (1992) · Personality and Social Psychology Bulletin · doi:10.1177/0146167292184017 · Cited by 30
Editorial comment.
Judd, C. M., & Gawronski, B. (2011) · Journal of Personality and Social Psychology · doi:10.1037/0022789 · Cited by 16
SCIENCE WITHOUT SENSE: The Risky Business of Public Health Research
Milloy, S. (2020) · Medical Entomology and Zoology · Cited by 8
Editorial.
Diener, E. (1998) · Journal of Personality and Social Psychology · doi:10.1037/h0092824 · Cited by 6
Harvard probe kept under wraps
Ledford, H. K. (2010) · Nature · doi:10.1038/466908a · Cited by 4
A false-positive error in search of selective reporting: A refutation of Francis
Balcetis, E., & Dunning, D. M. (2012) · i-Perception · Cited by 2
Replication, replication, replication
Morley, S., Masterson, C., & Main, C. J. (2017) · Routledge eBooks · doi:10.4324/9781315412931-7 · Cited by 2

Flagged first, then by citations

Sorted by citations

An opportunity cost model of subjective effort and task performance
Kurzban, R., Duckworth, A., Kable, J. W., & Myers, J. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12003196 · Cited by 1,381
Why self-control seems (but may not be) limited
Inzlicht, M., Schmeichel, B. J., & Macrae, C. N. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.12.009 · Cited by 1,026
Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770

See all 455

Authors: Schimmack, U.
Affiliations: University of Toronto (CA)
Corresponding author: Schimmack, Ulrich
ORCID: Schimmack, Ulrich
Venue: Psychological Methods
Publisher: American Psychological Association (APA)
Published: 2012
Volume: 17
Issue: 4
Pages: 551-566
ISSN: 1082-989X (ISSN-L)
1939-1463
DOI: 10.1037/a0029487
OpenAlex: W2164150544
Cited by: 455
References: 76
Citations by year: 2026: 8 · 2025: 10 · 2024: 14 · 2023: 15 · 2022: 24 · 2021: 55 · 2020: 27 · 2019: 56 · 2018: 36 · 2017: 47 · 2016: 42 · 2015: 48 · 2014: 38 · 2013: 29 · 2012: 5
FWCI: 43.66
Type: journal article
Language: en
Open access: closed
Locations: Psychological Methods (publishedVersion)
PubMed (publishedVersion)
Funders: Social Sciences and Humanities Research Council of Canada
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords: statistical power, research credibility, questionable research practices, non-significant results, sample size justification, replication study, extrasensory perception
MeSH terms: Humans, Psychology, Research Design, Statistics as Topic, Publication Bias, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The ironic effect of significant results on the credibility of multiple-study articles.

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks