Small Telescopes: Detectability and the Evaluation of Replication Results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our replications employed 2.5 times the sample size of the original paper (8).”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9)”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“This approach limits the sample size required to test for equivalence to 2.5 times the sample size of the original study.”
From Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses (2017) · cites this paper as Simonsohn (2015)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects, would on average have…”
Schwarz, N., & Clore, G. L. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.513 · Cited by 4,952
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990) · Journal of Political Economy · doi:10.1086/261737 · Cited by 3,194
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (2000) · Behavioral Law and Economics · doi:10.1017/cbo9781139175197.009 · Cited by 2,873
A 61-million-person experiment in social influence and political mobilization
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I., Marlow, C., Settle, J. E., & Fowler, J. H. (2012) · Nature · doi:10.1038/nature11421 · Cited by 2,575
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 2×
“In addition, because of publication bias, most true effects are overestimated in the published literature (Hedges, 1984; Ioannidis, 2008; Lane & Dunlap, 1978), so most sufficiently powered replication studies of true effects will obtain statistically significantly smaller estimates than the originals.”
Harvesting implicit group attitudes and beliefs from a demonstration web site.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002) · Group Dynamics: Theory, Research, and Practice · doi:10.1037/1089-2699.6.1.101 · Cited by 1,321
Washing Away Your Sins: Threatened Morality and Physical Cleansing
Zhong, C.-B., & Liljenquist, K. (2006) · Science · doi:10.1126/science.1130726 · Cited by 1,169
Introduction to Probability and Statistics
(2015) · Environmental Risk Analysis · doi:10.1201/b19097-5 · Cited by 1,153
The Logic of Scientific Discovery
Popper, K. (2005) · Routledge eBooks · doi:10.4324/9780203994627 · Cited by 1,142
Knetsch, J. L., & Sinden, J. A. (1984) · The Quarterly Journal of Economics · doi:10.2307/1885962 · Cited by 1,062
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“…than zero, and concluding that it is (basically) zero if we obtain a result that is statistically significantly smaller than that small effect (see, e.g., Cohen, 1988, pp. 16–17; Greenwald, 1975, pp. 16–19; Hodges & Lehmann, 1954; Kraemer, 1983; Rindskopf, 1997; Serlin & Lapsley, 1985, 1993).”
The Arrangement of Field Experiments
Fisher, R. A. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_8 · Cited by 964
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Verbal overshadowing of visual memories: Some things are better left unsaid
Schooler, J. W., & Engstler-Schooler, T. Y. (1990) · Cognitive Psychology · doi:10.1016/0010-0285(90)90003-m · Cited by 794
Introduction to Probability and Statistics from a Bayesian Viewpoint
Lindley, D. V. (1965) · Cambridge University Press eBooks · doi:10.1017/cbo9780511662973 · Cited by 760
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
Self-citationWhat If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 667
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
A Handbook for data analysis in the behavioral sciences: methodological issues
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0602 · Cited by 543
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
The Disparity Between Willingness to Accept and Willingness to Pay Measures of Value
Coursey, D. L., Hovis, J. L., & Schulze, W. D. (1987) · The Quarterly Journal of Economics · doi:10.2307/1884223 · Cited by 518
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · cited 2×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects,…”
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E.-J. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0036731 · Cited by 291
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241
Registered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223
Rationality in psychological research: The good-enough principle.
Serlin, R. C., & Lapsley, D. K. (1985) · American Psychologist · doi:10.1037//0003-066x.40.1.73 · Cited by 217
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166 · cited 1×
“Despite the notable shortcomings of this approach (see, e.g., Asendorpf et al., 2013; Cumming, 2008; Valentine et al., 2011), effects obtained in replication studies are currently evaluated almost exclusively on the basis of whether or not they are significantly different from zero.”
Estimating effect size: Bias resulting from the significance criterion in editorial decisions
Lane, D. M., & Dunlap, W. P. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00578.x · Cited by 164
Testing the Approximate Validity of Statistical Hypotheses
Hodges, J. L., & Lehmann, E. L. (1954) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1954.tb00169.x · Cited by 133
Fatherhood and Managerial Style: How a Male CEO’s Children Affect the Wages of His Employees
Dahl, M. S., Dezső, C. L., & Ross, D. G. (2012) · Administrative Science Quarterly · doi:10.1177/0001839212466521 · Cited by 131
Lucas, R. E., & Lawless, N. M. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0032124 · Cited by 119
Testing the Approximate Validity of Statistical Hypotheses
Hodges, J. L., & Lehmann, E. L. (2011) · Springer eBooks · doi:10.1007/978-1-4614-1412-4_15 · Cited by 117
Simonsohn, U. (2015) · The Winnower · doi:10.15200/winn.142559.90552 · Cited by 90
Self-citationRational appraisal of psychological research and the good-enough principle.
Serlin, R. C., & Lapsley, D. K. (1993) · Cited by 80
Bayesian methods for assessing importance of effects.
Rouanet, H. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.1.149 · Cited by 75
Theory of Estimation and Testing of Effect Sizes: Use in Meta-Analysis
Kraemer, H. C. (1983) · Journal of Educational Statistics · doi:10.2307/1164919 · Cited by 56
The Uncertain Universality of the Macbeth Effect with a Spanish Sample
Gámez, E., Gómez, J. M. D., & Marrero, H. (2011) · The Spanish Journal of Psychology · doi:10.5209/rev_sjop.2011.v14.n1.13 · Cited by 49
Testing 'Small', not Null, Hypotheses: Classical and Bayesian Approaches
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (2013) · doi:10.4324/9781315629049-26 · Cited by 27
Subjective Well-Being: Weather Matters; Climate Doesn'tPreprint
Feddersen, J., Metcalfe, R., & Wooden, M. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2182862 · Cited by 23
Registered Replication Reports
Simons, D. J. (2014) · APS observer · Cited by 16
Research works printed but not linked: 4
- Fisher R. A., 1926, Journal of the Ministry of Agriculture, 33, 503
- Jeffreys H., 1961, Theory of probability
- 10.3102/10769986008002093 · doi:10.3102/10769986008002093 · Paper page
- 10.1037/0003-066X.40.1.73 · doi:10.1037/0003-066x.40.1.73 · Paper page
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560
“Simonsohn (2015) recently proposed setting the SESOI as the effect size that an earlier study would have had 33% power to detect.”
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-03-23
- Volume
- 26
- Issue
- 5
- Pages
- 559-569
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2151509224
- Cited by
- 742
- References
- 52
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 59 · 2023: 67 · 2022: 64 · 2021: 66 · 2020: 111 · 2019: 79 · 2018: 76 · 2017: 63 · 2016: 48 · 2015: 16
- FWCI
- 61.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- effect size estimation, hypothesis testing, replication study, statistical power, underpowered studies, small effect sizes
- MeSH terms
- Astronomy, Data Interpretation, Statistical, Research Design, Reproducibility of Results, Models, Statistical, Sample Size, Telescopes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.