Small Telescopes: Detectability and the Evaluation of Replication Results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our replications employed 2.5 times the sample size of the original paper (8).”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9)”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“This approach limits the sample size required to test for equivalence to 2.5 times the sample size of the original study.”
From Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses (2017) · cites this paper as Simonsohn (2015)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects, would on average have…”
Schwarz, N., & Clore, G. L. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.513 · Cited by 4,952
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990) · Journal of Political Economy · doi:10.1086/261737 · Cited by 3,194
Research works printed but not linked: 4
- Fisher R. A., 1926, Journal of the Ministry of Agriculture, 33, 503
- Jeffreys H., 1961, Theory of probability
- 10.3102/10769986008002093 · doi:10.3102/10769986008002093 · Paper page
- 10.1037/0003-066X.40.1.73 · doi:10.1037/0003-066x.40.1.73 · Paper page
Cited by
Items 701–743 of 743
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705871
Line, E. N., Jaramillo, S., Goldwater, M. B., & Horne, Z. (2024) · Figshare · doi:10.6084/m9.figshare.26829020
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705880
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25596366
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705877
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705898
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705874
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26705901
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn-Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25596570
Geiser, A. E., & Nelson, L. D. (2025) · Journal of Consumer Research · doi:10.1093/jcr/ucaf067 · Cited by 2
Revisiting regulatory fit and its effect on honesty: A replication attempt
Ścigała, K. A., Schild, C., & Pfattheicher, S. (2025) · European Journal of Personality · doi:10.1177/08902070251410250 · Cited by 1
“Following recommendations by Simonsohn (2015), we aimed to recruit a sample 2.5 times higher than the sample of the original study.”
InfluentialTenney, E. R., Rochford, K. C., Stillwell, A., Liu, C. X., Tannenbaum, D., Hennecke, M., Stefanucci, J. K., Blair, B. A., Graham, J., & Bonner, B. L. (2026) · Psychological Science · doi:10.1177/09567976251412721
“We planned to obtain a sample size 2.5 times that of the original article (see Simonsohn, 2015).”
InfluentialGuérin, S. M. R., Karageorghis, C. I., Coeugnet, M. R., Bigliassi, M., & Delevoye, Y. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1166
Diverse groups look more moral in loose (but not tight) cultural contexts
Karataş, M., & Chin, S.-C. (. (2026) · Communications Psychology · doi:10.1038/s44271-026-00435-6
Houtin, L. (2018) · theses.fr (ABES) · doi:10.70675/7506be16z7782z40bez89efzb37929eb9a36 · Cited by 1
Guérin, S. M. R. (2021) · theses.fr (ABES) · doi:10.70675/ebe126c0ze4f5z41abza158z4ef9bd6d72f8
Registered Report: Can Agentic Black Women Get Ahead? An Experiment Revisited
Liu, C. X., Blair, B. A., & Tenney, E. R. (2026) · Psychological Science · doi:10.1177/09567976261436988 · Cited by 1
“We followed the small-telescopes method to determine the replicability of Livingston et al.’s (2012) key findings (Simonsohn, 2015).”
InfluentialNeural activation by vocal threat requires attention
Witteman, J., Axioti, S., & van Tol, A. (2026) · Cortex · doi:10.1016/j.cortex.2026.04.009
Who Cares More About Having Moral Traits? Evidence From 67 Countries
Paruzel‐Czachura, M., & McHugh, C. (2026) · Journal of Cross-Cultural Psychology · doi:10.1177/00220221261439284
“…poses challenges for interpretation, it also highlights a broader issue in the field: individual studies with limited samples, while widely accepted (Munafò et al., 2017; Simonsohn, 2015), may not provide a comprehensive understanding of psychological phenomena beyond the specific samples studied.”
Baess, P., Ayatollahi, S., Bermeitinger, C., Bogenschütz, L., Fenske, P., Hackländer, R. P. M., Hamzeloo, M., & Montoya, G. A. L. (2026) · Psychological Research · doi:10.1007/s00426-026-02299-1
“To address this, the current study closely replicates Stof - fer’s (1991) experiment, following the approach of previous replications (Weeks et al., 1996), but with nearly three times the original sample size (Brandt et al., 2014; Simonsohn, 2015).”
Psychology’s RenaissancePreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2022) · OSF Preprints (OSF Preprints)
Self-citationSodian, B., Schuwerk, T., Kristen, S., Kaltefleiter, L. J., Jarvers, I., & Wiesmann, C. G. (2022) · OSF Preprints (OSF Preprints)
Verschuère, B., Kleinberg, B., & Warmelink, L. (2023) · OSF Preprints (OSF Preprints)
The Effect of Animacy on Structural Priming: A Replication of Bock, Loebell and Morey (1992)Preprint
Zhang, C., Hartsuiker, R. J., Slim, M. S., Chen, X., & Muylle, M. (2023) · OSF Preprints (OSF Preprints)
Tønnesen, M. T., Michael, J., & Mitkidis, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6836276
Otterbring, T., Folwarczny, M., & Gąsiorowska, A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-22281-8.00004-7
Gender in action: Stereotype effects in the action observation–verb processing link
Bidet-Ildei, C., Alexopoulos, T., Selimbegović, L., Blandin, Y., & Chatard, A. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107146
Radin, D. I., & Cline, R. (2026) · Physics Essays · doi:10.4006/0836-1398-39.3.391
Arai, T., Miyagawa, Y., & Kanemasa, Y. (2026) · Journal of Family Violence · doi:10.1007/s10896-026-01128-8
Rethinking Type S and Type M Errors
Lakens, D., Mesquida, C., Xavier-Quintais, G., Rasti, S., Toffalini, E., & Altoè, G. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261432530
“For example, Simonsohn (2015) considered a test with less than 33% power severely underpowered, but with 33% power, the probability that a significant result is in the wrong direction is only 0.1%.”
InfluentialDonenfeld, J., Káldy, Z., & Blaser, E. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-026-02944-1
Koob, V., & Janczyk, M. (2026) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208261463136
“With n = 39 participants, we also exceed the (sometimes) recommended sample size for replication studies of n = 2.5 � n orig (Simonsohn, 2015).”
InfluentialLaulan, P., Millet, G., & Rimmele, U. (2026) · Psychophysiology · doi:10.1111/psyp.70348
“…d = 0.63, corresponding to the magnitude the present design is reasonably positioned to ad-judicate, and a more conservative secondary bound of d = 0.35, informed by prior work on age differences in emotional coherence (e.g., Lohani et al. 2018) and by the small-telescopes logic (Simonsohn 2015).”
InfluentialDurston, A. J., Fernandes, M. A., & Itier, R. J. (2026) · Cognition & Emotion · doi:10.1080/02699931.2026.2700438
Base Rate Information Improves Risk Estimation of Laypeople and Experts
Lehmann, R., Schäfer, T., & Hausam, J. (2026) · Sexual Abuse · doi:10.1177/10790632261470830
“%) and beta error (5%), corresponding to a 95% probability of detecting the expected population effect (e.g., Open Science Collaboration, 2012; Simonsohn, 2015).”
Schyns, B., Lester, G. V., & Hammond, M. M. (2026) · Journal of Management Scientific Reports · doi:10.1177/27550311261467341
“Tothisend,Simonsohn(2015)recommends that sample sizes for replication studies be 2.5 times those of the original study for detectability.”
InfluentialBao, S., Zhao, J., Yang, E., Thomas, L. D. W., Neth, H., Woike, J. K., & Hafenbrädl, S. (2026) · Journal of Management Scientific Reports · doi:10.1177/27550311261460804
“…participants who did not have a unique IP address, leaving a fi nal sample of 5,181 participants (2.4 times the number of observations per condition of the original study (following Simonsohn, 2015), 54.33% female, M age = 40.90 years, SD = 12.60, 59.25% completed at least a bachelor ’ s degree).”
InfluentialTrustworthy A/B Patterns and the Winner's Curse: Lessons from Eight Large-Scale Replications
Kohavi, R., Linowski, J., Vermeer, L., Andreev, A., Dodin, M., & Furuseth, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818498
A series of replication attempts of experimental awe-induction effects
Graziosi, M., Chirico, A., Clifton, J. D. W., Rohde, J. S., Chhabra, R., & Yaden, D. B. (2026) · PLoS ONE · doi:10.1371/journal.pone.0356728
“We therefore exceed the 2.5 times sample size rule of thumb described by Simonsohn (2015) [36].”
InfluentialEvaluation of replication success: A statistical perspective and extended analysis of the SCORE data
van Aert, R. C. M., van Assen, M. A. L. M., & López-Nicolás, R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0358126 · Cited by 1
“…are more approaches to study replicability based on, for example, meta-analysis methods [11–15], the prediction interval (PI; [16]), statistical power analysis [17], combining null hypothesis significance testing and equivalence testing [18], Bayes factors [19], and reverse Bayes analysis [20].”
Psychological Reactions to Subordinates’ LLM Use at Work
Meyers, S., Briker, R., & Bigman, Y. (2026) · The Leadership Quarterly · doi:10.1016/j.leaqua.2026.102004 · Cited by 1
Directed forgetting of political information revisited
Romagano, J., & Hupbach, A. (2026) · Learning and Motivation · doi:10.1016/j.lmot.2026.102368
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-03-23
- Volume
- 26
- Issue
- 5
- Pages
- 559-569
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2151509224
- Cited by
- 742
- References
- 52
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 59 · 2023: 67 · 2022: 64 · 2021: 66 · 2020: 111 · 2019: 79 · 2018: 76 · 2017: 63 · 2016: 48 · 2015: 16
- FWCI
- 61.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- effect size estimation, hypothesis testing, replication study, statistical power, underpowered studies, small effect sizes
- MeSH terms
- Astronomy, Data Interpretation, Statistical, Research Design, Reproducibility of Results, Models, Statistical, Sample Size, Telescopes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.