Small Telescopes: Detectability and the Evaluation of Replication Results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our replications employed 2.5 times the sample size of the original paper (8).”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9)”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“This approach limits the sample size required to test for equivalence to 2.5 times the sample size of the original study.”
From Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses (2017) · cites this paper as Simonsohn (2015)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects, would on average have…”
Schwarz, N., & Clore, G. L. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.513 · Cited by 4,952
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990) · Journal of Political Economy · doi:10.1086/261737 · Cited by 3,194
Research works printed but not linked: 4
- Fisher R. A., 1926, Journal of the Ministry of Agriculture, 33, 503
- Jeffreys H., 1961, Theory of probability
- 10.3102/10769986008002093 · doi:10.3102/10769986008002093 · Paper page
- 10.1037/0003-066X.40.1.73 · doi:10.1037/0003-066x.40.1.73 · Paper page
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560
“Simonsohn (2015) recently proposed setting the SESOI as the effect size that an earlier study would have had 33% power to detect.”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
What can be learned from growth mindset controversies?
Yeager, D. S., & Dweck, C. S. (2020) · American Psychologist · doi:10.1037/amp0000794 · Cited by 868
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Self-citationZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Soto, C. J. (2019) · Psychological Science · doi:10.1177/0956797619831612 · Cited by 470
“…show (from left to right) the sample size in the replication study (see Table 1), the sample size in the original study, a sample size 2.5 times as large as in the original study (Simonsohn, 2015), and the sample size required to provide 80% statistical power to detect the original effect size.”
InfluentialTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study
Gillen, B., Snowberg, E., & Yariv, L. (2019) · Journal of Political Economy · doi:10.1086/701681 · Cited by 350
Bayesian Benefits for the Pragmatic Researcher
Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016) · Current Directions in Psychological Science · doi:10.1177/0963721416643289 · Cited by 334
Replicating patterns of prospect theory for decision under risk
Ruggeri, K., Alí, S., Berge, M. L., Bertoldo, G., Bjørndal, L. D., Cortijos-Bernabeu, A., Davison, C., Demić, E., Esteban-Serna, C., Friedemann, M., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0886-x · Cited by 326
Inference of trustworthiness from intuitive moral judgments.
Everett, J. A. C., Pizarro, D. A., & Crockett, M. J. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000165 · Cited by 317
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 297
Panero, M. E., Weisberg, D. S., Black, J., Goldstein, T. R., Barnes, J. L., Brownell, H., & Winner, E. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000064 · Cited by 289
Improving inferences about null effects with Bayes factors and equivalence tests.
Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2020) · The Journals of Gerontology: Series B · doi:10.1093/geronb/gby065 · Cited by 279
“An equivalence test examines whether we can reject the effect size that was observed in the earlier study, or, as a more stringent test, an effect size the earlier study had a statistical power of 33% or 50% to detect (Lakens, Scheel, & Isager, 2018; Lenth, 2007; Simonsohn, 2015).”
Matheson, G. J. (2019) · PeerJ · doi:10.7717/peerj.6918 · Cited by 244
A tutorial on testing hypotheses using the Bayes factor.
Hoijtink, H., Mulder, J., van Lissa, C., & Gu, X. (2019) · Psychological Methods · doi:10.1037/met0000201 · Cited by 234
Power to Detect What? Considerations for Planning and Evaluating Sample Size
Giner-Sorolla, R., Montoya, A. K., Reifman, A., Carpenter, T., Lewis, N. A., Aberson, C. L., Bostyn, D. H., Conrique, B. G., Ng, B. W., Schoemann, A. M., et al. (2024) · Personality and Social Psychology Review · doi:10.1177/10888683241228328 · Cited by 221
“If inter-sectional populations are studied as a statistical interaction of categories, effects are likely to be weaker and more variable for a number of reasons (McClelland & Judd, 1993; Simonsohn, 2015).”
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 224
Low statistical power in biomedical science: a review of three human research domains
Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017) · Royal Society Open Science · doi:10.1098/rsos.160254 · Cited by 208
The replication crisis in psychology: An overview for theoretical and philosophical psychology.
Wiggins, B. J., & Christopherson, C. D. (2019) · Journal of Theoretical and Philosophical Psychology · doi:10.1037/teo0000137 · Cited by 207
The Pandora Effect: The Power and Peril of Curiosity
Hsee, C. K., & Ruan, B. (2016) · Psychological Science · doi:10.1177/0956797616631733 · Cited by 199
Replications of implicit theory of mind tasks with varying representational demands
Powell, L. J., Hobbs, K., Bardis, A., Carey, S., & Saxe, R. (2018) · Cognitive Development · doi:10.1016/j.cogdev.2017.10.004 · Cited by 183
Psychological reactions to human versus robotic job replacement
Granulo, A., Fuchs, C., & Puntoni, S. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0670-y · Cited by 170
Value-based attention but not divisive normalization influences decisions with multiple alternatives
Gluth, S., Kern, N., Kortmann, M., & Vitali, C. L. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0822-0 · Cited by 176
Smeekens, B. A., & Kane, M. J. (2016) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/aca0000046 · Cited by 170
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 165
Conservatives and liberals have similar physiological responses to threats
Bakker, B. N., Schumacher, G., Gothreau, C., & Arceneaux, K. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0823-z · Cited by 165
Mindfulness increases prosocial responses toward ostracized strangers through empathic concern.
Berry, D. R., Cairo, A. H., Goodman, R. J., Quaglia, J. T., Green, J. D., & Brown, K. W. (2018) · Journal of Experimental Psychology: General · doi:10.1037/xge0000392 · Cited by 166
Equivalence Tests: A Practical Primer for t-Tests, Correlations, and Meta-Analyses
Lakens, D. (2016) · doi:10.31234/osf.io/97gpc · Cited by 164
Conceptualizing and evaluating the replication of research results
Fabrigar, L. R., & Wegener, D. T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.009 · Cited by 165
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Cross-Validation Approaches for Replicability in Psychology
Koul, A., Becchio, C., & Cavallo, A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01117 · Cited by 143
A registered replication study on oxytocin and trust
Declerck, C. H., Boone, C., Pauwels, L., Vogt, B., & Fehr, E. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0878-x · Cited by 137
The Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Anderson, S. F., & Maxwell, S. E. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 131
How to quantify the evidence for the absence of a correlation
Wagenmakers, E.-J., Verhagen, J., & Ly, A. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0593-0 · Cited by 129
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 126
Conceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 124
Value-based attentional capture affects multi-alternative decision making
Gluth, S., Spektor, M. S., & Rieskamp, J. (2018) · eLife · doi:10.7554/elife.39659 · Cited by 140
“To test for the ‘replicability’ of this effect without reverting to the (equivocal) effect size of the original study, we conducted a recently proposed test of detectability that is independent of the reported effect size of the original study (Simonsohn, 2015).”
How to run an experimental auction: a review of recent advances
Canavari, M., Drichoutis, A. C., Lusk, J. L., & Nayga, R. M. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz038 · Cited by 140
“As a side note, a consequence of the inflation bias is that if a replication study is powered based on the effect size of the original study, the power of the replicated study will be lower than intended (Simonsohn, 2015).”
InfluentialCoyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 128
Camerer, C., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117
Crawford, J. T., Brandt, M. J., Inbar, Y., Chambers, J. R., & Motyl, M. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000074 · Cited by 114
Compassion magnifies third-party punishment.
Pfattheicher, S., Sassenrath, C., & Keller, J. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000165 · Cited by 124
Plaßmann, H., & Weber, B. (2015) · Journal of Marketing Research · doi:10.1509/jmr.13.0613 · Cited by 136
Geller, J., Toftness, A. R., Armstrong, P. I., Carpenter, S. K., Manz, C. L., Coffman, C. R., & Lamm, M. H. (2018) · Memory · doi:10.1080/09658211.2017.1397175 · Cited by 85
Blame It on the Self-Driving Car: How Autonomous Vehicles Can Alter Consumer Morality
Gill, T. (2020) · Journal of Consumer Research · doi:10.1093/jcr/ucaa018 · Cited by 112
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.001 · Cited by 108
Effects of charismatic leadership and rewards on individual performance
Meslec, N., Curșeu, P. L., Fodor, O. C., & Kenda, R. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101423 · Cited by 139
99% impossible: A valid, or falsifiable, internal meta-analysis.
Vosgerau, J., Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2019) · Journal of Experimental Psychology: General · doi:10.1037/xge0000663 · Cited by 98
Self-citationFalsifiability is not optional.
LeBel, E. P., Berger, D., Campbell, L., & Loving, T. J. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000106 · Cited by 105
Submentalizing or mentalizing in a Level 1 perspective-taking task: A cloak and goggles test.
Conway, J. R., Lee, D., Ojaghi, M., Catmur, C., & Bird, G. (2016) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0000319 · Cited by 120
Sanchez, C., Sundermeier, B., Gray, K., & Calin-Jageman, R. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172636 · Cited by 99
Predicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. (2019) · PLOS ONE · doi:10.1371/journal.pone.0225826 · Cited by 95
“[4, 35, 36]) should be seen as a complement to our work and could hopefully be used to improve future versions of this predictive model.”
The effects of facial attractiveness and trustworthiness in online peer-to-peer markets
Jaeger, B., Sleegers, W. W. A., Evans, A. M., Stel, M., & van Beest, I. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.11.004 · Cited by 100
“Simonsohn (2015) suggested that, as a rule of thumb, replication studies should aim for a sample size that is at least 2.5 times larger than the original study.”
InfluentialSchuwerk, T., Priewasser, B., Sodian, B., & Perner, J. (2018) · Royal Society Open Science · doi:10.1098/rsos.172273 · Cited by 93
Opportunities for increased reproducibility and replicability of developmental neuroimaging
Klapwijk, E. T., van den Bos, W., Tamnes, C. K., Raschle, N. M., & Mills, K. L. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100902 · Cited by 90
“Note that one way to circumvent this is the ‘small telescopes’ approach by Simonsohn (2015), which estimates whether the replication effect size is significantly smaller than an effect for which the original study had 33 % power to detect.”
Vazire, S. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615603955 · Cited by 106
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
Kidd, D., & Castano, E. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550618775410 · Cited by 87
“To directly assess consistency with the results of Experiment 5 in Kidd and Castano (2013), a “small telescopes” analysis was conducted to compare replication effect sizes with those in the original study (Simonsohn, 2015).”
A Brief Guide to Evaluate Replications
LeBel, E. P., Vanpaemel, W., Cheung, I., & Campbell, L. (2019) · Meta-Psychology · doi:10.15626/mp.2018.843 · Cited by 125
Laraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 103
“…to do with the “truthfulness” of the original study, so apparent “negative” results do not automatically indicate that the first study reported a false positive (Huitema, 1986b; Leek & Jager, 2017; Lynch, Bradlow, Huber, & Lehmann, 2015; Maxwell et al., 2015; Schmidt & Oh, 2016; Simonsohn, 2015).”
A Second Look at Automatic Theory of Mind: Reconsidering Kovács, Téglás, and Endress (2010)
Phillips, J., Ong, D. C., Surtees, A. D. R., Xin, Y., Williams, S. E., Saxe, R. R., & Frank, M. C. (2015) · Psychological Science · doi:10.1177/0956797614558717 · Cited by 105
Maniadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
Rieger, S., Göllner, R., Trautwein, U., & Roberts, B. W. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000037 · Cited by 84
People Are Slow to Adapt to the Warm Glow of Giving
O’Brien, E., & Kassirer, S. (2018) · Psychological Science · doi:10.1177/0956797618814145 · Cited by 98
“Sample size was guided by the recommendation to replicate effects by recruiting 2.5 times the original sample (Simonsohn, 2015).”
InfluentialEquivalence Testing for Psychological Research: A TutorialPreprint
Lakens, D., Scheel, A. M., & Isager, P. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/v3zkt · Cited by 125
Crawford, J. T., Brandt, M. J., Inbar, Y., & Mallinas, S. R. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000070 · Cited by 97
Wu, S., Liu, Y., Ruan, M., Chen, S., & Xie, X. (2025) · Scientific Reports · doi:10.1038/s41598-025-98385-2 · Cited by 167
A conceptual replication of ambidextrous leadership theory: An experimental approach
Klonek, F. E., Gerpott, F. H., & Parker, S. K. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101473 · Cited by 72
Chalkia, A., Schroyens, N., Leng, L., Vanhasbroeck, N., Zenses, A., Oudenhove, L. V., & Beckers, T. (2020) · Cortex · doi:10.1016/j.cortex.2020.04.017 · Cited by 82
“A sample size of 62 per group also clearly fulfills the suggestion that replication efforts include 2.5 times the original sample size, offered by Simonsohn (2015) on the basis of his “small telescopes” approach (which basically entails that if an effect is sufficiently large to have a minimal…”
InfluentialAwe and scientific explanation.
Valdesolo, P., Park, J. T., & Gottlieb, S. (2016) · Emotion · doi:10.1037/emo0000213 · Cited by 83
Replication and Reproducibility in Cross-Cultural Psychology
Milfont, T. L., & Klein, R. A. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117744892 · Cited by 73
Conditionals and inferential connections: A hypothetical inferential theory
Douven, I., Elqayam, S., Singmann, H., & van Wijnbergen‐Huitink, J. (2018) · Cognitive Psychology · doi:10.1016/j.cogpsych.2017.09.002 · Cited by 78
How prior information and police experience impact decisions to shoot.
Johnson, D. J., Cesario, J., & Pleskac, T. J. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000130 · Cited by 83
A two-lab direct replication attempt of Southgate, Senju and Csibra (2007)
Kampis, D., Karman, P., Csibra, G., Southgate, V., & Hernik, M. (2021) · Royal Society Open Science · doi:10.1098/rsos.210190 · Cited by 77
Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E.-J. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 64
“Moreover, there currently exists no consensus on which decision rule is superior for determining replication success (Cumming, 2008; Open Science Collaboration, 2015; Simonsohn, 2015; Verhagen & Wagenmakers, 2014).”
Sjouwerman, R., Niehaus, J., Kuhn, M., & Lonsdorf, T. B. (2016) · Psychophysiology · doi:10.1111/psyp.12761 · Cited by 64
Perceived crisis and reforms: Issues, explanations, and remedies.
De Boeck, P., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 64
More Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 74
Collova, J. R., Sutherland, C. A. M., & Rhodes, G. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000167 · Cited by 70
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 71
“This method, however, is poor at detecting false-positives (Simonsohn, 2015).”
InfluentialFailed Replication of Oxytocin Effects on Trust: The Envelope Task Case
Lane, A., Mikolajczak, M., Treinen, E., Samson, D., Corneille, O., de Timary, P., & Luminet, O. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137000 · Cited by 60
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
The effect of mood on judgments of subjective well-being: Nine tests of the judgment model.
Yap, S. C. Y., Wortman, J., Anusic, I., Baker, S., Scherer, L. D., Donnellan, M. B., & Lucas, R. E. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000115 · Cited by 77
Improving Psychological Science through Transparency and Openness: An Overview
Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018) · Perspectives on Behavior Science · doi:10.1007/s40614-018-00186-8 · Cited by 70
“Simonsohn (2015) recommends a “small telescopes test.””
InfluentialA New Standard for the Analysis and Design of Replication Studies
Held, L. (2019) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12493 · Cited by 80
“Standard significance of the replication study is often used as a criterion, but significance alone does not take the effect sizes of the original and replication study into account and can easily lead to conclusions opposite to what the evidence warrants (Simonsohn, 2015).”
InfluentialThe importance of awareness for understanding language.
Rabagliati, H., Robertson, A., & Carmel, D. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000348 · Cited by 70
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-03-23
- Volume
- 26
- Issue
- 5
- Pages
- 559-569
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2151509224
- Cited by
- 742
- References
- 52
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 59 · 2023: 67 · 2022: 64 · 2021: 66 · 2020: 111 · 2019: 79 · 2018: 76 · 2017: 63 · 2016: 48 · 2015: 16
- FWCI
- 61.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- effect size estimation, hypothesis testing, replication study, statistical power, underpowered studies, small effect sizes
- MeSH terms
- Astronomy, Data Interpretation, Statistical, Research Design, Reproducibility of Results, Models, Statistical, Sample Size, Telescopes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.