Small Telescopes: Detectability and the Evaluation of Replication Results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our replications employed 2.5 times the sample size of the original paper (8).”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9)”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“This approach limits the sample size required to test for equivalence to 2.5 times the sample size of the original study.”
From Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses (2017) · cites this paper as Simonsohn (2015)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects, would on average have…”
Schwarz, N., & Clore, G. L. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.513 · Cited by 4,952
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990) · Journal of Political Economy · doi:10.1086/261737 · Cited by 3,194
Research works printed but not linked: 4
- Fisher R. A., 1926, Journal of the Ministry of Agriculture, 33, 503
- Jeffreys H., 1961, Theory of probability
- 10.3102/10769986008002093 · doi:10.3102/10769986008002093 · Paper page
- 10.1037/0003-066X.40.1.73 · doi:10.1037/0003-066x.40.1.73 · Paper page
Cited by
Items 101–200 of 743
Power to Detect What? Considerations for Planning and Evaluating Sample SizePreprint
Giner‐Sorolla, R., Montoya, A. K., Aberson, C. L., Carpenter, T., Lewis, N. A., Bostyn, D. H., Conrique, B. G., Ng, B., Reifman, A. S., Schoemann, A. M., et al. (2023) · doi:10.31234/osf.io/rv3kw · Cited by 62
The Effects of Disgust on Moral Judgments: Testing Moderators
Johnson, D. J., Wortman, J., Cheung, F., Hein, M., Lucas, R. E., Donnellan, M. B., Ebersole, C. R., & Narr, R. K. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616654211 · Cited by 62
“We initially planned to collect a sample at least 2.5 times larger than the original sample size of 69 (i.e., 173 participants; Simonsohn, 2015), but continued data collection until the end of the semester to maximize power.”
InfluentialMeta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 67
“The p-curve analysis is an alternative approach to assessing for publication bias and may result in better estimates of the true effect of the data (Simonsohn, 2015).”
O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103313118 · Cited by 56
An a priori solution to the replication crisis
Trafimow, D. (2018) · Philosophical Psychology · doi:10.1080/09515089.2018.1490707 · Cited by 59
“According to Simonsohn (2015), it would be difficult to draw a conclusion about the finding itself, or about the research hypothesis under investigation.”
InfluentialBliese, P. D., & Wang, M. (2019) · Journal of Management · doi:10.1177/0149206319886909 · Cited by 60
“When observed power for cohesion is 58%, few researchers should be alarmed when a follow-up study fails to achieve significance, particularly if the replication did not increase the sample size to increase power (see Simonsohn, 2015).”
Mishra, S., Lalumière, M. L., & Williams, R. J. (2016) · Journal of Gambling Studies · doi:10.1007/s10899-016-9608-8 · Cited by 68
Stamos, A., Lange, F., Huang, S., & Dewitte, S. (2019) · Journal of Research in Personality · doi:10.1016/j.jrp.2019.103902 · Cited by 64
Short-sighted greed? Focusing on the future promotes reputation-based generosity
Sjåstad, H. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500003430 · Cited by 62
A preregistered replication of motivated numeracy
Persson, E., Andersson, D., Koppel, L., Västfjäll, D., & Tinghög, G. (2021) · Cognition · doi:10.1016/j.cognition.2021.104768 · Cited by 51
Donnellan, M. B., Lucas, R. E., & Cesario, J. (2015) · Emotion · doi:10.1037/a0036079 · Cited by 51
The Primacy Effect in Impression Formation: Some Replications and Extensions
Sullivan, J. L. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618771003 · Cited by 51
Evidence for Implicit—But Not Unconscious—Processing of Object-Scene Relations
Biderman, N., & Mudrik, L. (2017) · Psychological Science · doi:10.1177/0956797617735745 · Cited by 59
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 50
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P., & Carneiro, C. F. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 49
“…effect (e.g. two results finding a statistically significant difference in the same direction) might not be accompanied by reproducibility of the effect size; conversely, studies with effect sizes that are similar to each other might have different outcomes in significance tests (Simonsohn, 2015).”
Seven Pervasive Statistical Flaws in Cognitive Training Interventions
Moreau, D., Kirk, I. J., & Waldie, K. E. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00153 · Cited by 61
Kochari, A. R., & Flecken, M. (2018) · Language Cognition and Neuroscience · doi:10.1080/23273798.2018.1524500 · Cited by 63
“We will follow the informal suggestion of Simonsohn (2015) to consider an effect size too small for the original study to be considered trustworthy, if the upper bound of the its confidence interval rendered the original study a power of less than 33%.”
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2269357 · Cited by 57
“Several current practices in terms of null hypothesis significance testing require the use of the original effect size for assessing the replicability of original studies (Camerer et al., 2018; Errington et al., 2021; Open Science Collaboration, 2015; Simonsohn, 2015).”
Improving the Replicability of Psychological Science Through Pedagogy
Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., Keil, M., Lampinen, A. K., Raposo, S., Reynolds, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917740427 · Cited by 48
“Instructors must balance cost with the usefulness of replication research, given that larger sample sizes will result in more accurate estimates of the replication effect size (Button et al., 2013; Simonsohn, 2015).”
InfluentialPractical Methodological Reform Needs Good Theory
Gervais, W. M. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620977471 · Cited by 56
Identity interplay: The importance and challenges of consumer research on multiple identities
Forehand, M., Reed, A., & Clair, J. K. S. (2021) · Consumer Psychology Review · doi:10.1002/arcp.1066 · Cited by 56
McKay, B., Bacelar, M. F. B., Parma, J. O., Miller, M. W., & Carter, M. J. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2207255 · Cited by 59
“Nonetheless, the fi eld of motor learning is not past asking whether self-controlled practice and enhanced expectancies have any bene fi t, so concerns about estimating the magnitude of a potential bene fi t are premature (Simonsohn, 2015).”
Embodied power, testosterone, and overconfidence as a causal pathway to risk-taking
Ronay, R., Tybur, J. M., van Huijstee, D., & Morssinkhof, M. W. L. (2016) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2016.1248081 · Cited by 49
Chatard, A., Bocage-Barthélémy, Y., Selimbegović, L., & Guimond, S. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.05.005 · Cited by 59
Statistical Inference and the Replication Crisis
Colling, L. J., & Szűcs, D. (2019) · PhilPapers (PhilPapers Foundation) · doi:10.17863/cam.35452 · Cited by 41
Investigating the replicability of the social and behavioural sciences
Tyner, A. H., Abatayo, A. L., Daley, M., Field, S., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Miske, O., et al. (2026) · Nature · doi:10.1038/s41586-025-10078-y · Cited by 60
Expanding Statistical Frontiers in Sexual Science: Taxometric, Invariance, and Equivalence Testing
Sakaluk, J. K. (2019) · The Journal of Sex Research · doi:10.1080/00224499.2019.1568377 · Cited by 43
“In terms of specifying an effect size under H1 for TOSTs and/or Bayes factors, sexual scientists can look to a number of sources for inspiration, including theory (e.g., Hyde, 2005), meta-analysis (e.g., Petersen & Hyde, 2010), and even previous individual studies (see Simonsohn, 2015).”
Good intentions aren't good enough: Moral courage in opposing sexual harassment
Goodwin, R. D., Graham, J., & Diekmann, K. A. (2019) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2019.103894 · Cited by 52
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 67
Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance
McKay, B., Hussien, J., Vinh, M.-A., Mir-Orefice, A., Brooks, H., & Ste‐Marie, D. M. (2022) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2022.102165 · Cited by 43
Memory and decision making interact to shape the value of unchosen options
Biderman, N., & Shohamy, D. (2021) · Nature Communications · doi:10.1038/s41467-021-24907-x · Cited by 49
The norm-dependent effect of watching eyes on donation
Kawamura, Y., & Kusumi, T. (2017) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2017.05.003 · Cited by 40
The Practicalities of Running Randomized Evaluations
Glennerster, R. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.002 · Cited by 56
Statistical Inference and the Replication Crisis
Colling, L. J., & Szűcs, D. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0421-4 · Cited by 62
“Interestingly, an examination of replication attempts by Simonsohn (2015) suggests that in many cases, effect size estimates obtained from high-powered replications imply that the original studies reporting those effects were underpowered and, therefore, could not have reliably studied effects of those magnitudes.”
Ghelfi, E. A., Christopherson, C. D., Urry, H. L., Lenne, R. L., Legate, N., Fischer, M. A., Wagemans, F. M. A., Wiggins, B., Barrett, T., Bornstein, M., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919881152 · Cited by 56
“The minimum target for the overall sample for meta-analysis was 2.5 times the original sample of 57, or 143 subjects, which would be sufficient for a “small telescopes” analysis (Simonsohn, 2015).”
InfluentialRöseler, L., Schütz, A., Blank, P. A., Dück, M., Fels, S., Kupfer, J., Scheelje, L., & Seida, C. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104066 · Cited by 48
“A small effect is computed as the effect size for which the original study would have had only 33% power (i.e., d33%; Simonsohn, 2015, p. 4).”
Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D., & Warne, J. P. (2022) · Sports Medicine · doi:10.1007/s40279-022-01749-1 · Cited by 46
Domachowska, I., Heitmann, C., Deutsch, R., Goschke, T., Scherbaum, S., & Bolte, A. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.003 · Cited by 44
Linking metacognition and mindreading: Evidence from autism and dual-task investigations.
Nicholson, T., Williams, D. M., Lind, S. E., Grainger, C., & Carruthers, P. (2021) · Journal of Experimental Psychology: General · doi:10.1037/xge0000878 · Cited by 42
Rethinking research reproducibility
Dirnagl, U. (2019) · The EMBO Journal · doi:10.15252/embj.2018101117 · Cited by 35
“Confirmatory studies need to be sufficiently powered, which often means substantially larger sample sizes as in the original study (Simonsohn, 2015).”
No evidence for unethical amnesia for imagined actions: A failed replication and extension
Stanley, M. L., Yang, B. W., & Brigard, F. D. (2018) · Memory & Cognition · doi:10.3758/s13421-018-0803-y · Cited by 35
Heirene, R. M. (2020) · Addiction Research & Theory · doi:10.1080/16066359.2020.1751130 · Cited by 41
“Small telescopes: Proposed by Simonsohn (2015), the Small telescopes approach labels a replication as a failure if it rejects an effect that the original study was 33% powered to detect (d33%).”
Mok, A., & Cremer, D. D. (2015) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2015.1051038 · Cited by 39
Foo, Y. Z., Rhodes, G., & Simmons, L. W. (2016) · Behavioral Ecology · doi:10.1093/beheco/arw188 · Cited by 35
Lateral Eye Movements Do Not Increase False-Memory Rates: A Failed Direct-Replication Study
van Schie, K., & Leer, A. (2019) · Clinical Psychological Science · doi:10.1177/2167702619859335 · Cited by 42
“The sample size was set at 2.5 times the original sample (as recommended by Simonsohn, 2015) because sample estimations based on published (inflated) effect sizes may be misleading (Simonsohn, Nelson, & Simmons, 2014).”
InfluentialAssessing heterogeneity and power in replications of psychological experiments.
Schauer, J. M., & Hedges, L. V. (2020) · Psychological Bulletin · doi:10.1037/bul0000232 · Cited by 41
Kim, I., Hammond, M. D., & Milfont, T. L. (2020) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2020.101547 · Cited by 39
Free to blame? Belief in free will is related to victim blaming
Genschow, O., & Vehlow, B. (2021) · Consciousness and Cognition · doi:10.1016/j.concog.2020.103074 · Cited by 46
The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 45
“The “small-telescopes” approach of Simonsohn (2015) is based on the idea of detectability in that it tests whether the effect obtained in a replication study is smaller than an effect size that the original study had only 33% power to detect.”
Enhanced or impoverished recruitment of top-down attentional control of inhibition in test anxiety
Wei, H., Beuckelaer, A. D., & Zhou, R. (2021) · Biological Psychology · doi:10.1016/j.biopsycho.2021.108070 · Cited by 35
Context Moderates Affirmation Effects on the Ethnic Achievement Gap
Protzko, J., & Aronson, J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616646426 · Cited by 39
An Introduction to Replication Research in Gifted Education: Shiny and New Is Not the Same as Useful
Makel, M. C., & Plucker, J. A. (2015) · Gifted Child Quarterly · doi:10.1177/0016986215578747 · Cited by 32
Exact replication: Foundation of science or game of chance?
Piper, S. K., Grittner, U., Rex, A., Riedel, N., Fischer, F. H., Nadon, R., Siegerink, B., & Dirnagl, U. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000188 · Cited by 36
Jekel, M., Fiedler, S., Torras, R. A., Mischkowski, D., Dorrough, A. R., & Glöckner, A. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719868149 · Cited by 39
“Students may also define effect sizes according to alternative methods such as equivalence testing (Lakens, 2017) or at least rely on useful heuristics (e.g., 2.5 times the original sample size; Simonsohn, 2015) when deciding how many participants to collect.”
InfluentialSims, T. L., Raposo, S., Bailenson, J. N., & Carstensen, L. L. (2020) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000275 · Cited by 44
Testing the generalizability of the white leadership standard in the post-Obama era
Ubaka, A., Lü, X., & Gutierrez, L. J. (2022) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101591 · Cited by 37
Targeting Next Generations to Change the Common Practice of Underpowered Research
Crutzen, R., & Peters, G.-J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01184 · Cited by 39
How Polysemy Affects Concreteness Ratings: The Case of Metaphor
Reijnierse, W. G., Burgers, C. F., Bolognesi, M., & Krennmayr, T. (2019) · Cognitive Science · doi:10.1111/cogs.12779 · Cited by 41
“…of previous findings has sufficient statistical power, it has been recommended to include at least 2.5 times the number of original observations (Simonsohn, 2015), which means that we needed at least 75 observations per word per group (non-metaphor, metaphor), leading to a total of 150…”
InfluentialBrain‐wide inferiority and equivalence tests in fMRI group analyses: Selected applications
Gerchen, M. F., Kirsch, P., & Feld, G. B. (2021) · Human Brain Mapping · doi:10.1002/hbm.25664 · Cited by 36
“…set should be used for replication of an external effect from another study, our procedures would for example allow to implement a voxel-wise small telescope approach in which one could tests for the existence of effects that an original study could have meaningfully examined (Simonsohn, 2015).”
A field experiment shows that subtle linguistic cues might not affect voter behavior
Gerber, A. S., Huber, G. A., Biggers, D. R., & Hendry, D. J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513727113 · Cited by 35
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 49
The Uneven Impact of Generative AI on Entrepreneurial PerformancePreprint
Otis, N. G., Clarke, R. P., Delecourt, S., Holtz, D., & Koning, R. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4671369 · Cited by 110
How strong is the association between social media use and false consensus?
Bunker, C. J., & Varnum, M. E. W. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106947 · Cited by 30
Moody, D. L. B., Leibel, D. K., Darden, T. M., Ashe, J. J., Waldstein, S. R., Katzel, L. I., Liu, H. B., Weng, N., Evans, M. K., & Zonderman, A. B. (2018) · Biological Psychology · doi:10.1016/j.biopsycho.2018.12.004 · Cited by 34
“Given that underpowered studies can produce biased findings (Crutzen & Peters, 2017; Simonsohn, 2015), ensuring that studies are powered adequately for the analyses being conducted is an important consideration.”
Jurchiș, R., Costea, A., Dienes, Z., Miclea, M., & Opre, A. (2020) · Journal of Experimental Psychology: General · doi:10.1037/xge0000734 · Cited by 30
Future-Orientation (as Trait and State) Promotes Reputation-Protective Choice in Moral Dilemmas
Vonasch, A. J., & Sjåstad, H. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619899257 · Cited by 38
“Inspired by the “small telescopes” approach (Simonsohn, 2015) and recent calls for large-scale internal replications in psychological research, the recruited sample was 2.5 times larger per cell as in the original studies.”
InfluentialVigué‐Guix, I., Fernández, L. M., Cuello, M. T., Ruzzoli, M., & Soto‐Faraco, S. (2020) · European Journal of Neuroscience · doi:10.1111/ejn.14931 · Cited by 36
“In a new study, it would then be reasonable to expect a minimum effect size equal to 33% of the original effect size, assuming the effect in the original study were true (Simonsohn, 2015).”
Genetic diversity in terrestrial subsurface ecosystems impacted by geological degassing
Bornemann, T. L. V., Adam, P. S., Turzynski, V., Schreiber, U., Figueroa-Gonzalez, P. A., Rahlff, J., Köster, D., Schmidt, T. C., Schunk, R., Krauthausen, B., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27783-7 · Cited by 38
The Affect Misattribution Procedure: In Search of Prejudice Effects
Teige‐Mocigemba, S., Becker, M., Sherman, J. W., Reichardt, R., & Klauer, K. C. (2017) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000364 · Cited by 33
The Weapons Identification Task: Recommendations for adequately powered research
Rivers, A. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0177857 · Cited by 34
Askelund, A. D., Wootton, R. E., Torvik, F. A., Lawn, R. B., Ask, H., Corfield, E. C., Magnus, M. C., Reichborn‐Kjennerud, T., Magnus, P. M., Andreassen, O. A., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03361-8 · Cited by 39
Siegelman, M., Blank, I., Mineroff, Z., & Fedorenko, E. G. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2019.06.003 · Cited by 36
“And small effects, especially observed in underpowered studies, are less likely to be real (e.g., Gelman and Carlin, 2014; Simonsohn, 2015; Open Science Collaboration, 2015).”
Does sleep protect memories against interference? A failure to replicate
Bailes, C. A., Caldwell, M., Wamsley, E. J., & Tucker, M. A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0220419 · Cited by 35
“The sample used in this replication (N = 97), which was more than double the sample size of the original study [36], yields statistical power = 0.”
No Evidence of Within-Domain Moral Licensing in the Environmental Domain
Urban, J., Braun Kohlová, M., & Bahník, Š. (2021) · Environment and Behavior · doi:10.1177/0013916520942604 · Cited by 26
McKay, B., & Ste‐Marie, D. M. (2020) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2020.1795056 · Cited by 32
“Using the two-one-sided-test approach (Lakens, 2017; Simonsohn, 2015), the present experiment can reject learning effect sizes as large or larger than d = .555, t(122.96) = − 1.666, p = .0492.”
InfluentialPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
New Statistical Metrics for Multisite Replication Projects
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12572 · Cited by 28
“Existing work (Andrews and Kasy, 2019; Etz and Vandekerckhove, 2016; Patil et al., 2016; Simonsohn, 2015; Verhagen and Wagenmakers, 2014) has proposed analytic approaches for a single replication of a single study or designs in which numerous original studies across a discipline or domain are each…”
InfluentialAcademy Awards Speeches Reflect Social Status, Cinematic Roles, and Winning Expectations
Markowitz, D. M. (2018) · Journal of Language and Social Psychology · doi:10.1177/0261927x17751012 · Cited by 31
“First, in the spirit of recent and continued calls for replication in the social sciences (Munafò et al., 2017; Simonsohn, 2015), this study reports a positive replication for the finding that low social status is associated with an amplified rate of self-references than high social status.”
Who Should Do Replication Labor?
Romero, F. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918803619 · Cited by 29
“Moreover, if the replication study is properly powered and preregistered and the original experiment is not, then the former is likely more informative (Simonsohn, 2015).”
Wagge, J., Baciu, C., Banas, K., Nadler, J. T., Schwarz, S., Weisberg, Y. J., IJzerman, H. R., Legate, N., & Grahe, J. E. (2019) · Collabra Psychology · doi:10.1525/collabra.177 · Cited by 34
Hopp, C., Wentzel, D., & Rose, S. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101437 · Cited by 30
The effect of animacy on structural Priming: A replication of Bock, Loebell and Morey (1992)
Chen, X., Hartsuiker, R. J., Muylle, M., Slim, M. S., & Zhang, C. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104354 · Cited by 29
The Sceptical Bayes Factor for the Assessment of Replication Success
Pawel, S., & Held, L. (2022) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12491 · Cited by 34
“Instead, a variety of statistical methods, frequentist (Simonsohn, 2015; Patil et al., 2016; Hedges and Schauer, 2019; Mathur and VanderWeele, 2020), Bayesian (Bayarri and Mayoral, 2002a,b; Verhagen and Wagenmakers, 2014; Johnson et al.”
Caton, N. R., Pearson, S. G., & Dixson, B. J. W. (2022) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2022.04.002 · Cited by 26
Balzarini, R. N., Dobson, K., Chin, K. A., & Campbell, L. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.11.003 · Cited by 27
Collins, T. P., Crawford, J. T., & Brandt, M. J. (2017) · Social Psychology · doi:10.1027/1864-9335/a000300 · Cited by 33
Of two minds or one? A registered replication of Rydell et al. (2006)
Heycke, T., Gehrmann, S. M., Haaf, J. M., & Stahl, C. (2018) · Cognition & Emotion · doi:10.1080/02699931.2018.1429389 · Cited by 30
Greener grass or sour grapes? How people value future goals after initial failure
Sjåstad, H., Baumeister, R. F., & Ent, M. R. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.103965 · Cited by 28
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
van Zwet, E. W., & Goodman, S. N. (2022) · Statistics in Medicine · doi:10.1002/sim.9406 · Cited by 31
Assessing and Improving Robustness of Psychological Research Findings in Four Steps
Nuijten, M. B. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_17 · Cited by 28
More for Us or More for Me? Social Dominance as Parochial Egoism
Halali, E., Dorfman, A., Jun, S., & Halevy, N. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617732819 · Cited by 28
Straub, E., Dames, H., Kiesel, A., & Dignath, D. (2022) · Acta Psychologica · doi:10.1016/j.actpsy.2022.103497 · Cited by 30
Improving Inferences about Null Effects with Bayes Factors and Equivalence TestsPreprint
Lakens, D., McLatchie, N. M., Isager, P. M., Scheel, A. M., & Dienes, Z. P. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qtzwr · Cited by 36
The moral foreign language effect is stable across presentation modalities
Muda, R., Pieńkosz, D., Francis, K., & Białek, M. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021820935072 · Cited by 30
Dimensions of Perception: 3D Real-Life Objects Are More Readily Detected Than Their 2D Images
Korisky, U., & Mudrik, L. (2021) · Psychological Science · doi:10.1177/09567976211010718 · Cited by 32
When you can, scale up: Large-scale study shows no effect of tDCS in an ambiguous risk-taking task
Russo, R., Twyman, P., Cooper, N. R., Fitzgerald, P. B., & Wallace, D. R. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.08.008 · Cited by 32
Balakrishnan, A., Palma, P. A., Patenaude, J., & Campbell, L. (2017) · Scientific Data · doi:10.1038/sdata.2016.120 · Cited by 26
Prommegger, B., Thatcher, J. B., Wiesche, M., & Krcmar, H. (2020) · European Journal of Information Systems · doi:10.1080/0960085x.2020.1841573 · Cited by 41
“You may find instructions for comparing replica studies to be helpful in this regard (Simonsohn, 2015).”
Grosso, S. S., Schuwerk, T., Kaltefleiter, L. J., & Sodian, B. (2018) · Infant Behavior and Development · doi:10.1016/j.infbeh.2018.09.012 · Cited by 25
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-03-23
- Volume
- 26
- Issue
- 5
- Pages
- 559-569
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2151509224
- Cited by
- 742
- References
- 52
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 59 · 2023: 67 · 2022: 64 · 2021: 66 · 2020: 111 · 2019: 79 · 2018: 76 · 2017: 63 · 2016: 48 · 2015: 16
- FWCI
- 61.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- effect size estimation, hypothesis testing, replication study, statistical power, underpowered studies, small effect sizes
- MeSH terms
- Astronomy, Data Interpretation, Statistical, Research Design, Reproducibility of Results, Models, Statistical, Sample Size, Telescopes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.