Small Telescopes: Detectability and the Evaluation of Replication Results
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our replications employed 2.5 times the sample size of the original paper (8).”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“Selective reporting of measures, treatments, and analyses, and the p-hacking they enable (7) has been identified as an existential threat to behavioral science (8, 9)”
From Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity (2021) · cites this paper as [8]
In the citing paper
“This approach limits the sample size required to test for equivalence to 2.5 times the sample size of the original study.”
From Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses (2017) · cites this paper as Simonsohn (2015)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“If the original study is powered to 50%, an optimistic scenario (Button et al., 2013; Cohen, 1962; Rossi, 1990; Sedlmeier & Gigerenzer, 1989), the resulting upward bias in effect size would be severe enough that replications with samples sizes set to obtain 80% power, given the reported effects, would on average have…”
Schwarz, N., & Clore, G. L. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.3.513 · Cited by 4,952
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Experimental Tests of the Endowment Effect and the Coase Theorem
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1990) · Journal of Political Economy · doi:10.1086/261737 · Cited by 3,194
Research works printed but not linked: 4
- Fisher R. A., 1926, Journal of the Ministry of Agriculture, 33, 503
- Jeffreys H., 1961, Theory of probability
- 10.3102/10769986008002093 · doi:10.3102/10769986008002093 · Paper page
- 10.1037/0003-066X.40.1.73 · doi:10.1037/0003-066x.40.1.73 · Paper page
Cited by
Items 301–400 of 743
Conscientiousness makes effort less frustrating
Bates, T. C. (2024) · Personality and Individual Differences · doi:10.1016/j.paid.2024.112690 · Cited by 20
Yu, J. E., & Chattopadhyay, D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642796 · Cited by 23
Lehmann, R., & Vogt, B. (2024) · Brain Informatics · doi:10.1186/s40708-024-00232-z · Cited by 21
Replicability and the Psychology of Science
Clark, C. J., Honeycutt, N., & Jussim, L. J. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_3 · Cited by 14
Replicability Across Multiple Studies
Bogomolov, M., & Heller, R. (2023) · Statistical Science · doi:10.1214/23-sts892 · Cited by 29
“Simonsohn (2015) argues that this is the case for many studies in psychology.”
Boyce, V., Mathur, M. B., & Frank, M. C. (2023) · Royal Society Open Science · doi:10.1098/rsos.231240 · Cited by 22
Is comfort food really good for the soul? A replication of Troisi and Gabriel's (2011) Study 2
Ong, L. S., IJzerman, H., & Leung, A. K. ‐. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00314 · Cited by 14
Is narcissism associated with baseline cortisol in men and women?
Wardecker, B. M., Chopik, W. J., LaBelle, O. P., & Edelstein, R. S. (2016) · Journal of Research in Personality · doi:10.1016/j.jrp.2016.07.006 · Cited by 15
Does study duration have opposite effects on recognition and repetition priming?
Berry, C. J., Ward, E. V., & Shanks, D. R. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2017.07.004 · Cited by 16
Jacobson, E. D., & Simpson, A. M. (2018) · Mathematics Education Research Journal · doi:10.1007/s13394-018-0242-x · Cited by 14
Trafimow, D. (2019) · Journal for the Theory of Social Behaviour · doi:10.1111/jtsb.12211 · Cited by 12
Smartphone bans and workplace performance
Chadi, A., Mechtel, M., & Mertins, V. (2022) · Experimental Economics · doi:10.1007/s10683-021-09715-w · Cited by 19
“An alternative explanation could be that our field experiment lacks stastical power to detect effects (for discussions, see, e.g., Maniadis et al. 2014; Simonsohn 2015), which might be particularly relevant for the number of conducted interviews.”
Blain, S. D., Taylor, S. F., Lasagna, C. A., Angstadt, M., Rutherford, S. E., Peltier, S., Diwadkar, V. A., & Tso, I. F. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.08.004 · Cited by 20
Thron, M., Wöll, A., Doller, L., Quittmann, O. J., Härtel, S., Ruf, L., & Altmann, S. (2024) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000004801 · Cited by 14
Brohmer, H., Hofer, G., Bauch, S. A., Beitner, J., Berkessel, J. B., Corcoran, K., García, D., Gruber, F. M., Giuliani, F., Jauk, E., et al. (2024) · International Review of Social Psychology · doi:10.5334/irsp.522 · Cited by 22
The “Next” Effect: When a Better Future Worsens the Present
O’Brien, E. (2021) · Social Psychological and Personality Science · doi:10.1177/19485506211034972 · Cited by 13
“How- ever, because enjoyment can be surprisingly difficult to influence (O’Brien & Roney, 2017; O’Brien & Smith, 2019), we predetermined *250/cell (2.5 this number: Simonsohn, 2015) for Experiments 1 and 3, which have relatively scalable designs.”
InfluentialFisher, M., & Mormann, M. M. (2022) · Journal of Consumer Research · doi:10.1093/jcr/ucac006 · Cited by 12
Ben-Arye, R. C., & Halali, E. (2023) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2023.102193 · Cited by 14
DeVoe, S. E., & House, J. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.11.004 · Cited by 12
Lavallee, J. P., Gíusto, B. D., & Yu, T. (2019) · Climatic Change · doi:10.1007/s10584-019-02402-z · Cited by 14
“Current best practice in terms of replications call for a sample size of at least 2.5 times that of the original study (Simonsohn 2015).”
InfluentialBarlev, M., Mermelstein, S., Cohen, A. S., & German, T. C. (2019) · Cognitive Science · doi:10.1111/cogs.12784 · Cited by 14
“…and/or effect sizes, and difficulties of interpreting replications due to differences in power between the original and replication experiments (Simonsohn, 2015; Verhagen & Wagenmakers, 2014), we complemented the previous two criteria to evaluating replication success by an Evidence Updating…”
“Need” personality constructs and preferences for different types of self-relevant feedback
Vaughan‐Johnston, T. I., & Jacobson, J. A. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109671 · Cited by 15
Anecdotes impact medical decisions even when presented with statistical information or decision aids
Line, E. N., Jaramillo, S., Goldwater, M. B., & Horne, Z. (2024) · Cognitive Research Principles and Implications · doi:10.1186/s41235-024-00577-3 · Cited by 16
“…confirmatory studies), based on the mean effect size observed in meta-meta-analyses of typical social psychological effects (Richard et al., 2003) and observations about differentiating genuine replication failure from replication failure due to insufficient statistical power (Simonsohn, 2015).”
Dufner, M., Brümmer, M., Chung, J. M., Drewke, P. M., Blaison, C., & Schmukle, S. C. (2017) · Psychological Science · doi:10.1177/0956797617734315 · Cited by 12
da Silva, M. R. D., Rusz, D., & Postma-Nilsenová, M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207578 · Cited by 14
“This sample size was sufficient for detecting the smallest effect size of interest (SESOI) according to Simonsohn (2015)’s [79] recommendation to set the smallest effect size of interest so that the original experiment had 33% power to reject the null hypothesis if this effect size was true.”
Cohen, A. S., Chun, R., & Sznycer, D. (2020) · Royal Society Open Science · doi:10.1098/rsos.191922 · Cited by 20
The role of memory in counterfactual valuation.
Biderman, N., Gershman, S. J., & Shohamy, D. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001364 · Cited by 16
Lee, J. B., Thompson, K. M. A., Teixeira, A. L. D. S., Burr, J. F., & Millar, P. J. (2023) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00775.2022 · Cited by 13
Adaptive scaling of reward in episodic memory: a replication study
Mason, A., Ludwig, C. J. H., & Farrell, S. (2016) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2016.1233439 · Cited by 14
Goldschmied, N. P., Sheptock, M., Kim, K., & Galily, Y. (2016) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2016.1237980 · Cited by 13
The role of trust in the social heuristics hypothesis
Montealegre, A., & Jiménez‐Leal, W. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216329 · Cited by 12
“, setting the smallest effect size of interest to the effect that a study had 33% power to detect) [31, 34].”
Krueger, J. I., & Heck, P. R. (2018) · Collabra Psychology · doi:10.1525/collabra.108 · Cited by 12
Habituation does not rescue depletion: Two tests of the ego-depletion effect
Alós‐Ferrer, C., Ritschel, A., García-Segarra, J., & Achtziger, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.11.001 · Cited by 12
Sexual unfaithfulness can be judged with some accuracy from men's but not women's faces
Foo, Y. Z., Loncarevic, A., Simmons, L. W., Sutherland, C. A. M., & Rhodes, G. (2019) · Royal Society Open Science · doi:10.1098/rsos.181552 · Cited by 12
Realizing the promises of replication studies in education
Morrison, K. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2020.1838300 · Cited by 15
No evidence for binding of items to task-irrelevant backgrounds in visual working memory
Udale, R. C., Farrell, S., & Kent, C. D. (2017) · Memory & Cognition · doi:10.3758/s13421-017-0727-y · Cited by 11
Replicating different roles of intent across moral domains
Sweetman, J., & Newman, G. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190808 · Cited by 11
Learning disorders and difficulties: From a categorical to a dimensional perspective
Caviola, S., Greiff, S., & Toffalini, E. (2024) · Learning and Individual Differences · doi:10.1016/j.lindif.2024.102490 · Cited by 18
Otterbring, T., Festila, A., & Folwarczny, M. (2021) · Current Research in Ecological and Social Psychology · doi:10.1016/j.cresp.2021.100023 · Cited by 15
“Moderation analyses typially require a larger sample size, but given our conservative approach, ssuming a relatively low probability of Type II errors, quadrupling the ample size required to detect a main effect should be sufficient (cf. erugini et al., 2018 ; Simonsohn, 2015 ).”
Do Experimental Manipulations of Pathogen Avoidance Motivations Influence Conformity?
van Leeuwen, F., Jaeger, B., Sleegers, W. W. A., & Petersen, M. B. (2023) · Personality and Social Psychology Bulletin · doi:10.1177/01461672231160655 · Cited by 12
“This sample size would also allow for a small telescopes analysis (Simonsohn, 2015), which in this case would require a total sample of 358.”
Coppens, L. C., Verkoeijen, P. P. J. L., Bouwmeester, S., & Rikers, R. M. J. P. (2016) · BMC Psychology · doi:10.1186/s40359-016-0127-2 · Cited by 16
Cognitive consequences of formal clothing: the effects of clothing versus thinking of clothing
Burger, A. M., & Bless, H. (2017) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2017.1396185 · Cited by 11
Sunami, N., Nadzan, M. A., & Jaremka, L. M. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618762301 · Cited by 14
An examination of the sequential trial effect on experiences of agency in the Simon task
Wang, Y., Damen, T. G. E., & Aarts, H. (2018) · Consciousness and Cognition · doi:10.1016/j.concog.2018.10.005 · Cited by 10
“Based on previous recommendations, we aimed to include at least 2.5 times more subjects than in the first study (Simonsohn, 2015).”
InfluentialOrder processing of number symbols is influenced by direction, but not format
Wong, B., Bull, R., Ansari, D., Watson, D. M., & Liem, G. A. D. (2021) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218211026800 · Cited by 12
Internal Meta-Analysis Makes False-Positives Easier To Produce and Harder To CorrectPreprint
Vosgerau, J., Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3271372 · Cited by 12
Self-citationGeological degassing enhances microbial metabolism in the continental subsurfacePreprint
Bornemann, T. L. V., Adam, P. S., Turzynski, V., Schreiber, U., Figueroa-Gonzalez, P. A., Rahlff, J., Köster, D., Schmidt, T. C., Schunk, R., Krauthausen, B., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.980714 · Cited by 10
A Bayes Factor for Replications of ANOVA ResultsPreprint
Harms, C. (2018) · OSF Preprints (OSF Preprints) · Cited by 9
Ghelfi, E. A., Christopherson, C. D., Urry, H. L., Lenne, R. L., Legate, N., Fischer, M. A., Wagemans, F. M. A., Wiggins, B. J., Barrett, T., Bornstein, M., et al. (2018) · doi:10.31234/osf.io/349pk · Cited by 10
Open Science in Applied Linguistics: An Introduction to MetasciencePreprint
Al‐Hoorie, A. H., & Hiver, P. V. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/76dak · Cited by 12
Sitting in Judgment: How Body Posture Influences Deception Detection and Gazing Behavior
Zloteanu, M., Krumhuber, E. G., & Richardson, D. C. (2021) · Behavioral Sciences · doi:10.3390/bs11060085 · Cited by 13
Evaluating replicability of ten influential research on sensory marketing
Motoki, K., & Iseki, S. (2022) · Frontiers in Communication · doi:10.3389/fcomm.2022.1048896 · Cited by 8
“The sample size was determined to be at least 2.5 times the sample size of any original paper (Simonsohn, 2015; O’Donnell et al., 2021).”
InfluentialGurunandan, K., Cooper, E., Tibon, R., Henson, R. N., & Greve, A. (2023) · Memory · doi:10.1080/09658211.2023.2262188 · Cited by 9
“…that one would generally need 2.5 times their sample size (i.e., 62 per group) to have an 80% chance of concluding that the e ff ect is not “ unde-tectably small ” (Simonsohn, 2015), which would correspond to a Cohens d = 0.35 for 33% power with Coutanche and Thompson-Schill ’ s N = 25 sample size.”
InfluentialReplication Success Under Questionable Research Practices—a Simulation Study
Freuli, F., Held, L., & Heyard, R. (2023) · Statistical Science · doi:10.1214/23-sts904 · Cited by 20
“This criterion however ignores the effect size of the original and the replication studies and has other shortcomings [Simonsohn, 2015].”
Do red objects enhance sexual attractiveness? No evidence from two large replications
Pollet, T. V., Costello, J., Groeneboom, L., Peperkoorn, L. S., & Wu, J. (2018) · Displays · doi:10.1016/j.displa.2018.10.008 · Cited by 9
Opitz, T., Schuwerk, T., Paulus, M., Kloo, D., Osterhaus, C., Lesch, K., & Sodian, B. (2021) · Developmental Science · doi:10.1111/desc.13100 · Cited by 9
Świątkowski, W., & Dompnier, B. (2021) · Social Psychology of Education · doi:10.1007/s11218-021-09623-0 · Cited by 9
“The number of participants for Samples 1 and 2 were determined to conveniently detect the effect size of performance goals’ effects on achievement based on Murayama and Elliot (2012; Study 3): we almost tripled the sample size of the original study (Simonsohn, 2015).”
Balzus, L., Klawohn, J., Elsner, B., Schmidt, S. H., Brandt, S. A., & Kathmann, N. (2022) · NeuroImage Clinical · doi:10.1016/j.nicl.2022.103113 · Cited by 14
“As recommended by Simonsohn (2015) for studies building on previous work, we defined the SESOI as the effect size that the study by Reinhart and Woodman (2014) had 33% power to detect.”
Mah, E. Y., Grannon, K. E. L., Campbell, A. C., Tamburri, N., Jamieson, R. K., & Lindsay, D. S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1146200 · Cited by 11
Zhu, M., & Feldman, G. (2023) · Collabra Psychology · doi:10.1525/collabra.77608 · Cited by 11
Sorokowski, P., Kowal, M., Hussain, S., Haideri, R. A., Misiak, M., Chatzipentidis, K., Mahmut, M. K., Małecki, W., Dąbrowski, J., Frąckowiak, T., et al. (2023) · Ethics & Behavior · doi:10.1080/10508422.2023.2248327 · Cited by 13
“Therefore, studying moral judgment across cultures is essential to fully understand whether the effect hypothesized above is specific to a particular culture or more universal, especially in light of the current replication crisis in psychology (Maxwell et al., 2015; Simonsohn, 2015).”
Communication increases collaborative corruption
Tønnesen, M. T., Elbæk, C. T., Pfattheicher, S., & Mitkidis, P. (2024) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2024.104603 · Cited by 11
“We follow Simonsohn (2015), and define a small effect as the effect that would give the original study a power of 33% ( d 33% ).”
Task demands determine comparison strategy in whole probe change detection.
Udale, R. C., Farrell, S., & Kent, C. D. (2018) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0000490 · Cited by 10
Moving beyond unwise replication practices: The case of romantic motivation.
Sundie, J. M., Beal, D. J., Neuberg, S. L., & Kenrick, D. T. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000527 · Cited by 10
A Brief Social-Belonging Intervention in the Workplace: Evidence from a Field Experiment
Mobasseri, S., Srivastava, S. B., & Kray, L. J. (2019) · Academy of Management Discoveries · doi:10.5465/amd.2018.0115 · Cited by 13
Gyurkovics, M., Kovács, M., Jaquiery, M., Pálfi, B., Děchtěrenko, F., & Aczél, B. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-020-02021-2 · Cited by 9
“In our replication, target N per task was 2.5 times that of the original target (2.5 × 50 = 125), following the guidelines suggested by Simonsohn (2015).”
InfluentialNeves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., de Abreu, M. B., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · Memórias do Instituto Oswaldo Cruz · doi:10.1590/0074-02760200328 · Cited by 11
Huang, Y., Shen, S., Yang, S., Kuang, Y., Li, Y.-X., & Li, S. (2021) · Symmetry · doi:10.3390/sym13101928 · Cited by 9
Scheifele, C., Steffens, M. C., & van Laar, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260950 · Cited by 12
Testing ANOVA Replications by Means of the Prior Predictive p-Value
Zondervan-Zwijnenburg, M. A. J., van de Schoot, R., & Hoijtink, H. (2022) · Meta-Psychology · doi:10.15626/mp.2019.1615 · Cited by 10
Visual Attention and Poor Sleep Quality
Abdolalizadeh, A., & Nabavi, S. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.850372 · Cited by 11
“The second approach is proposed by Simonsohn (2015) based on the effect sizes of a prior study.”
Shen, S., Huang, Y., Jiang, C., & Li, S. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500003442 · Cited by 11
How does language affect spatial attention? Deconstructing the prime-target relationship
Shaki, S., & Fischer, M. H. (2023) · Memory & Cognition · doi:10.3758/s13421-022-01390-3 · Cited by 10
“In light of frequent failures to replicate the spatial interference effect (Petrova et al., 2018) we aimed to test 80 participants in each of the main experiments (cf. Simonsohn, 2015).”
On the reliability of retrieval-induced forgetting
Rowland, C. A., Bates, L. E., & DeLosh, E. L. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01343 · Cited by 39
Inconsistent Effect of Arousal on Early Auditory Perception
Bolders, A. C., Band, G. P. H., & Stallen, P. J. M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00447 · Cited by 9
Mayiwar, L., & Lai, L. M. H. (2019) · Social Psychology · doi:10.1027/1864-9335/a000388 · Cited by 9
Design-Based Approaches to Causal Replication Studies
Wong, V. C., Anglin, K., & Steiner, P. M. (2021) · Prevention Science · doi:10.1007/s11121-021-01234-7 · Cited by 18
“…Robustness checks ✓ Independent reproducibility Yes Yes Yes Yes 1 3 et al., 2017; Anderson & Maxwell, 2017; Hedges & Schauer, 2019; Schauer & Hedges, 2020; Simonsohn, 2015), there are likely design-based approaches to replication that may be effective for addressing these…”
Schütt, E., Mackenzie, I. G., Kaup, B., & Dudschig, C. (2022) · Psychological Research · doi:10.1007/s00426-022-01650-6 · Cited by 12
“Given that primary studies in behavioral research tend to overestimate effect sizes (Fanelli & Ioannidis, 2013), we then also considered the advice of Simonsohn (2015) that replications should have 2.5 times as many observations as the original study.”
InfluentialSun, Y., Shen, L., Pan, Z., & Qian, S. (2025) · Communication Research · doi:10.1177/00936502241308599 · Cited by 10
“Additionally, replications of underpowered studies may also suffer from insufficient power, as sample size planning is often based on published effect sizes, which are likely biased (Simonsohn, 2015).”
Kleinberg, B., Warmelink, L., Arntz, A. R., & Verschuère, B. (2018) · Applied Cognitive Psychology · doi:10.1002/acp.3439 · Cited by 7
“54) at face value can be misleading because most published effect sizes are overestimations of the true effect (Gelman & Carlin, 2014; Simonsohn, 2015).”
InfluentialResearch Replication Prediction Using Weakly Supervised Learning
Luo, T., Li, X., Wang, H., & Liu, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.132 · Cited by 12
An evaluation of statistical methods for aggregate patterns of replication failure
Schauer, J. M., Fitzgerald, K. G., Peko-Spicer, S., Whalen, M. C. R., Zejnullahi, R., & Hedges, L. V. (2021) · The Annals of Applied Statistics · doi:10.1214/20-aoas1387 · Cited by 8
Kaltefleiter, L. J., Schuwerk, T., Wiesmann, C. G., Kristen, S., Jarvers, I., & Sodian, B. (2022) · Developmental Science · doi:10.1111/desc.13224 · Cited by 12
Han, E. J., Choi, J. S., & Na, J. (2023) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2023.104477 · Cited by 12
Louvet, E., Ehrke, F., Gaubert, C., & Tran, H. L. (2023) · British Journal of Social Psychology · doi:10.1111/bjso.12664 · Cited by 13
Increasing the Power of Two-Sample T-tests in Health Psychology Using a Compositional Data Approach
Lehmann, R., & Vogt, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43075-6_29 · Cited by 8
Stahl, C., Béna, J., Aust, F., Mierop, A., & Corneille, O. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02081-w · Cited by 5
“We collected a sample of N = 60 participants (students at UCLouvain), as this is approximately 2x the sample size recruited by H¨utter et al. (2012) (as recommended by the rule of thumb for replications by Simonsohn, 2015).”
The Effect of Brooding About Societal Problems on Conspiracy Beliefs: A Registered Report
Liekefett, L., Sebben, S., & Becker, J. C. (2024) · Collabra Psychology · doi:10.1525/collabra.92995 · Cited by 9
Modification of episodic memories by novel learning: a failed replication study
van Schie, K., van Veen, S. C., van den Hout, M., & Engelhard, I. M. (2017) · European Journal of Psychotraumatology · doi:10.1080/20008198.2017.1315291 · Cited by 9
Die New Statistics in der Psychologie: Status quo und Zukunft der Datenanalyse
Schäfer, T. (2018) · Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie · doi:10.1026/0049-8637/a000184 · Cited by 8
Are affective factors related to individual differences in facial expression recognition?
Alharbi, S. A. H., Button, K. S., Zhang, L., O’Shea, K. J., Fasolt, V., Lee, A. J., DeBruine, L. M., & Jones, B. C. (2020) · Royal Society Open Science · doi:10.1098/rsos.190699 · Cited by 8
“Simonsohn [10] recommends that sample sizes in replication studies be approximately two and a half times the sample size of the original study.”
Schulze, C., & Buttelmann, D. (2021) · Cognitive Development · doi:10.1016/j.cogdev.2021.101054 · Cited by 9
Ciria, L. F., Quintero, M. J., López, F. J., Luque, D., Cobos, P. L., & Morís, J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256210 · Cited by 10
“According to Simonsohn’s proposal, the target effect size would be the effect size that can be detected with a power of 33% in the original study.”
Theoretical false positive psychology
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02098-w · Cited by 13
“Moreover, because both effect sizes are similar despite being small, the original effect is also validated from the “small telescopes” perspective (Simonsohn, 2015).”
Miyagawa, Y. (2022) · Personality and Individual Differences · doi:10.1016/j.paid.2022.111954 · Cited by 11
McLean, C., Want, S. C., & Dyson, B. J. (2015) · Cognition · doi:10.1016/j.cognition.2015.01.002 · Cited by 8
Philipp-Muller, A. Z., & MacDonald, G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.11.010 · Cited by 9
Probability matching does not decrease under cognitive load: A preregistered failure to replicate
Schulze, C., James, G. M., Koehler, D. J., & Newell, B. R. (2019) · Memory & Cognition · doi:10.3758/s13421-018-0888-3 · Cited by 10
“5 times that of the original study (Simonsohn, 2015) and, based on estimation of the original effect size, suggests a power of .”
InfluentialInteractive effects of power and donation target on charitable giving
Wang, Y., Feng, H., Qiu, S., & Cui, L. (2020) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2223 · Cited by 8
Katzir, M., Cohen, S., & Halali, E. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104192 · Cited by 10
Metadata
- Authors
- Simonsohn, U.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Simonsohn, Uri
- ORCID
- Simonsohn, Uri
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-03-23
- Volume
- 26
- Issue
- 5
- Pages
- 559-569
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2151509224
- Cited by
- 742
- References
- 52
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 59 · 2023: 67 · 2022: 64 · 2021: 66 · 2020: 111 · 2019: 79 · 2018: 76 · 2017: 63 · 2016: 48 · 2015: 16
- FWCI
- 61.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- effect size estimation, hypothesis testing, replication study, statistical power, underpowered studies, small effect sizes
- MeSH terms
- Astronomy, Data Interpretation, Statistical, Research Design, Reproducibility of Results, Models, Statistical, Sample Size, Telescopes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.