The Value of Direct Replication
Abstract
Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Almost none of the findings in the intranasal oxytocin literature underwent direct replication attempts.
“almost none of the findings in the literature underwent direct replication attempts”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [26]
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 703 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · see citing works
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597 · see citing works
Priming, Replication, and the Hardest Science
Cesario, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513470 · Cited by 320 · see citing works
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · see citing works
Life after P-HackingPreprint
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2205186 · Cited by 210 · see citing works
The importance of negative results in psychological research.
Smart, R. (1964) · The Canadian Psychologist · doi:10.1037/h0083036 · Cited by 156 · see citing works
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101 · see citing works
An Intersensory Interaction Account of Priming Effects—and Their Absence
Klatzky, R. L., & Creswell, J. D. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513468 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986 · see citing works
Self-citationReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Ferguson, C. J. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615592234 · Cited by 524 · see citing works
Applications of transcranial direct current stimulation for understanding brain function
Filmer, H. L., Dux, P. E., & Mattingley, J. B. (2014) · Trends in Neurosciences · doi:10.1016/j.tins.2014.08.003 · Cited by 519 · see citing works
Valian, V. V. (2014) · Bilingualism Language and Cognition · doi:10.1017/s1366728914000522 · Cited by 469 · see citing works
Insufficient effort responding: Examining an insidious confound in survey data.
Huang, J. L. W., Liu, M., & Bowling, N. A. (2014) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 421 · see citing works
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918797607 · Cited by 377 · see citing works
Replicating patterns of prospect theory for decision under risk
Ruggeri, K., Alì, S., Berge, M. L., Bertoldo, G., Bjørndal, L. D., Cortijos‐Bernabeu, A., Davison, C., Demić, E., Serna, C. E., Friedemann, M., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0886-x · Cited by 329 · see citing works
Does Oxytocin Increase Trust in Humans? A Critical Review of Research
Nave, G., Camerer, C., & McCullough, M. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615600138 · Cited by 309 · see citing works
Green consumerism: The influence of product attributes and values on purchasing intentions
Schuitema, G., & de Groot, J. I. M. (2014) · Journal of Consumer Behaviour · doi:10.1002/cb.1501 · Cited by 261 · see citing works
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251 · see citing works
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra Psychology · doi:10.1525/collabra.158 · Cited by 233 · see citing works
Musical Preferences Predict Personality: Evidence From Active Listening and Facebook Likes
Nave, G., Minxha, J., Greenberg, D. M., Kosiński, M., Stillwell, D. J., & Rentfrow, P. J. (2018) · Psychological Science · doi:10.1177/0956797618761659 · Cited by 209 · see citing works
Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences
Köhler, T., & Cortina, J. M. (2019) · Journal of Management · doi:10.1177/0149206319843985 · Cited by 204 · see citing works
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 201 · see citing works
The case for formal methodology in scientific reform
Devezer, B., Navarro, D., Vandekerckhove, J., & Buzbas, E. O. (2021) · Royal Society Open Science · doi:10.1098/rsos.200805 · Cited by 195 · see citing works
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187 · see citing works
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 185 · see citing works
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 182 · see citing works
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · see citing works
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167 · see citing works
Conceptualizing and evaluating the replication of research results
Fabrigar, L. R., & Wegener, D. T. (2015) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.009 · Cited by 168 · see citing works
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 168 · see citing works
Strategic Intelligence: The Cognitive Capability to Anticipate Competitor Behavior
Levine, S. S., Bernard, M. M., & Nagel, R. (2017) · Strategic Management Journal · doi:10.1002/smj.2660 · Cited by 156 · see citing works
Construct validity and the validity of replication studies: A systematic review.
Flake, J. K., Davidson, I. J., Wong, O. C. Y., & Pek, J. C. X. (2022) · American Psychologist · doi:10.1037/amp0001006 · Cited by 154 · see citing works
Linden, A. H., & Hönekopp, J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964193 · Cited by 148 · see citing works
Lane, A., Luminet, O., Nave, G., & Mikolajczak, M. (2016) · Journal of Neuroendocrinology · doi:10.1111/jne.12384 · Cited by 134 · see citing works
On Replication in Communication Science
McEwan, B. L., Carpenter, C. J., & Westerman, D. (2018) · Communication Studies · doi:10.1080/10510974.2018.1464938 · Cited by 134 · see citing works
Anderson, S. F., & Maxwell, S. E. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 133 · see citing works
Strengthening the Practice of Exercise and Sport-Science Research
Halperin, I., Vigotsky, A. D., Foster, C., & Pyne, D. B. (2017) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2017-0322 · Cited by 141 · see citing works
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 122 · see citing works
Reflections on the replication corner: In praise of conceptual replications
Lynch, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2015.09.006 · Cited by 122 · see citing works
No Replication, No Trust? How Low Replicability Influences Trust in Psychology
Wingen, T., Berkessel, J. B., & Englich, B. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619877412 · Cited by 121 · see citing works
Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli
Westfall, J., Judd, C. M., & Kenny, D. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614564879 · Cited by 122 · see citing works
Increasing clarity where it is needed most: articulating and evaluating theoretical contributions
DeAndrea, D. C., & Holbert, R. L. (2017) · Annals of the International Communication Association · doi:10.1080/23808985.2017.1304163 · Cited by 119 · see citing works
Recommendations for Replication Research in Special Education
Coyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 117 · see citing works
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G., & Clarke, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 106 · see citing works
On the Other Side of the Mirror: Priming in Cognitive and Social Psychology
Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014) · Social Cognition · doi:10.1521/soco.2014.32.supp.12 · Cited by 112 · see citing works
Self-citationDisfluent fonts don’t help people solve math problems.
Meyer, A., Frederick, S., Burnham, T. C., Pinto, J. D. G., Boyer, T. W., Ball, L. J., Pennycook, G., Ackerman, R., Thompson, V. A., & Schuldt, J. P. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000049 · Cited by 109 · see citing works
Morehead, K., Dunlosky, J. T., & Rawson, K. A. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09468-2 · Cited by 109 · see citing works
Schaerer, M., du Plessis, C., Yap, A. J., & Thau, S. (2018) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.08.004 · Cited by 105 · see citing works
Definitely maybe: can unconscious processes perform the same functions as conscious processes?
Hesselmann, G., & Moors, P. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00584 · Cited by 101 · see citing works
Why did I do that? Explaining actions activated outside of awareness
Gantman, A. P., Adriaanse, M. A., Gollwitzer, P. M., & Oettingen, G. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1260-5 · Cited by 97 · see citing works
Predicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225826 · Cited by 89 · see citing works
Machery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 93 · see citing works
Sanchis‐Segura, C., Ibáñez, M. V., Aguirre, N., Cruz-Gómez, Á. J., & Forn, C. (2020) · Scientific Reports · doi:10.1038/s41598-020-69361-9 · Cited by 92 · see citing works
Arrested Theory Development: The Misguided Distinction Between Exploratory and Confirmatory Research
Szollosi, A., & Donkin, C. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966796 · Cited by 95 · see citing works
Fitzsimmons, S. R., Liao, Y., & Thomas, D. C. (2016) · Journal of International Business Studies · doi:10.1057/s41267-016-0053-9 · Cited by 95 · see citing works
Understanding Priming Effects in Social Psychology: An Overview and Integration
Molden, D. C. (2014) · Social Cognition · doi:10.1521/soco.2014.32.supp.243 · Cited by 93 · see citing works
A Validity-Based Framework for Understanding Replication in Psychology
Fabrigar, L. R., Wegener, D. T., & Petty, R. E. (2020) · Personality and Social Psychology Review · doi:10.1177/1088868320931366 · Cited by 94 · see citing works
Mas, J. M., Dunst, C. J., Balcells‐Balcells, A., García‐Ventura, S., Giné, C. G., & Cañadas, M. (2019) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2019.103495 · Cited by 86 · see citing works
Postretrieval new learning does not reliably induce human memory updating via reconsolidation
Hardwicke, T. E., Taqi, M., & Shanks, D. R. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1601440113 · Cited by 83 · see citing works
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 86 · see citing works
Schaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87 · see citing works
Keating, D. M., & Totzkay, D. (2019) · Annals of the International Communication Association · doi:10.1080/23808985.2019.1632218 · Cited by 74 · see citing works
Backlash against gender stereotype-violating preschool children
Sullivan, J. L., Moss-Racusin, C. A., Lopez, M. J., & Williams, K. N. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195503 · Cited by 79 · see citing works
From Repeatability to Reproducibility and Corroboration
Feitelson, D. G. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723875 · Cited by 75 · see citing works
Bonin, P., Méot, A., & Bugaïska, A. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1014-y · Cited by 77 · see citing works
Roelofs, C. L., Krepel, N., Corlier, J., Carpenter, L. L., Fitzgerald, P. B., Daskalakis, Z. J., Tendolkar, I., Wilson, A. C., Downar, J. H., Bailey, N. W., et al. (2020) · Clinical Neurophysiology · doi:10.1016/j.clinph.2020.10.017 · Cited by 75 · see citing works
Smeekes, A. (2015) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2015.07.001 · Cited by 74 · see citing works
Hendriks, F., Kienhues, D., & Bromme, R. (2020) · Public Understanding of Science · doi:10.1177/0963662520902383 · Cited by 73 · see citing works
Guill, K., Lüdtke, O., & Köller, O. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2016.10.001 · Cited by 65 · see citing works
Dodge, K. A., Goodman, W. B., Bai, Y., O’Donnell, K., & Murphy, R. A. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.14522 · Cited by 72 · see citing works
Do subtle reminders of money change people’s political views?
Rohrer, D., Pashler, H., & Harris, C. R. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000058 · Cited by 72 · see citing works
The malleability of developmental trends in neutral and negative memory illusions.
Otgaar, H., Howe, M. L., Brackmann, N., & Smeets, T. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000127 · Cited by 71 · see citing works
Verkuyten, M., Kollar, R., Gale, J., & Yogeeswaran, K. (2021) · Personality and Individual Differences · doi:10.1016/j.paid.2021.111217 · Cited by 69 · see citing works
The replication crisis: How might philosophy and theory of psychology be of use?
Morawski, J. G. (2019) · Journal of Theoretical and Philosophical Psychology · doi:10.1037/teo0000129 · Cited by 70 · see citing works
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R., & Xu, M.-R. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 64 · see citing works
Morgan‐Short, K., Marsden, E. J., Heil, J., Issa, B. I., Leow, R. P., Mikhaylova, A., Mikołajczak, S., Moreno, N., Slabakova, R., & Szudarski, P. (2018) · Language Learning · doi:10.1111/lang.12292 · Cited by 68 · see citing works
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 57 · see citing works
Liu, Y., & Dewitte, S. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2021.102472 · Cited by 65 · see citing works
Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives?
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. F., & Puhlmann, L. M. C. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000116 · Cited by 65 · see citing works
The role and value of replication in empirical software engineering results
Shepperd, M., Ajienka, N., & Counsell, S. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.006 · Cited by 65 · see citing works
Rawson, K. A., Vaughn, K. E., & Carpenter, S. K. (2014) · Memory & Cognition · doi:10.3758/s13421-014-0477-z · Cited by 63 · see citing works
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 60 · see citing works
Benefits of Open and High-Powered Research Outweigh Costs
LeBel, E. P., Campbell, L., & Loving, T. J. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000049 · Cited by 63 · see citing works
Failed Replication of Oxytocin Effects on Trust: The Envelope Task Case
Lane, A., Mikolajczak, M., Treinen, E., Samson, D., Corneille, O., de Timary, P., & Luminet, O. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137000 · Cited by 61 · see citing works
“Replication studies add great value to science [34; 35; 36] and are crucially needed in the OT research field if we want to dispose of a solid theoretical background for interpreting the significant and non-significant effects of OT.”
BackgroundMcGonagle, A. K., Huang, J. L. W., & Walsh, B. M. (2015) · Applied Psychology · doi:10.1111/apps.12058 · Cited by 61 · see citing works
Increasing propensity to mind‐wander by transcranial direct current stimulation? A registered report
Boayue, N. M., Csifcsák, G., Aslaksen, P. M., Turi, Z., Antal, A., Groot, J. M., Hawkins, G. E., Forstmann, B. U., Opitz, A., Thielscher, A., et al. (2019) · European Journal of Neuroscience · doi:10.1111/ejn.14347 · Cited by 61 · see citing works
Körner, U., Röer, J. P., Buchner, A., & Bell, R. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2017.05.005 · Cited by 60 · see citing works
Flake, J. K. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898962 · Cited by 58 · see citing works
Strategy mediation in working memory training in younger and older adults
Forsberg, A., Fellman, D., Laine, M., Johnson, W., & Logie, R. H. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021820915107 · Cited by 58 · see citing works
Azaryahu, L., Courey, S. J., Elkoshi, R., & Adi‐Japha, E. (2019) · Developmental Science · doi:10.1111/desc.12882 · Cited by 58 · see citing works
Injunctive Norms and Alcohol Consumption: A Revised Conceptualization
Krieger, H., Neighbors, C., Lewis, M. A., LaBrie, J. W., Foster, D. W., & Larimer, M. E. (2016) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.13037 · Cited by 58 · see citing works
Better together: Advancing life course research through multi-cohort analytic approaches
O’Connor, M., Spry, E., Patton, G. C., Moreno‐Betancur, M., Arnup, S. J., Downes, M., Goldfeld, S. R., Burgner, D. P., & Olsson, C. A. (2022) · Advances in Life Course Research · doi:10.1016/j.alcr.2022.100499 · Cited by 53 · see citing works
Theory Building, Replication, and Behavioral Priming
Locke, E. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691614567231 · Cited by 58 · see citing works
Open Science Promotes Diverse, Just, and Sustainable Research and Educational Outcomes
Grahe, J. E., Cuccolo, K., Leighton, D. C., & Alvarez, L. D. C. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719869164 · Cited by 58 · see citing works
Science is not a signal detection problem
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914237117 · Cited by 54 · see citing works
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works
Roelle, J., & Renkl, A. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000365 · Cited by 58 · see citing works
Starr, L. R., Hershenberg, R., Shaw, Z. A., Li, Y. I., & Santee, A. C. (2019) · Emotion · doi:10.1037/emo0000630 · Cited by 55 · see citing works
Weisz, J. R., Bearman, S. K., Ugueto, A. M., Herren, J., Evans, S. C., Cheron, D. M., Alleyne, A., Weissman, A. S., Tweed, J. L., Pollack, A. A., et al. (2019) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2019.1655757 · Cited by 57 · see citing works
Güttinger, S. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-019-0269-1 · Cited by 60 · see citing works
Siegel, J. A., Huellemann, K. L., Hillier, C. C., & Campbell, L. (2019) · Body Image · doi:10.1016/j.bodyim.2019.12.003 · Cited by 56 · see citing works
- Authors
- Simons, D. J.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Simons, Daniel J.
- ORCID
- Simons, Daniel J.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-01
- Volume
- 9
- Issue
- 1
- Pages
- 76-80
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2077240377
- Cited by
- 630
- References
- 18
- Citations by year
- 2026: 18 · 2025: 36 · 2024: 29 · 2023: 44 · 2022: 51 · 2021: 69 · 2020: 91 · 2019: 75 · 2018: 67 · 2017: 47 · 2016: 36 · 2015: 48 · 2014: 18
- FWCI
- 48.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- direct replication, reproducibility, statistical power, replication study, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review