The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
The Value of Direct Replication
Abstract
Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“almost none of the findings in the literature underwent direct replication attempts”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [26]
Citations
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 758
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Research works printed but not linked: 1
- Roediger H. L., 2012, Observer, 25, 27
Cited by
Sorted by citations
Items 501–600 of 634
Calderon, S., Giolla, E. M., Ask, K., & Luke, T. J. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/wqbhd · Cited by 2
Brauer, K. (2024) · International Journal of Applied Positive Psychology · doi:10.1007/s41042-024-00204-y · Cited by 2
Questionable Interpretive Practices
Jussim, L. J., Stevens, S. T., & Anglin, S. M. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0009 · Cited by 1
Mind the (Construct-Measurement) Gap
Hodson, G. (2023) · Psychological Inquiry · doi:10.1080/1047840x.2023.2274389 · Cited by 1
Online Moral Conformity Revisited
Fedorov, A. A., & Steshova, A. A. (2025) · International Journal of Psychology · doi:10.1002/ijop.70033 · Cited by 2
Braun, M. N., Bülow, S., Müller-Klein, A., Link-Dorner, U., Michael, T., & Lass‐Hennemann, J. (2025) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2025.106716 · Cited by 1
Why We Disagree About the Success of Replications
Sarisoy, J. (2025) · Journal for General Philosophy of Science · doi:10.1007/s10838-024-09709-1 · Cited by 1
Bush, R., Staiger, P. K., McNeill, I. M., Brown, R., Lubman, D. I., & McNair, R. (2025) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2025.102690 · Cited by 1
Why satisfied buyers might not be (attitudinally or behaviorally) loyal
Pleshko, L. P., & Dawes, J. G. (2025) · International Journal of Quality and Service Sciences · doi:10.1108/ijqss-03-2025-0057 · Cited by 2
Summak, M. S., & Kalender, B. (2025) · SAGE Open · doi:10.1177/21582440251369583 · Cited by 1
What do agreeable people think about? spontaneous thoughts as a component of personality
Fereidouni, H., Asad, M. R., & Robinson, M. D. (2025) · Journal of Research in Personality · doi:10.1016/j.jrp.2025.104675 · Cited by 1
Commentary to Kaiser and Wilson (2026)
Genschow, O., & Crusius, J. (2026) · European Psychologist · doi:10.1027/1016-9040/a000572 · Cited by 1
The Impact of Invalidating Emotional Experiences in Childhood on Forgiveness in Close Relationships
Leggett, P. A., Lamarche, V. M., & Klabunde, M. (2026) · Psychological Reports · doi:10.1177/00332941261436657 · Cited by 1
Context matters: but how much? Replicating coopetition performance drivers
Klimas, P., Czakon, W., Kawa, A., Stańczyk, S., & Sachpazidu-Wójcicka, K. (2026) · Review of Managerial Science · doi:10.1007/s11846-026-01022-z · Cited by 1
Emotional costs of service labor: do consumers care?
Moran, N., & Ronen, S. (2021) · Journal of Services Marketing · doi:10.1108/jsm-03-2020-0085 · Cited by 1
Towards more credible conceptual replications under heteroscedasticity and unbalanced designs
Mateu, P. F., Applegate, B. E., & Coryn, C. L. S. (2023) · Quality & Quantity · doi:10.1007/s11135-023-01657-0 · Cited by 1
Laurin, R., & Pellet, J. (2023) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2023.2189466 · Cited by 1
Lee, H., Joung, D., Kwon, Y. J., & Park, B. (2023) · Journal of people, plants, and environment · doi:10.11628/ksppe.2023.26.5.535 · Cited by 1
Closing the Gap: A Conceptual Replication of an Early Reading Intervention
Solheim, O. J., Foldnes, N., Walgermo, B. R., Bjerke, K. K., Birkedal, I. K., Uppstad, P. H., & Lundetræ, K. (2024) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2024.2333734 · Cited by 2
Koppang, H., Hærem, T., Mayiwar, L., & Pineda, J. A. (2024) · Royal Society Open Science · doi:10.1098/rsos.231575 · Cited by 1
Coll‐Martín, T., Carretero‐Dios, H., & Lupiáñez, J. (2024) · Collabra Psychology · doi:10.1525/collabra.121392 · Cited by 4
Breit, F., Tamburelli, M., Gruffydd, I., & Brasca, L. (2024) · Linguistics Beyond and Within (LingBaW) · doi:10.31743/lingbaw.18005 · Cited by 1
The Prevalence of Replication Studies in Human Factors (2020-2024)
Doane, R., Andrade, F. M. G., Marra-Dotts, J., Li, B., Groetsch, S., Lichter, K. A., Do, A., Hyk, A., Baker, S., Johnson, J. D., et al. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251376984 · Cited by 1
Allen, M. S., McRae, K., & Walter, E. E. (2018) · Body Image · doi:10.1016/j.bodyim.2018.12.008 · Cited by 1
Mack, S. (2019) · Dialnet (Universidad de la Rioja) · Cited by 1
The “Mnemonic Time-Travel Effect”
Mieth, L., Bell, R., & Buchner, A. (2019) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000461 · Cited by 1
In search of scope: A response to Ruiz et al. (2020)
Hulbert‐Williams, L., Pendrous, R., Hochard, K. D., & Hulbert-Williams, N. (2020) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2020.10.008 · Cited by 1
Direct and conceptual replication in numerical cognition
Cipora, K., & Soltanlou, M. (2021) · Journal of Numerical Cognition · doi:10.5964/jnc.7799 · Cited by 1
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 1
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 1
Hiserodt, M., Fitzgerald, H. E., García, J., Hoyt, D. L., Milligan, M. A., & Otto, M. W. (2022) · Current Psychology · doi:10.1007/s12144-022-03059-1 · Cited by 1
Li, M. (2021) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/6354814 · Cited by 1
Optimal design of psychological experiments with multiple random factors
Westfall, J. (2015) · CU Scholar (University of Colorado Boulder) · Cited by 1
An investigation of teaching behaviour in primates and birds
Troisi, C. A. (2017) · St Andrews Research Repository (St Andrews Research Repository) · Cited by 1
Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological SciencePreprint
Nuijten, M. B. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qtk7e · Cited by 1
Bridging the gap between embodied cognition and cognitive load theory
Skulmowski, A. (2019) · Qucosa (Saxon State and University Library Dresden) · Cited by 1
Heterogeneity of research results: New perspectives on psychological science
Linden, A. H. (2019) · Northumbria Research Link (Northumbria University) · Cited by 1
Science Needs Systematic Replicability AuditsPreprint
Allard, A., & Vazire, S. (2021) · doi:10.31234/osf.io/wusdr · Cited by 1
Jennnings, F. J. (2018) · MOspace Institutional Repository (University of Missouri) · doi:10.32469/10355/68905 · Cited by 1
Tang, C., Owsley, N. C., Wein, T., & Lab, B. L. B. (2021) · doi:10.31222/osf.io/4vpsz · Cited by 1
Monteith, M. J. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619895728 · Cited by 1
Replicable quality across every channel: the Busara research agenda on methods in the Global SouthPreprint
Mughogho, W., Owsley, N. C., Wein, T., & Lab, B. L. B. (2021) · doi:10.31222/osf.io/2hn6v · Cited by 1
Increasing propensity of mind wandering using tDCS? A Registered ReportPreprint
Boayue, N. M., Csifcsák, G., Aslaksen, P. M., Turi, Z., Antal, A., Groot, J. M., Hawkins, G. E., Forstmann, B. U., Thielscher, A., Opitz, A., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dfex3 · Cited by 1
Editorial to special issue on direct replications in legal and criminological psychology
Meijer, E. H., & Verschuère, B. (2024) · Legal and Criminological Psychology · doi:10.1111/lcrp.12267 · Cited by 1
Half a century of Lighting Research & Technology: A bibliometric review
Chen, Q., Ru, T., Zhai, D., Huang, X., Li, Y., Qian, L., Wang, Y., & Zhou, G. (2019) · Sage Journals Data · doi:10.25384/sage.c.4680548 · Cited by 1
Violent video games and social behaviour
Tear, M. J. (2015) · The University of Queensland · doi:10.14264/uql.2015.784
Under the Influence and Unaware
Balcetis, E., & Granot, Y. (2015) · Wiley eBooks · doi:10.1002/9781118468333.ch11
Influencing consumer resistance through priming: the role of mindsets and motivational orientation
Harman, B. (2016) · Arrow@dit (Dublin Institute of Technology)
Meegesleept in de wereld van het verhaal?
Jansen, C., Nederhoff, S., & Ooms, J. (2017) · Tijdschrift voor taalbeheersing · doi:10.5117/tvt2017.1.jans
Westwood, S. J., Olson, A., Miall, R. C., & Romani, C. (2017) · Cortex · doi:10.1016/j.cortex.2017.08.012
Papadimos, T. J. (2018) · International Journal of Critical Illness and Injury Science · doi:10.4103/ijciis.ijciis_55_18
Statistical methods for evidence synthesisPreprint
Mathur, M. B., & VanderWeele, T. J. (2018) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/kd6ja
Understanding Replication: Trust But Verify
Suter, W. N., & Suter, P. M. (2019) · Home Health Care Management & Practice · doi:10.1177/1084822319850501 · Cited by 1
Functional opponency in working memory capacity predicts cognitive flexibility in problem solving.
Stockum, C. A. V. (2019) · ThinkIR: The University of Louisville's Institutional Repository (University of Louisville) · doi:10.18297/etd/3277
Theory of Causality and Modern Mainstream Psychology
Toomela, A. (2019) · SpringerBriefs in psychology · doi:10.1007/978-3-030-31449-1_7
Bustos, C., Canihuante, N. Z., Salcedo, P., & Mújica, A. D. (2020) · Psykhe (Santiago) · doi:10.7764/psykhe.29.2.1498 · Cited by 4
Radžvilas, M., Pretis, F. D., Peden, W., Tortoli, D., & Osimani, B. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2011.07797
Independent replication of classic trials in neurosurgery: A missing validation practice
Kerr, N., Kerr, J., Sheridan, M. M. P., & Holcombe, A. O. (2020) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2020.10.028
Fortier, P., & Schmidt, L. A. (2021) · Methods in Psychology · doi:10.1016/j.metip.2021.100045 · Cited by 3
Doing Science: Methods and PhilosophyPreprint
van Dongen, N. N. N. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/yegvs
Two steps forward, one step back
Krepel, N. (2021) · Research Publications (Maastricht University) · doi:10.26481/dis.20210618nk
Editorial: Direct and Conceptual Replication in Numerical CognitionPreprint
Cipora, K., & Soltanlou, M. (2021) · doi:10.31234/osf.io/t7h3x
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/uqz2e
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/mjvwu
Liao, T. F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.004
Replication crisisPreprint
Quang-Loc, N. (2022) · doi:10.31219/osf.io/4wd5x
Wiechert, S., Ben‐Shakhar, G., Verschuère, B., Pertzov, Y., Wessel, I., & Fawcett, J. (2022) · doi:10.31219/osf.io/jnk78
Smart but evil? Student-teachers perception of educational researchers’ epistemic trustworthiness.Preprint
Merk, S., & Rosman, T. (2018) · doi:10.31219/osf.io/cduqe
What’s the room got to do with it?
Hoogesteyn, K. (2020) · doi:10.26481/dis.20200706kh
Elman, C., Gerring, J., & Mahoney, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.019
Zhang & Smith_Does Power Reduce Temporal DiscountingPreprint
Zhang, M., & Smith, P. K. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/3fv45
Are affective factors related to individual differences in facial expression recognition?Preprint
Alharbi, S. A., Button, K. S., Bagshaw, A., Zhang, L., O’Shea, K. J., Fasolt, V., Lee, A. J., DeBruine, L. M., & Jones, B. C. (2019) · doi:10.31234/osf.io/fg8yz
Multiplicity Control vs Replication: Making an Obvious Choice Even More ObviousPreprint
Hunter, A., Beribisky, N., Farmus, L., & Cribbie, R. A. (2019) · doi:10.31219/osf.io/hbe48
Of intersectionality, replicability, and holistic perspectives
Mack, S. (2019) · Issues in Hispanic and Lusophone linguistics · doi:10.1075/ihll.21.12mac
Automatic Prejudice and Weapon Identification: A Study with Students and Police Officers
Dantas, G. S., Alves, M. V., & Pereira, M. E. (2022) · Race and Social Problems · doi:10.1007/s12552-022-09373-4
Kim, I. K. (2022) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.21220838
Transparency and Open Science Practices in Research on Emerging AdulthoodPreprint
Nguyen, P. L. L., & Syed, M. (2023) · doi:10.31234/osf.io/ckdws
Behavioral Public Administration: Recognizing, Celebrating and Reflecting on Research Trends
Wright, B. E. (2023) · Springer eBooks · doi:10.1007/978-981-16-9917-7_4
Kuiper, R. M., & Clapper, E. (2023) · doi:10.31234/osf.io/qv76x
Konaszewski, K., Muszyńska, J., Skalski‐Bednarz, S. B., & Surżykiewicz, J. (2023) · Polska Myśl Pedagogiczna · doi:10.4467/24504564pmp.23.016.18245
Good theory from bad ingredients?Preprint
Field, S. M., Kaznatcheev, A., van Dongen, N. N. N., Volz, L., & Wright, C. (2023) · doi:10.31219/osf.io/sz9y7
Verkoeijen, P. P. J. L., Koppenol-Gonzalez, G. V., van Peppen, L. M., Broeren, M. M. D. H. J., Heijltjes, A. E. G., Kuijpers, R. E., Nobelen, J. T. L. M., Tillema, M., de Moor, M. H. M., & Arends, L. R. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241270604
Why are replication rates so low?
Vu, P. (2024) · Journal of Econometrics · doi:10.1016/j.jeconom.2024.105868
Bell, R., & Buchner, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-80296-3
Coyle, T., Murphy, J., & Warne, J. P. (2024) · Communications in Kinesiology · doi:10.51224/cik.2024.65
Transparency and Open Science Practices in Research on Emerging Adulthood
Nguyen, P., & Syed, M. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197677797.003.0002
Wiechert, S., Ben‐Shakhar, G., Verschuère, B., Pertzov, Y., Wessel, I., & Fawcett, J. M. (2022) · doi:10.31219/osf.io/jnk78_v1
Examining the availability/findability of stimuli employed in social media and body image research
Smailes, D., Aleksandra, A., Coakley, M., Mair, S., & Ventress, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0324514
The Epistemic Function of Replication: Mapping the Domains of the DebatePreprint
Orkeny, B. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/h8swj_v1
The Epistemic Aims of the Humanities
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0003
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0002
Philosophy of the Humanities: What and Why?
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0001
Authority and Deference in the Humanities
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0004
The Future of the Philosophy of the Humanities
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0010
Perspectives, Social Constructions, Standpoints, and Truth in the Humanities
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0006
Grimm, S. R., Peels, R., & van Woudenberg, R. (2025) · doi:10.1093/9780197749647.003.0008
Metadata
- Authors
- Simons, D. J.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Simons, Daniel J.
- ORCID
- Simons, Daniel J.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-01
- Volume
- 9
- Issue
- 1
- Pages
- 76-80
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2077240377
- Cited by
- 630
- References
- 18
- Citations by year
- 2026: 18 · 2025: 36 · 2024: 29 · 2023: 44 · 2022: 51 · 2021: 69 · 2020: 91 · 2019: 75 · 2018: 67 · 2017: 47 · 2016: 36 · 2015: 48 · 2014: 18
- FWCI
- 48.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- direct replication, reproducibility, statistical power, replication study, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review