The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index.
Replicability, Robustness, and Reproducibility in Psychological Science
Open accessPublished version of a preprint
Abstract
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using a common approach to analyze data from multiple informants, time points, and developmental domains, we can address concerns about the robustness of scientific findings (20,21).”
From A Comprehensive Multilevel Analysis of the Bucharest Early Intervention Project: Causal Effects on Recovery From Early Severe Deprivation (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research into replicability in other disciplines has also found that findings with stronger initial evidence (such as larger effect sizes and/or smaller p-values) is more likely to replicate”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Independent verification is central to the scientific method (Nosek et al., 2022).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Nosek et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“A substantial proportion of replications in recent largescale efforts have failed to support the original finding (Nosek et al., 2022).”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Nosek et al., 2022
In the citing paper
“This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
In the citing paper
“However, particular challenges arise when applying these principles to LLMs, particularly proprietary ones, whose black-box nature impedes the reproducibility of findings that pertain to them.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,559
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
(2002) · Cited by 13,211
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citation
Research works printed but not linked: 17
- Artner R, 2020, Psychol. Methods.
- Cent. Open Sci, 2020, Center for Open Science
- Cohen J., 1973, Am. Educ. Res. J., 10, 225
- 10.1037/0033-2909.112.1.155 · DOI · Paper page
- Crisp RJ, 2014, Soc. Psychol., 45, 303
- Epskamp S, 2018, Statistical Software
- Errington TM, 2021, eLife
- Ferguson MJ, 2014, Soc. Psychol., 45, 301
- Gelman A, 2013, The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
- Marcus A, 2020, WIRED
- Meyer MN, 2014, Slate Magazine
- Mischel W., 2008, APS Observer
- Nosek B., 2019, Center for Open Science
- Nosek B., 2019, Center for Open Science
- Schwarz N, 2014, Soc. Psychol., 45, 305
- Sterling TD., 1959, J. Am. Stat. Assoc., 54, 30
- Yong E, 2012, National Geo-graphic
Cited by
Sorted by citations
Items 701–772 of 772
Theories and Research in Personal Relationships
Campbell, L. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009381949.003
Tenney, E. R., Rochford, K. C., Stillwell, A., Liu, C. X., Tannenbaum, D., Hennecke, M., Stefanucci, J. K., Blair, B. A., Graham, J., & Bonner, B. L. (2026) · Psychological Science · doi:10.1177/09567976251412721
Awuni, P. M. A., Mbinta, J. F., & Zobazie, C. B. (2026) · Bulletin of the National Research Centre/Bulletin of the National Research Center · doi:10.1186/s42269-026-01401-2
Synthetic data as a method for increasing reproducibility and transparency in educational research
Grund, S., Lüdtke, O., & Robitzsch, A. (2026) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-026-01396-6 · Cited by 2
Javadi, V. S., Róg, F., Aksa, A., Trippas, J. R., Vakulenko, S., & Flek, L. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787921 · Cited by 5
Reproducibility in Corpus Linguistics
Schweinberger, M. (2026) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00967-4
Balducci, C., Perinelli, E., Avanzi, L., Atroszko, P. A., Charzyńska, E., & Menghini, L. (2026) · Work & Stress · doi:10.1080/02678373.2026.2631986
Empiirinen kirjallisuuden ja lukemisen tutkimus: Menetelmiä ja käytäntöjä
Ovaska, A., Harinen, K., Kinnunen, E., Lahtinen, T., Löytty, O., & Suomalainen, R. (2026) · doi:10.21435/tl.304 · Cited by 1
Quantum-resistant timestamping for open science: a non-technical guide
Krägeloh, C. U., Bartholomew, E. J., & Medvedev, O. N. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2639409
Hussain, S. T. (2026) · Journal of Paleolithic Archaeology · doi:10.1007/s41982-026-00261-6
Deng, Y.-N., Wu, S.-Q., Luo, X.-F., Tang, B.-B., Feng, Y.-C., Feng, H., Guo, H.-L., Li, Y., Shi, L., Zhi-rui, X. U., et al. (2026) · Evidence-Based Chinese Medicine and Technology Assessment · doi:10.26599/ecmta.2026.9570031
The Replication Crisis in Psychology: Causes, Consequences, and Strategies for Overcoming It
Гуляс, І. А. (2026) · PSIHOLOGÌÂ Ì SUSPÌLʹSTVO · doi:10.35774/pis2026.01.083
Gelsleichter, Y. A., Banzi, R., Naudet, F., Vinatier, C., Kertész, I., & Varga, M. (2026) · F1000Research · doi:10.12688/f1000research.172013.2
A computational account of how affective states influence economic decisions
Heerema, R. (2023) · theses.fr (ABES) · doi:10.70675/a319dd46z4623z4805z9223zb939a0acf611
Why It Is Time to Use Multiverse Analysis in Industrial and Organizational Psychology
Mazei, J., & Hüffmeier, J. (2026) · Zeitschrift für Arbeits- und Organisationspsychologie A&O · doi:10.1026/0932-4089/a000473
Registered Report: Can Agentic Black Women Get Ahead? An Experiment Revisited
Liu, C. X., Blair, B. A., & Tenney, E. R. (2026) · Psychological Science · doi:10.1177/09567976261436988
Brown, M., Sacco, D. F., Raible, C., Wallace, K. J., & Lemacks, J. L. (2026) · Accountability in Research · doi:10.1080/08989621.2026.2659673
Lopes, E. L., Mesquita, E., & Herrero, E. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-29863-9.00089-1
Anthropological and Psychological Approaches to Culture, Identity, and Essentialism
Moya, C., & Wagner, W. (2026) · Theory and history in the human and social sciences · doi:10.1007/978-3-032-11918-6_1
Tackling Math Anxiety: Are Short Interventions Effective?
Cho, K. W. (2026) · The Journal of Experimental Education · doi:10.1080/00220973.2026.2666579
Using synthetic data to improve the reproducibility of statistical results in psychological researchPreprint
Lüdtke, O., Robitzsch, A., & Grund, S. (2023) · OSF Preprints (OSF Preprints)
Following the money: an econometric investigation into health valuations
Ásgeirsdóttir, T. L., & Kárason, V. A. (2026) · Scandinavian Journal of Economics · doi:10.1111/sjoe.70026
Miles, A., Samim, Y., & Khan, S. (2026) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2026.104947 · Cited by 1
Sell, J., & Webster, M. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009483353.006
Hidden processes of workflow in cognitive developmental psychology
Girouard‐Hallam, L. N., & Gelman, S. A. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0611 · Cited by 1
Pavani, J., Colombo, D., & Dauvier, B. (2026) · Affective Science · doi:10.1007/s42761-026-00372-x
Staller, M. S., & Körner, S. (2026) · Frontiers in Sociology · doi:10.3389/fsoc.2026.1758071
Stopping and changing expected and unexpected movements
Weber, S., Vucak, N., Salomoni, S. E., Ross, A. J., Coleman, E., & Hinder, M. R. (2026) · Biological Psychology · doi:10.1016/j.biopsycho.2026.109312
Die Differentielle und Persönlichkeitspsychologie im Jahr 2025
Rauthmann, J. F., Armbruster, D., Asselmann, E., Berkessel, J. B., Bleidorn, W., Buecker, S., Gärtner, A., Leising, D., Luhmann, M., Phan, L. V., et al. (2026) · Psychologische Rundschau · doi:10.1026/0033-3042/a000737
Self-citationHuang, X. (2026) · Transactions in GIS · doi:10.1111/tgis.70285
Lee, C. J. (2026) · Social Epistemology · doi:10.1080/02691728.2026.2669578
Pénzes, I., Oti, K., & van Hooren, S. (2026) · The Arts in Psychotherapy · doi:10.1016/j.aip.2026.102495
Thorne, T., Milyavskaya, M., & Leduc‐Cummings, I. (2026) · European Journal of Personality · doi:10.1177/08902070261463054
Embodied movement markers of alliance ruptures in psychotherapy: A machine learning study
Tchizick, A., Laskorunskyi, O., Dolev‐Amit, T., Peysachov, G., Pizman, P., Deres-Cohen, K., Dattner, I., & Zilcha‐Mano, S. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2689627
Beck, E. D., Willroth, E. C., Delius, J. A. M., Bennett, D. A., Barnes, L. L., James, B. D., Lipton, R. B., Katz, M., Hassing, L. B., Huisman, M. A., et al. (2026) · Personality and Social Psychology Bulletin · doi:10.1177/01461672261451901 · Cited by 2
bio.tools: an expanded web service for research software in the life sciences
Mendes, A., Ienasescu, H., Andersen, C. A. T., Calhoun, E., Chmura, P. J., Collier, K., Goga, I., Gustafsson, O., Kalaš, M., Kierkegaard, M., et al. (2026) · Nucleic Acids Research · doi:10.1093/nar/gkag420
Does open data promote the impact of research articles in psychology?
Pan, X., Li, K., Kuang, Q., & Ma, Z. (2026) · Journal of Information Science · doi:10.1177/01655515261461729
Wirth, J. H., Hales, A. H., Wicks, S. G., & Okdie, B. M. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261437994
Kowalski, J. (2025) · Przegląd Psychologiczny · doi:10.31648/przegldpsychologiczny.12958
Estimating the replicability of Brazilian biomedical sciencePreprint
Initiative, T. B. R. (2026) · eLife · doi:10.7554/elife.111001
Estimating the replicability of Brazilian biomedical sciencePreprint
Initiative, T. B. R. (2026) · eLife · doi:10.7554/elife.111001.1
Visual similarity and lexical access in an Arabic-derived script: evidence from Persian
Changizi, S., Fernández-López, M., & Perea, M. (2026) · Language and Cognition · doi:10.1017/langcog.2026.10095
Many AI analysts, one dataset: Navigating the agentic data science multiverse
Bertrán, M., Fogliato, R., & Wu, Z. S. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2606495123 · Cited by 1
Uptake and Implementation of Multiverse-style Analyses Across 613 StudiesPreprint
Nepomuceno, A., Ghosal, A., Sandoval‐Lentisco, A., & Ioannidis, J. P. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.15.738584
Nall, L., Zinko, R., Ford, A., & Ballentine, W. (2025) · Marketing of Scientific and Research Organizations · doi:10.2478/minib-2025-0016
Ravelo, Y., González-Méndez, R., Covelli, M., Tarasi, L., de Fatis, C. T., Romei, V., de la Rosa, O. M. A., Urrutia, M., & Marrero, H. (2026) · Current Psychology · doi:10.1007/s12144-026-09866-0
Grönroos, R., Celuch, M., Savolainen, I., Kaakinen, M., & Oksanen, A. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107491
Opportunities and risks of big data for the methods and culture of psychological science
Ibasco, G. C., Götz, F. M., Clark, L., & Tracy, J. L. (2026) · Nature Reviews Psychology · doi:10.1038/s44159-026-00603-9 · Cited by 1
Open practices across science, software, education and other domains
Meluso, J., Chambers, C. R., Littauer, R., Llamas, N. A., Lingo, E. L., Mhangami, M., Pitt, B., Splitter, V., & Wang, H. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02526-0
Robust and replicable effects of ageing on resting state brain electrophysiology measured with MEGPreprint
Quinn, A. J., Pitt, J., Kohl, O., Gohil, C., van Es, M. W. J., Nobre, A. C., & Woolrich, M. W. (2026) · eLife · doi:10.7554/elife.108829.2
Gao, H., Evans, T., Brown, G., Meissel, K., & Chase, J. G. (2026) · Educational Studies in Mathematics · doi:10.1007/s10649-026-10547-9
León, P. M. S., & Cotrina, L. K. C. (2026) · Journal of Sustainable Business · doi:10.1186/s40991-026-00151-8
Gorman, C. A., Meriac, J. P., Himmler, J. R., Boitet, L. M., & Contreras, T. F. (2026) · Personnel Review · doi:10.1108/pr-07-2025-0838
The impact of ambient light on road traffic crashes: A study of analysis robustness
Fotios, S. A., Uttley, J., & Sun, H. (2026) · Lighting Research & Technology · doi:10.1177/14771535261474463
Papadakis, Z., & Stamatis, A. (2026) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2026.1902953
Guidelines for Meta‐Analytic Structural Equation Modeling: Steps, Resources, and Illustrations
Waheed, H., & Lim, W. M. (2026) · Psychology and Marketing · doi:10.1002/mar.70266
Human–AI Collaboration is an Underspecified Intervention: A Reporting Guide for Cumulative Research
Rafner, J. F. (2026) · Creativity Research Journal · doi:10.1080/10400419.2026.2726243 · Cited by 1
Yang, C., Lim, J. H., Rho, E., Dong, Q., Lin, X., & Zhang, Y. (2026) · Journal of School Psychology · doi:10.1016/j.jsp.2026.101590
Jo, D., Im, S., Spencer, S. D., & Masuda, A. (2026) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-026-10314-3
Identifying Alternatives to Retraction of Controversial Research Findings
Brown, M., Sacco, D. F., Namuth, A., Clark, C. J., & Honeycutt, N. (2026) · Science Communication · doi:10.1177/10755470261486044
Reproducibility in biomedical research
Zhong, J., D’Angelo, D., Lyu, C., Xu, I., Enns, P. K., Wang, F., & Kaushal, R. (2026) · Nature Medicine · doi:10.1038/s41591-026-04667-1
Ben-Menachem, J., Galper, A., & Fishman, N. (2026) · Sociological Methods & Research · doi:10.1177/00491241261490850
Benkendorff, A., Sina, E., Bergt, A. C., Stadelmaier, J., & Meerpohl, J. J. (2026) · BMJ Open · doi:10.1136/bmjopen-2026-117895
DeCoster, J., Beliveau, L., Iselin, A.-M. R., & Gallucci, M. (2026) · International Journal of Behavioral Development · doi:10.1177/01650254261491094
A Call for Greater Transparency in Piloting
Handley-Miner, I. J., Bochynska, A., Buttliere, B., Chung, K.-L., Elsherif, M., Graham, C. J., Heiser, N. H., Loenneker, H., Sætrevik, B., Seetahul, Y., et al. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261487966 · Cited by 1
The question that wouldn’t die: A history of research on social interactions
Votinov, M. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-34080-2.00005-6
Matarese, V. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-40622-5.00015-3
Data Extraction Errors and Diagnostic Conclusions of Meta-Analyses
Huang, Z., Tian, Y., Gong, P., Zhang, T., & Tang, H. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.40850
Metadata
- Authors
- Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
- Affiliations
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Wisconsin–Madison (US)
- University of California, Davis (US)
- Grand Valley State University (US)
- Stockholm School of Economics (SE)
- The University of Melbourne (AU)
- Illinois State University (US)
- Center for Open Science, Charlottesville, Virginia 22903, USA
- Tilburg University (NL)
- Leipzig University (DE)
- University of Groningen (NL)
- Eindhoven University of Technology (NL)
- University of Colorado Anschutz (US)
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2022-01-04
- Volume
- 73
- Issue
- 1
- Pages
- 719-748
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W3161588210
- Cited by
- 770
- References
- 253
- Citations by year
- 2026: 109 · 2025: 192 · 2024: 180 · 2023: 177 · 2022: 92
- FWCI
- 19.02
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation
- Patient-Centered Outcomes Research Institute
- Arnold Ventures
- European Commission (HORIZON2020, 841188)
- University of Melbourne
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Templeton World Charity Foundation
- Templeton Religion Trust
- National Institutes of Health
- Agency for Healthcare Research and Quality
- Australian Research Council (FT150100297)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, replicability, reproducibility, robustness, research practices, replication study, research incentives
- MeSH terms
- Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
