The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index.
Replicability, Robustness, and Reproducibility in Psychological Science
Open accessPublished version of a preprint
Abstract
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using a common approach to analyze data from multiple informants, time points, and developmental domains, we can address concerns about the robustness of scientific findings (20,21).”
From A Comprehensive Multilevel Analysis of the Bucharest Early Intervention Project: Causal Effects on Recovery From Early Severe Deprivation (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research into replicability in other disciplines has also found that findings with stronger initial evidence (such as larger effect sizes and/or smaller p-values) is more likely to replicate”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Independent verification is central to the scientific method (Nosek et al., 2022).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Nosek et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“A substantial proportion of replications in recent largescale efforts have failed to support the original finding (Nosek et al., 2022).”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Nosek et al., 2022
In the citing paper
“However, particular challenges arise when applying these principles to LLMs, particularly proprietary ones, whose black-box nature impedes the reproducibility of findings that pertain to them.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,559
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
(2002) · Cited by 13,211
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citation
Research works printed but not linked: 17
- Artner R, 2020, Psychol. Methods.
- Cent. Open Sci, 2020, Center for Open Science
- Cohen J., 1973, Am. Educ. Res. J., 10, 225
- 10.1037/0033-2909.112.1.155 · DOI · Paper page
- Crisp RJ, 2014, Soc. Psychol., 45, 303
- Epskamp S, 2018, Statistical Software
- Errington TM, 2021, eLife
- Ferguson MJ, 2014, Soc. Psychol., 45, 301
- Gelman A, 2013, The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
- Marcus A, 2020, WIRED
- Meyer MN, 2014, Slate Magazine
- Mischel W., 2008, APS Observer
- Nosek B., 2019, Center for Open Science
- Nosek B., 2019, Center for Open Science
- Schwarz N, 2014, Soc. Psychol., 45, 305
- Sterling TD., 1959, J. Am. Stat. Assoc., 54, 30
- Yong E, 2012, National Geo-graphic
Cited by
Sorted by citations
Items 101–200 of 772
Proliferation of measures contributes to advancing psychological science
Iliescu, D., Greiff, S., Ziegler, M. D., Nye, C. D., Geisinger, K. F., Sellbom, M., Samuel, D. B., & Saklofske, D. H. (2024) · Communications Psychology · doi:10.1038/s44271-024-00065-w · Cited by 19
Can large language models help predict results from a complex behavioural science study?
Lippert, S., Dreber, A., Johannesson, M., Tierney, W., Cyrus-Lai, W., Uhlmann, E. L., Emotion Expression Collaboration, & Pfeiffer, T. (2024) · Royal Society Open Science · doi:10.1098/rsos.240682 · Cited by 32
Self-citationThe COVID-19 pandemic and social cognitive outcomes in early childhood
Scott, R. M., Nguyentran, G., & Sullivan, J. Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-80532-w · Cited by 20
Luijken, K., Lohmann, A. W., Alter, U., Gonzalez, J. C., Clouth, F. J., Fossum, J. L., Hesen, L., Huizing, A., Ketelaar, J., Montoya, A. K., et al. (2024) · Royal Society Open Science · doi:10.1098/rsos.231003 · Cited by 20
Stability and Change in Gender Identity and Sexual Orientation Across Childhood and Adolescence
deMayo, B., Gallagher, N. M., Leshin, R. A., & Olson, K. R. (2025) · Monographs of the Society for Research in Child Development · doi:10.1111/mono.12479 · Cited by 22
Boukrouh, S., Karouach, F., Aayadi, S. E., Amiri, B. E., Hornick, J., Nilahyane, A., & Hirich, A. (2026) · Archives animal breeding/Archiv für Tierzucht · doi:10.5194/aab-69-101-2026 · Cited by 21
The logical structure of experiments lays the foundation for a theory of reproducibility
Buzbas, E. O., Devezer, B., & Baumgaertner, B. O. (2023) · Royal Society Open Science · doi:10.1098/rsos.221042 · Cited by 22
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 20
Cognitive training for mental health problems
Hallion, L. S., Hsu, K. J., & Schleider, J. L. (2024) · Nature Mental Health · doi:10.1038/s44220-023-00185-y · Cited by 20
Is biomedical research self-correcting? Modelling insights on the persistence of spurious science
Grimes, D. R. (2024) · Royal Society Open Science · doi:10.1098/rsos.231056 · Cited by 18
Dimant, E., Clemente, E. G., Pieper, D. J., Dreber, A., Gelfand, M. J., Hallsworth, M., Holzwarth, A., & Tantia, P. (2021) · Scientific Reports · doi:10.1038/s41598-022-10524-1 · Cited by 22
Self-citationA Unified Framework for Understanding and Intervening on False News SharingPreprint
Gollwitzer, A., Tump, A. N., Martel, C., Deffner, D., Sultan, M., Kurvers, R. H. J. M., & Hertwig, R. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/pxn29 · Cited by 19
Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102327 · Cited by 22
Chan, M., & Feldman, G. (2025) · Royal Society Open Science · doi:10.1098/rsos.250290 · Cited by 17
Resilience and small business performance: An empirical investigation
Sangboon, K., & Schjoedt, L. (2024) · Journal of the International Council for Small Business · doi:10.1080/26437015.2024.2382119 · Cited by 21
Stokoe, E. H., Raymond, G., & Whitehead, K. A. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-020124-023147 · Cited by 24
Validity and Transparency in Quantifying Open-Ended Data
Conry‐Murray, C., Waltzer, T., DeBernardi, F. C., Fossum, J. L., Haasova, S., Matthews, M. S., O’Mahony, A., Moreau, D., Baum, M. A., Karhulahti, V., et al. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241275217 · Cited by 25
The replication crisis has led to positive structural, procedural, and community changesPreprint
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M. M., Breznau, N., Robertson, O. M., Kalandadze, T., et al. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/r6cvx · Cited by 18
When Do Interaction/Moderation Effects Stabilize in Linear Regression?
Castillo, A. J., Miller, J. D., Vize, C. E., Baranger, D. A. A., & Lynam, D. R. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251407860 · Cited by 16
Doing reliable research in comparative psychology: Challenges and proposals for improvement.
Tecwyn, E. C. (2021) · Journal of comparative psychology · doi:10.1037/com0000291 · Cited by 17
Reer, A., Wiebe, A., Wang, X., & Rieger, J. W. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1086802 · Cited by 18
Successful reproduction of a large EEG study across software packages
Kabbara, A., Forde, N., Maumet, C., & Hassan, M. (2023) · Neuroimage Reports · doi:10.1016/j.ynirp.2023.100169 · Cited by 16
Beffara, B., Hadj‐Bouziane, F., Hamed, S. B., Boehler, C. N., Chelazzi, L., Santandrea, E., & Macaluso, E. (2023) · Human Brain Mapping · doi:10.1002/hbm.26520 · Cited by 20
One Decade Into the Replication Crisis, How Have Psychological Results Changed?
Bogdan, P. C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251323480 · Cited by 17
Behavioral consequences of intergroup sensitivity
Thürmer, J. L., & McCrea, S. M. (2022) · Social and Personality Psychology Compass · doi:10.1111/spc3.12716 · Cited by 16
Meier, M., Lonsdorf, T. B., LUPIEN, S. J., Stalder, T., Laufer, S., Sicorello, M., Linz, R., & Puhlmann, L. M. C. (2022) · Comprehensive Psychoneuroendocrinology · doi:10.1016/j.cpnec.2022.100144 · Cited by 18
A Connected World: System-Level Support Through Biosensors
McLamore, E. S., & Datta, S. (2023) · Annual Review of Analytical Chemistry · doi:10.1146/annurev-anchem-100322-040914 · Cited by 17
Torka, A., Mazei, J., Bosco, F. A., Cortina, J. M., Götz, M., Kepes, S., O’Boyle, E. H., & Hüffmeier, J. (2023) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2023.2206571 · Cited by 20
A toolbox to evaluate the trustworthiness of published findings
Adler, S. J., Röseler, L., & Schöniger, M. K. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114189 · Cited by 21
The limitations of machine learning models for predicting scientific replicability
Crockett, M. J., Bai, X., Kapoor, S., Messeri, L. R., & Narayanan, A. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2307596120 · Cited by 15
Loenneker, H. D., Buchanan, E. M., Martinovici, A., Primbs, M. A., Elsherif, M. M., Baker, B. J., Dudda, L. A., Đurđević, D. F., Mišić, K., Peetz, H. K., et al. (2023) · Cortex · doi:10.1016/j.cortex.2023.11.012 · Cited by 19
Motivation and Ability: Unpacking Underperforming Firms’ Risk Taking
Ref, O., Hu, S., Milyavsky, M., Feldman, N. E., & Shapira, Z. (2024) · Organization Science · doi:10.1287/orsc.2020.13953 · Cited by 16
Hardwicke, T. E., Thibault, R. T., Clarke, B., Moodie, N., Crüwell, S., Schiavone, S. R., Handcock, S. A., Nghiem, K. A., Mody, F., Eerola, T., et al. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241283477 · Cited by 29
Self-citationPalmer, L., Brooks, G. A., & Clark, L. (2025) · BMC Psychology · doi:10.1186/s40359-025-02766-1 · Cited by 20
Liu, M., & Henry, A. (2025) · Studies in Second Language Acquisition · doi:10.1017/s0272263125000166 · Cited by 14
Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines
Rahal, R.-M., Hamann, H., Brohmer, H., & Pethig, F. (2022) · Research Ideas and Outcomes · doi:10.3897/rio.8.e89980 · Cited by 17
Gutiérrez, A. C. Q., Lindegger, D. J., Heravi, A. T., Stojanov, T., Sykora, M. D., Elayan, S., Mooney, S. J., Naslund, J. A., Fadda, M., & Gruebner, O. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021473 · Cited by 17
Narvekar, N., Leno, V. C., Pasco, G., Ali, J. B., Johnson, M. H., Charman, T., Jones, E. J. H., & The STAARS Team (2024) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13941 · Cited by 19
Open Access, Scholarly Communication, and Open Science in Psychology: An Overview for Researchers
Mullen, L. B. (2024) · SAGE Open · doi:10.1177/21582440231205390 · Cited by 18
Are the Signs of Factor Loadings Arbitrary in Confirmatory Factor Analysis? Problems and Solutions
Tang, D., Boker, S. M., & Tong, X. (2024) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2024.2351102 · Cited by 23
Australian and Italian Psychologists’ View of Replication
Agnoli, F., Fraser, H. S., Thorn, F. S., & Fidler, F. M. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211039218 · Cited by 13
Self-citationSharing is caring: Ethical implications of transparent research in psychology.
Bosma, C. M., & Granger, A. M. (2022) · American Psychologist · doi:10.1037/amp0001002 · Cited by 19
Reproducibility in Small- N Treatment Research: A Tutorial Using Examples From Aphasiology
Cavanaugh, R., Quique, Y. M., Swiderski, A. M., Kallhoff, L., Terhorst, L. A., Wambaugh, J. L., Hula, W. D., & Evans, W. S. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00333 · Cited by 16
The Chinese Open Science Network (COSN): Building an Open Science Community From Scratch
Jin, H., Wang, Q., Yang, Y., Zhang, H., Gao, M. M., Jin, S., Chen, Y., Xu, T., Zheng, Y.-R., Ji, C., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221144986 · Cited by 21
Challenges and Solutions to the Measurement of Neurocognitive Mechanisms in Developmental Settings
Pezzoli, P., Parsons, S. D., Kievit, R., Astle, D. E., Huys, Q. J. M., Steinbeis, N., & Viding, E. M. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.03.011 · Cited by 15
Research Design and Issues of Validity
Brewer, M. B., & Crano, W. D. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009170123.007 · Cited by 15
Estimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13
Heyard, R., Pawel, S., Frese, J., Voelkl, B., Würbel, H., McCann, S., Held, L., Wever, K. E., Hartmann, H., Townsin, L., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242076 · Cited by 21
Reproducibility and robustness of economics and political science research
Brodeur, A., Mikola, D., Cook, N., Fiala, L., Brailey, T., Briggs, R. C., de Gendre, A., Dupraz, Y., Gabani, J., Gauriot, R., et al. (2026) · Nature · doi:10.1038/s41586-026-10251-x · Cited by 18
Self-citationPapageorgopoulou, E., Jones, E. J. H., Johnson, M. H., Charman, T., Green, J., Wan, M. W., & The BASIS‐STAARS Team (2024) · Autism Research · doi:10.1002/aur.3190 · Cited by 15
Tincani, M., Gilroy, S. P., & Dowdy, A. (2024) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.2909 · Cited by 15
Statistical Perspectives on Reproducibility: Definitions and Challenges
Simkus, A., Coolen‐Maturi, T., Coolen, F. P. A., & Bendtsen, C. (2025) · Journal of Statistical Theory and Practice · doi:10.1007/s42519-025-00459-x · Cited by 14
Replicability Across Multiple Studies
Bogomolov, M., & Heller, R. (2023) · Statistical Science · doi:10.1214/23-sts892 · Cited by 16
The social psychology of everyday life
(1993) · Choice Reviews Online · doi:10.5860/choice.30-4092 · Cited by 13
Nature's path to thinking about others and the surrounding environmentPreprint
Schertz, K. E., Kotabe, H., Meidenbauer, K. L., Layden, E. A., Zhen, J., Bowman, J. E., Lakhtakia, T., Lyu, M., Paraschos, O. A., Janey, E. A., et al. (2022) · doi:10.1016/j.jenvp.2023.102046 · Cited by 13
AITurk: Using ChatGPT for Social Science ResearchPreprint
Qin, X., Huang, M., & Ding, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4922861 · Cited by 14
Hallford, D. J., Woolfit, M., Follett, A., Jones, E. D., Harrison, O., & Austin, D. (2024) · Memory · doi:10.1080/09658211.2024.2333510 · Cited by 15
Reliably measuring learning-dependent distractor suppression with eye tracking
Kim, A. J., Grégoire, L., & Anderson, B. A. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02552-8 · Cited by 11
Nostalgia encourages exploration and fosters uncertainty in response to AI technology
Dang, J., Sedikides, C., Wildschut, T., & Liu, L. (2024) · British Journal of Social Psychology · doi:10.1111/bjso.12843 · Cited by 13
Tincani, M., Travers, J. C., Dowdy, A., Slocum, T. A., & Deitrich, R. (2025) · Perspectives on Behavior Science · doi:10.1007/s40614-025-00441-9 · Cited by 17
Ostropolets, A., Albogami, Y., Conover, M. M., Banda, J. M., Baumgartner, W. A., Blacketer, C., Desai, P., DuVall, S. L., Fortin, S., Gilbert, J. P., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad009 · Cited by 12
Systematic reviewers' perspectives on replication of systematic reviews: A survey
Nguyen, P., McKenzie, J. E., Hamilton, D. G., Moher, D., Tugwell, P., Fidler, F. M., Haddaway, N., Higgins, J. P. T., Kanukula, R., Karunananthan, S., et al. (2023) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12009 · Cited by 12
Self-citationOpen Science Standards at Journals that Inform Evidence-Based Policy
Grant, S., Mayo‐Wilson, E., Kianersi, S., Naaman, K., & Henschel, B. (2023) · Prevention Science · doi:10.1007/s11121-023-01543-z · Cited by 20
Hensel, P. G. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1981870 · Cited by 16
Open and reproducible neuroimaging: from study inception to publicationPreprint
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · doi:10.31219/osf.io/pu5vb · Cited by 12
A Problem in Theory and More: Measuring the Moderating Role of Culture in Many Labs 2Preprint
Schimmelpfennig, R., Spicer, R., White, C., Gervais, W. M., Norenzayan, A., Heine, S. J., Henrich, J., & Muthukrishna, M. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hmnrx · Cited by 12
den Abeele, E. V., Beuckels, E., Hudders, L., & Vanwesenbeeck, I. (2024) · International Journal of Advertising · doi:10.1080/02650487.2024.2371183 · Cited by 11
Dreyfuss, J. M., Djordjilović, V., Pan, H., Bussberg, V., MacDonald, A. M., Narain, N. R., Kiebish, M. A., Blüher, M., Tseng, Y., & Lynes, M. D. (2024) · Communications Biology · doi:10.1038/s42003-024-06646-z · Cited by 13
How can we make sound replication decisions?
Davis‐Stober, C. P., Sarafoglou, A., Aczél, B., Chandramouli, S., Errington, T. M., Field, S. M., Fishbach, A., Freire, J., Ioannidis, J. P. A., Oberauer, K., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401236121 · Cited by 13
Decision Under Uncertainty: State of the Science
Gilboa, I. (2025) · Annual Review of Economics · doi:10.1146/annurev-economics-090924-041522 · Cited by 12
Reflections on adversarial collaboration from the adversaries: was it worth it?
Isch, C. G., Tetlock, P. E., & Clark, C. J. (2025) · Theory and Society · doi:10.1007/s11186-025-09634-2 · Cited by 13
Ten Strategies to Foster Open Science in Psychology and Beyond
Alessandroni, N., & Byers‐Heinlein, K. (2022) · Collabra Psychology · doi:10.1525/collabra.57545 · Cited by 11
Open Science in Developmental Science
Gennetian, L. A., Frank, M. C., & Tamis‐LeMonda, C. S. (2022) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-120920-042335 · Cited by 12
Advancing the psychology of social class with large-scale replications in four countries
Batruch, A., Sommet, N., & Autin, F. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02234-1 · Cited by 22
Perspectives: replication is more than meets the eye
Bergkvist, L., Keyzer, F. D., & Buzeta, C. (2023) · International Journal of Advertising · doi:10.1080/02650487.2023.2245278 · Cited by 11
Franco, F. O., Oliveira, J. S., Portolese, J., Sumiya, F. M., Silva, A., Machado‐Lima, A., Nunes, F. L. S., & Brentani, H. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02389-9 · Cited by 10
The Replicability Crisis and Human Agency in the Neo-Structured World
Сорокин, П. С., & Mironenko, I. A. (2025) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-024-09887-z · Cited by 11
Systemic Gamification Theory (SGT): A Holistic Model for Inclusive Gamified Digital Learning
Coelho, F., & Abreu, A. M. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9070070 · Cited by 16
A framework for assessing the trustworthiness of scientific research findings
Nosek, B. A., Allison, D. B., Jamieson, K. H., Mcnutt, M. K., Nielsen, A. B., & Wolf, S. M. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2536736123 · Cited by 16
Self-citationWu, J. F., Nivargi, R., Lanka, S. S. T., Menon, A. M., Modukuri, S. A., Nakshatri, N. S., Wei, X., Wang, Z., Caverlee, J., Rajtmajer, S. M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04580 · Cited by 22
From significance testing to estimation and Open Science: How esci can help
Calin-Jageman, R. J., & Cumming, G. (2024) · International Journal of Psychology · doi:10.1002/ijop.13132 · Cited by 9
Addante, R. J., Clise, E., Waechter, R., Bengson, J., Drane, D. L., & Perez-Caban, J. (2024) · iScience · doi:10.1016/j.isci.2024.111439 · Cited by 8
A Systematic Review on the Evolution of Power Analysis Practices in Psychological Research
Vankelecom, L., Schacht, O., Laroy, N., Loeys, T., & Moerkerke, B. (2025) · Psychologica Belgica · doi:10.5334/pb.1318 · Cited by 17
A Credibility Revolution for Relationship Science: Where Can We Step Up Our Game?
Joel, S., Eastwick, P. W., & Khera, D. S. (2025) · Social and Personality Psychology Compass · doi:10.1111/spc3.70042 · Cited by 10
Bhargava, H. K., Brown, S. A., Ghose, A., Gupta, A., Leidner, D. E., & Wu, D. J. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3721846 · Cited by 9
Issues in Grounded Cognition and How to Solve Them – the Minimalist Account
Friedrich, J., Fischer, M. H., & Raab, M. (2025) · Journal of Cognition · doi:10.5334/joc.444 · Cited by 9
Costa, C., Pezzetta, R., Toffalini, E., Grassi, M., Cona, G., Miniussi, C., Bauer, P. J., Borgomaneri, S., Brysbaert, M., Chambers, C., et al. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02705-3 · Cited by 10
Self-citationA Study on Reproducibility and Replicability of Table Structure Recognition Methods
Ajayi, K., Choudhury, M. H., Rajtmajer, S. M., & Wu, J. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41679-8_1 · Cited by 9
An advanced method to streamline p-hacking
Sarstedt, M., & Adler, S. J. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.113942 · Cited by 10
Failing to replicate predicts citation declines in psychology
Clark, C. J., Connor, P. R., & Isch, C. G. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2304862120 · Cited by 8
The problem of a hammer: Eyewitness identification research relies on the wrong comparisons.
Hyman, I. E. (2021) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2021.06.004 · Cited by 9
Nuijten, M. B., & Wicherts, J. M. (2023) · doi:10.31234/osf.io/bxau9 · Cited by 9
Self-citationPerils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/tg79n · Cited by 10
Testing Replicability and Generalizability of the Time on Task Effect
Krämer, R. J., Koch, M., Levacher, J., & Schmitz, F. (2023) · Journal of Intelligence · doi:10.3390/jintelligence11050082 · Cited by 8
Replication Studies Using Secondary or Nonexperimental Datasets
Huang, F. L., & Huang, A. B. (2024) · School Psychology Review · doi:10.1080/2372966x.2024.2346781 · Cited by 8
A Systematic Review of Aphantasia: Concept, Measurement, Neural Basis, and Theory Development
Jin, F., Hsu, S., & Li, Y. (2024) · Vision · doi:10.3390/vision8030056 · Cited by 14
Paterson, J. L., Walters, M. A., Brown, R. J., & Carrasco, D. (2024) · Journal of Interpersonal Violence · doi:10.1177/08862605241286455 · Cited by 8
Rubin, M. (2025) · European Journal for Philosophy of Science · doi:10.1007/s13194-024-00629-x · Cited by 13
Skubera, M., Korbmacher, M., Evans, T. R., Azevedo, F., & Pennington, C. R. (2025) · Royal Society Open Science · doi:10.1098/rsos.241726 · Cited by 16
Mundinger, C., Schulz, N. K. E., Singh, P., Janz, S., Schurig, M., Seidemann, J., Kurtz, J., Müller, C., Schielzeth, H., von Kortzfleisch, V. T., et al. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003019 · Cited by 8
Metadata
- Authors
- Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
- Affiliations
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Wisconsin–Madison (US)
- University of California, Davis (US)
- Grand Valley State University (US)
- Stockholm School of Economics (SE)
- The University of Melbourne (AU)
- Illinois State University (US)
- Center for Open Science, Charlottesville, Virginia 22903, USA
- Tilburg University (NL)
- Leipzig University (DE)
- University of Groningen (NL)
- Eindhoven University of Technology (NL)
- University of Colorado Anschutz (US)
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2022-01-04
- Volume
- 73
- Issue
- 1
- Pages
- 719-748
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W3161588210
- Cited by
- 770
- References
- 253
- Citations by year
- 2026: 109 · 2025: 192 · 2024: 180 · 2023: 177 · 2022: 92
- FWCI
- 19.02
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation
- Patient-Centered Outcomes Research Institute
- Arnold Ventures
- European Commission (HORIZON2020, 841188)
- University of Melbourne
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Templeton World Charity Foundation
- Templeton Religion Trust
- National Institutes of Health
- Agency for Healthcare Research and Quality
- Australian Research Council (FT150100297)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, replicability, reproducibility, robustness, research practices, replication study, research incentives
- MeSH terms
- Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
