The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index.
Replicability, Robustness, and Reproducibility in Psychological Science
Open accessPublished version of a preprint
Abstract
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using a common approach to analyze data from multiple informants, time points, and developmental domains, we can address concerns about the robustness of scientific findings (20,21).”
From A Comprehensive Multilevel Analysis of the Bucharest Early Intervention Project: Causal Effects on Recovery From Early Severe Deprivation (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research into replicability in other disciplines has also found that findings with stronger initial evidence (such as larger effect sizes and/or smaller p-values) is more likely to replicate”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Independent verification is central to the scientific method (Nosek et al., 2022).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Nosek et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“A substantial proportion of replications in recent largescale efforts have failed to support the original finding (Nosek et al., 2022).”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Nosek et al., 2022
In the citing paper
“This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
In the citing paper
“However, particular challenges arise when applying these principles to LLMs, particularly proprietary ones, whose black-box nature impedes the reproducibility of findings that pertain to them.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,559
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
(2002) · Cited by 13,211
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citation
Research works printed but not linked: 17
- Artner R, 2020, Psychol. Methods.
- Cent. Open Sci, 2020, Center for Open Science
- Cohen J., 1973, Am. Educ. Res. J., 10, 225
- 10.1037/0033-2909.112.1.155 · DOI · Paper page
- Crisp RJ, 2014, Soc. Psychol., 45, 303
- Epskamp S, 2018, Statistical Software
- Errington TM, 2021, eLife
- Ferguson MJ, 2014, Soc. Psychol., 45, 301
- Gelman A, 2013, The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
- Marcus A, 2020, WIRED
- Meyer MN, 2014, Slate Magazine
- Mischel W., 2008, APS Observer
- Nosek B., 2019, Center for Open Science
- Nosek B., 2019, Center for Open Science
- Schwarz N, 2014, Soc. Psychol., 45, 305
- Sterling TD., 1959, J. Am. Stat. Assoc., 54, 30
- Yong E, 2012, National Geo-graphic
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationBergdahl, J., Latikka, R., Celuch, M., Savolainen, I., Soares Mantere, E., Savela, N., & Oksanen, A. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102013 · Cited by 204
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
A synthesis of evidence for policy from behavioural science during COVID-19
Ruggeri, K., Stock, F., Haslam, S. A., Capraro, V., Boggio, P. S., Ellemers, N., Cichocka, A. K., Douglas, K. M., Rand, D. G., van der Linden, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06840-9 · Cited by 121
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111
Klein, R. A., Cook, C. L., Ebersole, C. R., Vitiello, C., Nosek, B. A., Hilgard, J., Ahn, P. H., Brady, A. J., Chartier, C. R., Christopherson, C. D., et al. (2022) · Collabra Psychology · doi:10.1525/collabra.35271 · Cited by 193
Self-citationCredibility Beyond Replicability: Improving the Four Validities in Psychological Science
Vazire, S., Schiavone, S. R., & Bottesini, J. G. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214211067779 · Cited by 115
Self-citationOpen and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 112
Kinds of Replication: Examining the Meanings of “Conceptual Replication” and “Direct Replication”
Derksen, M., & Morawski, J. G. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211041116 · Cited by 104
The concept of inhibition in bilingual control.
Declerck, M., & Koch, I. (2022) · Psychological Review · doi:10.1037/rev0000367 · Cited by 96
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 110
Why are people antiscience, and what can we do about it?
Philipp-Muller, A. Z., Lee, S. W. S., & Petty, R. E. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120755119 · Cited by 115
Rodríguez-Nieto, G., Seer, C., Sidlauskaite, J., Vleugels, L., Van Roy, A., Hardwick, R., & Swinnen, S. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119665 · Cited by 89
Functional neuroimaging in psychiatry and the case for failing better
Nour, M. M., Liu, Y., & Dolan, R. J. (2022) · Neuron · doi:10.1016/j.neuron.2022.07.005 · Cited by 92
Music Training and Nonmusical Abilities
Schellenberg, E. G., & Lima, C. F. (2023) · Annual Review of Psychology · doi:10.1146/annurev-psych-032323-051354 · Cited by 107
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
Why most psychological research findings are not even wrong
Scheel, A. M. (2022) · Infant and Child Development · doi:10.1002/icd.2295 · Cited by 111
Self-citationReproducibility in Neuroimaging Analysis: Challenges and Solutions
Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 95
Survey of open science practices and attitudes in the social sciences
Ferguson, J., Littman, R., Christensen, G., Paluck, E. L., Swanson, N., Wang, Z., Miguel, E. A., Birke, D. J., & Pezzuto, J.-H. (2023) · Nature Communications · doi:10.1038/s41467-023-41111-1 · Cited by 112
The five pillars of computational reproducibility: bioinformatics and beyond
Ziemann, M., Poulain, P., & Bora, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad375 · Cited by 106
A discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 99
Platform-controlled social media APIs threaten open science
Davidson, B. I., Wischerath, D., Racek, D., Parry, D. A., Godwin, E., Hinds, J. M., van der Linden, D., Roscoe, J. F., Ayravainen, L. E. M., & Cork, A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01750-2 · Cited by 75
Lechat, B., Loffler, K. A., Wallace, D. M., Reynolds, A. C., Appleton, S., Scott, H., Vakulin, A., Lovato, N., Adams, R., Eckert, D. J., et al. (2022) · Nature and Science of Sleep · doi:10.2147/nss.s379252 · Cited by 75
Higgins, W. C., Kaplan, D. M., Deschrijver, E., & Ross, R. M. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2023.102378 · Cited by 80
Renton, A., Dao, T. T., Johnstone, T., Civier, O., Sullivan, R. P., White, D., Lyons, P., Slade, B., Abbott, D. F., Amos, T. J., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02145-x · Cited by 75
Generalization Bias in Science
Peters, U., Krauss, A., & Braganza, O. (2022) · Cognitive Science · doi:10.1111/cogs.13188 · Cited by 63
Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66
Current research practices on pro-environmental behavior: A survey of environmental psychologists
Brick, C., Nielsen, K. S., Berger, S., Henn, L., Wolske, K. S., Lange, F., Hanss, D., Bauer, J. M., Aldoh, A., Sachisthal, M. S. M., et al. (2024) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2024.102375 · Cited by 85
Study design features increase replicability in brain-wide association studies
Kang, K., Seidlitz, J., Bethlehem, R. A. I., Xiong, J., Jones, M. T., Mehta, K. P., Keller, A. S., Tao, R., Randolph, A., Larsen, B., et al. (2024) · Nature · doi:10.1038/s41586-024-08260-9 · Cited by 66
Good scientific practice in EEG and MEG research: Progress and perspectives
Niso, G., Krol, L. R., Combrisson, E., Dubarry, A., Elliott, M., Clément, F., Héjja‐Brichard, Y., Herbst, S. K., Jerbi, K., Ković, V., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119056 · Cited by 68
Boosting: Empowering Citizens with Behavioral Science
Herzog, S. M., & Hertwig, R. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-020924-124753 · Cited by 73
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 66
Self-citationPownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z. M., Grinschgl, S., Elsherif, M. M., Gilligan-Lee, K. A., Oliveira, C. M., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221255 · Cited by 68
Adversarial Collaboration: The Next Science Reform
Clark, C. J., & Tetlock, P. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-29148-7_32 · Cited by 60
Baumeister, R. F., Tice, D. M., & Bushman, B. J. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221121815 · Cited by 53
Survey on Open Science Practices in Functional Neuroimaging
Paret, C., Unverhau, N., Feingold, F. W., Poldrack, R. A., Stirner, M., Schmahl, C. G., & Sicorello, M. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119306 · Cited by 51
Brennan, B., Kelman, A. R., & Belser, A. (2023) · Psychedelic Medicine · doi:10.1089/psymed.2023.0007 · Cited by 50
DeKay, M. L., Rubinchik, N., Li, Z., & de Boeck, P. A. L. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221079611 · Cited by 51
(Why) Are Open Research Practices the Future for the Study of Language Learning?
Marsden, E. J., & Morgan‐Short, K. (2023) · Language Learning · doi:10.1111/lang.12568 · Cited by 49
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2269357 · Cited by 57
The Emotional Rewards of Prosocial Spending Are Robust and Replicable in Large Samples
Aknin, L. B., Dunn, E. W., & Whillans, A. V. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221121100 · Cited by 47
Heino, M. T. J., Proverbio, D., Marchand, G. C., Resnicow, K., & Hankonen, N. E. (2022) · Health Psychology Review · doi:10.1080/17437199.2022.2146598 · Cited by 47
de Oliveira, G. S., & Baggs, E. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009303538 · Cited by 57
Conspiracy theories and climate change: A systematic review
Tam, K., & Chan, H. (2023) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2023.102129 · Cited by 59
Taboos and Self-Censorship Among U.S. Psychology Professors
Clark, C. J., Fjeldmark, M., Lu, L., Baumeister, R. F., Ceci, S. J., Frey, K., Miller, G. F., Reilly, W., Tice, D. M., von Hippel, W., et al. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916241252085 · Cited by 51
Investigating the replicability of the social and behavioural sciences
Tyner, A. H., Abatayo, A. L., Daley, M., Field, S., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Miske, O., et al. (2026) · Nature · doi:10.1038/s41586-025-10078-y · Cited by 54
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., Phan, L. V., Schmitt, M., Scheel, A. M., Schubert, A., et al. (2022) · doi:10.31234/osf.io/rgh5b · Cited by 40
Self-citationSingle-Night Diagnosis of Sleep Apnea Contributes to Inconsistent Cardiovascular Outcome Findings
Lechat, B., Nguyen, D. P., Reynolds, A. C., Loffler, K. A., Escourrou, P., McEvoy, R. D., Adams, R., CATCHESIDE, P. G., & Eckert, D. J. (2023) · CHEST Journal · doi:10.1016/j.chest.2023.01.027 · Cited by 43
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Tzavella, L., Bendixen, T., Handcock, S. A., Köneke, V. E., & Ioannidis, J. P. A. (2022) · Royal Society Open Science · doi:10.1098/rsos.220139 · Cited by 38
Self-citationBest Practices for Ethical Conduct of Misinformation Research
Greene, C. M., Laurent, C. D. S., Murphy, G., Prike, T., Hegarty, K., & Ecker, U. K. H. (2022) · European Psychologist · doi:10.1027/1016-9040/a000491 · Cited by 50
Toward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 50
Kohrs, F. E., Auer, S., Bannach‐Brown, A., Fiedler, S., Haven, T. L., Heise, V., Holman, C., Azevedo, F., Bernard, R., Bleier, A., et al. (2023) · eLife · doi:10.7554/elife.89736 · Cited by 55
Kilinc‐Ata, N., Kaya, E., & Barut, A. (2024) · Journal of the Knowledge Economy · doi:10.1007/s13132-024-02452-4 · Cited by 37
A manifesto for a globally diverse, equitable, and inclusive open science
Ghai, S., Thériault, R., Forscher, P., Shoda, Y., Syed, M., Puthillam, A., Peng, H. C., Basnight-Brown, D., Majid, A., Azevedo, F., et al. (2025) · Communications Psychology · doi:10.1038/s44271-024-00179-1 · Cited by 32
Bavelaar, L., Visser, M., Walshe, C., Preston, N. J., Kaasalainen, S., Sussman, T., Cornally, N., Hartigan, I., Loučka, M., Giulio, P. D., et al. (2023) · Age and Ageing · doi:10.1093/ageing/afad027 · Cited by 39
Factors influencing trust in algorithmic decision-making: an indirect scenario-based experiment
Marmolejo‐Ramos, F., Marrone, R. L., Korolkiewicz, M. W., Gabriel, F. C., Siemens, G., Joksimovíc, S., Yamada, Y., Mori, Y., Rahwan, T., Sahakyan, M., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1465605 · Cited by 45
Panteleimon Ekkekakis, A., Swinton, P. A., & Tiller, N. B. (2023) · Sports Medicine · doi:10.1007/s40279-023-01880-7 · Cited by 39
Cui, F., Huang, X., Jing, Y., Luo, Y., Liu, J., & Gu, R. (2022) · Cerebral Cortex · doi:10.1093/cercor/bhac017 · Cited by 32
Using preregistration as a tool for transparent fNIRS study design
Schroeder, P. A., Artemenko, C., Kosie, J. E., Cockx, H., Stute, K., Pereira, J., Klein, F., & Mehler, D. M. A. (2023) · Neurophotonics · doi:10.1117/1.nph.10.2.023515 · Cited by 36
Bucher, A., Blazek, E. S., & Symons, C. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.05.007 · Cited by 41
Yu, C., & Wellman, H. M. (2024) · Developmental Review · doi:10.1016/j.dr.2024.101162 · Cited by 36
King, L. S., Guyon-Harris, K. L., Valadez, E. A., Radulescu, A., Fox, N. A., Nelson, C. A., Zeanah, C. H., & Humphreys, K. L. (2023) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20220672 · Cited by 29
No Man’s Hand: Artificial Intelligence Does Not Improve Police Report Writing Speed
Adams, I. T., Barter, M., McLean, K. D., Boehme, H. M., & Geary, I. A. (2024) · Journal of Experimental Criminology · doi:10.1007/s11292-024-09644-7 · Cited by 42
Boukrouh, S., Mnaouer, I., de Souza, P. M., Hornick, J., Nilahyane, A., Amiri, B. E., & Hirich, A. (2025) · Archives animal breeding/Archiv für Tierzucht · doi:10.5194/aab-68-223-2025 · Cited by 36
Gernigon, C., Hartigh, R. J. R. D., Vallacher, R. R., & van Geert, P. L. C. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231187324 · Cited by 35
The pervasive impact of ignorance
Kirfel, L., & Phillips, J. (2022) · Cognition · doi:10.1016/j.cognition.2022.105316 · Cited by 29
Zettersten, M., Yurovsky, D., Xu, T. L., Uner, S., Tsui, A. S. M., Schneider, R. M., Saleh, A. N., Meylan, S. C., Marchman, V. A., Mankewitz, J., et al. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01906-4 · Cited by 30
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28
Smallset Timelines: A Visual Representation of Data Preprocessing Decisions
Lucchesi, L. R., Kuhnert, P., Davis, J. L., & Xie, L. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533175 · Cited by 28
Wojtczuk‐Turek, A., Turek, D., Edgar, F., Klein, H. J., Bosak, J., Okay‐Somerville, B., Fu, N., Raeder, S., Jurek, P., Lupina‐Wegener, A., et al. (2024) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.2815 · Cited by 34
Uher, J. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1009893 · Cited by 33
Wilkinson, S., Ribeiro, L., Krägeloh, C. U., Bergomi, C. R., Parsons, M., Siegling, A. B., Tschacher, W., Kupper, Z., & Medvedev, O. N. (2023) · Mindfulness · doi:10.1007/s12671-023-02099-3 · Cited by 31
Questionable Metascience Practices
Rubin, M. (2023) · Journal of Trial and Error · doi:10.36850/mr4 · Cited by 40
Toward open science in PLS-SEM: Assessing the state of the art and future perspectives
Adler, S. J., Sharma, P. N., & Radomir, L. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114291 · Cited by 40
The Moderating Role of Culture in the Generalizability of Psychological Phenomena
Schimmelpfennig, R., Spicer, R., White, C., Gervais, W. M., Norenzayan, A., Heine, S. J., Henrich, J., & Muthukrishna, M. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231225163 · Cited by 33
The challenges of open data sharing for qualitative researchers
Lamb, D., Russell, A. M., Morant, N. J., & Stevenson, F. A. (2024) · Journal of Health Psychology · doi:10.1177/13591053241237620 · Cited by 26
Chien, C., Hsu, S., & Lin, T.-H. (2024) · Social Psychology of Education · doi:10.1007/s11218-024-09905-3 · Cited by 30
Practical Implications of Sum Scores Being Psychometrics’ Greatest Accomplishment
McNeish, D. (2024) · Psychometrika · doi:10.1007/s11336-024-09988-z · Cited by 29
Assessing and Improving Robustness of Psychological Research Findings in Four Steps
Nuijten, M. B. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_17 · Cited by 26
Self-citationMass Reproducibility and Replicability: A New HopePreprint
Brodeur, A., Mikola, D., & Cook, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790780 · Cited by 31
Clarke, B., Alley, L. J., Ghai, S., Flake, J. K., Rohrer, J. M., Simmons, J. P., Schiavone, S. R., & Vazire, S. (2024) · Social and Personality Psychology Compass · doi:10.1111/spc3.12979 · Cited by 33
Self-citationMaier, M., Bartos̆, F., Oh, M., Wagenmakers, E., Shanks, D. R., & Harris, A. J. L. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/r8nyu · Cited by 28
Brysbaert, M. (2024) · Cognitive Research Principles and Implications · doi:10.1186/s41235-024-00540-2 · Cited by 35
Opportunities for improving data sharing and FAIR data practices to advance global mental health
Sadeh, Y., Denejkina, A., Karyotaki, E., Lenferink, L. I. M., & Kassam‐Adams, N. (2023) · Cambridge Prisms Global Mental Health · doi:10.1017/gmh.2023.7 · Cited by 23
Type I Error Rates are Not Usually Inflated
Rubin, M. (2024) · Journal of Trial and Error · doi:10.36850/4d35-44bd · Cited by 30
Center for Open Science: Strategic PlanPreprint
Nosek, B. A., Shaw, L. C., Errington, T. M., Pfeiffer, N., & Zaringhalam, M. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/x2w9h · Cited by 30
Self-citationBinge-Watching, Self-Determination, and Well-Being
Erdmann, E., & Dienlin, T. (2022) · Journal of Media Psychology Theories Methods and Applications · doi:10.1027/1864-1105/a000334 · Cited by 26
Grund, S., Lüdtke, O., & Robitzsch, A. (2022) · Psychological Methods · doi:10.1037/met0000526 · Cited by 27
Moeller, J., Dietrich, J., Neubauer, A. B., Brose, A., Kühnel, J., Dehne, M., Jaehne, M. F., Schmiedek, F., Bellhäuser, H., Malmberg, L., et al. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/5wsna · Cited by 26
Toward open science in marketing research
Deer, L., Adler, S. J., Datta, H., Mizik, N., & Sarstedt, M. (2024) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.12.005 · Cited by 28
Solana, P., & Santiago, J. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104834 · Cited by 24
Otterbring, T., Samuelsson, P., Arsenovic, J. F., Elbæk, C. T., & Folwarczny, M. (2022) · European Journal of Marketing · doi:10.1108/ejm-04-2022-0250 · Cited by 22
Davis, E. B., Lacey, E. K., Heydt, E. J., LaBouff, J. P., Barker, S. B., van Elk, M., & Ladd, K. L. (2022) · Psychology of Religion and Spirituality · doi:10.1037/rel0000485 · Cited by 22
The social psychology of everyday life
Hofmann, W., & Grigoryan, L. K. (2023) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2023.06.001 · Cited by 30
A guide for social science journal editors on easing into open science
Silverstein, P., Elman, C., Montoya, A. K., McGillivray, B. C., Pennington, C. R., Harrison, C. H., Steltenpohl, C. N., Röer, J. P., Corker, K. S., Charron, L., et al. (2024) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00141-5 · Cited by 28
Self-citationOpen developmental science: An overview and annotated reading list
Kalandadze, T., & Hart, S. A. (2022) · Infant and Child Development · doi:10.1002/icd.2334 · Cited by 23
Unconscious integration: Current evidence for integrative processing under subliminal conditions
Zher‐Wen, & Yu, R. (2023) · British Journal of Psychology · doi:10.1111/bjop.12631 · Cited by 26
Metadata
- Authors
- Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
- Affiliations
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Wisconsin–Madison (US)
- University of California, Davis (US)
- Grand Valley State University (US)
- Stockholm School of Economics (SE)
- The University of Melbourne (AU)
- Illinois State University (US)
- Center for Open Science, Charlottesville, Virginia 22903, USA
- Tilburg University (NL)
- Leipzig University (DE)
- University of Groningen (NL)
- Eindhoven University of Technology (NL)
- University of Colorado Anschutz (US)
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2022-01-04
- Volume
- 73
- Issue
- 1
- Pages
- 719-748
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W3161588210
- Cited by
- 770
- References
- 253
- Citations by year
- 2026: 109 · 2025: 192 · 2024: 180 · 2023: 177 · 2022: 92
- FWCI
- 19.02
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation
- Patient-Centered Outcomes Research Institute
- Arnold Ventures
- European Commission (HORIZON2020, 841188)
- University of Melbourne
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Templeton World Charity Foundation
- Templeton Religion Trust
- National Institutes of Health
- Agency for Healthcare Research and Quality
- Australian Research Council (FT150100297)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, replicability, reproducibility, robustness, research practices, replication study, research incentives
- MeSH terms
- Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
