The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index.
Replicability, Robustness, and Reproducibility in Psychological Science
Open accessPublished version of a preprint
Abstract
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using a common approach to analyze data from multiple informants, time points, and developmental domains, we can address concerns about the robustness of scientific findings (20,21).”
From A Comprehensive Multilevel Analysis of the Bucharest Early Intervention Project: Causal Effects on Recovery From Early Severe Deprivation (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research into replicability in other disciplines has also found that findings with stronger initial evidence (such as larger effect sizes and/or smaller p-values) is more likely to replicate”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Independent verification is central to the scientific method (Nosek et al., 2022).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Nosek et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“A substantial proportion of replications in recent largescale efforts have failed to support the original finding (Nosek et al., 2022).”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Nosek et al., 2022
In the citing paper
“However, particular challenges arise when applying these principles to LLMs, particularly proprietary ones, whose black-box nature impedes the reproducibility of findings that pertain to them.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–256 of 256
Giner‐Sorolla, R. (2018) · European Review of Social Psychology · doi:10.1080/10463283.2018.1542902 · Cited by 52
Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018) · PeerJ Preprints · doi:10.7287/peerj.preprints.26968v1 · Cited by 50
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
Self-citationThe Reputational Consequences of Failed Replications and Wrongness Admission among Scientists
Fetterman, A. K., & Sassenberg, K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0143723 · Cited by 47
The Prior Odds of Testing a True Effect in Cognitive and Social Psychology
Wilson, B. M., & Wixted, J. T. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918767122 · Cited by 47
The best time to argue about what a replication means? Before you do it
Nosek, B. A., & Errington, T. M. (2020) · Nature · doi:10.1038/d41586-020-02142-6 · Cited by 47
Self-citationGordon, M., Viganola, D., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Twardy, C. R., et al. (2020) · Royal Society Open Science · doi:10.1098/rsos.200566 · Cited by 47
Self-citationJanke, S., Daumiller, M. H., & Rudert, S. C. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618790227 · Cited by 46
Krishna, A., & Peter, S. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0203470 · Cited by 46
Brief Notes: Statistical Power Analysis and Research Results
Cohen, J. (1973) · American Educational Research Journal · doi:10.3102/00028312010003225 · Cited by 46
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., et al. (2020) · doi:10.31222/osf.io/9sz2y · Cited by 44
Self-citationGordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44
Self-citationGhelfi, E. A., Christopherson, C. D., Urry, H. L., Lenne, R. L., Legate, N., Fischer, M. A., Wagemans, F. M. A., Wiggins, B., Barrett, T., Bornstein, M., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919881152 · Cited by 42
Schwarz, N., & Strack, F. (2014) · Cited by 36
Probabilistic forecasting of replication studies
Pawel, S., & Held, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231416 · Cited by 35
An N-pact factor for clinical psychological research.
Reardon, K. W., Smack, A. J., Herzhoff, K., & Tackett, J. L. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000435 · Cited by 35
Verify original results through reanalysis before replicating
Nuijten, M. B., Bakker, M., Maassen, E., & Wicherts, J. M. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000791 · Cited by 33
Self-citationResearch in the History of Economic Thought and Methodology
Fiorito, L., Scheall, S. D., & Suprinyak, C. E. (2018) · Research in the history of economic thought and methodology · doi:10.1108/s0743-4154201836a · Cited by 32
Dutch agency launches first grants programme dedicated to replication
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.20287 · Cited by 29
New Statistical Metrics for Multisite Replication Projects
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12572 · Cited by 29
Predicting and reasoning about replicability using structured groupsPreprint
Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Wilkinson, D. P., Hemming, V., Bush, M., Fraser, H. S., Thorn, F. S., McBride, M. F., et al. (2021) · doi:10.31222/osf.io/vtpmb · Cited by 28
Self-citationWagge, J., Baciu, C., Banas, K., Nadler, J. T., Schwarz, S., Weisberg, Y. J., IJzerman, H., Legate, N., & Grahe, J. E. (2019) · Collabra Psychology · doi:10.1525/collabra.177 · Cited by 28
Schnall, S., Johnson, D. J., Cheung, F., & Brent Donnellan, M. (2014) · Social Psychology · doi:10.1027/1864-9335/a000204 · Cited by 25
Peirce and the Trivialization of the Self-Corrective Thesis
Laudan, L. (1981) · Springer eBooks · doi:10.1007/978-94-015-7288-0_14 · Cited by 25
A multi-semester classroom demonstration yields evidence in support of the facial feedback effect.
Marsh, A. A., Rhoads, S. A., & Ryan, R. M. (2018) · Emotion · doi:10.1037/emo0000532 · Cited by 25
Brand, A., & Bradley, M. T. (2016) · Psychological Reports · doi:10.1177/0033294115625265 · Cited by 25
Petty, R. E., & Cacioppo, J. T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.12.007 · Cited by 23
Swift, J. K., Christopherson, C. D., Bird, M. O., Zöld, A. L., & Goode, J. (2020) · Training and Education in Professional Psychology · doi:10.1037/tep0000322 · Cited by 22
Is there evidence for cross-domain congruency sequence effect? A replication of Kan et al . (2013)
Aczél, B., Kovács, M., Bognar, M., Pálfi, B., Hartanto, A., Onie, S., Tiong, L. E., & Evans, T. R. (2021) · Royal Society Open Science · doi:10.1098/rsos.191353 · Cited by 21
Isager, P. M., van Aert, R. C. M., Bahník, Š., et al. (2020) · doi:10.31222/osf.io/2gurz · Cited by 20
Credibility Beyond Replicability: Improving the Four Validities in Psychological SciencePreprint
Vazire, S., Schiavone, S. R., & Bottesini, J. G. (2020) · doi:10.31234/osf.io/bu4d3 · Cited by 20
Self-citationData sharing practices in randomized trials of addiction interventions
Vassar, M. B., Jellison, S., Wendelbo, H., & Wayant, C. (2019) · Addictive Behaviors · doi:10.1016/j.addbeh.2019.106193 · Cited by 20
Ferguson, M. J., Carter, T. J., & Hassin, R. R. (2014) · Cited by 19
Hughes, B. W. (2018) · Cited by 19
Observe, hypothesize, test, repeat: Luttrell, Petty and Xu (2017) demonstrate good science
Ebersole, C. R., Alaei, R., Atherton, O. E., Bernstein, M. J., Brown, M., Chartier, C. R., Chung, L. Y., Hermann, A. D., Joy-Gaba, J. A., Line, M., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.12.005 · Cited by 15
Self-citationLeighton, D. C., Legate, N., LePine, S., Anderson, S. F., & Grahe, J. E. (2018) · Psi Chi Journal of Psychological Research · doi:10.24839/2325-7342.jn23.2.98 · Cited by 15
(1978) · Issue · doi:10.2307/1166676 · Cited by 14
Support for the replicability of imagined contact effects
Crisp, R. J., Miles, E., & Husnu, S. (2014) · Sussex Research Online (University of Sussex) · Cited by 14
Towards wide-scale adoption of open science practices: The role of open science communitiesPreprint
Armeni, K., Brinkman, L., Carlsson, R., et al. (2020) · doi:10.31222/osf.io/7gct9 · Cited by 13
Reflection on the Professor-Priming Replication Report
Dijksterhuis, A. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618755705 · Cited by 12
Competition for novelty reduces information sampling in a research game - a registered report
Tiokhin, L., & Derex, M. (2019) · Royal Society Open Science · doi:10.1098/rsos.180934 · Cited by 11
A Multi-Site Collaborative Study of the Hostile Priming Effect
McCarthy, R. J., Gervais, W. M., Aczél, B., Al‐Kire, R. L., Aveyard, M. E., Baraldo, S. M., Baruh, L., Basch, C., Baumert, A., Behler, A. M. C., et al. (2021) · Collabra Psychology · doi:10.1525/collabra.18738 · Cited by 10
Kossmeier, M., Vilsmeier, J. K., Dittrich, R., Fritz, T., Kolmanz, C., Plessen, C. Y., Slowik, A., Tran, U. S., & Voracek, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000384 · Cited by 10
Meet the ‘data thugs' out to expose shoddy and questionable research
(2018) · AAAS Articles DO Group · doi:10.1126/science.aat3133 · Cited by 8
Fox, N. W., Honeycutt, N., & Jussim, L. J. (2018) · doi:10.31234/osf.io/3v7hx · Cited by 7
McCarthy, R. J., Hartnett, J., Heider, J. D., Scherer, C. R., Wood, S., Nichols, A., Edlund, J. E., & Walker, W. (2018) · International Review of Social Psychology · doi:10.5334/irsp.133 · Cited by 6
Commentary and Rejoinder on and
Shih, M. J., Pittinsky, T. L., Moon, A., & Roeder, S. S. (2014) · Social Psychology · doi:10.1027/1864-9335/a000207 · Cited by 5
Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. D. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/5zvmh · Cited by 5
Self-citationMoran, T., Hughes, S., Hussey, I., Vadillo, M. A., Olson, M. A., Aust, F., Bading, K. C., Balas, R., Benedict, T., Corneille, O., et al. (2020) · doi:10.31234/osf.io/kavx4 · Cited by 4
Wagge, J., Baciu, C., Banas, K., et al. (2018) · doi:10.31234/osf.io/chax8 · Cited by 3
Reflections on positive stereotypes research and on replications.
Shih, M. J., & Pittinsky, T. L. (2014) · Cited by 3
Support for Open Science Practices in Emotion Science: A Survey StudyPreprint
van den Akker, O. R., Scherer, L. D., Wicherts, J. M., & Koole, S. L. (2020) · doi:10.31234/osf.io/ub4wc · Cited by 3
Self-citationMischaracterizing replication studies leads to erroneous conclusionsPreprint
Nosek, B. A., & Gilbert, E. A. (2017) · doi:10.31234/osf.io/nt4d3 · Cited by 2
Self-citationClean Data: Statistical Artifacts Wash Out Replication Efforts
Schnall, S. (2014) · Apollo (University of Cambridge) · Cited by 2
Post-replication citation patterns in psychology: Four case studies
Hardwicke, T. E. (2021) · Code Ocean · doi:10.24433/co.4225975.v1 · Cited by 1
Self-citation
Research works printed but not linked: 17
- Artner R, 2020, Psychol. Methods.
- Cent. Open Sci, 2020, Center for Open Science
- Cohen J., 1973, Am. Educ. Res. J., 10, 225
- 10.1037/0033-2909.112.1.155 · DOI · Paper page
- Crisp RJ, 2014, Soc. Psychol., 45, 303
- Epskamp S, 2018, Statistical Software
- Errington TM, 2021, eLife
- Ferguson MJ, 2014, Soc. Psychol., 45, 301
- Gelman A, 2013, The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
- Marcus A, 2020, WIRED
- Meyer MN, 2014, Slate Magazine
- Mischel W., 2008, APS Observer
- Nosek B., 2019, Center for Open Science
- Nosek B., 2019, Center for Open Science
- Schwarz N, 2014, Soc. Psychol., 45, 305
- Sterling TD., 1959, J. Am. Stat. Assoc., 54, 30
- Yong E, 2012, National Geo-graphic
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationBergdahl, J., Latikka, R., Celuch, M., Savolainen, I., Soares Mantere, E., Savela, N., & Oksanen, A. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102013 · Cited by 204
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Metadata
- Authors
- Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
- Affiliations
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Wisconsin–Madison (US)
- University of California, Davis (US)
- Grand Valley State University (US)
- Stockholm School of Economics (SE)
- The University of Melbourne (AU)
- Illinois State University (US)
- Center for Open Science, Charlottesville, Virginia 22903, USA
- Tilburg University (NL)
- Leipzig University (DE)
- University of Groningen (NL)
- Eindhoven University of Technology (NL)
- University of Colorado Anschutz (US)
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2022-01-04
- Volume
- 73
- Issue
- 1
- Pages
- 719-748
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W3161588210
- Cited by
- 770
- References
- 253
- Citations by year
- 2026: 109 · 2025: 192 · 2024: 180 · 2023: 177 · 2022: 92
- FWCI
- 19.02
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation
- Patient-Centered Outcomes Research Institute
- Arnold Ventures
- European Commission (HORIZON2020, 841188)
- University of Melbourne
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Templeton World Charity Foundation
- Templeton Religion Trust
- National Institutes of Health
- Agency for Healthcare Research and Quality
- Australian Research Council (FT150100297)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, replicability, reproducibility, robustness, research practices, replication study, research incentives
- MeSH terms
- Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
