The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index.
Replicability, Robustness, and Reproducibility in Psychological Science
Open accessPublished version of a preprint
Abstract
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using a common approach to analyze data from multiple informants, time points, and developmental domains, we can address concerns about the robustness of scientific findings (20,21).”
From A Comprehensive Multilevel Analysis of the Bucharest Early Intervention Project: Causal Effects on Recovery From Early Severe Deprivation (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research into replicability in other disciplines has also found that findings with stronger initial evidence (such as larger effect sizes and/or smaller p-values) is more likely to replicate”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Nosek et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Independent verification is central to the scientific method (Nosek et al., 2022).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Nosek et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“A substantial proportion of replications in recent largescale efforts have failed to support the original finding (Nosek et al., 2022).”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Nosek et al., 2022
In the citing paper
“This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
In the citing paper
“However, particular challenges arise when applying these principles to LLMs, particularly proprietary ones, whose black-box nature impedes the reproducibility of findings that pertain to them.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,559
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
(2002) · Cited by 13,211
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationConfirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Evaluating Effect Size in Psychological Research: Sense and Nonsense
Funder, D. C., & Ozer, D. J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847202 · Cited by 3,295
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationPromoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
The Aim and Structure of Physical Theory
Duhem, P. M. M. (1954) · doi:10.1515/9780691233857 · Cited by 2,128
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 2,061
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Self-citationHARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Strack, F., Martin, L. L., & Stepper, S. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.5.768 · Cited by 1,898
Gergen, K. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034436 · Cited by 1,845
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Self-citationNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Self-citationShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,102
A Multilab Preregistered Replication of the Ego-Depletion Effect
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H. J. E. M., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brand, M., Brewer, G., Bruyneel, S. D., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616652873 · Cited by 1,097
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Self-citationAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Self-citationReproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924
Role of consciousness and accessibility of death-related thoughts in mortality salience effects.
Greenberg, J., Pyszczynski, T., Solomon, S., Simon, L., & Breus, M. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.67.4.627 · Cited by 901
Effects of need for cognition on message evaluation, recall, and persuasion.
Cacioppo, J., Petty, R., & Morris, K. J. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.4.805 · Cited by 899
Outcome Bias in Decision Evaluation
Baron, J., & Hershey, J. C. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.4.569 · Cited by 873
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773
Self-citationSzucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
Muthukrishna, M., & Henrich, J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0522-1 · Cited by 745
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700
Self-citationMaxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0912-z · Cited by 651
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
Intellectual Humility: Owning Our Limitations
Whitcomb, D., Battaly, H., Baehr, J., & Howard‐Snyder, D. (2017) · Philosophy and Phenomenological Research · doi:10.1111/phpr.12228 · Cited by 610
Rational Expectations and the Aggregation of Diverse Information in Laboratory Security Markets
Plott, C. R., & Sunder, S. (1988) · Econometrica · doi:10.2307/1911360 · Cited by 608
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000083 · Cited by 594
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Self-citationNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Cognitive and Interpersonal Features of Intellectual Humility
Leary, M. R., Diebels, K. J., Davisson, E. K., Jongman-Sereno, K. P., Isherwood, J. C., Raimi, K. T., Deffler, S. A., & Hoyle, R. H. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167217697695 · Cited by 539
Soto, C. J. (2019) · Psychological Science · doi:10.1177/0956797619831612 · Cited by 537
Small is beautiful: In defense of the small-N design
Smith, P. L., & Little, D. R. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1451-8 · Cited by 511
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
Self-citationLakens, D., Adolfi, F., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S., Baguley, T. S., Becker, R. B., Benning, S. D., Bradford, D. E., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0311-x · Cited by 508
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474
Replication, falsification, and the crisis of confidence in social psychology
Earp, B. D., & Trafimow, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00621 · Cited by 472
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468
Self-citationThe ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
The hindsight bias: A meta-analysis
Christensen-Szalanski, J. J. J., & Fobian, C. S. (1991) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(91)90010-q · Cited by 448
Nosek, B. A., & Errington, T. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000691 · Cited by 444
Self-citationRegistered Replication Report: Strack, Martin, & Stepper (1988)
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674458 · Cited by 430
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
Self-citationAnderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 398
Self-citationMoshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918797607 · Cited by 377
Self-citationStatistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376
On the scientific superiority of conceptual replications for scientific progress
Crandall, C. S., & Sherman, J. W. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.002 · Cited by 370
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
Frank, M. C., Bergelson, E., Bergmann, C., Cristià, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Struhl, M. K., Levelt, C. C., et al. (2017) · Infancy · doi:10.1111/infa.12182 · Cited by 353
Ioannidis, J., & Trikalinos, T. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 352
Quantifying Sources of Variability in Infancy Research Using the Infant-Directed-Speech Preference
Frank, M. C., Alcock, K., Arias‐Trejo, N., Aschersleben, G., Baldwin, D. A., Barbu, S., Bergelson, E., Bergmann, C., Black, A. K., Blything, R. P., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919900809 · Cited by 343
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.5.646 · Cited by 340
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Self-citationUsing prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Self-citationInterpreting the predictions of prediction markets
Manski, C. F. (2006) · Economics Letters · doi:10.1016/j.econlet.2006.01.004 · Cited by 330
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Veer, A. V. '., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 326
Self-citationStatistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323
Priming, Replication, and the Hardest Science
Cesario, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513470 · Cited by 320
Data Sharing in Psychology: A Survey on Barriers and Preconditions
Houtkoop, B. L., Chambers, C., Macleod, M. R., Bishop, D., Nichols, T. E., & Wagenmakers, E. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917751886 · Cited by 320
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Three ways to make replication mainstream
Gantman, A. P., Gomila, R., Martinez, J. E., Matias, J., Paluck, E., Starck, J. G., Wu, S., & Yaffe, N. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 302
Changing order: replication and induction in scientific practice
Collins, H. M. (1985) · CERN Document Server (European Organization for Nuclear Research) · Cited by 298
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Self-citationThe Generalizability CrisisPreprint
Yarkoni, T. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/jqw35 · Cited by 290
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036731 · Cited by 287
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
A Unified Framework to Quantify the Credibility of Scientific Findings
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918787489 · Cited by 236
A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect
Vohs, K. D., Schmeichel, B. J., Lohmann, S., Gronau, Q. F., Finley, A. J., Ainsworth, S. E., Alquist, J. L., Baker, M. D., Brizi, A., Bunyi, A., et al. (2021) · Psychological Science · doi:10.1177/0956797621989733 · Cited by 233
Registered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 204
Self-citationSafeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203
Registered Replication Report: Rand, Greene, and Nowak (2012)
Bouwmeester, S., Verkoeijen, P. P. J. L., Aczél, B., Barbosa, F., Bègue, L., Brañas‐Garza, P., Chmura, T., Cornelissen, G., Døssing, F. S., Espín, A. M., et al. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693624 · Cited by 199
Klein, R. A., Cook, C. L., Ebersole, C. R., Vitiello, C., Nosek, B. A., Hilgard, J., Ahn, P. H., Brady, A. J., Chartier, C. R., Christopherson, C. D., et al. (2022) · Collabra Psychology · doi:10.1525/collabra.35271 · Cited by 193
Self-citationResponse to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189
Self-citationA Multilab Replication of the Ego Depletion Effect
Dang, J., Barker, P. A., Baumert, A., Bentvelzen, M., Berkman, E. T., Buchholz, N., Buczny, J., Chen, Z., Cristofaro, V. D., de Vries, L. P., et al. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619887702 · Cited by 189
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
Self-citationMisguided Effort With Elusive Implications
Baumeister, R. F., & Vohs, K. D. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616652878 · Cited by 183
An updated meta-analysis of the ego depletion effect
Dang, J. (2017) · Psychological Research · doi:10.1007/s00426-017-0862-x · Cited by 180
Normative Dissonance in Science: Results from a National Survey of U.S. Scientists
Anderson, M. S., Martinson, B., & de Vries, R. (2007) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2007.2.4.3 · Cited by 174
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Self-citationReproducible research in sport and exercise psychology: The role of sample sizes
Schweizer, G., & Furley, P. (2016) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2015.11.005 · Cited by 168
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
What’s next for Registered Reports?
Chambers, C. (2019) · Nature · doi:10.1038/d41586-019-02674-6 · Cited by 154
Experiences of collaborative research
Kahneman, D. (2003) · American Psychologist · doi:10.1037/0003-066x.58.9.723 · Cited by 149
What a massive database of retracted papers reveals about science publishing's ‘death penalty'
(2018) · AAAS Articles DO Group · doi:10.1126/science.aav8384 · Cited by 144
HARKing's Threat to Organizational Research: Evidence From Primary and Meta‐Analytic Sources
Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2015) · Personnel Psychology · doi:10.1111/peps.12111 · Cited by 139
Specification curve analysisPreprint
Schweinsberg, M., Amireh, H., Tierney, W., Mohamed, Z., & Uhlmann, E. L. (2019) · Cited by 138
An excess of positive results: Comparing the standard Psychology literature with Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2020) · doi:10.31234/osf.io/p6e9c · Cited by 136
Self-citationRethinking Reproducibility as a Criterion for Research Quality
Leonelli, S. (2018) · Research in the history of economic thought and methodology · doi:10.1108/s0743-41542018000036b009 · Cited by 136
Open Science Practices are on the Rise: The State of Social Science (3S) SurveyPreprint
Christensen, G., Wang, Z., Paluck, E. L., Swanson, N., Birke, D. J., Miguel, E. A., & Littman, R. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5rksu · Cited by 135
Towards wide-scale adoption of open science practices: The role of open science communities
Armeni, K., Brinkman, L., Carlsson, R., Eerland, A., Fijten, R., Fondberg, R., Heininga, V. E., Heunis, S., Koh, W. Q., Masselink, M., et al. (2021) · Science and Public Policy · doi:10.1093/scipol/scab039 · Cited by 133
The Value of Preregistration for Psychological Science: A Conceptual AnalysisPreprint
Lakens, D. (2019) · doi:10.31234/osf.io/jbh4w · Cited by 130
Brown, N. J. L., & Heathers, J. A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550616673876 · Cited by 127
Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920918872 · Cited by 127
Klein, R. A., Cook, C. L., Ebersole, C. R., et al. (2019) · doi:10.31234/osf.io/vef2c · Cited by 125
Underreporting in Psychology Experiments
Franco, A., Malhotra, N. A., & Simonovits, G. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 119
Noah, T., Schul, Y., & Mayo, R. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000121 · Cited by 119
Registered Replication Report on Mazar, Amir, and Ariely (2008)
Verschuère, B., Meijer, E. H., Jim, A., Hoogesteyn, K., Orthey, R., McCarthy, R. J., Skowronski, J. J., Acar, O. A., Aczél, B., Bakos, B. E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918781032 · Cited by 118
Charting the future of social psychology on stormy seas: Winners, losers, and recommendations
Baumeister, R. F. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.02.003 · Cited by 116
Metastudies for robust tests of theory
Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C., Boeck, P. D., & Vandekerckhove, J. (2018) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1708285114 · Cited by 115
Credibility Beyond Replicability: Improving the Four Validities in Psychological Science
Vazire, S., Schiavone, S. R., & Bottesini, J. G. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214211067779 · Cited by 115
Self-citationMotyl, M. S., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B., Prims, J., Sun, J., Washburn, A. N., Wong, K., et al. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000084 · Cited by 114
Devezer, B., Nardin, L. G., Baumgaertner, B. O., & Buzbas, E. O. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216125 · Cited by 113
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E. C., Zhu, L. L., Diermeier, D. A., Heinze, J. E., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.001 · Cited by 110
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
Self-citationMany Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
Self-citationPredicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 101
Self-citationMachery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 99
Estimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 99
99% impossible: A valid, or falsifiable, internal meta-analysis.
Vosgerau, J., Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2019) · Journal of Experimental Psychology: General · doi:10.1037/xge0000663 · Cited by 98
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Self-citationBoth Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 97
I nvestigate D iscuss E stimate A ggregate for structured expert judgement
Hanea, A., McBride, M., Burgman, M., Wintle, B., Fidler, F., Flander, L., Twardy, C. R., Manning, B., & Mascaro, S. (2017) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2016.02.008 · Cited by 95
Self-citationPredicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225826 · Cited by 94
Self-citationIsager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 94
Cunningham, M. R., & Baumeister, R. F. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01639 · Cited by 93
The what, why, and how of born-open data
Rouder, J. N. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0630-z · Cited by 92
Registered Replication Report: Dijksterhuis and van Knippenberg (1998)
O'Donnell, M. P., Nelson, L. D., Ackermann, E., Aczél, B., Akhtar, A., Aldrovandi, S., Alshaif, N., Andringa, R., Aveyard, M. E., Babinčák, P., et al. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618755704 · Cited by 92
Feest, U. (2019) · Philosophy of Science · doi:10.1086/705451 · Cited by 92
Reproducibility of individual effect sizes in meta-analyses in psychology
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., Olsson-Collentine, A., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233107 · Cited by 90
Self-citationBuilding a collaborative psychological science: Lessons learned from ManyBabies 1.
Byers‐Heinlein, K., Bergmann, C., Davies, C. E., Frank, M. C., Hamlin, J. K., Struhl, M. K., Kominsky, J. F., Kosie, J. E., Lew‐Williams, C., Liu, L., et al. (2020) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000216 · Cited by 86
Washburn, A. N., Hanson, B. E., Motyl, M. S., Skitka, L. J., Yantis, C., Wong, K., Sun, J., Prims, J., Mueller, A. B., Melton, Z. J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918757427 · Cited by 85
Commentary: A Multilab Preregistered Replication of the Ego-Depletion Effect
Dang, J. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01155 · Cited by 84
Low replicability can support robust and efficient science
Lewandowsky, S., & Oberauer, K. (2020) · Nature Communications · doi:10.1038/s41467-019-14203-0 · Cited by 83
Cheung, I. K. M., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutoğlu, B., Bahník, Š., Bowen, J. D., Bredow, C. A., Bromberg, C., Caprariello, P. A., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616664694 · Cited by 82
Using OSF to Share Data: A Step-by-Step Guide
Soderberg, C. K. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918757689 · Cited by 81
Heterogeneity in direct replications in psychology and its association with effect size.
Olsson-Collentine, A., Wicherts, J. M., & van Assen, M. A. L. M. (2020) · Psychological Bulletin · doi:10.1037/bul0000294 · Cited by 81
Where Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81
Self-citationScientific progress despite irreproducibility: A seeming paradox
Shiffrin, R. M., Börner, K., & Stigler, S. M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711786114 · Cited by 79
Reflection on the Smiling Registered Replication Report
Strack, F. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674460 · Cited by 78
Questionable science and reproducibility in electrical brain stimulation research
Héroux, M. E., Loo, C. K. A., Taylor, J. L., & Gandevia, S. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175635 · Cited by 78
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020) · Psychological Methods · doi:10.1037/met0000365 · Cited by 77
Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. D. (2022) · Personality Science · doi:10.5964/ps.6029 · Cited by 75
Self-citationMischel, W. (2008) · APS observer · Cited by 74
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 73
Self-citationStatistical analyses for studying replication: Meta-analytic perspectives.
Hedges, L. V., & Schauer, J. M. (2018) · Psychological Methods · doi:10.1037/met0000189 · Cited by 73
Defense Advanced Research Projects Agency
Karsten, P. (2005) · doi:10.4135/9781412952460.n78 · Cited by 72
Scientists’ Reputations Are Based on Getting It Right, Not Being Right
Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002460 · Cited by 67
Self-citationCreative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 67
Self-citationWolff, W., Baumann, L., & Englert, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199554 · Cited by 65
Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)
Colling, L. J., Szűcs, D., Marco, D. D., Cipora, K., Ulrich, R., Nuerk, H., Soltanlou, M., Bryce, D., Chen, S.-C., Schroeder, P. A., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920903079 · Cited by 64
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R., & Xu, M.-R. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 64
statcheck: Extract Statistics from Articles and Recompute P-Values
Nuijten, M. B., & Epskamp, S. (2014) · doi:10.32614/cran.package.statcheck · Cited by 63
Self-citationLaypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 62
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
Self-citationEerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J., Aucoin, P., Berger, S. A., Birt, A. R., Capezza, N. M., Carlucci, M. E., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615605826 · Cited by 61
Ellemers, N., Fiske, S. T., Abele, A. E., Koch, A. S., & Yzerbyt, V. Y. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1906720117 · Cited by 59
Gervais, W. M., Jewell, J. A., Najle, M. B., & Ng, B. K. L. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615584199 · Cited by 58
Improving social and behavioral science by making replication mainstream: A response to commentaries
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000961 · Cited by 58
The “Goldilocks Zone”: (Too) many confidence intervals in tests of mediation just exclude zero.
Götz, M., O’Boyle, E. H., Gonzalez-Mule, E. N., Banks, G. C., & Bollmann, S. (2020) · Psychological Bulletin · doi:10.1037/bul0000315 · Cited by 58
Association between contextual dependence and replicability in psychology may be spurious
Inbar, Y. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1608676113 · Cited by 56
Research Culture and Reproducibility
Munafo, M. R., Chambers, C., Collins, A., Fortunato, L., & Macleod, M. R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.12.002 · Cited by 56
Science is not a signal detection problem
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914237117 · Cited by 56
Registered Replication Report on Srull and Wyer (1979)
McCarthy, R. J., Skowronski, J. J., Verschuère, B., Meijer, E. H., Jim, A., Hoogesteyn, K., Orthey, R., Acar, O. A., Aczél, B., Bakos, B. E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918777487 · Cited by 55
Research works printed but not linked: 17
- Artner R, 2020, Psychol. Methods.
- Cent. Open Sci, 2020, Center for Open Science
- Cohen J., 1973, Am. Educ. Res. J., 10, 225
- 10.1037/0033-2909.112.1.155 · DOI · Paper page
- Crisp RJ, 2014, Soc. Psychol., 45, 303
- Epskamp S, 2018, Statistical Software
- Errington TM, 2021, eLife
- Ferguson MJ, 2014, Soc. Psychol., 45, 301
- Gelman A, 2013, The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
- Marcus A, 2020, WIRED
- Meyer MN, 2014, Slate Magazine
- Mischel W., 2008, APS Observer
- Nosek B., 2019, Center for Open Science
- Nosek B., 2019, Center for Open Science
- Schwarz N, 2014, Soc. Psychol., 45, 305
- Sterling TD., 1959, J. Am. Stat. Assoc., 54, 30
- Yong E, 2012, National Geo-graphic
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationBergdahl, J., Latikka, R., Celuch, M., Savolainen, I., Soares Mantere, E., Savela, N., & Oksanen, A. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102013 · Cited by 204
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Metadata
- Authors
- Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
- Affiliations
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Wisconsin–Madison (US)
- University of California, Davis (US)
- Grand Valley State University (US)
- Stockholm School of Economics (SE)
- The University of Melbourne (AU)
- Illinois State University (US)
- Center for Open Science, Charlottesville, Virginia 22903, USA
- Tilburg University (NL)
- Leipzig University (DE)
- University of Groningen (NL)
- Eindhoven University of Technology (NL)
- University of Colorado Anschutz (US)
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2022-01-04
- Volume
- 73
- Issue
- 1
- Pages
- 719-748
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W3161588210
- Cited by
- 770
- References
- 253
- Citations by year
- 2026: 109 · 2025: 192 · 2024: 180 · 2023: 177 · 2022: 92
- FWCI
- 19.02
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation
- Patient-Centered Outcomes Research Institute
- Arnold Ventures
- European Commission (HORIZON2020, 841188)
- University of Melbourne
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Templeton World Charity Foundation
- Templeton Religion Trust
- National Institutes of Health
- Agency for Healthcare Research and Quality
- Australian Research Council (FT150100297)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, replicability, reproducibility, robustness, research practices, replication study, research incentives
- MeSH terms
- Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
