Comment on “Estimating the reproducibility of psychological science.”
Abstract
A paper from the Open Science Collaboration (Research Articles, 28 August 2015, aac4716) attempting to replicate 100 published studies suggests that the reproducibility of psychological science is surprisingly low. We show that this article contains three statistical errors and provides no support for such a conclusion. Indeed, the data are consistent with the opposite conclusion, namely, that the reproducibility of psychological science is quite high.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Follow-up investigations did not support the claim that replication failures in the Reproducibility Project: Psychology were due to deficiencies in replication quality.
“follow-up investigations did not support the claim that replication failures were due to deficiencies in replication quality”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Gilbert et al., 2016
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Shnabel, N., & Nadler, A. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.116 · Cited by 618 · see citing works
Why do implicit and explicit attitude tests diverge? The role of structural fit.
Payne, B. K., Burkley, M. A., & Stokes, M. B. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.16 · Cited by 401 · see citing works
Detecting the Snake in the Grass
LoBue, V., & DeLoache, J. S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02081.x · Cited by 382 · see citing works
Masicampo, E. J., & Baumeister, R. F. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02077.x · Cited by 331 · see citing works
Dynamics of self-regulation: How (un)accomplished goal actions affect motivation.
Koo, M., & Ayelet Fishbach, B. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.2.183 · Cited by 291 · see citing works
Why people are reluctant to tempt fate.
Risen, J. L., & Gilovich, T. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.293 · Cited by 138 · see citing works
Replication Data for: Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. K., King, G., Pettigrew, S., & Wilson, T. (2016) · Harvard Dataverse · doi:10.7910/dvn/5lkvh2 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Muthukrishna, M., & Henrich, J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0522-1 · Cited by 676 · see citing works
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · see citing works
Behavioural science is unlikely to change the world without a heterogeneity revolution
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01143-3 · Cited by 547 · see citing works
Small is beautiful: In defense of the small-N design
Smith, P. L., & Little, D. R. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1451-8 · Cited by 466 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
Effects of tDCS on motor learning and memory formation: A consensus and critical position paper
Buch, E. R., Santarnecchi, E., Antal, A., Jan Born, A., Celnik, P. A., Claßen, J., Gerloff, C. P., Hallett, M., Hummel, F. C., Nitsche, M. A., et al. (2017) · Clinical Neurophysiology · doi:10.1016/j.clinph.2017.01.004 · Cited by 351 · see citing works
Frank, M. C., Bergelson, E., Bergmann, C., Cristià, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Struhl, M. K., Levelt, C. C., et al. (2017) · Infancy · doi:10.1111/infa.12182 · Cited by 333 · see citing works
Jeremy Freese, A., & Peterson, D. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053450 · Cited by 323 · see citing works
Predicting risk and outcomes for frail older adults: an umbrella review of frailty screening tools
Apóstolo, J., Cooke, R. W. I., Campos, E. B., Santana, S., Marcucci, M., Cano, A., Vollenbroek-Hutten, M. M. R., Germini, F., & Holland, C. (2017) · The JBI Database of Systematic Reviews and Implementation Reports · doi:10.11124/jbisrir-2016-003018 · Cited by 241 · see citing works
The experimental evidence for parapsychological phenomena: A review.
Cardeña, E. (2018) · American Psychologist · doi:10.1037/amp0000236 · Cited by 221 · see citing works
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215 · see citing works
Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences
Köhler, T., & Cortina, J. M. (2019) · Journal of Management · doi:10.1177/0149206319843985 · Cited by 204 · see citing works
Reliability of the chronic mild stress model of depression: A user survey
Willner, P. (2017) · Neurobiology of Stress · doi:10.1016/j.ynstr.2016.08.001 · Cited by 198 · see citing works
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189 · see citing works
What is science’s crisis really about?
Saltelli, A., & Funtowicz, S. (2017) · Futures · doi:10.1016/j.futures.2017.05.010 · Cited by 188 · see citing works
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
Caird, J. K., Simmons, S. M., Wiley, K., Johnston, K. A., & Horrey, W. J. (2018) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720817748145 · Cited by 180 · see citing works
The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 170 · see citing works
A Review on Blockchain Technology and Blockchain Projects Fostering Open Science
Leible, S., Steffen Schlager, C., Schubotz, M., & Gipp, B. (2019) · Frontiers in Blockchain · doi:10.3389/fbloc.2019.00016 · Cited by 176 · see citing works
Rutjens, B. T., Heine, S. J., Sutton, R. M., & van Harreveld, F. (2017) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2017.08.001 · Cited by 154 · see citing works
Leuz, C. (2018) · Accounting and Business Research · doi:10.1080/00014788.2018.1470151 · Cited by 153 · see citing works
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 152 · see citing works
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 147 · see citing works
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Anderson, S. F., & Maxwell, S. E. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 133 · see citing works
Open Science Practices are on the Rise: The State of Social Science (3S) SurveyPreprint
Christensen, G., Wang, Z., Paluck, E. L., Swanson, N., Birke, D. J., Miguel, E. A., & Littman, R. (2019) · doi:10.31222/osf.io/5rksu · Cited by 127 · see citing works
The Emergence of Contextual Social Psychology
Pettigrew, T. F. (2018) · Personality and Social Psychology Bulletin · doi:10.1177/0146167218756033 · Cited by 122 · see citing works
No Replication, No Trust? How Low Replicability Influences Trust in Psychology
Wingen, T., Berkessel, J. B., & Englich, B. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619877412 · Cited by 121 · see citing works
Replication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 118 · see citing works
What Is Meant by “Replication” and Why Does It Encounter Resistance in Economics?
Duvendack, M., Palmer‐Jones, R., & Reed, W. R. (2017) · American Economic Review · doi:10.1257/aer.p20171031 · Cited by 118 · see citing works
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117 · see citing works
Motyl, M. S., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B., Prims, J., Sun, J., Washburn, A. N., Wong, K., et al. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000084 · Cited by 109 · see citing works
Predicting Ideological Prejudice
Brandt, M. J. (2017) · Psychological Science · doi:10.1177/0956797617693004 · Cited by 105 · see citing works
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105 · see citing works
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95 · see citing works
Machery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 95 · see citing works
Z-curve 2.0: Estimating Replication Rates and Discovery Rates
Bartos̆, F., & Schimmack, U. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2720 · Cited by 92 · see citing works
A statistical definition for reproducibility and replicabilityPreprint
Patil, P., Peng, R. D., & Leek, J. T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/066803 · Cited by 90 · see citing works
Laraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 87 · see citing works
Survey of open science practices and attitudes in the social sciences
Ferguson, J., Littman, R., Christensen, G., Paluck, E. L., Swanson, N., Wang, Z., Miguel, E. A., Birke, D. J., & Pezzuto, J.-H. (2023) · Nature Communications · doi:10.1038/s41467-023-41111-1 · Cited by 86 · see citing works
Maniadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 84 · see citing works
Some Advances in the Design of Survey Experiments
Sniderman, P. M. (2018) · Annual Review of Political Science · doi:10.1146/annurev-polisci-042716-115726 · Cited by 84 · see citing works
Perception science in the age of deep neural networks.
VanRullen, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00142 · Cited by 82 · see citing works
Video can make behavioural science more reproducible
Gilmore, R. O., & Adolph, K. E. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0128 · Cited by 81 · see citing works
Preregistration Is Neither Sufficient nor Necessary for Good Science
Pham, M. T., & Oh, T. T. (2020) · Journal of Consumer Psychology · doi:10.1002/jcpy.1209 · Cited by 85 · see citing works
Rizzo, M. J., & Whitman, G. (2019) · Cambridge University Press eBooks · doi:10.1017/9781139061810 · Cited by 125 · see citing works
Harris, B. N. (2019) · General and Comparative Endocrinology · doi:10.1016/j.ygcen.2019.113355 · Cited by 76 · see citing works
Pre-registration: Weighing costs and benefits for researchers
Logg, J. M., & Dorison, C. A. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.05.006 · Cited by 69 · see citing works
Self-correction in science: The diagnostic and integrative motives for replication
Peterson, D., & Panofsky, A. L. (2021) · Social Studies of Science · doi:10.1177/03063127211005551 · Cited by 73 · see citing works
Leuz, C. (2018) · National Bureau of Economic Research · doi:10.3386/w24535 · Cited by 72 · see citing works
Statistical analyses for studying replication: Meta-analytic perspectives.
Hedges, L. V., & Schauer, J. M. (2018) · Psychological Methods · doi:10.1037/met0000189 · Cited by 69 · see citing works
The quest for an optimal alpha
Miller, J., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 63 · see citing works
Point estimates, Simpson’s paradox, and nonergodicity in biological sciences
Mangalam, M., & Kelty‐Stephen, D. G. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.017 · Cited by 68 · see citing works
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R., & Xu, M.-R. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 64 · see citing works
Replicator degrees of freedom allow publication of misleading failures to replicate
Bryan, C. J., Yeager, D. S., & O’Brien, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910951116 · Cited by 70 · see citing works
Reproducibility in Accounting Research: Views of the Research Community
Hail, L., Lang, M. H., & Leuz, C. (2020) · Journal of Accounting Research · doi:10.1111/1475-679x.12305 · Cited by 67 · see citing works
Perceived crisis and reforms: Issues, explanations, and remedies.
de Boeck, P. A. L., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 67 · see citing works
More Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 64 · see citing works
Bechky, B. A. (2019) · Administrative Science Quarterly · doi:10.1177/0001839219855329 · Cited by 64 · see citing works
Odle‐Dusseau, H. N., Matthews, R. A., & Wayne, J. H. (2018) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12216 · Cited by 61 · see citing works
Behavioral Public Administration: Combining Insights from Public Administration and Psychology
Grimmelikhuijsen, S., Jilke, S., Olsen, A. L., & Tummers, L. G. (2016) · Public Administration Review · doi:10.1111/puar.12609 · Cited by 61 · see citing works
Improving Psychological Science through Transparency and Openness: An Overview
Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018) · Perspectives on Behavior Science · doi:10.1007/s40614-018-00186-8 · Cited by 61 · see citing works
The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science
Jeremy Freese, A., & Peterson, D. (2018) · Sociological Theory · doi:10.1177/0735275118794987 · Cited by 60 · see citing works
Gehlbach, H., & Robinson, C. D. (2021) · Educational Psychology · doi:10.1080/00461520.2021.1898961 · Cited by 46 · see citing works
How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence
Barnes, J. C., TenEyck, M. F., Pratt, T. C., & Cullen, F. T. (2019) · Justice Quarterly · doi:10.1080/07418825.2018.1495252 · Cited by 59 · see citing works
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 53 · see citing works
Flake, J. K. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898962 · Cited by 58 · see citing works
Science is not a signal detection problem
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914237117 · Cited by 56 · see citing works
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works
The Weak Spots in Contemporary Science (and How to Fix Them)
Wicherts, J. M. (2017) · Animals · doi:10.3390/ani7120090 · Cited by 53 · see citing works
Improving the rigor of psychophysiology research
Baldwin, S. A. (2016) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2016.04.006 · Cited by 53 · see citing works
RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 90 · see citing works
RetractedNull hypothesis significance testing and Type I error: The domain problem
Trafimow, D., & Earp, B. (2017) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2017.01.002 · Cited by 57 · see citing works
Infants’ false belief understanding: A non-replication of the helping task
Crivello, C., & Poulin‐Dubois, D. (2017) · Cognitive Development · doi:10.1016/j.cogdev.2017.10.003 · Cited by 52 · see citing works
A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2017.55 · Cited by 52 · see citing works
Open science, closed doors: The perils and potential of open science for research in practice
Guzzo, R. A., Schneider, B., & Nalbantian, H. R. (2022) · Industrial and Organizational Psychology · doi:10.1017/iop.2022.61 · Cited by 52 · see citing works
Communicating What We Know and What Isn’t So: Science Communication in Psychology
Lewis, N. A., & Wai, J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964062 · Cited by 50 · see citing works
Transparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 57 · see citing works
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 56 · see citing works
Open Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50 · see citing works
Seven Pervasive Statistical Flaws in Cognitive Training Interventions
Moreau, D., Kirk, I. J., & Waldie, K. E. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00153 · Cited by 49 · see citing works
Pagell, M. (2020) · Journal of Operations Management · doi:10.1002/joom.1120 · Cited by 47 · see citing works
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48 · see citing works
Replication in computing education research: researcher attitudes and experiences
Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A. (2016) · Proceedings of the 16th Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999554 · Cited by 48 · see citing works
What Can We Learn from Many Labs Replications?
Stroebe, W. (2019) · Basic and Applied Social Psychology · doi:10.1080/01973533.2019.1577736 · Cited by 34 · see citing works
Sharma, G., Biswas, S. S., Mishra, J., Navik, U., Kandimalla, R. J. L., Reddy, P. H., Bhatti, G. K., & Bhatti, J. S. (2023) · Life Sciences · doi:10.1016/j.lfs.2023.121882 · Cited by 46 · see citing works
Revisiting the revisit: added evidence for a social chemosignal in human emotional tears
Sobel, N. (2016) · Cognition & Emotion · doi:10.1080/02699931.2016.1177488 · Cited by 47 · see citing works
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C., Wicherts, J., & van Assen, M. (2017) · Collabra Psychology · doi:10.1525/collabra.71 · Cited by 46 · see citing works
Improving the Replicability of Psychological Science Through Pedagogy
Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., Keil, M., Lampinen, A. K., Raposo, S., Reynolds, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917740427 · Cited by 46 · see citing works
Practical Methodological Reform Needs Good Theory
Gervais, W. M. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620977471 · Cited by 37 · see citing works
Replication initiatives will not salvage the trustworthiness of psychology
Coyne, J. C. (2016) · BMC Psychology · doi:10.1186/s40359-016-0134-3 · Cited by 44 · see citing works
- Authors
- Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D.
- Affiliations
- Harvard University (US)
- University of Virginia (US)
- Corresponding author
- Gilbert, Daniel T.
- Venue
- Science
- Published
- 2016-03-04
- Volume
- 351
- Issue
- 6277
- Pages
- 1037-1037
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2290831829
- Cited by
- 474
- References
- 10
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 16 · 2023: 26 · 2022: 33 · 2021: 50 · 2020: 57 · 2019: 80 · 2018: 74 · 2017: 71 · 2016: 45 · 2015: 2
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- NSF (BCS-1423747)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducibility, replication, Open Science Collaboration, statistical errors, meta-analysis, replication study
- MeSH terms
- Psychology, Publishing, Research, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review