Where Are the Self-Correcting Mechanisms in Science?
Open accessAbstract
It is often said that science is self-correcting, but the replication crisis suggests that self-correction mechanisms have fallen short. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction, such as journal-based peer review and institutional committees, have been inadequate. We describe more verifiable indicators of a field’s commitment to self-correction. These fall under the broad headings of 1) transparency, which is already the subject of many reform efforts and 2) critical appraisal, which has received less attention and which we focus on here. Only by obtaining Observable Self-Correction Indicators (OSCIs) can we begin to evaluate the claim that “science is self-correcting.” We expect that the veracity of this claim varies across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vazire and Holcombe note that a scientific community that encourages transparency while discouraging criticism is not one that prioritizes self‐correction.
“Vazire and Holcombe [46] note that a scientific community that encourages transparency while discouraging criticism is not one that prioritizes self‐correction.”
From Quality issue management and disclosure in forensic science: A survey of practice and perceptions (2025) · cites this paper as [46]
Cites
THE STRUCTURE OF SCIENTIFIC REVOLUTIONS
McGuire, J. E. (1963) · Philosophical Books · doi:10.1111/j.1468-0149.1963.tb00813.x · Cited by 8,544 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 7× · see citing works
“In their paper “Signaling the Trustworthiness of Science,” Jamieson et al. (2019) discuss both transparency-related norms and a “culture of critique” as fundamental to maintaining the trustworthiness of science.”
BackgroundInfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Science as Social Knowledge: Values and Objectivity in Scientific Inquiry
Longino, H. E. (1990) · doi:10.1515/9780691209753 · Cited by 3,092 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,930 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · see citing works
Boneau, C. A. (2010) · Wiley eBooks · doi:10.1002/9780470479216.corpsy0730 · Cited by 1,288 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1× · see citing works
“Multiple fields of science have a very poor record of publishing negative and null results (Sterling et al., 1995; Fanelli, 2010, 2012; Greenwald, 1975; Hubbard & Armstrong, 1992; Franco et al., 2014; Sterling, 1959; Sterling, Rosenbaum, & Wenkam, 1995), suggesting that the published literature is…”
BackgroundSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2× · see citing works
“Multiple fields of science have a very poor record of publishing negative and null results (Sterling et al., 1995; Fanelli, 2010, 2012; Greenwald, 1975; Hubbard & Armstrong, 1992; Franco et al., 2014; Sterling, 1959; Sterling, Rosenbaum, & Wenkam, 1995), suggesting that the published literature is biased and incomplet…”
BackgroundResultReproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924 · see citing works
WHY SUMMARIES OF RESEARCH ON PSYCHOLOGICAL THEORIES ARE OFTEN UNINTERPRETABLE
Meehl, P. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.195 · Cited by 901 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
Prejudice Reduction: Progress and Challenges
Paluck, E. L., Porat, R., Clark, C. S., & Green, D. P. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-071620-030619 · Cited by 686 · cited 1× · see citing works
“In the social sciences, (Paluck et al., 2021) recently reviewed the literature on prejudice reduction and identified several concerning patterns in the types of designs and methods used, and conclusions drawn.”
BackgroundPhilosophical Transactions of the Royal Society
Wightman, W. P. D. (1961) · Nature · doi:10.1038/192023a0 · Cited by 626 · see citing works
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608 · see citing works
WaveLab and Reproducible Research
Buckheit, J. B., & Donoho, D. L. (1995) · Lecture notes in statistics · doi:10.1007/978-1-4612-2544-7_5 · Cited by 551 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
An introduction to Docker for reproducible research, with examples from the R environmentPreprint
Boettiger, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0846 · Cited by 372 · see citing works
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 359 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
Defending science--within reason: between scientism and cynicism
Haack, S. (2004) · Choice Reviews Online · doi:10.5860/choice.41-4006 · Cited by 296 · see citing works
"How Does It Work?" versus "What Are the Laws?": Two Conceptions of Psychological Explanation
Cummins, R. (2000) · The MIT Press eBooks · doi:10.7551/mitpress/2930.003.0009 · Cited by 296 · see citing works
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271 · see citing works
The fight against fake-paper factories that churn out sham science
Else, H., & Van Noorden, R. (2021) · Nature · doi:10.1038/d41586-021-00733-5 · Cited by 251 · see citing works
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246 · see citing works
Trust and mistrust in Americans’ views of scientific experts
Funk, C., Hefferon, M., Kennedy, B., & Johnson, C. (2019) · Cited by 242 · see citing works
COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time
Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019) · Trials · doi:10.1186/s13063-019-3173-2 · Cited by 233 · cited 1× · see citing works
“However, anecdotal reports suggest that the bar for publishing reports of errors in a scientific paper can be very high (Heathers, 2015; Pickett, 2020; Friedman et al., in press; Goldacre et al., 2019).”
BackgroundWhat errors do peer reviewers detect, and does training improve their ability to detect them?
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2008.080062 · Cited by 226 · cited 1× · see citing works
“Reviewers were given different levels of training to assess whether intervention could help, but for all groups, the mean number of errors detected was three or fewer (Schroter et al., 2008).”
BackgroundNonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177 · see citing works
Normative Dissonance in Science: Results from a National Survey of U.S. Scientists
Anderson, M. S., Martinson, B., & de Vries, R. (2007) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2007.2.4.3 · Cited by 174 · see citing works
Registered Reports: Realigning incentives in scientific publishing
Chambers, C., Dienes, Z. P., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.022 · Cited by 171 · see citing works
Gene name errors are widespread in the scientific literature
Ziemann, M., Eren, Y., & El-Osta, A. (2016) · Genome Biology · doi:10.1186/s13059-016-1044-7 · Cited by 168 · cited 1× · see citing works
“Similarly, in an analysis of genetics articles published between 2005 and 2015, a team of researchers found that about 1 in 5 articles hadMicrosoft Excel files with gene lists that are affected by gene name errors (e.g., gene names that Excel automatically converts to dates; Ziemann et al., 2016).”
BackgroundBrown, N. J. L., & Heathers, J. A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550616673876 · Cited by 127 · cited 1× · see citing works
“The Statcheck (Nuijten & Polanin, 2020) and GRIM (Brown & Heathers, 2017) algorithms are two examples.”
MethodSignaling the trustworthiness of science
Jamieson, K. H., Mcnutt, M. K., Kiermer, V., & Sever, R. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1913039116 · Cited by 89 · see citing works
Sztompka, P. (2007) · Journal of Classical Sociology · doi:10.1177/1468795x07078038 · Cited by 71 · see citing works
Are null results becoming an endangered species in marketing?
Hubbard, R., & Armstrong, J. (1992) · Marketing Letters · doi:10.1007/bf00993992 · Cited by 71 · see citing works
Computational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69 · see citing works
Ellison, A. E. (1975) · The Journal of Sports Medicine · doi:10.1177/036354657500300407 · Cited by 63 · see citing works
An “alarming” and “exceptionally high” rate of COVID-19 retractions?
Abritis, A. J., Marcus, A., & Oransky, I. (2020) · Accountability in Research · doi:10.1080/08989621.2020.1793675 · Cited by 59 · see citing works
Questionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 59 · see citing works
Self-citationTheory Development Requires an Epistemological Sea Change
van Rooij, I., & Baggio, G. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1853477 · Cited by 50 · see citing works
Franco, A., Malhotra, N., & Simonovits, G. (2015) · Political Analysis · doi:10.1093/pan/mpv006 · Cited by 44 · see citing works
Open science and reform practices in organizational behavior research over time (2011 to 2019)
Tenney, E. R., Costa, E., Allard, A., & Vazire, S. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.10.015 · Cited by 35 · see citing works
Self-citationEthics and Statistics: Honesty and Transparency Are Not Enough
Gelman, A. (2017) · CHANCE · doi:10.1080/09332480.2017.1302720 · Cited by 31 · see citing works
Bakker, B. N., Jaidka, K., Dorr, T. G., et al. (2020) · doi:10.31234/osf.io/7uyn5 · Cited by 20 · see citing works
“Social” Objectivity and the Objectivity of Values
Smith, T. S. (2017) · University of Pittsburgh Press eBooks · doi:10.2307/j.ctt5vkg7t.12 · Cited by 12 · see citing works
Fiske, S. T., Schacter, D. L., & Taylor, S. E. (2016) · Annual Review of Psychology · doi:10.1146/annurev-ps-67-121415-100001 · Cited by 6 · see citing works
Improving science through online commentary
Eagleman, D. M., & Holcombe, A. O. (2003) · Nature · doi:10.1038/423015a · Cited by 5 · see citing works
Self-citationPublishers launch joint effort to tackle altered images in research papers
Noorden, R. V. (2020) · Nature · doi:10.1038/d41586-020-01410-9 · Cited by 4 · see citing works
Delete offensive language? Change recommendations? Some editors say it’s OK to alter peer reviews
O’Grady, C. (2020) · Science · doi:10.1126/science.abf4690 · Cited by 4 · see citing works
Mayo‐Wilson, E., Grant, S., Supplee, L. H., et al. (2021) · doi:10.31222/osf.io/b3wju · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Self-citationIs biomedical research self-correcting? Modelling insights on the persistence of spurious science
Grimes, D. R. (2024) · Royal Society Open Science · doi:10.1098/rsos.231056 · Cited by 18 · see citing works
An analysis of retracted papers in Computer Science
Shepperd, M., & Yousefi, L. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285383 · Cited by 16 · see citing works
Understanding ‘error’ in the forensic sciences: A primer
Martire, K. A., Chin, J. M., Davis, C., Edmond, G., Growns, B., Gorski, S., Kemp, R. I., Lee, Z., Verdon, C. M., Jansen, G., et al. (2024) · Forensic Science International: Synergy · doi:10.1016/j.fsisyn.2024.100470 · Cited by 9 · see citing works
Quality issue management and disclosure in forensic science: A survey of practice and perceptions
Heavey, A. L., Houck, M. M., Turbett, G. R., & Lewis, S. W. (2025) · Journal of Forensic Sciences · doi:10.1111/1556-4029.70121 · Cited by 10 · see citing works
Transparency and diversity to advance legal-scientific communication
Chin, J. M., & Baker, J. B. (2024) · Forensic Science International: Synergy · doi:10.1016/j.fsisyn.2024.100557 · Cited by 1 · see citing works
- Authors
- Vazire, S., & Holcombe, A. O.
- Affiliations
- The University of Melbourne (AU)
- The University of Sydney (AU)
- Corresponding author
- Vazire, Simine
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2021-08-12
- Volume
- 26
- Issue
- 2
- Pages
- 212-223
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W3161748415
- Semantic Scholar
- cf711f3733c86221242af49e483114177ed662c8
- Cited by
- 81
- References
- 63
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 22 · 2023: 14 · 2022: 9 · 2021: 9 · 2020: 7
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Review of General Psychology (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- scientific self-correction, replication crisis, critical appraisal, research transparency, peer review, scientific credibility, biomedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
