Reproducibility failures are essential to scientific inquiry
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Proceedings of the National Academy of Sciences (PNAS), a peer reviewed journal of the National Academy of Sciences (NAS) - an authoritative source of high-impact, original research that broadly spans the biological, physical, and social sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“All of these issues amalgamate into a critical problem, which is not inconsequential and which substantially hinders the ability of science to self-correct [25, 26].”
From Media and social media attention to retracted articles according to Altmetric (2021) · cites this paper as [26]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
The Principles of Quantum Mechanics
Dirac, P. A. M., & Polkinghorne, J. C. (1958) · Physics Today · doi:10.1063/1.3062610 · Cited by 5,409
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Perceptrons: An Introduction to Computational Geometry
Minsky, M., & Papert, S. A. (1969) · Cited by 3,742
Schwartz, L. (1966) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 3,513
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
The Principles of Quantum Mechanics.
Ettlinger, H. J., & Dirac, P. A. B. (1931) · The American Mathematical Monthly · doi:10.2307/2300785 · Cited by 1,333
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Théorie Analytique de la Chaleur
Fourier, J. B. J. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511693229 · Cited by 1,168
Every planar map is four colorable
Appel, K., & Haken, W. (1976) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1976-14122-5 · Cited by 1,109
Thalidomide-induced Teratogenesis: History and Mechanisms
Vargesson, N. (2015) · Birth Defects Research Part C Embryo Today Reviews · doi:10.1002/bdrc.21096 · Cited by 912
Ignorance: How it drives science
Firestein, S. (2012) · Cited by 609
Formal Proof—The Four- Color Theorem
Gonthier, G. (2008) · Cited by 601
Robertson, N., Sanders, D., Seymour, P., & Thomas, R. (1997) · Journal of Combinatorial Theory, Series B · doi:10.1006/jctb.1997.1750 · Cited by 598
Ringel, G. (1974) · Springer eBooks · doi:10.1007/978-3-642-65759-7 · Cited by 515
(1964) · Properties and Operations · doi:10.1016/b978-1-4832-2976-8.50001-5 · Cited by 491
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Ignorance: how it drives science
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6848 · Cited by 414
DEFINITION AND SIMPLEST PROPERTIES OF GENERALIZED FUNCTIONS
GEL'FAND, I. M., & SHILOV, G. E. (1964) · Properties and Operations · doi:10.1016/b978-1-4832-2976-8.50007-6 · Cited by 291
Every Planar Map is Four Colorable
Appel, K., & Haken, W. (1989) · Contemporary Mathematics · doi:10.1090/conm/098 · Cited by 290
On the Geographical Problem of the Four Colours
Kempe, A. B. (1879) · American Journal of Mathematics · doi:10.2307/2369235 · Cited by 266
Dirichlet, G. L. (1829) · Journal für die reine und angewandte Mathematik (Crelles Journal) · doi:10.1515/crll.1829.4.157 · Cited by 216
Life cycle of translational research for medical interventions
Contopoulos-Ioannidis, D. G., Alexiou, G. A., Gouvias, T. C., & Ioannidis, J. P. A. (2008) · Science · doi:10.1126/science.1160622 · Cited by 207
Reproducible research can still be wrong: Adopting a prevention approach
Leek, J. T., & Peng, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1421412111 · Cited by 193
A formal proof of the four color theoremPreprint
Xiang, L. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0905.3713 · Cited by 163
A long journey to reproducible results
Lithgow, G. J., Driscoll, M., & Phillips, P. (2017) · Nature · doi:10.1038/548387a · Cited by 157
Strain differences in fear-motivated behavior of rats
Rex, A., Sondern, U., Voigt, J., Franck, S., & Fink, H. (1996) · Pharmacology Biochemistry and Behavior · doi:10.1016/0091-3057(95)02128-0 · Cited by 135
Failure: why science is so successful
(2016) · Choice Reviews Online · doi:10.5860/choice.194830 · Cited by 128
Evolution of the Function Concept: A Brief Survey
Kleiner, I. (1989) · The College Mathematics Journal · doi:10.1080/07468342.1989.11973245 · Cited by 114
Evolution of the Function Concept: A Brief Survey, Israel Kleiner
(2009) · Spectrum · doi:10.5948/upo9781614445043.004 · Cited by 100
Dirichlet, P. G. L. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139237338.010 · Cited by 88
Crease, R. P. (2006) · Physics World · doi:10.1088/2058-7058/19/6/26 · Cited by 6
Cited by
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
Animal models of depression: pros and cons
Harro, J. (2018) · Cell and Tissue Research · doi:10.1007/s00441-018-2973-0 · Cited by 117
On the role of theory and modeling in neuroscience
Levenstein, D., Alvarez, V. A., Amarasingham, A., Azab, H., Chen, Z. S., Gerkin, R. C., Hasenstaub, A., Iyer, R., Jolivet, R. B., Marzen, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.13825 · Cited by 93
Self-citationOn the Role of Theory and Modeling in Neuroscience
Levenstein, D., Alvarez, V. A., Amarasingham, A., Azab, H., Chen, Z. S., Gerkin, R. C., Hasenstaub, A., Iyer, R., Jolivet, R. B., Marzen, S., et al. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.1179-22.2022 · Cited by 92
Self-citationRinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing
Penders, B., Holbrook, J. B., & de Rijcke, S. (2019) · Publications · doi:10.3390/publications7030052 · Cited by 85
Metadata
- Authors
- Redish, A. D., Kummerfeld, E., Morris, R. L., & Love, A. C.
- Affiliations
- University of Minnesota (US)
- Stanford University (US)
- Corresponding author
- Redish, A. David
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-05-15
- Volume
- 115
- Issue
- 20
- Pages
- 5042-5046
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5960342
- OpenAlex
- W2803831040
- Cited by
- 66
- References
- 37
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 3 · 2023: 22 · 2022: 9 · 2021: 7 · 2020: 10 · 2019: 7 · 2018: 2
- FWCI
- 9.23
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
- Keywords
- scientific inquiry, reproducibility, research credibility, scientific rigor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.