Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Abstract
Concerns regarding the reliability of biomedical research outcomes were precipitated by two independent reports from the pharmaceutical industry that documented a lack of reproducibility in preclinical research in the areas of oncology, endocrinology, and hematology. Given their potential impact on public health, these concerns have been extensively covered in the media. Assessing the magnitude and scope of irreproducibility is limited by the anecdotal nature of the initial reports and a lack of quantitative data on specific failures to reproduce published research. Nevertheless, remediation activities have focused on needed enhancements in transparency and consistency in the reporting of experimental methodologies and results. While such initiatives can effectively bridge knowledge gaps and facilitate best practices across established and emerging research disciplines and therapeutic areas, concerns remain on how these improve on the historical process of independent replication in validating research findings and their potential to inhibit scientific innovation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is a reproducibility crisis in science.
“However, the reproducibility crisis (Baker, 2016;Jarvis and Williams, 2016) in science alone should prompt and cement such quality assurance.”
From Toward good in vitro reporting standards (2019) · cites this paper as Jarvis and Williams, 2016
Cites
The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments
Bustin, S. A., Benes, V., Garson, J. A., Hellemans, J., Huggett, J., Kubista, M., Mueller, R., Nolan, T., Pfaffl, M. W., Shipley, G. L., et al. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.112797 · Cited by 16,303 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
SYRCLE’s risk of bias tool for animal studies
Hooijmans, C. R., Rovers, M. M., de Vries, R. B., Leenaars, M., Ritskes-Hoitinga, M., & Langendam, M. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-43 · Cited by 4,415 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1× · see citing works
“Misconduct Adding to the concerns regarding reproducibility are reports of growing rates of retractions of published biomedical research findings due to unethical behavior or scientific misconduct [31,58–61].”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
The oligopoly of academic publishers in the digital era
Larivière, V., Haustein, S., & Mongeon, P. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127502 · Cited by 1,240 · see citing works
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232 · cited 2× · see citing works
“Given that more than 25 million biomedical research papers have been published since the 1940s [35], obvious questions arise regarding how representative the currently available data on irreproducibility are and also whether they provide a global assessment of the reproducibility issues identified to date.”
BackgroundA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · see citing works
Bohannon, J. (2013) · Science · doi:10.1126/science.342.6154.60 · Cited by 934 · see citing works
Kilkenny, C., Parsons, N., Kadyszewski, E., Festing, M. F. W., Cuthill, I. C., Fry, D., Hutton, J., & Altman, D. G. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007824 · Cited by 860 · see citing works
The promise and peril of chemical probes
Arrowsmith, C. H., Audia, J. E., Austin, C., Baell, J., Bennett, J., Blagg, J., Bountra, C., Brennan, P. E., Brown, P. J., Bunnage, M. E., et al. (2015) · Nature Chemical Biology · doi:10.1038/nchembio.1867 · Cited by 839 · see citing works
Reproducibility crisis: Blame it on the antibodies
Baker, M. (2015) · Nature · doi:10.1038/521274a · Cited by 830 · see citing works
Predatory’ open access: a longitudinal study of article volumes and market characteristics
Shen, C., & Björk, B.-C. (2015) · BMC Medicine · doi:10.1186/s12916-015-0469-2 · Cited by 819 · see citing works
Rescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · see citing works
Animal models of human disease: challenges in enabling translation
McGonigle, P., & Ruggeri, B. (2014) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.08.006 · Cited by 610 · see citing works
Taylor, C. F., Field, D., Sansone, S.-A., Aerts, J., Apweiler, R., Ashburner, M., Ball, C. A., Binz, P.-A., Bogue, M., Booth, T., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt.1411 · Cited by 558 · see citing works
Why has the number of scientific retractions increased?
Steen, R. G., Casadevall, A., & Fang, F. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068397 · Cited by 415 · see citing works
Vasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271 · see citing works
Hvistendahl, M. (2013) · Science · doi:10.1126/science.342.6162.1035 · Cited by 256 · see citing works
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608 · see citing works
Bustin, S., & Nolan, T. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12801 · Cited by 113 · see citing works
Toward good in vitro reporting standards
Hartung, T. (2019) · ALTEX · doi:10.14573/altex.1812191 · Cited by 63 · see citing works
“However, the reproducibility crisis (Baker, 2016; Jarvis and Williams, 2016) in science alone should prompt and cement such quality assurance.”
BackgroundAligning physiology with psychology: Translational neuroscience in neuropsychiatric drug discovery
McArthur, R. A. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.02.004 · Cited by 42 · see citing works
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28 · see citing works
Self-citationWasted Money in United States Biomedical and Agricultural Animal Research
Keen, J. (2019) · Animal Experimentation: Working Towards a Paradigm Change · doi:10.1163/9789004391192_011 · Cited by 11 · see citing works
- Authors
- Jarvis, M. F., & Williams, M.
- Affiliations
- AbbVie (United States) (US)
- Northwestern University (US)
- Corresponding author
- Jarvis, Michael F.
- Venue
- Trends in Pharmacological Sciences
- Publisher
- Elsevier BV
- Published
- 2016-04
- Volume
- 37
- Issue
- 4
- Pages
- 290-302
- ISSN
- 0165-6147 (ISSN-L)
- OpenAlex
- W2232091047
- Cited by
- 129
- References
- 25
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 7 · 2023: 13 · 2022: 10 · 2021: 15 · 2020: 13 · 2019: 20 · 2018: 11 · 2017: 19 · 2016: 8
- FWCI
- 4.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Trends in Pharmacological Sciences (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- irreproducibility, preclinical research, research reliability, reproducibility, independent replication, research transparency, reporting quality, knowledge gaps, scientific innovation
- MeSH terms
- Animals, Humans, Perception, Reproducibility of Results, Guidelines as Topic, Knowledge, Uncertainty, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
