The Economics of Reproducibility in Preclinical Research
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible-in the United States alone. We outline a framework for solutions and a plan for long-term improvements in reproducibility rates that will help to accelerate the discovery of life-saving therapies and cures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Studies reporting the prevalence of irreproducibility. Source: Begley and Ellis [6], Prinz et al. [7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al. [9].](https://figures.cochat.ai/paper-figures/a30bd9bd65dc4bedb5db19b413eb814f/the-economics-of-reproducibility-in-preclinical-research-figure-1.jpg)
![Fig 2 Estimated US preclinical research spend and categories of errors that contribute to irreproducibility. Note that the percentage value of error for each category is the midpoint of the high and low prevalence estimates for that category divided (weighted) by the sum of all midpoint error rates (see S1 Dataset). Source: Chakma et al. [18] and the American Association for the Advancement of Science (AAAS) [19].](https://figures.cochat.ai/paper-figures/207a0a58df5c4620a819d7aa1f4aa960/the-economics-of-reproducibility-in-preclinical-research-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Besides, reliably determining actual translational success rates is unmanageable as long as the current status of reporting of preclinical research leaves room for improvement [89], and non-reproducibility is such a critical issue in both animal [1] and human [90] studies.”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The S&B tool therefore directly addresses the larger problem of material reagents and standards representing the major source of incorrect published results from pre-clinical research [3, 4].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current drug development is handicapped by high attrition rates; many molecules that were promising during preclinical development fail during subsequent clinical testing [1].”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [16]
Experimental — accuracy not yet measured
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Ostrom, E. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 18,859
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“[7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · cited 1×
“Much has been written about the alarming number of preclinical studies that were later found to be irreproducible [1,2].”
BackgroundBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Partial Identification of Probability Distributions
(2003) · Springer Series in Statistics · doi:10.1007/b97478 · Cited by 698
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Publication bias in reports of animal stroke studies leads to major overstatement of efficacy
Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W., & Macleod, M. R. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000344 · Cited by 600 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundThe Costs of using Unauthenticated, Over-Passaged Cell Lines: How Much more Data do we Need?
Hughes, P., Marshall, D., Reid, Y., Parkes, H., & Gelber, C. (2007) · BioTechniques · doi:10.2144/000112598 · Cited by 434
Climbing atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research
Furman, J. L., & Stern, S. (2011) · American Economic Review · doi:10.1257/aer.101.5.1933 · Cited by 344 · cited 1×
“Nonetheless, economic research also has shown that standardization and auditing of biological materials—through biological resource centers—can enhance cumulative production of scientific knowledge by improving both availability and reliability of research inputs [28].”
BackgroundBaker, D., Lidster, K., Sottomayor, A., & Amor, S. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001756 · Cited by 305 · cited 1×
“Despite the emergence of a wide variety of reporting guidelines to improve reporting of biomedical research methods and results, to date, compliance levels and their impact to improve reproducibility have been disappointing [26].”
BackgroundVasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271 · cited 2×
“For example, some studies define reproducibility as the ability to replicate the same results demonstrated in a particular study using precisely the same methods and materials [11]; others evaluate whether the study’s methodology and results were presented in sufficient detail to allow replication or reanalysis [8].”
BackgroundResultDistinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · cited 1×
“The definition of reproducibility may also vary depending upon whether a particular study is confirmatory (designed to test basic theories through rigorous study design and analysis) or exploratory (primarily aimed at developing theories and frameworks for further study) [12].”
BackgroundFinancial costs and personal consequences of research misconduct resulting in retracted publications
Stern, A. M., Casadevall, A., Steen, R. G., & Fang, F. C. (2014) · eLife · doi:10.7554/elife.02956 · Cited by 189 · cited 1×
“[21], even in cases of retracted publications due to scientific misconduct, which is not a major source of irreproducibility [13,22], “it is conceivable that some of the research resulting in a retracted article still provides useful information for other nonretracted”
BackgroundReplication and reproducibility in spinal cord injury research
Steward, O., Popovich, P. G., Dietrich, W. D., & Kleitman, N. (2012) · Experimental Neurology · doi:10.1016/j.expneurol.2011.06.017 · Cited by 186 · cited 1×
“For example, some studies define reproducibility as the ability to replicate the same results demonstrated in a particular study using precisely the same methods and materials [11]; others evaluate whether the study’s methodology and results were presented in sufficient detail to allow replication or reanalysis [8].”
ResultFixing problems with cell lines
Lorsch, J. R., Collins, F. S., & Lippincott-Schwartz, J. (2014) · Science · doi:10.1126/science.1259110 · Cited by 179
The Oxford handbook of the digital economy
(2013) · Choice Reviews Online · doi:10.5860/choice.50-3382 · Cited by 159
Climbing Atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research
Furman, J. L., & Stern, S. (2006) · National Bureau of Economic Research · doi:10.3386/w12523 · Cited by 138
Asia's Ascent — Global Trends in Biomedical R&D Expenditures
Chakma, J., Sun, G. H., Steinberg, J. D., Sammut, S. M., & Jagsi, R. (2014) · New England Journal of Medicine · doi:10.1056/nejmp1311068 · Cited by 130 · cited 5×
“8B in the United States [18] is spent annually on life sciences research, with the pharmaceutical industry being the largest funder at 61.”
BackgroundInfluentialTracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 116 · cited 2×
“Indeed, there are many different perspectives on the size of this problem, and published estimates of irreproducibility range from 51% [5] to 89% [6] (Fig 1).”
BackgroundLearned Non-Rigid Object Motion is a View-Invariant Cue to Recognizing Novel Objects
Chuang, L. L., Vuong, Q. C., & Bülthoff, H. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00026 · Cited by 113
The Increasing Urgency for Standards in Basic Biologic Research
Freedman, L. P., & Inglese, J. (2014) · Cancer Research · doi:10.1158/0008-5472.can-14-0925 · Cited by 89 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundSelf-citationOpen Peer Review by a Selected-Papers Network
Lee, C. J. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00001 · Cited by 85
Shi, J., Xia, Y., Song, Q., Zhou, X., Mizokami, A., Keller, E. T., Zhang, J., & Lü, Y. (2014) · Cancer Research · doi:10.1158/1538-7445.am2014-1 · Cited by 47
Reproducibility of Results in Preclinical Studies: A Perspective From the Bone Field
Manolagas, S. C., & Kronenberg, H. M. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2293 · Cited by 44
The Impact of Preclinical Irreproducibility on Drug Development
Freedman, L. P., & Gibson, M. (2014) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.9 · Cited by 42 · cited 1×
“, PubMed Commons, PubPeer), data sharing, and open access publishing that accelerate the identification of irreproducible studies [4].”
BackgroundSelf-citationFarrell, J., & Simcoe, T. S. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780195397840.013.0002 · Cited by 33 · cited 1×
“require a coordinated response are best solved by internally organized and driven, dynamic, and self-regulating collaborations of key stakeholders who establish and enforce their respective rules of engagement [32,33].”
BackgroundSelf-citationChristakis, D., & Zimmerman, F. J. (2013) · JAMA · doi:10.1001/jama.2013.281337 · Cited by 31
Gunn, W. (2014) · Nature · doi:10.1038/505483b · Cited by 12
Science Isn't Science If It Isn't Reproducible
Roth, K. A., & Cox, A. E. (2014) · American Journal Of Pathology · doi:10.1016/j.ajpath.2014.11.001 · Cited by 8 · cited 2×
“For example, the National Institutes of Health (NIH) have recently announced a list of Principles and Guidelines for Reporting Preclinical Research [24], which over 100 journals have joined as cosignatories and that builds on previous recommendations by Landis et al. [25] to improve methodological reporting of animal…”
BackgroundInfluentialCampbell, P. (2014) · Nature · doi:10.1038/505483f · Cited by 1
Buehring (2004) · doi:10.1290/1543-706x(2004)40<211:clchaa>2.0.co;2
Research works printed but not linked: 1
- CF Manski, 2003, Springer Series in Statistics, 178
Cited by
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Moving Forward: Challenges and Directions for Psychopathological Network Theory and Methodology
Fried, E. I., & Cramer, A. O. J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617705892 · Cited by 938
Genetic and transcriptional evolution alters cancer cell line drug response
Ben-David, U., Siranosian, B., Ha, G., Tang, H., Oren, Y., Hinohara, K., Strathdee, C. A., Dempster, J., Lyons, N. J., Burns, R., et al. (2018) · Nature · doi:10.1038/s41586-018-0409-3 · Cited by 931
Zhou, N., & Bao, J. (2020) · Database · doi:10.1093/database/baaa021 · Cited by 788
“We acknowledge that this assumption may introduce some errors because unreliable scientific claims might exist in original studies, especially in life sciences research which is beset by low reproducibility rates (16).”
Background
Metadata
- Authors
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S.
- Affiliations
- Global Biological Standards Institute (US)
- Boston University (US)
- Corresponding author
- Freedman, Leonard P.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-06-09
- Volume
- 13
- Issue
- 6
- Pages
- e1002165
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4461318
- OpenAlex
- W585341442
- Cited by
- 1,055
- References
- 35
- Citations by year
- 2026: 48 · 2025: 45 · 2024: 69 · 2023: 92 · 2022: 93 · 2021: 96 · 2020: 142 · 2019: 116 · 2018: 109 · 2017: 93 · 2016: 103 · 2015: 49
- FWCI
- 188.58
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- OpenBU (Boston University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Keywords
- reproducibility, preclinical research, research costs, research waste, reproducibility framework
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.