The Economics of Reproducibility in Preclinical Research
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible-in the United States alone. We outline a framework for solutions and a plan for long-term improvements in reproducibility rates that will help to accelerate the discovery of life-saving therapies and cures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Studies reporting the prevalence of irreproducibility. Source: Begley and Ellis [6], Prinz et al. [7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al. [9].](https://figures.cochat.ai/paper-figures/a30bd9bd65dc4bedb5db19b413eb814f/the-economics-of-reproducibility-in-preclinical-research-figure-1.jpg)
![Fig 2 Estimated US preclinical research spend and categories of errors that contribute to irreproducibility. Note that the percentage value of error for each category is the midpoint of the high and low prevalence estimates for that category divided (weighted) by the sum of all midpoint error rates (see S1 Dataset). Source: Chakma et al. [18] and the American Association for the Advancement of Science (AAAS) [19].](https://figures.cochat.ai/paper-figures/207a0a58df5c4620a819d7aa1f4aa960/the-economics-of-reproducibility-in-preclinical-research-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Besides, reliably determining actual translational success rates is unmanageable as long as the current status of reporting of preclinical research leaves room for improvement [89], and non-reproducibility is such a critical issue in both animal [1] and human [90] studies.”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The S&B tool therefore directly addresses the larger problem of material reagents and standards representing the major source of incorrect published results from pre-clinical research [3, 4].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current drug development is handicapped by high attrition rates; many molecules that were promising during preclinical development fail during subsequent clinical testing [1].”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [16]
Experimental — accuracy not yet measured
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Ostrom, E. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 18,859
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“[7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · cited 1×
“Much has been written about the alarming number of preclinical studies that were later found to be irreproducible [1,2].”
BackgroundBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Partial Identification of Probability Distributions
(2003) · Springer Series in Statistics · doi:10.1007/b97478 · Cited by 698
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Research works printed but not linked: 1
- CF Manski, 2003, Springer Series in Statistics, 178
Cited by
Items 101–200 of 1,003
https://www.altex.org/index.php/altex/article/view/1339
Krebs, A., Waldmann, T., Wilks, M. F., van Vugt-Lussenburg, B. M. A., van der Burg, B., Terron, A., Steger-Hartmann, T., Ruegg, J., Rovida, C., Pedersen, E., et al. (2019) · ALTEX · doi:10.14573/altex.1909271 · Cited by 77
Data management plans: the missing perspective
Williams, M., Bagwell, J., & Nahm Zozus, M. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.004 · Cited by 75
Mascia, A., Cirafici, A. M., Bongiovanni, A., Colotti, G., Lacerra, G., Carlo, M. D., Digilio, F. A., Liguori, G. L., Lanati, A., & Kisslinger, A. (2020) · Accreditation and Quality Assurance · doi:10.1007/s00769-020-01441-9 · Cited by 77
How we can make ecotoxicology more valuable to environmental protection
Hanson, M. L., Wolff, B. A., Green, J. W., Kivi, M., Panter, G. H., Warne, M. S. J., Ågerstrand, M., & Sumpter, J. P. (2016) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2016.07.160 · Cited by 85
“For example, it has been estimated that irreproducible results in the biomedical literature cost 28 billion USD in America alone, each year (Freedman et al., 2015).”
BackgroundPoor replication validity of biomedical association studies reported by newspapers
Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172650 · Cited by 76
“Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8– 11], and iii) clinical trials [12–16].”
ResultIntroduction to the EQIPD quality system
Bespalov, A., Bernard, R., Gilis, A., Gerlach, B., Guillén, J., Castagné, V., Lefevre, I. A., Ducrey, F., Monk, L., Bongiovanni, S., et al. (2021) · eLife · doi:10.7554/elife.63294 · Cited by 76
Harris, B. N. (2019) · General and Comparative Endocrinology · doi:10.1016/j.ygcen.2019.113355 · Cited by 83
3S - Systematic, systemic, and systems biology and toxicology
Smirnova, L. (2018) · ALTEX · doi:10.14573/altex.1804051 · Cited by 72
“A significant portion of irreproducible science could be avoided by using evidence-based approaches (Hartung, 2013; Freedman et al., 2015).”
BackgroundThe role of replication studies in ecology
Fraser, H. S., Barnett, A., Parker, T. H., & Fidler, F. M. (2020) · Ecology and Evolution · doi:10.1002/ece3.6330 · Cited by 91
Jha, B. S., Farnoodian, M., & Bharti, K. (2020) · Stem Cells Translational Medicine · doi:10.1002/sctm.20-0242 · Cited by 76
“firming the manufacturing process reproducibility is a crucial part of the preclinical studies.(73,74) Once the research-grade process is translated into a GMP-facility, successful cGMP-compliant manufacturing”
MethodThe researchers’ view of scientific rigor—survey on the conduct and reporting of in vivo research
Reichlin, T. S., Vogt, L., & Würbel, H. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165999 · Cited by 72
“Based on estimates of irreproducibility in preclinical research, up to USD 28B/year may be spent in the US alone on irreproducible preclinical research [19].”
BackgroundAuthorization of Animal Experiments Is Based on Confidence Rather than Evidence of Scientific Rigor
Vogt, L., Reichlin, T. S., Nathues, C., & Würbel, H. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000598 · Cited by 73
Standardization in synthetic biology: an engineering discipline coming of age
Decoene, T., Paepe, B. D., Maertens, J., Coussement, P., Peters, G., Maeseneire, S. L. D., & Mey, M. D. (2017) · Critical Reviews in Biotechnology · doi:10.1080/07388551.2017.1380600 · Cited by 76
“address the enormous cellular complexity [3,8,9] and contribute to reproducibility in biological experimentation [10,11].”
BackgroundCritical review of reporting of the data analysis step in metabolomics
Considine, E. C., Thomas, G. R., Boulesteix, A., Khashan, A. S., & Kenny, L. C. (2017) · Metabolomics · doi:10.1007/s11306-017-1299-3 · Cited by 81
The quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 89
Peer review: The current landscape and future trends
Jubb, M. (2016) · Learned Publishing · doi:10.1002/leap.1008 · Cited by 75
“Nevertheless, peer review clearly sometimes fails to prevent the publication of poor-quality or flawed research, nor has it ensured that published research is reproducible by independent researchers (although there is much debate about replicability, especially in health and life sciences: e.g. Begley & Ellis, 2012; M…”
Preclinical Pain Research: Can We Do Better?
Clark, J. D. (2016) · Anesthesiology · doi:10.1097/aln.0000000000001340 · Cited by 75
Computational Methodologies for Real-Space Structural Refinement of Large Macromolecular Complexes
Goh, B. C., Hadden‐Perilla, J. A., Bernardi, R. C., Singharoy, A., McGreevy, R., Rudack, T., Cassidy, C. K., & Schulten, K. J. (2016) · Annual Review of Biophysics · doi:10.1146/annurev-biophys-062215-011113 · Cited by 75
“Reproducibility is a topic of intense debate in the biomedical and life sciences community (15, 52).”
BackgroundMoser, G., & Huppertz, B. (2017) · Placenta · doi:10.1016/j.placenta.2017.02.007 · Cited by 76
“Otherwise the percentage of irreproducible results will further increase [36].”
BackgroundStriving for transparent and credible research: practical guidelines for behavioral ecologists
Ihle, M., Winney, I. S., Krystalli, A., & Croucher, M. (2017) · Behavioral Ecology · doi:10.1093/beheco/arx003 · Cited by 75
“…the same code on different computers), automatization, and appropriate documentation, are undeni-ably a hindrance to our productivity (Markowetz 2015) (Box 1, b) and can lead to major retractions (Hall and Salipante 2007; Schweppe et al. 2008; Pryke et al. 2014; Freedman et al. 2015; Neimark 2015).”
BackgroundSystematic heterogenization for better reproducibility in animal experimentation
Richter, S. H. (2017) · Lab Animal · doi:10.1038/laban.1330 · Cited by 78
Davies, G. F., Greenhough, B., Hobson‐West, P., Kirk, R. G. W., Applebee, K., Bellingan, L., Berdoy, M., Buller, H. J., Cassaday, H. J., Davies, K., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158791 · Cited by 92
“Furthermore, debates on reproducibility and bias, relevant to the benefits of laboratory animal research, indicate how individual, institutional and commercial pressures on scientists may influence the selection of data and presentation of results [29–33].”
BackgroundRogers, J., Churilov, L. P., Hannan, A. J., & Renoir, T. (2016) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2016.12.007 · Cited by 80
The impact of structural genomics: the first quindecennial
Grabowski, M., Niedzialkowska, E., Zimmerman, M., & Minor, W. (2016) · Journal of Structural and Functional Genomics · doi:10.1007/s10969-016-9201-5 · Cited by 68
“According to a recent study, the annual cost of irreproducibility in life science amounts to a staggering sum of $28 billion for the US alone [111].”
BackgroundA close look onto structural models and primary ligands of metallo-β-lactamases
Raczynska, J. E., Shabalin, I. G., Minor, W., Wlodawer, A., & Jaskólski, M. (2018) · Drug Resistance Updates · doi:10.1016/j.drup.2018.08.001 · Cited by 67
“Not only are large resources of time, manpower, and money (estimated at $28 billion a year in the US alone (Freedman et al., 2015)) invested in potentially useless endeavors, but also false hopes may be raised for patients and attention may be diverted in the erroneous directions.”
BackgroundKan, V., Vargo, E., Machover, N., Ishii, H., Pan, S., Chen, W. '., & Kakehi, Y. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025952 · Cited by 66
Nanomedicine: is It Lost in translation?
Greish, K. F., Mathur, A., Bakhiet, M., & Taurin, S. (2018) · Therapeutic Delivery · doi:10.4155/tde-2017-0118 · Cited by 71
DuBois, J. M., & Antes, A. L. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001966 · Cited by 77
“31”
Dirnagl, U., Duda, G. N., Grainger, D. W., Reinke, P., & Roubenoff, R. (2022) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2022.114118 · Cited by 72
“Translational failures in this view result from faulty experimental designs or conduct [52] and the intrinsic lack of reproducibility of biomedical research [55].”
BackgroundToward good in vitro reporting standards
Hartung, T. (2019) · ALTEX · doi:10.14573/altex.1812191 · Cited by 65
“Freedman et al. (2015) have estimated how this translates to wasted money: “the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US $28B)/year spent on preclinical research that is not reproducible – in the United States…”
Shin, H. S., Shin, H., & Shudo, Y. (2021) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2021.673683 · Cited by 72
Stukalin, Y., Lan, A., & Einat, H. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.01.034 · Cited by 66
Centenary of the death of Elie Metchnikoff: a visionary and an outstanding team leader
Cavaillon, J., & Legout, S. (2016) · Microbes and Infection · doi:10.1016/j.micinf.2016.05.008 · Cited by 65
Bridging the translational innovation gap through good biomarker practice
van Gool, A. J., Biétrix, F., Caldenhoven, E., Zatloukal, K., Scherer, A., Litton, J., Meijer, G. A., Blomberg, N., Smith, A., Mons, B., et al. (2017) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2017.72 · Cited by 62
Made in Academia: The Effect of Institutional Origin on Inventors’ Attention to Science
Bikard, M. A. (2018) · Organization Science · doi:10.1287/orsc.2018.1206 · Cited by 72
“Not only is scientific knowledge generally complex and tacit (Polanyi 1958; Zucker and Darby 1996), but the literature is vast and fast-changing (Jones 2009; Alberts 2010) and the reliability of published findings is uncertain (Freedman, Cockburn, and Simcoe 2015).”
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Brain and Behavior · doi:10.1002/brb3.1141 · Cited by 60
Oxytocin selectively modulates brain response to stimuli probing social synchrony
Levy, J., Goldstein, A., Zagoory-Sharon, O., Weisman, O., Schneiderman, I., Eidelman-Rothman, M., & Feldman, R. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2015.09.066 · Cited by 61
“It is important to note that while these concerns are certainly valid and OT findings should be interpreted with caution, such problems are not unique to intranasal OT research and may point to general issues the scientific community must address (Freedman et al., 2015; Ioannidis, 2005).”
BackgroundMoving forward: challenges and directions for psychopathological network theory and methodologyPreprint
Fried, E. I., & Cramer, A. O. J. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mh3cf · Cited by 63
Augustine, S., Cheng, W., Avey, M. T., Chan, M. L. M., Lingappa, S. M. C., Hutton, B., & Thébaud, B. (2019) · Stem Cells Translational Medicine · doi:10.1002/sctm.19-0193 · Cited by 64
“Irreproducibility remains a challenge in preclinical research when upward of 50% findings fail replication resulting in over $28 billion US/y in preclinical research waste.(89) Reasons for irreproducibility could be observer bias or confirmation bias.”
BackgroundWhy (and how) we should publish negative data
Nimpf, S., & Keays, D. A. (2019) · EMBO Reports · doi:10.15252/embr.201949775 · Cited by 72
Introduction to precision medicine in COPD
Leung, J. M., Obeidat, M., Sadatsafavi, M., & Sin, D. D. (2019) · European Respiratory Journal · doi:10.1183/13993003.02460-2018 · Cited by 60
Hoffmann, S., Schönbrodt, F. D., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.201925 · Cited by 85
“It has been estimated that in preclinical research alone, approximately $28 billion dollars are spent every year on research findings that are not replicable1 [10].”
BackgroundBiofilm over teeth and restorations: What do we need to know?
Lin, N. J. (2017) · Dental Materials · doi:10.1016/j.dental.2017.03.003 · Cited by 63
Festing, M. F. W. (2020) · Scientific Reports · doi:10.1038/s41598-020-74538-3 · Cited by 66
To Err is Human; To Reproduce Takes Time
Scott, S. L., Gunnoe, T. B., Fornasiero, P., & Crudden, C. M. (2022) · ACS Catalysis · doi:10.1021/acscatal.2c00967 · Cited by 64
Reproducible image handling and analysis
Miura, K., & Nørrelykke, S. F. (2021) · The EMBO Journal · doi:10.15252/embj.2020105889 · Cited by 65
“With a reported 50 % of preclinical studies not being replicable the economic cost, attributable to irreproducible data analysis and reporting alone, is estimated at $ 7 B/year in the US (Freedman, Cockburn, & Simcoe, 2015 ).”
BackgroundLabbé, C., Grima, N., Gautier, T., Favier, B., & Byrne, J. A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0213266 · Cited by 55
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
BackgroundTen Basic Rules of Antibody Validation
Weller, M. G. (2018) · Analytical Chemistry Insights · doi:10.1177/1177390118757462 · Cited by 62
The Experimental Design Assistant
du Sert, N. P., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I. C., Fry, D., Karp, N. A., Macleod, M. R., Moon, L. D. F., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4462 · Cited by 58
The Antibody Society’s antibody validation webinar series
Voskuil, J. L. A., Bandrowski, A., Begley, C. G., Bradbury, A. R. M., Chalmers, A. D., Gomes, A. V., Hardcastle, T., Lund-Johansen, F., Plückthun, A., Roncador, G., et al. (2020) · mAbs · doi:10.1080/19420862.2020.1794421 · Cited by 54
“However, this is probably a substantial underestimate of the real cost of poorly validated research antibodies, given that the ubiquitous use of antibodies for many/most research procedures and the cost of irreproducible research in 2015 was estimated to be 28 USD billion per annum in the USA alone.(14)”
BackgroundRelevance of Antibody Validation for Flow Cytometry
Kalina, T., Lundsten, K., & Engel, P. (2019) · Cytometry Part A · doi:10.1002/cyto.a.23895 · Cited by 60
“Antibodies are believed to be, in part, responsible for inconsistent experimental results and the publication of inaccurate data in the scientific literature (12,13).”
ResultSingh, V. P., Pratap, K., Sinha, J., Desiraju, B. K., Bahal, D., & Kukreti, R. (2016) · International Journal of Immunopathology and Pharmacology · doi:10.1177/0394632016671728 · Cited by 63
“554 International Journal of Immunopathology and Pharmacology 29(4)”
BackgroundMethodInfluentialMichel, M. C., Cardozo, L., Chermansky, C., Cruz, F., Igawa, Y., Lee, K., Sahai, A., Wein, A. J., & Andersson, K. (2023) · Pharmacological Reviews · doi:10.1124/pharmrev.121.000523 · Cited by 58
Evaluation of Sex Differences in Preclinical Pharmacology Research: How Far Is Left to Go?
Allegra, S., Chiara, F., Grazia, D. D., Gaspari, M., & Francia, S. D. (2023) · Pharmaceuticals · doi:10.3390/ph16060786 · Cited by 57
From big data to important information
Bar-Yam, Y. (2016) · Complexity · doi:10.1002/cplx.21785 · Cited by 64
“Careful methodologies are needed to ensure that errors do not take place and multiple critiques of statistically based experiments suggest that errors are common in science [73–76].”
Launch of an Infrastructure for Health Research: BBMRI-ERIC
Litton, J. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2018.0027 · Cited by 52
“22,23 One study found that only half of all reagents mentioned in > 200 recent publications from a range of journals and fields could be adequately identified, indicating a failure of researchers to comprehensively report the re-agents they use and of editors and reviewers to require such reporting.”
Assay Guidance Manual : Quantitative Biology and Pharmacology in Preclinical Drug Discovery
Coussens, N. P., Sittampalam, G. S., Guha, R., Brimacombe, K. R., Grossman, A., Chung, T. D. Y., Weidner, J. R., Riss, T., Trask, O. J., Auld, D. S., et al. (2018) · Clinical and Translational Science · doi:10.1111/cts.12570 · Cited by 52
Ten simple rules on how to write a standard operating procedure
Hollmann, S., Frohme, M., Endrullat, C., Kremer, A., D’Elia, D., Regierer, B., Nechyporenko, A. S., & CA15110, O. B. O. C. A. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008095 · Cited by 52
Harnessing QbD, Programming Languages, and Automation for Reproducible Biology
Sadowski, M. I., Grant, C., & Fell, T. S. (2015) · Trends in biotechnology · doi:10.1016/j.tibtech.2015.11.006 · Cited by 60
“One recent study estimated that the annual economic cost of irreproducible research in the life sciences is $28 billion [20].”
BackgroundManaging risks in drug discovery: reproducibility of published findings
Kannt, A., & Wieland, T. (2016) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-016-1216-8 · Cited by 57
“the USA alone, 28 billion dollars are annually spent on preclinical research that is not reproducible (Freedman et al. 2015).”
BackgroundInfluentialHeterogenising study samples across testing time improves reproducibility of behavioural data
Bodden, C., von Kortzfleisch, V. T., Karwinkel, F., Kaiser, S., Sachser, N., & Richter, S. H. (2019) · Scientific Reports · doi:10.1038/s41598-019-44705-2 · Cited by 61
Gutbier, S., May, P., Berthelot, S., Krishna, A., Trefzer, T., Behbehani, M., Efremova, L., Delp, J., Gstraunthaler, G., Waldmann, T., et al. (2018) · Archives of Toxicology · doi:10.1007/s00204-018-2326-5 · Cited by 49
“Therefore, an important issue concerning a perceived reproducibility crisis of science (Freedman 2015; Freedman et al.”
BackgroundKöse, M. F., & Korkmaz, M. (2019) · Higher Education Research & Development · doi:10.1080/07294360.2019.1634679 · Cited by 49
The Challenge of Reproducibility and Accuracy in Nutrition Research: Resources and Pitfalls
Sorkin, B. C., Kuszak, A. J., Williamson, J. S., Hopp, D. C., & Betz, J. M. (2016) · Advances in Nutrition · doi:10.3945/an.115.010595 · Cited by 52
Quality control of protein reagents for the improvement of research data reproducibility
de Marco, A., Berrow, N. S., Lebendiker, M., García-Alai, M., Knauer, S. H., Mendez, B. L., Matagne, A., Parret, A. H. A., Remans, K., Uebel, S. F. W., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-23167-z · Cited by 56
“They have been embraced by a wide community of specialists (a full list of these researchers can be found on ARBRE-MOBIEU and P4EU website) and comprise three parts: (1) minimal information, (2) minimal QC tests, and (3) extended QC tests.”
BackgroundMethodOptimized design and analysis of preclinical intervention studies in vivo
Laajala, T. D., Jumppanen, M., Huhtaniemi, R., Fey, V., Kaur, A., Knuuttila, M., Aho, E., Oksala, R., Westermarck, J., Mäkelä, S., et al. (2016) · Scientific Reports · doi:10.1038/srep30723 · Cited by 48
“Even if this may lead to unexpectedly high number of test animals, it is widely acknowledged that underpowered or otherwise poorly designed studies are not only unethical but also contribute to both delays and increased costs of drug development process 4,9 .”
New Challenges for Biobanks: Accreditation to the New ISO 20387:2018 Standard Specific for Biobanks
Blasio, P. D., & Biunno, I. (2021) · BioTech · doi:10.3390/biotech10030013 · Cited by 47
“Quality of samples [15–17] can be the bottleneck for diagnosis and research studies in case the BMaD [10] are collected by several Biobanks which use different standards [10,15,18].”
BackgroundMethodDesigning, conducting, and reporting reproducible animal experiments
Wilson, E., Ramage, F. J., Wever, K. E., Sena, E. S., Macleod, M. R., & Currie, G. L. (2023) · Journal of Endocrinology · doi:10.1530/joe-22-0330 · Cited by 44
The Reproducibility Crisis and Academic Libraries
Sayre, F., & Riegelman, A. (2018) · College & Research Libraries · doi:10.5860/crl.79.1.2 · Cited by 59
Advanced In Vitro Models for Preclinical Drug Safety: Recent Progress and Prospects
Nair, D. G., & Weiskirchen, R. (2024) · Current Issues in Molecular Biology · doi:10.3390/cimb47010007 · Cited by 53
Zeiss, C. J., Allore, H. G., & Beck, A. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0171790 · Cited by 43
Kitney, R. I., Adeogun, M., Fujishima, Y., Goñi‐Moreno, Á., Johnson, R., Maxon, M. E., Steedman, S., Ward, S., Winickoff, D. E., & Philp, J. C. (2019) · Trends in biotechnology · doi:10.1016/j.tibtech.2019.03.017 · Cited by 45
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44
Tradition, not science, is the basis of animal model selection in translational and applied research
Veening-Griffioen, D. H., Ferreira, G. S., Boon, W. P. C., Wied, C. C. G.-D., Schellekens, H., Moors, E. H. M., & Meer, P. J. K. V. (2020) · ALTEX · doi:10.14573/altex.2003301 · Cited by 44
“, 2018) neither did the level of reproducibility of animal studies (Freedman et al., 2015).”
BackgroundZebrafish (Danio rerio) meets bioethics: the 10Rs ethical principles in research
Canedo, A., Saiki, P. T. O., Santos, A. L., Carneiro, K. D. S., de Souza, A. M., Qualhato, G., Brito, R. D. S., Mello-Andrade, F., & Rocha, T. L. (2022) · Ciência Animal Brasileira · doi:10.1590/1809-6891v22e-70884 · Cited by 58
Biobank sustainability: current status and future prospects
Doucet, M., Yuille, M., Georghiou, L., & Dagher, G. (2017) · Journal of Biorepository Science for Applied Medicine · doi:10.2147/bsam.s100899 · Cited by 47
“It has been reported that about 30% of unreproducible results are related to an inappropriate quality of samples and products, which is responsible for ~ €9 billion ($8 billion) per year of unnecessary expenditure in the US alone.(27) A funded national/international strategy aiming to deliver robust diagnosis and repr…”
BackgroundMiles, B., & Lee, P. L. (2018) · SLAS Technology · doi:10.1177/2472630318784506 · Cited by 45
Mede, N. G., Schäfer, M. S., Ziegler, R., & Weißkopf, M. (2020) · Public Understanding of Science · doi:10.1177/0963662520954370 · Cited by 63
“Poor reproducibility(1) rates were also found in medicine (Freedman et al., 2015), economics (Camerer et al.”
BackgroundStrong Inference in Mathematical Modeling: A Method for Robust Science in the Twenty-First Century
Ganusov, V. V. (2016) · Frontiers in Microbiology · doi:10.3389/fmicb.2016.01131 · Cited by 49
“If these findings can be extrapolated to the whole field of biomedical research one study estimates that over $28B are wasted on unreproducible studies, and half of those expenditures are suggested to result from inappropriate study design and data analysis (Freedman et al., 2015).”
BackgroundResultAnalyzing crowdsourced ratings of speech-based take-over requests for automated driving
Bazilinskyy, P., & de Winter, J. (2017) · Applied Ergonomics · doi:10.1016/j.apergo.2017.05.001 · Cited by 53
“Small samples are a prime cause of poor replicability, a message that has now transpired to many fi elds, including medicine (Arrowsmith, 2011; Begley and Ellis, 2012; Freedman et al., 2015), economics (Ioannidis and Doucouliagos, 2013), and neuro-science (Button et al., 2013).”
BackgroundCurrent State of Modeling Human Psychiatric Disorders Using Zebrafish
Costa, F. V., Kolesnikova, T. O., Galstyan, D. S., Ilyin, N. P., de Abreu, M. S., Petersen, E. V., Demin, K. A., Yenkoyan, K. B., & Kalueff, A. V. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24043187 · Cited by 45
Torres, L., & Tiersch, T. R. (2018) · Journal of the World Aquaculture Society · doi:10.1111/jwas.12541 · Cited by 41
A supporting ecosystem to mature extracellular vesicles into clinical application
Wever, O. D., & Hendrix, A. (2019) · The EMBO Journal · doi:10.15252/embj.2018101412 · Cited by 43
“Although any efforts to improve reproducibility will require a measured investment in capital and time, the long-term benefits to society derived from increased scientific fidelity will greatly exceed the upfront costs (Freedman et al , 2015).”
Exercise interventions for low back pain are poorly reported: a systematic review
Davidson, S. R. E., Kamper, S. J., Haskins, R., Robson, E. K., Gleadhill, C., da Silva, P. V., Williams, A. J., Yu, Z., & Williams, C. M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.020 · Cited by 44
Nakagawa, S., Lagisz, M., Yang, Y., & Drobniak, S. M. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002423 · Cited by 43
A Biobank's Journey: Implementation of a Quality Management System and Accreditation to ISO 20387
Mouttham, L., Garrison, S. J., Archer, D. L., & Castelhano, M. G. (2020) · Biopreservation and Biobanking · doi:10.1089/bio.2020.0068 · Cited by 40
SECIMTools: a suite of metabolomics data analysis tools
Kirpich, A., Ibarra, M., Moskalenko, O. V., Fear, J. M., Gerken, J., Mi, X., Ashrafi, A., Morse, A. M., & McIntyre, L. M. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2134-1 · Cited by 44
Signaling of nicotinic acetylcholine receptors in mononuclear phagocytes
Richter, K., & Grau, V. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106727 · Cited by 40
Yıldırım, Z., Swanson, K., Wu, X., Zou, J., & Wu, J. C. (2024) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-022724-095035 · Cited by 49
“In the United States alone, approximately $28 billion is spent on preclinical research annually (3).”
Mullane, K., & Williams, M. J. (2015) · Biochemical Pharmacology · doi:10.1016/j.bcp.2015.07.002 · Cited by 42
“This probably results from investigator hubris and complacency, that has led to preventable translation failures in cancer drug discovery [32] and avoidable waste that has reached staggering proportions [17,33].”
BackgroundMiller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 43
“As a consequence, individual studies would be less replicable due to statistical variability (Freedman et al., 2015).”
BackgroundFlorian, D. C., Odziomek, M., Ock, C. L., Chen, H., & Guelcher, S. A. (2020) · Scientific Reports · doi:10.1038/s41598-020-70465-5 · Cited by 42
What is the optimum design for my animal experiment?
Karp, N. A., & Fry, D. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100126 · Cited by 45
Evidence-based medicine in times of crisis
Djulbegović, B., & Guyatt, G. H. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.002 · Cited by 42
“One is that optimism regarding important benefits is likely to result in disappointment: clinical success from interventions suggested by apparently promising results from preclinical studies is extraordinarily infrequent [7], as are successes following low-quality clinical evidence.”
Fergusson, D. A., Avey, M. T., Barron, C. C., Bocock, M., Biefer, K., Boet, S., Bourque, S. L., Conic, I., Chen, K., Dong, Y., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215221 · Cited by 40
“Poorly designed, executed, and reported preclinical studies have contributed to the replication crisis, and increase waste of research funding, laboratory animals, and personnel time [8, 9].”
BackgroundSafeguarding Structural Data Repositories against Bad Apples
Minor, W., Dauter, Z., Helliwell, J. R., Jaskólski, M., & Wlodawer, A. (2016) · Structure · doi:10.1016/j.str.2015.12.010 · Cited by 38
“The price tag of the irreproducibility of published results was recently estimated to be in the vicinity of 28 billion dollars per annum (Freedman et al., 2015), almost equivalent to the entire annual budget of the NIH.”
BackgroundPellizzon, M., & Ricci, M. R. (2018) · Current Protocols in Toxicology · doi:10.1002/cptx.55 · Cited by 38
Kan, V., Vargo, E., Machover, N., Ishii, H., Pan, S., Chen, W. '., & Kakehi, Y. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · doi:10.48550/arxiv.1605.01148 · Cited by 36
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility
Hrynaszkiewicz, I. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_290 · Cited by 43
“An economic analysis in 2015 estimated that irreproducible preclinical research costs US $28 billion per year (Freedman et al. 2015).”
BackgroundLinking Engineered Cells to Their Digital Twins: A Version Control System for Strain Engineering
Tellechea‐Luzardo, J., Winterhalter, C., Widera, P., Kozyra, J., de Lorenzo, V., & Krasnogor, N. (2020) · ACS Synthetic Biology · doi:10.1021/acssynbio.9b00400 · Cited by 39
Metadata
- Authors
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S.
- Affiliations
- Global Biological Standards Institute (US)
- Boston University (US)
- Corresponding author
- Freedman, Leonard P.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-06-09
- Volume
- 13
- Issue
- 6
- Pages
- e1002165
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4461318
- OpenAlex
- W585341442
- Cited by
- 1,055
- References
- 35
- Citations by year
- 2026: 48 · 2025: 45 · 2024: 69 · 2023: 92 · 2022: 93 · 2021: 96 · 2020: 142 · 2019: 116 · 2018: 109 · 2017: 93 · 2016: 103 · 2015: 49
- FWCI
- 188.58
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- OpenBU (Boston University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Keywords
- reproducibility, preclinical research, research costs, research waste, reproducibility framework
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.