The Economics of Reproducibility in Preclinical Research
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible-in the United States alone. We outline a framework for solutions and a plan for long-term improvements in reproducibility rates that will help to accelerate the discovery of life-saving therapies and cures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Studies reporting the prevalence of irreproducibility. Source: Begley and Ellis [6], Prinz et al. [7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al. [9].](https://figures.cochat.ai/paper-figures/a30bd9bd65dc4bedb5db19b413eb814f/the-economics-of-reproducibility-in-preclinical-research-figure-1.jpg)
![Fig 2 Estimated US preclinical research spend and categories of errors that contribute to irreproducibility. Note that the percentage value of error for each category is the midpoint of the high and low prevalence estimates for that category divided (weighted) by the sum of all midpoint error rates (see S1 Dataset). Source: Chakma et al. [18] and the American Association for the Advancement of Science (AAAS) [19].](https://figures.cochat.ai/paper-figures/207a0a58df5c4620a819d7aa1f4aa960/the-economics-of-reproducibility-in-preclinical-research-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Besides, reliably determining actual translational success rates is unmanageable as long as the current status of reporting of preclinical research leaves room for improvement [89], and non-reproducibility is such a critical issue in both animal [1] and human [90] studies.”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The S&B tool therefore directly addresses the larger problem of material reagents and standards representing the major source of incorrect published results from pre-clinical research [3, 4].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current drug development is handicapped by high attrition rates; many molecules that were promising during preclinical development fail during subsequent clinical testing [1].”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [16]
Experimental — accuracy not yet measured
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Ostrom, E. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 18,859
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“[7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · cited 1×
“Much has been written about the alarming number of preclinical studies that were later found to be irreproducible [1,2].”
BackgroundBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Partial Identification of Probability Distributions
(2003) · Springer Series in Statistics · doi:10.1007/b97478 · Cited by 698
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Research works printed but not linked: 1
- CF Manski, 2003, Springer Series in Statistics, 178
Cited by
Items 301–400 of 1,003
From Gutenberg to Open Science: An Unfulfilled Odyssey
Triggle, C. R., & Triggle, D. J. (2016) · Drug Development Research · doi:10.1002/ddr.21369 · Cited by 22
“Clearly the problem of reproducibility of data is now recognized as a serious issue that needs to be addressed [Ioannidis, 2005; Collins and Tabak, 2014; Begley and Ioannidis, 2015; Freedman et al., 2015; Jarvis and Williams, 2016].”
BackgroundIn vitro disease models 4.0 via automation and high-throughput processing
Eggert, S., & Hutmacher, D. W. (2019) · Biofabrication · doi:10.1088/1758-5090/ab296f · Cited by 25
“Irreproducible research results in enormous economic consequences, such as the waste of grant funding [19]”
BackgroundEnhancing reproducibility using interprofessional team best practices
Rolland, B., Burnside, E. S., Voils, C. I., Shah, M. N., & Brasier, A. R. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.512 · Cited by 24
“In the US alone, cumulative costs of irreproducible preclinical research exceed 50% of research expenditures, resulting in the economic impact of ~$28B annually [3].”
BackgroundBlueprints for measuring natural behavior
Puścian, A., & Knapska, E. (2022) · iScience · doi:10.1016/j.isci.2022.104635 · Cited by 24
“It has been argued that in animal research the prevalence of irreproducible data can be as high as 50 to 90% (Prinz et al., 2011; Collins and Tabak, 2014; Freedman et al., 2015).”
BackgroundReprohackathons: promoting reproducibility in bioinformatics through training
Cokelaer, T., Cohen‐Boulakia, S., & Lemoine, F. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad227 · Cited by 30
“In the past two decades, reproducibility has become a major concern in many disciplines, starting with social and psychological sciences (Open Science Collaboration 2015) and spreading to other domains such as pre-clinical research (Freedman et al. 2015), and computational and life sciences (Baker 2016; Cohen-Boulakia…”
BackgroundCatalá-López, F., Ridao-López, M., Tejedor-Romero, L., Caulley, L., Hutton, B., Husereau, D., Alonso‐Arroyo, A., Bernal‐Delgado, E., Drummond, M., & Moher, D. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.10.024 · Cited by 21
EXPLORING THE REPRODUCIBILITY OF PROBABILISTIC CAUSAL MOLECULAR NETWORK MODELS
COHAIN, A., DIVARANIYA, A. A., ZHU, K., SCARPA, J. R., KASARSKIS, A., ZHU, J., CHANG, R., DUDLEY, J. T., & SCHADT, E. E. (2017) · Biocomputing 2017 · doi:10.1142/9789813207813_0013 · Cited by 23
“%) of studies not reproduced are due to data-analysis and reporting issues 17 .”
BackgroundMale zebra finches have limited ability to identify high-fecundity females
Wang, D., Kempenaers, N., Kempenaers, B., & Forstmeier, W. (2017) · Behavioral Ecology · doi:10.1093/beheco/arx037 · Cited by 23
Bridging the Gap between Reproducibility and Translation: Data Resources and Approaches
Zeiss, C. J., & Johnson, L. K. (2017) · ILAR Journal · doi:10.1093/ilar/ilx017 · Cited by 21
“However, if the goal of animal research is to promote human health, and failed efficacy is seen as an obstacle to this goal, the sustainability of the current model is questionable from both an economic (Freedman et al. 2015) and ethical (Russell and Burch 1959) perspective.”
BackgroundSalandra, R. (2018) · Research Policy · doi:10.1016/j.respol.2018.04.005 · Cited by 24
“In preclinical research, USD$28b are estimated to be spent every year in the US on studies that are irreproducible, leading to high costs and delays in the development of new drugs (Freedman et al., 2015).”
BackgroundTranslational medicine in neuromuscular disorders: from academia to industry
Cowling, B. S., & Thielemans, L. A. (2019) · Disease Models & Mechanisms · doi:10.1242/dmm.041434 · Cited by 25
“However, data generated in one lab with one model may not always be reproducible, and up to 50% of studies fail to be duplicated (Freedman et al., 2015).”
Haselimashhadi, H., Mason, J., Mallon, A., Smedley, D. P., Meehan, T. F., & Parkinson, H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242933 · Cited by 22
“One example is irreproducibility in results from animal experiments that is cited as a major contributor to explain many drugs failure in the development pipeline [39].”
BackgroundTritz, Z. P., Ayasoufi, K., & Johnson, A. J. (2021) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdab066 · Cited by 24
Large inherent variability in data derived from highly standardised cell culture experiments
Reddin, I. G., Fenton, T. R., Wass, M. N., & Michaelis, M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106671 · Cited by 23
Toward a common standard for data and specimen provenance in life sciences
Wittner, R., Holub, P., Mascia, C., Frexia, F., Müller, H., Plass, M., Allocca, C. M., Betsou, F., Burdett, T., Cancio, I., et al. (2023) · Learning Health Systems · doi:10.1002/lrh2.10365 · Cited by 30
“Hence, the urgent need for trustworthy documentation of the data lineage and specimens is evident, especially when considering the serious impact of irreproducible or even flawed scientific results on health, economics, and political decisions.(12-16)”
BackgroundBad habits–good goals? Meta-analysis and translation of the habit construct to alcoholism
Giannone, F., Ebrahimi, C., Endraß, T., Hansson, A. C., Schlagenhauf, F., & Sommer, W. H. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02965-1 · Cited by 26
“Importantly, our meta-analysis offers a framework exemplifying how to address the reproducibility crisis in preclinical research [75, 76], potentially leading to the adoption of more rigorous experimental designs.”
MethodChauvin, A., Truchot, J., Bafeta, A., Pateron, D., Plaisance, P. I., & Yordanov, Y. (2017) · Internal and Emergency Medicine · doi:10.1007/s11739-017-1770-1 · Cited by 24
“To avoid waste of research with lack of information on methods, authors, editors, and peer-reviewers must pay better attention to the reporting of keys elements of the reproducibility [119].”
MethodChallenges for the Sustainability of University-Run Biobanks
Kongsholm, N. C. H., Christensen, S. T., Herrmann, J. R., Larsen, L. A., Minssen, T., Pedersen, L. B., Neethu, R., Tommerup, N., Tupasela, A., & Schovsbo, J. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2018.0054 · Cited by 20
“Freedman et al. 37 have suggested that a major challenge facing biomedical research using tissue samples is the lack of reproducibility of research results.”
Linsen, L., Vanhees, K., Vanoppen, E., Ulenaers, K., Driessens, S., Penders, J., Somers, V. A., Stinissen, P., & Rummens, J. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00224 · Cited by 20
Affimers as an alternative to antibodies for protein biomarker enrichment
Tans, R., van Rijswijck, D. M. H., Davidson, A., Hannam, R., Ricketts, B., Tack, C. J., Wessels, H. J. C. T., Gloerich, J., & van Gool, A. J. (2020) · Protein Expression and Purification · doi:10.1016/j.pep.2020.105677 · Cited by 22
Between two stools: preclinical research, reproducibility, and statistical design of experiments
Reynolds, P. S. (2022) · BMC Research Notes · doi:10.1186/s13104-022-05965-w · Cited by 23
“Advances in therapeutic agents or diagnostics development are more than offset by multimillion-dollar losses in investment, and ultimately unsustainable research and development costs [6, 11, 12].”
BackgroundProxyeconomics, a theory and model of proxy-based competition and cultural evolution
Braganza, O. (2022) · Royal Society Open Science · doi:10.1098/rsos.211030 · Cited by 13
How Will Open Science Impact on University-Industry Collaboration?
Chataway, J., Parks, S., & Smith, E. (2017) · Foresight-Russia · doi:10.17323/2500-2597.2017.2.44.53 · Cited by 31
Kudithipudi, S., & Jeltsch, A. (2016) · Cell chemical biology · doi:10.1016/j.chembiol.2016.07.013 · Cited by 26
Pratap, A., Homiar, A., Waninger, L., Herd, C., Suver, C., Volponi, J. J., Anguera, J. A., & Areán, P. A. (2022) · Scientific Data · doi:10.1038/s41597-022-01633-7 · Cited by 24
“Recent meta-studies have shown that up to 50% of reviewed studies were not reproducible, costing taxpayers billions of dollars every year 26,27 .”
BackgroundTagne, A. M., Cassina, N., Furgiuele, A., Storelli, E., Cosentino, M., & Marino, F. (2020) · Journal of Academic Ethics · doi:10.1007/s10805-020-09359-0 · Cited by 20
“In any case, both RM and QRP pose a threat to biomedical research as they waste resources (Gammon and Franzini 2013; Freedman et al. 2015), undermine public trust in science (Arrowsmith 2011; Prinz et al. 2011; Resnik and Shamoo 2017), tarnish the reputation of researchers and the credibility of…”
BackgroundWeggemans, M., ter Haar, N. M., Prakken, B. J., & ten Cate, O. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004130 · Cited by 19
Reproducibility: Team up with industry
Edwards, A. M. (2016) · Nature · doi:10.1038/531299a · Cited by 19
Zebrafish Xenograft: An Evolutionary Experiment in Tumour Biology
Wyatt, R. A., Trieu, N., & Crawford, B. D. (2017) · Genes · doi:10.3390/genes8090220 · Cited by 22
“Beside the problems already associated with working with cell cultures [82,83], drug development using cancer cell lines frequently yields compounds that fail phase III clinical trials [84,85].”
BackgroundWashetine, K., Heeke, S., Bonnetaud, C., Kara-Borni, M., Ilié, M., Lassalle, S., Butori, C., Long-Mira, É., Marquette, C., Cohen, C., et al. (2018) · Cancers · doi:10.3390/cancers10070220 · Cited by 20
“The variability of the quality of the samples obtained from center to center can lead to inconsistent results that are often not reproducible, and for which the economic impact is substantial [14].”
BackgroundGrüter, B. E., Croci, D. M., Schöpf, S., Nevzati, E., d’Allonzo, D., Lattmann, J., Roth, T. A., Bircher, B., Muroi, C., Dutilh, G., et al. (2020) · Translational Stroke Research · doi:10.1007/s12975-020-00801-4 · Cited by 19
Weld, G., West, P., Glenski, M., Arbour, D. T., Rossi, R. A., & Althoff, T. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19362 · Cited by 31
Eicosanoid Content in Fetal Calf Serum Accounts for Reproducibility Challenges in Cell Culture
Niederstaetter, L., Neuditschko, B., Brunmair, J., Janker, L., Bileck, A., Favero, G. D., & Gerner, C. (2021) · Biomolecules · doi:10.3390/biom11010113 · Cited by 20
“Important decisions such as the choice of drug candidates to be evaluated in clinical studies may be based on such experiments [5].”
BackgroundSamota, E. K., & Davey, R. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.678554 · Cited by 23
“While irreproducibility is not confined to biology and medical sciences [5], irreproducible biomedical experiments pose a strong financial burden on society; an estimated $28 billion was spent on irreproducible biomedical science in 2015 in the USA alone [6].”
BackgroundTutorial: Correction of shifts in single-stage LC-MS(/MS) data
Mitra, V., Smilde, A. K., Bischoff, R., & Horvatovich, P. L. (2017) · Analytica Chimica Acta · doi:10.1016/j.aca.2017.09.039 · Cited by 21
Benign and malignant scientific irreproducibility
Fiala, C., & Diamandis, E. P. (2018) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2018.03.015 · Cited by 19
Experiences in integrated data and research object publishing using GigaDB
Edmunds, S., Li, P., Hunter, C., Zhe, X. S., Davidson, R., Nogoy, N., & Goodman, L. (2016) · International Journal on Digital Libraries · doi:10.1007/s00799-016-0174-6 · Cited by 20
“wasted just on irreproducible preclinical research [24], and will be much cheaper and more effective if reproducible”
BackgroundLima, T. S. M., de Souza, W., Geaquinto, L. R. O., Sanches, P. L., Stępień, E. Ł., Meneses, J., Gortari, E. F., Meisner-Kober, N. C., Himly, M., Granjeiro, J. M., et al. (2022) · Nanomaterials · doi:10.3390/nano12071231 · Cited by 22
Improving laboratory animal genetic reporting: LAG-R guidelines
Teboul, L., Amos-Landgraf, J. M., Benavides, F. J., Birling, M., Brown, S. D. M., Bryda, E. C., Bunton-Stasyshyn, R. K. A., Chin, H.-J. G., Crispo, M., Delerue, F., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-49439-y · Cited by 23
“This wastessigni fi cant researchresources 15 , including those used to reconstitute missing information on both genetic background and genetic alterations 2 , as well as those used to re-establish genotyping assays 16 .”
MethodExperts fail to reliably detect AI-generated histological data
Hartung, J., Reuter, S., Kulow, V. A., Fähling, M., Spreckelsen, C., & Mrowka, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-73913-8 · Cited by 21
“Taken together, we find that participants classified images faster during correct as opposed to incorrect trials, and that this effect cannot be explained by prior experience with histological images.”
The Genotype–Phenotype Relationships in the Light of Natural Selection
Chen, H., Wu, C., & He, X. (2017) · Molecular Biology and Evolution · doi:10.1093/molbev/msx288 · Cited by 19
“Thus, contextdependent findings, a major confounding factor in current molecular cell biology (Freedman et al. 2015), appear to be an FIG.”
BackgroundInfluentialAbove a swamp: A theory of high-quality scientific production
Kiri, B., Lacetera, N., & Zirulia, L. (2018) · Research Policy · doi:10.1016/j.respol.2018.01.011 · Cited by 20
“Associated economic costs may be substantial: according to Freedman et al. (2015) US 28 billon dollars per year are spent in the US on preclinical research that is not reproducible.”
Refinement on the Way Towards Replacement: Are We Doing What We Can?
Herrmann, K. (2019) · BRILL eBooks · doi:10.1163/9789004391192_002 · Cited by 39
“Garner (2008). Home Improvement: C57BL/6J Mice Given More Naturalistic Nesting Materials Build Better Nests.”
BackgroundAutomation in biobanking from a laboratory medicine perspective
Baber, R., & Kiehntopf, M. (2019) · Journal of Laboratory Medicine · doi:10.1515/labmed-2019-0151 · Cited by 18
Challenges and considerations for reproducibility of STARR-seq assays
Das, M., Hossain, A., Banerjee, D., Praul, C. A., & Girirajan, S. (2023) · Genome Research · doi:10.1101/gr.277204.122 · Cited by 19
“We also delineate several features specific to STARR-seq in the context of general reproducibility of biological experiments (Freedman et al. 2015).”
BackgroundBorn to be wild: utilizing natural microbiota for reliable biomedical research
Bruno, P., Schüler, T., & Rosshart, S. P. (2024) · Trends in Immunology · doi:10.1016/j.it.2024.11.013 · Cited by 20
Reer, A., Wiebe, A., Wang, X., & Rieger, J. W. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1086802 · Cited by 20
“Similarly, a meta-analysis of past studies on the cost of nonreproducible research has revealed that in the US over 50% of the preclinical research cannot be reproduced and therefore complicates cumulative knowledge acquisition (Freedman et al., 2015).”
BackgroundFossen, F. M., & Neyse, L. (2023) · Entrepreneurship Theory and Practice · doi:10.1177/10422587231211005 · Cited by 19
“For example, the costs associated with irre-producible preclinical research are estimated at US$28 billion a year in the United States (Freedman et al., 2015).”
BackgroundRisk of bias in exercise science: A systematic review of 340 studies
Preobrazenski, N., McCaig, A., Turner, A., Kushner, M., Pacitti, L. J., Mendolia, P., MacDonald, B., Storoschuk, K. L., Bouck, T., Zaza, Y., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109010 · Cited by 25
“These poor reporting practices persist despite multiple publications detailing the economic 28 and scientific ramifications 11,29–33 of failing to reduce bias and improve reproducibility.”
BackgroundCode sharing in ecology and evolution increases citation rates but remains uncommon
Maitner, B., Andrade, P. E. S., Lei, L., Kass, J. M., Owens, H. L., Barbosa, G. C. G., Boyle, B. L., Castorena, M., Enquist, B. J., Feng, X., et al. (2024) · Ecology and Evolution · doi:10.1002/ece3.70030 · Cited by 22
“This low rate of code sharing undoubtedly hinders scientific progress and likely has far-reaching financial consequences, since a lack of reproducibility means that coding must be continuously redone for common analytical tasks (Freedman et al., 2015).”
BackgroundA compiler for cyber-physical digital microfluidic biochips
Curtis, C., Grissom, D. T., & Brisk, P. (2018) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1145/3168826 · Cited by 17
“Many of the languages that have been developed for programmable biochemistry address the $28 billion reproducibility crisis in the life sciences [100].”
BackgroundPrecision medicine: Driving the evolution of biobanking quality
Barnes, R. O., & Watson, P. H. (2020) · Healthcare Management Forum · doi:10.1177/0840470419898874 · Cited by 17
“result in any progress toward better detection or treatment of disease.(22) In their analysis, they found that the single largest contributor to irreproducibility was “biological reagents and reference materials.”
BackgroundBuilding confidence in digital health through metrology
Smith, N. A. S., Sinden, D., Thomas, S. A., Romanchikova, M., Talbott, J. E., & Adeogun, M. (2020) · British Journal of Radiology · doi:10.1259/bjr.20190574 · Cited by 17
Yarborough, M. A. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-106014 · Cited by 19
Berrow, N. S., de Marco, A., Lebendiker, M., García-Alai, M., Knauer, S. H., Mendez, B. L., Matagne, A., Parret, A. H. A., Remans, K., Uebel, S. F. W., et al. (2021) · European Biophysics Journal · doi:10.1007/s00249-021-01528-2 · Cited by 19
“However, there are many examples of the inability to accurately reproduce published data (Begley and Ioannidis 2015; Marco et al. 2021; Freedman et al. 2015; Bradbury and Plückthun 2015; Baker 2016).”
Biofilms vs. cities and humans vs. aliens – a tale of reproducibility in biofilms
Azevedo, N. F., Allkja, J., & Goeres, D. M. (2021) · Trends in Microbiology · doi:10.1016/j.tim.2021.05.003 · Cited by 19
“For example, it was estimated that 50% of the preclinical research in the USA, at a cost of approximately US$28 billion/year, is not reproducible [5].”
BackgroundZaragozí, B., Trilles, S., & Carrión, J. T. N. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9030138 · Cited by 21
“Some more striking figures estimate that 85% biomedical research efforts are wasted [7], while [9] have calculated that—only in the USA—28,000 million dollars are spent every year on research that is not reproducible.”
BackgroundScientific Misconduct: The Elephant in the Lab. A Response to Parker et al.
Clark, T. D., Binning, S. A., Raby, G. D., Speers‐Roesch, B., Sundin, J., Jutfelt, F., & Roche, D. G. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.09.006 · Cited by 17
“Globally, the annual science budget wasted on irreproducible research is estimated in the tens of billions of dollars [7].”
BackgroundBiddle, M., Stylianou, P., Rekas, M., Wright, A. K. A., Sousa, J. V. R., Ruddy, D. A., Stefana, M. I., Kmiecik, K., Bandrowski, A. E., Kahn, R., et al. (2024) · mAbs · doi:10.1080/19420862.2024.2323706 · Cited by 18
Aspiroz, L. S., Mennecozzi, M., Batlle‐Morera, L., Corneo, B., Healy, L. E., Kotter, M., Kurtz, A., Ludwig, T. E., Mummery, C. L., Pera, M. F., et al. (2025) · Stem Cell Reports · doi:10.1016/j.stemcr.2025.102531 · Cited by 20
Dose prescription and reporting in stereotactic body radiotherapy: A multi-institutional study
Das, I. J., Yadav, P., Andersen, A. D., Chen, Z. J., Huang, L., Langer, M., Lee, C., Li, L., Popple, R. A., Rice, R. K., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.109571 · Cited by 16
Estrogen receptor beta expression and role in cancers
Monteiro, F. L., Stepanauskaitė, L., Archer, A., & Williams, C. (2024) · The Journal of Steroid Biochemistry and Molecular Biology · doi:10.1016/j.jsbmb.2024.106526 · Cited by 16
The need for a formalised system of Quality Control for environmental policy-science
Larcombe, P., & Ridd, P. (2017) · Marine Pollution Bulletin · doi:10.1016/j.marpolbul.2017.11.038 · Cited by 16
“Freedman et al. (2015) estimated that the cumulative prevalence of irreproducible preclinical research exceeds 50%, which in the United States alone, results in approximately US$28 billion per annum spent on research that is not reproducible.”
BackgroundA Blockchain-Based Approach to Provenance and Reproducibility in Research Workflows
Wittek, K., Wittek, N., Lawton, J., Dohndorf, I., Weinert, A., & Ionita, A. (2021) · IEEE International Conference on Blockchain and Cryptocurrency (ICBC) · doi:10.1109/icbc51069.2021.9461139 · Cited by 25
“billion USD loss per year and a big proportion of the errors are induced by study design, data analysis and reporting [12].”
BackgroundA grafted ovarian fragment rescues host fertility after chemotherapy
Batchvarov, I. S., Taylor, R. W., Bustamante-Marin, X. M., Czerwinski, M. J., Johnson, E. S., Kornbluth, S. A., & Capel, B. (2016) · Molecular Human Reproduction · doi:10.1093/molehr/gaw064 · Cited by 16
Recent progress toward the identification of anti-viral immune mechanisms in decapod crustaceans
Hauton, C. (2017) · Journal of Invertebrate Pathology · doi:10.1016/j.jip.2017.01.002 · Cited by 16
“…to present sufficient meta-data 281 for any experiment (for example, the MISA guidelines argued in Hauton et al., 2015) so that any 282 differences between replicated experiments conducted by different research teams can be 283 identified and rationalised (Freedman et al., 2015; Baker, 2016).”
McKay, B., Castellanos, M. E., Ebell, M. H., Whalen, C. C., & Handel, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0217219 · Cited by 17
Meta-Research: Broadening the Scope of PLOS Biology
Kousta, S. T., Ferguson, C., & Ganley, E. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002334 · Cited by 18
“Freedman and colleagues estimated that over 50% of preclinical research can’t be replicated, placing the approximate annual cost of irreproducibility in the US alone at US$28 billion [4].”
BackgroundBetter reporting for better research: a checklist for reproducibility
Kenall, A., Edmunds, S., Goodman, L., Bal, L., Flintoft, L., Shanahan, D. R., & Shipley, T. (2015) · GigaScience · doi:10.1186/s13742-015-0071-8 · Cited by 15
Improving Research Standards to Restore Trust in Intranasal Oxytocin
Carson, D. S., Yuan, H., & Labuschagne, I. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.08.031 · Cited by 15
Mulhall, P., Taggart, L., McAloon, T., & COATES, V. E. (2020) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12838 · Cited by 15
Quality of interventional animal experiments in Chinese journals: compliance with ARRIVE guidelines
Zhao, B. Q., Jiang, Y., Zhang, T., Shang, Z., Zhang, W., Hu, K., Chen, F., Mei, F., Gao, Q., Zhao, L., et al. (2020) · BMC Veterinary Research · doi:10.1186/s12917-020-02664-1 · Cited by 24
Meier, M., Lonsdorf, T. B., LUPIEN, S. J., Stalder, T., Laufer, S., Sicorello, M., Linz, R., & Puhlmann, L. M. C. (2022) · Comprehensive Psychoneuroendocrinology · doi:10.1016/j.cpnec.2022.100144 · Cited by 20
Miao, X., Hu, J., Chai, C., Tang, H., Zhao, Z., Luo, W., Zhou, W., & Xu, H. (2022) · Cancer Cell International · doi:10.1186/s12935-022-02840-3 · Cited by 15
“Several studies are not reproducible because of the misidentification, contamination, genetic drift, and clonal evolution of cell lines, thereby hindering cancer research [13].”
BackgroundThe relationship between gender and pharmacology
Younes, S. (2024) · Current Research in Pharmacology and Drug Discovery · doi:10.1016/j.crphar.2024.100192 · Cited by 20
“Factors such as rat age and strain could impact the results, and variables like the time of day the data were collected and methodological differences between laboratories could also influence the data (Voelkl et al., 2018, 2020; Freedman et al., 2015).”
BackgroundNtai, A., Baronchelli, S., Spada, A., Moles, A., Guffanti, A., Blasio, P. D., & Biunno, I. (2017) · Biopreservation and Biobanking · doi:10.1089/bio.2016.0097 · Cited by 15
Linsen, L., Landuyt, K. V., & Ectors, N. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2019.00309 · Cited by 15
“The integrity and quality of the biospecimens used for research, has been identified as one of the key factors that contribute to improved reproducibility (3).”
BackgroundCellular Metrology: Scoping for a Value Proposition in Extra- and Intracellular Measurements
Faruqui, N., Kummrow, A., Fu, B., Divieto, C., Rojas-Cornejo, F., Kisulu, F., Cavalcante, J. J. V., Wang, J., Campbell, J. J., Martins, J. L., et al. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00456 · Cited by 16
“The choice of an appropriate cell line is a fundamental variability in cell analysis (Freedman, 2015; Lin-Gibson et al., 2016a).”
BackgroundImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2018) · Cancer Reports · doi:10.1002/cnr2.1150 · Cited by 21
“Attempts to reproduce published results costs the United States approximately $28B annually (Freedman et al., 2015; Freedman, Venugopalan, & Wisman, 2017), yet poor descriptions of the published studies lead to a majority of studies becoming non-replicable (Glasziou et al., 2014).”
Evaluation of indicators supporting reproducibility and transparency within cardiology literature
Anderson, J., Wright, B. D., Rauh, S., Tritz, D. J., Horn, J., Parker, I., Bergeron, D., Cook, S., & Vassar, M. (2020) · Heart · doi:10.1136/heartjnl-2020-316519 · Cited by 18
Cancer blues? A promising judgment bias task indicates pessimism in nude mice with tumors
Resasco, A., MacLellan, A., Ayala, M. Á., Kitchenham, L., Edwards, A. M., Lâm, S., Dejardin, S., & Mason, G. J. (2021) · Physiology & Behavior · doi:10.1016/j.physbeh.2021.113465 · Cited by 17
da Cruz, I. B. M., Bonotto, N. C. D. A., Turra, B. O., Teixeira, C. F., Azzolin, V. F., Ribeiro, E. A. M., Piccoli, J. D. C. E., & Barbisan, F. (2023) · Toxicology in Vitro · doi:10.1016/j.tiv.2023.105637 · Cited by 16
Gurdita, A., Vovko, H., & Ungrin, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147140 · Cited by 16
Understanding of researcher behavior is required to improve data reliability
Wass, M. N., Ray, L., & Michaelis, M. (2019) · GigaScience · doi:10.1093/gigascience/giz017 · Cited by 22
Sivakova, O. V., Pokrovskaya, M. S., Metelskaya, V. A., Limonova, A. S., Meshkov, A. N., & Drapkina, O. M. (2019) · Russian Journal of Preventive Medicine · doi:10.17116/profmed20192206295 · Cited by 15
Securing the Exchange of Synthetic Genetic Constructs Using Digital Signatures
Gallegos, J. E., Kar, D. M., Ray, I., Ray, I., & Peccoud, J. (2020) · ACS Synthetic Biology · doi:10.1021/acssynbio.0c00401 · Cited by 15
How animal ethics committees make decisions – a scoping review of empirical studies
Milford, A., Clercq, E. D., Louis-Maerten, E., Geneviève, L. D., & Elger, B. S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318570 · Cited by 15
“This holds great significance given the emerging evidence of poor reproducibility or translatability of completed animal research in recent years due to poor experimental design [57,58].”
Mascaro, L., & Facer‐Childs, E. R. (2020) · Physiology News · doi:10.36866/pn.118.36 · Cited by 14
Light and water are not simple conditions: fine tuning of animal housing in male C57BL/6 mice
Langgartner, D., Foertsch, S., Füchsl, A. M., & Reber, S. O. (2016) · Stress · doi:10.1080/10253890.2016.1254186 · Cited by 14
“Absence of experimental reproducibility between different laboratories poses a prominent biomedical research problem (Freedman et al., 2015).”
BackgroundThe path to VICTORy – a beginner's guide to success using commercial research antibodies
Goodman, S. L. (2018) · Journal of Cell Science · doi:10.1242/jcs.216416 · Cited by 14
“Estimates imply that worldwide $365 million–$2 billion a year are spent buying cr-Abs (Bird, 2012; Bradbury and Plückthun, 2015b; Freedman et al., 2015) – and ‘a billion here, a billion there, and pretty soon you’re talking about real money’ (Dirksen, 1961).”
BackgroundBiotechnology and Industry 4.0: The professionals of the future
Massabni, A. C., & da Silva, G. J. (2019) · International Journal of Advances in Medical Biotechnology - IJAMB · doi:10.25061/2595-3931/ijamb/2019.v2i2.39 · Cited by 14
Translating Laboratory Tests into Clinical Practice: A Conceptual Framework
Nagler, M. (2020) · Hämostaseologie · doi:10.1055/a-1227-8008 · Cited by 15
The need for standardisation in life science research - an approach to excellence and trust.
Hollmann, S., Kremer, A., Baebler, Š., Trefois, C., Gruden, K., Rudnicki, W. R., Tong, W., Gruca, A., Bongcam‐Rudloff, E., Evelo, C. T., et al. (2020) · F1000Research · doi:10.12688/f1000research.27500.2 · Cited by 14
Quality assurance and best research practices for non-regulated veterinary clinical studies
Davies, R., London, C. A., Lascelles, B. D. X., & Conzemius, M. G. (2017) · BMC Veterinary Research · doi:10.1186/s12917-017-1153-x · Cited by 17
Microscopy, Immuno-Histochemistry, Digital Imaging, and Quantitative Microscopy
Goodwin, P. C., Johnson, B., & Frevert, C. W. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-802900-8.00004-x · Cited by 14
Jaljuli, I., Kafkafi, N., Giladi, E., Golani, I., Gozes, I., Chesler, E. J., Bogue, M. A., & Benjamini, Y. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002082 · Cited by 15
A Review of the Effects of Some Extrinsic Factors on Mice Used in Research
Gozalo, A. S., & Elkins, W. R. (2023) · Comparative Medicine · doi:10.30802/aalas-cm-23-000028 · Cited by 19
Testing of putative antiseizure medications in a preclinical Dravet syndrome zebrafish model
Whyte‐Fagundes, P., Vance, A., Carroll, A., Figueroa, F., Manukyan, C., & Baraban, S. C. (2024) · Brain Communications · doi:10.1093/braincomms/fcae135 · Cited by 13
Zhou, J., Li, Y., Lou, J., Wang, Y., Kan, Z., Neugschwandtner, R. W., Li, F., Liu, J., Dong, K., Xue, Y., et al. (2024) · Plants · doi:10.3390/plants13131738 · Cited by 13
“On the other hand, many allelochemicals can be released from straw decomposition, such as phenolic acids and short-and long-chain fatty acids, which could decrease specific soil-borne fungal pathogens under a full amount of straw return [47].”
BackgroundReproducibility analysis of bioimpedance-based self-developed live cell assays
Vizvári, Z., Győrfi, N., Maczko, G., Varga, R., Jakabfi-Csepregi, R., Sari, Z., Füredi, A., Bajtai, E., Vajda, F., Tadić, V., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-67061-2 · Cited by 15
Metadata
- Authors
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S.
- Affiliations
- Global Biological Standards Institute (US)
- Boston University (US)
- Corresponding author
- Freedman, Leonard P.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-06-09
- Volume
- 13
- Issue
- 6
- Pages
- e1002165
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4461318
- OpenAlex
- W585341442
- Cited by
- 1,055
- References
- 35
- Citations by year
- 2026: 48 · 2025: 45 · 2024: 69 · 2023: 92 · 2022: 93 · 2021: 96 · 2020: 142 · 2019: 116 · 2018: 109 · 2017: 93 · 2016: 103 · 2015: 49
- FWCI
- 188.58
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- OpenBU (Boston University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Keywords
- reproducibility, preclinical research, research costs, research waste, reproducibility framework
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.