The Economics of Reproducibility in Preclinical Research
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible-in the United States alone. We outline a framework for solutions and a plan for long-term improvements in reproducibility rates that will help to accelerate the discovery of life-saving therapies and cures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Studies reporting the prevalence of irreproducibility. Source: Begley and Ellis [6], Prinz et al. [7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al. [9].](https://figures.cochat.ai/paper-figures/a30bd9bd65dc4bedb5db19b413eb814f/the-economics-of-reproducibility-in-preclinical-research-figure-1.jpg)
![Fig 2 Estimated US preclinical research spend and categories of errors that contribute to irreproducibility. Note that the percentage value of error for each category is the midpoint of the high and low prevalence estimates for that category divided (weighted) by the sum of all midpoint error rates (see S1 Dataset). Source: Chakma et al. [18] and the American Association for the Advancement of Science (AAAS) [19].](https://figures.cochat.ai/paper-figures/207a0a58df5c4620a819d7aa1f4aa960/the-economics-of-reproducibility-in-preclinical-research-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Besides, reliably determining actual translational success rates is unmanageable as long as the current status of reporting of preclinical research leaves room for improvement [89], and non-reproducibility is such a critical issue in both animal [1] and human [90] studies.”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The S&B tool therefore directly addresses the larger problem of material reagents and standards representing the major source of incorrect published results from pre-clinical research [3, 4].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current drug development is handicapped by high attrition rates; many molecules that were promising during preclinical development fail during subsequent clinical testing [1].”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [16]
Experimental — accuracy not yet measured
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Ostrom, E. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 18,859
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“[7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · cited 1×
“Much has been written about the alarming number of preclinical studies that were later found to be irreproducible [1,2].”
BackgroundBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Partial Identification of Probability Distributions
(2003) · Springer Series in Statistics · doi:10.1007/b97478 · Cited by 698
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Research works printed but not linked: 1
- CF Manski, 2003, Springer Series in Statistics, 178
Cited by
Items 201–300 of 1,003
Wiik, E., Jones, J. P. G., Pynegar, E., Bottazzi, P., Asquith, N., Gibbons, J., & Kontoleon, A. V. (2020) · Conservation Biology · doi:10.1111/cobi.13508 · Cited by 37
Embedding data provenance into the Learning Health System to facilitate reproducible research
Ćurčin, V. (2016) · Learning Health Systems · doi:10.1002/lrh2.10019 · Cited by 44
Luxenhofer, R. (2015) · Nanomedicine · doi:10.2217/nnm.15.139 · Cited by 42
Fostering reproducibility in industry-academia research
Jasny, B. R., Wigginton, N. S., Mcnutt, M. K., Bubela, T. M., Buck, S., Cook-Deegan, R. M., Gardner, T. S., Hanson, B., Hustad, C. M., Kiermer, V., et al. (2017) · Science · doi:10.1126/science.aan4906 · Cited by 40
Gupta, A., Gautam, P., Wennerberg, K., & Aittokallio, T. A. (2020) · Communications Biology · doi:10.1038/s42003-020-0765-z · Cited by 42
Greever, C., Ramirez‐Aguilar, K. A., & Connelly, J. (2020) · FEBS Letters · doi:10.1002/1873-3468.13932 · Cited by 35
Research – A blockchain of knowledge?
Ducrée, J. (2020) · Blockchain Research and Applications · doi:10.1016/j.bcra.2020.100005 · Cited by 35
Bäckryd, E., Themistocleous, A. C., Larsson, A. O., Gordh, T., Rice, A. S. C., Tesfaye, S. H., Bennett, D., & Gerdle, B. U. C. (2021) · Pain · doi:10.1097/j.pain.0000000000002451 · Cited by 36
“In the context of a purported reproducibility crisis, 7,8,26,34,35,45,54 this is arguably no small advantage.”
Are Leadership and Management Essential for Good Research? An Interview Study of Genetic Researchers
Antes, A. L., Mart, A., & DuBois, J. M. (2016) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264616668775 · Cited by 45
“This is notable given the sizeable scientific workforce, the immense investments made in research, and the social consequences of the work (Freedman, Cockburn, & Simcoe, 2015; Hurley, 2003; Ladd, Lappé, McCormick, Boyce, & Cho, 2009; McCormick, Boyce, Ladd, & Cho, 2012).”
Freedman, L. P., Gibson, M. C., Bradbury, A. R. M., Buchberg, A. M., Davis, D., Dolled-Filhart, M. P., Lund-Johansen, F., & Rimm, D. L. (2016) · BioTechniques · doi:10.2144/000114431 · Cited by 34
Self-citationReproducibility crisis in science or unrealistic expectations?
França, T. F. A., & Monserrat, J. M. (2018) · EMBO Reports · doi:10.15252/embr.201846008 · Cited by 39
Best (but oft forgotten) practices: sample size planning for powerful studies
Anderson, S. F. (2019) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqz058 · Cited by 38
Lenzen, S. (2017) · Diabetes/Metabolism Research and Reviews · doi:10.1002/dmrr.2915 · Cited by 37
“Failures to successfully translate not only delay the introduction of new effective therapies for patients with serious diseases, which might be curable with such a new therapy, but they also result in huge financial losses for the pharmaceutical companies when long and expensive clinical trials do not successfully me…”
BackgroundMuradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
An, G. C. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0497-0 · Cited by 40
“…terms) has been used to describe the phenomenon that a series of foundational studies across a range of disciplines have failed in what can be considered a fundamental aspect of science: that they can be reproduced (Ioannidis 2005; Baker 2016; Prinz et al. 2011; Kaiser 2017; Freedman et al. 2015).”
BackgroundAutomated Home-Cage Testing as a Tool to Improve Reproducibility of Behavioral Research?
Richter, S. H. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00383 · Cited by 38
“Thereby, particular attention has been given to animal research, where irreproducibility prevalence rates have been estimated to range between 50 and 90% (Prinz et al., 2011; Collins and Tabak, 2014; Freedman et al., 2015).”
BackgroundFinding the Path to Biobank Sustainability Through Sound Business Planning
Henderson, M. K., Simeon‐Dubach, D., & Albert, M. (2015) · Biopreservation and Biobanking · doi:10.1089/bio.2015.29039.mh · Cited by 33
“estimates the annual cost of this type of irreproducibility in the United States alone as high as $28 billion.(4) There are some key messages throughout the articles in this issue that are worth highlighting.”
An incentive-based approach for improving data reproducibility
Rosenblatt, M. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5003 · Cited by 34
Jaric, I., Voelkl, B., Clerc, M., Schmid, M. W., Novak, J., Rosso, M., Rufener, R., von Kortzfleisch, V. T., Richter, S. H., Buettner, M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001837 · Cited by 32
“Thus, the prevalence of irreproducible findings in preclinical research was estimated to be greater than 50% of the published findings [4].”
Antibody characterization is critical to enhance reproducibility in biomedical research
Kahn, R. A., Virk, H., Laflamme, C., Houston, D. W., Polinski, N. K., Meijers, R., Levey, A. I., Saper, C. B., Errington, T. M., Turn, R. E., et al. (2024) · eLife · doi:10.7554/elife.100211 · Cited by 32
Self-citationBlockchain for Science and Knowledge Creation
Bartling, S. H. (2018) · Springer eBooks · doi:10.1007/978-3-662-57611-3_10 · Cited by 35
Integrated Chemometrics and Statistics to Drive Successful Proteomics Biomarker Discovery
Suppers, A., van Gool, A. J., & Wessels, H. J. C. T. (2018) · Proteomes · doi:10.3390/proteomes6020020 · Cited by 39
Issues in Reproducible Simulation Research
Fitzpatrick, B. G. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0496-1 · Cited by 36
Wang, X., Zhang, S., Liu, Y., Du, J., & Huang, H. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120698 · Cited by 67
INFRAFRONTIER: a European resource for studying the functional basis of human disease
Raess, M., de Castro, A. A., Gailus‐Durner, V., Fessele, S., & de Angelis, M. H. (2016) · Mammalian Genome · doi:10.1007/s00335-016-9642-y · Cited by 33
“By setting standard operation procedures and strict quality control measures, they help reduce the irreproducibility of scientific results in preclinical research, the costs of which have been recently estimated to be 28 billion $ per year in the US alone (Freedman et al. 2015).”
BackgroundLaboratory mice engrafted with natural gut microbiota possess a wildling-like phenotype
Runge, S., von Zedtwitz, S., Maucher, A., Bruno, P., Osbelt, L., Zhao, B., Gernand, A. M., Lesker, T. R., Gräwe, K., Rogg, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60554-2 · Cited by 35
Lalu, M. M., Montroy, J. R., Begley, C. G., Bubela, T. M., Hunniford, V. T., Ripsman, D., Wesch, N. L., Kimmelman, J., Macleod, M. R., Moher, D., et al. (2020) · F1000Research · doi:10.12688/f1000research.23663.2 · Cited by 35
The Culture of Cell Culture Practices and Authentication—Results from a 2015 Survey
Freedman, L. P., Gibson, M. C., Wisman, R., Ethier, S. P., Soule, H. R., Reid, Y. A., & Neve, R. M. (2015) · BioTechniques · doi:10.2144/000114344 · Cited by 34
Self-citationHaslacher, H., Szekeres, T., Gerner, M. C., Ponweiser, E., Repl, M., Wagner, O. F., & Perkmann, T. (2016) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2016-0608 · Cited by 33
Petiti, J., Caria, S., Revel, L., Pegoraro, M., & Divieto, C. (2024) · Cells · doi:10.3390/cells13231959 · Cited by 30
Open data in drug discovery and development: lessons from malaria
Wells, T. N. C., Willis, P. A., Burrows, J. N., & van Huijsduijnen, R. H. (2016) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2016.154 · Cited by 30
“Without quality standards, flawed data are introduced into analyses, which has a profound cost to society 1 .”
BackgroundQuality Assurance in Biobanking for Pre-Clinical Research
Simeon‐Dubach, D., Zeisberger, S. M., & Hoerstrup, S. P. (2016) · Transfusion Medicine and Hemotherapy · doi:10.1159/000448254 · Cited by 33
Halim, A. (2018) · Current Drug Discovery Technologies · doi:10.2174/1570163815666180925095433 · Cited by 32
Applying the science of measurement to biology: Why bother?
Coxon, C. H., Longstaff, C., & Burns, C. J. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000338 · Cited by 33
Witjes, V. M., Boleij, A., & Halffman, W. (2020) · Animals · doi:10.3390/ani10122415 · Cited by 39
“Independent replication studies that fail to confirm the original result contribute significantly to the rising concern about irreproducibility amongst the animal research community [7,18].”
BackgroundResultA biologist’s guide to planning and performing quantitative bioimaging experiments
Senft, R. A., Diaz-Rohrer, B. B., Colarusso, P., Swift, L. H., Jamali, N., Jambor, H. K., Pengo, T., Brideau, C., Llopis, P. D. L. M. M., Uhlmann, V., et al. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002167 · Cited by 38
“The reproducibility crisis [56,57] has spawned community-led initiatives to tackle the problems that come along with big data.”
BackgroundQiao, C., Meng, F., Jang, I.-H., Jo, H., Chen, Y. E., & Zhang, J. (2016) · Physiological Genomics · doi:10.1152/physiolgenomics.00025.2016 · Cited by 36
“Despite this, a tremendous amount of EC research has been conducted under this condition, which may not be translatable to subsequent in vivo studies because of the missing biomechanical milieu (14).”
BackgroundIrreproducible text-book “knowledge”: The effects of color bands on zebra finch fitness
Wang, D., Forstmeier, W., Ihle, M., Khadraoui, M., Jerónimo, S., Martin, K., & Kempenaers, B. (2018) · Evolution · doi:10.1111/evo.13459 · Cited by 35
Improving the conduct, reporting, and appraisal of animal research
Ritskes-Hoitinga, M. J., & Wever, K. E. (2018) · BMJ · doi:10.1136/bmj.j4935 · Cited by 32
Technical pitfalls when collecting, cryopreserving, thawing, and stimulating human T-cells
Browne, D. J., Miller, C., & Doolan, D. L. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1382192 · Cited by 37
“Indeed, a relatively recent large meta-analysis proposed that, at best, around 50% of preclinical biomedical research was reproducible (109).”
BackgroundDucrée, J., Etzrodt, M., Bartling, S. H., Walshe, R., Harrington, T. S., Wittek, N., Posth, S., Wittek, K., Ionita, A., Prinz, W., et al. (2021) · Frontiers in Blockchain · doi:10.3389/fbloc.2021.631648 · Cited by 31
“Freedman et al. (2015) demonstrated that in the United States, the high rate of irreproducibility, estimated at 50% in preclinical research, induced high economical loss of approximately USD28 billion per annum.”
BackgroundCuenca‐Martínez, F., Touche, R. L., Varangot‐Reille, C., Sardinoux, M., Bahier, J., Suso‐Martí, L., & Fernández‐Carnero, J. (2022) · Physical Therapy · doi:10.1093/ptj/pzac040 · Cited by 31
A reaction norm perspective on reproducibility
Voelkl, B., & Würbel, H. (2021) · Theory in Biosciences · doi:10.1007/s12064-021-00340-y · Cited by 37
“1 3 Several potential causes for poor reproducibility have been proposed, including lack of scientific rigour, low statistical power, publication bias, analytical flexibility and perverse incentives in research—leading in some cases to outright fraud (Loken and Gelman 2017; Freedman et al. 2015; Ioannidis 2005).”
BackgroundIrreproducible biology research costs put at $28 billion per year
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.17711 · Cited by 29
Wyndham, J. M., Weisenberg, N., McCarty, C. A., Goldman, G. T., Brown, M., & Borenstein, J. (2022) · doi:10.1126/aaas.ade8203 · Cited by 32
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
Arroyo-Araujo, M., Graf, R., Maco, M., van Dam, E., Schenker, E., Drinkenburg, W. H. I. M., Koopmans, B., de Boer, S. F., Cullum‐Doyle, M., Noldus, L. P. J. J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-47981-0 · Cited by 34
Critical Analysis of Non-Thermal Plasma-Driven Modulation of Immune Cells from Clinical Perspective
Smolková, B., Frtús, A., Uzhytchak, M., Lunová, M., Kubinová, Š., Dejneka, A., & Lunov, O. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21176226 · Cited by 31
“It is estimated, that low reproducibility rates within life science research result in approximately $28 billion USD/year being wasted on irreproducible preclinical research in the United States [155].”
BackgroundOldoni, E., Saunders, G. I., Biétrix, F., García‐Bermejo, L., Niehues, A., Hoen, P. A. C. ’., Nordlund, J., Hajdúch, M., Scherer, A., Kivinen, K., et al. (2022) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2022.974799 · Cited by 33
“The AMRI RIs are already committed to quality in science (Freedman et al., 2015) and lead various actions addressing reproducibility, best practice guidelines, benchmarking, standards and reference materials for generation, sharing and management of multiomics data and metadata.”
BackgroundDumas-Mallet, E., Button, K. S., Boraud, T., Munafo, M. R., & Gonon, F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158064 · Cited by 31
“The reproducibility of preclinical studies has been recently reviewed [5]; five studies estimated the prevalence of irreproducible results in preclinical studies to be between 51 and 89%.”
BackgroundBorries, C., Sandel, A. A., Koenig, A., Fernández‐Duque, E., Kamilar, J. M., Amoroso, C. R., Barton, R. A., Bray, J., Fiore, A. D., Gilby, I. C., et al. (2016) · Evolutionary Anthropology Issues News and Reviews · doi:10.1002/evan.21502 · Cited by 30
Pre-analytics and tumor heterogeneity
Bonin, S., & Stanta, G. (2019) · New Biotechnology · doi:10.1016/j.nbt.2019.09.007 · Cited by 29
“Overall, it has been estimated that around 1/3rd of the irreproducibility can be related to pre-analytical conditions [25].”
BackgroundIn Situ Target Engagement Studies in Adherent Cells
Axelsson, H., Almqvist, H., Otrocka, M., Vallin, M., Lundqvist, S., Hansson, P., Karlsson, U., Lundbäck, T., & Seashore-Ludlow, B. A. (2018) · ACS Chemical Biology · doi:10.1021/acschembio.7b01079 · Cited by 30
Riegman, P. H. J., Becker, K., Zatloukal, K., Pazzagli, M., Schröder, U., & Oelmüller, U. (2019) · New Biotechnology · doi:10.1016/j.nbt.2019.06.007 · Cited by 29
Data Management Practices in Academic Library Learning Analytics: A Critical Review
Briney, K. A. (2019) · Journal of Librarianship and Scholarly Communication · doi:10.7710/2162-3309.2268 · Cited by 36
Mische, S. M., Fisher, N. C., Meyn, S. M., Sol‐Church, K., Hegstad‐Davies, R. L., Weis‐Garcia, F., Adams, M., Ashton, J. M., Delventhal, K. M., Dragon, J. A., et al. (2020) · Journal of Biomolecular Techniques JBT · doi:10.7171/jbt.20-3101-003 · Cited by 32
Dai, H., Young, H. P., Durant, T. J. S., Gong, G., Kang, M., Krumholz, H. M., Schulz, W. L., & Jiang, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03662 · Cited by 34
“While improving the transparency of data acquisition and analysis requires a multifaceted approach [9-16], the use of emerging cryptographic technologies, such as blockchain, may reduce the risk of data manipulation and boost the confidence in conclusions made by the scientific community.”
Backgroundvan der Naald, M., Chamuleau, S. A. J., Menon, J. M. L., de Leeuw, W., de Haan, J., Duncker, D. J., & Wever, K. E. (2022) · BMJ Open Science · doi:10.1136/bmjos-2021-100259 · Cited by 26
A community-led initiative for training in reproducible research
Auer, S., Haelterman, N. A. T., Weissgerber, T. L., Erlich, J. C., Susilaradeya, D., Julkowska, M., Gazda, M. A., Schwessinger, B., Jadavji, N. M., Reproducibility for Everyone Team, et al. (2021) · eLife · doi:10.7554/elife.64719 · Cited by 28
“their importance, studies have shown that it can be quite challenging to reproduce and replicate peer-reviewed results (Baker and Penny, 2016; Freedman et al., 2015).”
BackgroundReproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 43
“A recent study estimated the cost of funding irreproducible research at approximately $28 billion a year in the U.S. alone [27], [28].”
BackgroundMechanotransductive Receptor Piezo1 as a Promising Target in the Treatment of Neurological Diseases
Bryniarska, N., Kubiak, A., & Basta‐Kaim, A. (2022) · Current Neuropharmacology · doi:10.2174/1570159x20666220927103454 · Cited by 26
“On the other hand, Piezo1/2 boom should lead researchers to pru-dence about results presented in new scientific publications – in the context of also physiological meaning, data reproducibility and their scientific merit [40, 41].”
BackgroundYAMP: a containerized workflow enabling reproducibility in metagenomics research
Visconti, A., Martin, T. C., & Falchi, M. (2018) · GigaScience · doi:10.1093/gigascience/giy072 · Cited by 31
“Indeed, as outlined in the Background 250 Section, a lack of reproducibility is nowadays ubiquitous, and, 251 besides undermining the credibility of scienti c research, it has 252 an economical cost, quanti ed, for instance, in US$28B/year 253 for preclinical research [41].”
BackgroundThe other functions of the sodium pump
Askari, A. (2019) · Cell Calcium · doi:10.1016/j.ceca.2019.102105 · Cited by 26
Damerow, J., Varadharajan, C., Boye, K., Brodie, E., Burrus, M., Chadwick, K. D., Crystal‐Ornelas, R., Elbashandy, H., Alves, R. E., Ely, K., et al. (2021) · Data Science Journal · doi:10.5334/dsj-2021-011 · Cited by 26
“…that require communities to change behavior and provide standardized data will require a coordinated effort, which is best addressed by collaborations of key stakeholders who establish community consensus, enforce guidelines, and help solve problems (Farrell and Simcoe, 2012; Freedman et al, 2015).”
BackgroundGetting personal with the “reproducibility crisis”: interviews in the animal research community
Fitzpatrick, B. G., Koustova, E., & Wang, Y. (2018) · Lab Animal · doi:10.1038/s41684-018-0088-6 · Cited by 31
Menke, J., Eckmann, P., Ozyurt, I. B., Roelandse, M., Anderson, N., Grethe, J., Gamst, A., & Bandrowski, A. (2022) · Journal of Medical Internet Research · doi:10.2196/37324 · Cited by 25
BioBlocks: Programming Protocols in Biology Made Easier
Gupta, V., Irimia, J., Pau, I., & Rodríguez‐Patón, A. (2017) · ACS Synthetic Biology · doi:10.1021/acssynbio.6b00304 · Cited by 25
Bonafiglia, J. T., Islam, H., Preobrazenski, N., & Gurd, B. J. (2021) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2021.03.005 · Cited by 30
“There is a growing awareness of a “reproducibility crisis” in preclinical and clinical research(1,2) that has widespread societal and financial ramifications.(1,3) Many research groups(1,4,5) attribute poor reproducibility to shortcomings in key aspects of study design (e.”
BackgroundResearch integrity requires to be aware of good and questionable research practices
Boisgontier, M. P. (2022) · European Rehabilitation Journal · doi:10.52057/erj.v2i1.24 · Cited by 28
“Unfortunately, studies have shown that the prevalence of irreproducible preclinical research ranges from 51% to 89% [6].”
BackgroundOpen Science Practices in Communication Sciences and Disorders: A Survey
Amin, M. E., Borders, J. C., Long, H. L., Keller, M. A., & Kearney, E. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00062 · Cited by 28
Koçak, B., Yardımcı, A., Yuzkan, S., Keleş, A., Altun, O., Bulut, E., Bayrak, O. N., & Okumus, A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.11.030 · Cited by 23
Treuting, P. M., Snyder, J. M., Ikeno, Y., Schofield, P. N., Ward, J. M., & Sundberg, J. P. (2016) · Veterinary Pathology · doi:10.1177/0300985815620629 · Cited by 28
“Systematically employed, consistently applied pathology end points will increase accuracy, reproducibility, and translatability of preclinical interventional aging studies.(12,21)”
MethodSnyder, H. M., Shineman, D. W., Friedman, L. G., Hendrix, J. A., Khachaturian, A. S., Guillou, I. L., Pickett, J., Refolo, L. M., Sancho, R. M., & Ridley, S. H. (2016) · Alzheimer s & Dementia · doi:10.1016/j.jalz.2016.07.001 · Cited by 25
Embracing Transparency Through Data Sharing
Boué, S., Byrne, M., Hayes, A. W., Hoeng, J., & Peitsch, M. C. (2018) · International Journal of Toxicology · doi:10.1177/1091581818803880 · Cited by 26
“The core drivers contributing to irreproducibility are reporting and publication bias, underpowered studies, a lack of open access to methods and data, and a lack of clearly defined standards and guidelines in areas, such as reagent validation.(13) Growing concerns in the scientific community regarding funds spent on…”
BackgroundMethodInfluentialGlial cells as drug targets: What does it take?
Möller, T., & Boddeke, H. W. G. M. (2016) · Glia · doi:10.1002/glia.22993 · Cited by 25
“A recent analysis estimated that the cumulative (total) prevalence of irreproducible preclinical research in the US alone exceeds 50% of all published studies, resulting in approximately US$28,000,000,000 p.a. wasted on faulty research (Freedman et al., 2015).”
BackgroundReproducible Bioconductor workflows using browser-based interactive notebooks and containers
Almugbel, R., Hung, L., Hu, J., Almutairy, A., Ortogero, N., Tamta, Y., & Yeung, K. Y. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx120 · Cited by 24
“While these steps go a long way to documenting computational workflows, it is estimated that more than 25% of computational workflows cannot be reproduced [2].”
BackgroundStryjek, R., Parsons, M. H., Fendt, M., Święcicki, J., & Bębas, P. (2021) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2021.109303 · Cited by 24
Leman, J. K., Lyskov, S., Lewis, S. M., Adolf‐Bryfogle, J., Alford, R. F., Barlow, K. A., Ben-Aharon, Z., Farrell, D. P., Fell, J. S., Hansen, W. A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-27222-7 · Cited by 25
“Use case (2): scientific test framework facilitates bug fixes and maintenance.”
BackgroundInfluentialPosition paper on management of personal data in environment and health research in Europe
Govarts, E., Gilles, L., Bopp, S. K., Holub, P., Matalonga, L., Vermeulen, R. C. H., Vrijheid, M., Beltrán, S., Hartlev, M., Sarah, J., et al. (2022) · Environment International · doi:10.1016/j.envint.2022.107334 · Cited by 25
Ellis, R. J. (2022) · eNeuro · doi:10.1523/eneuro.0017-22.2022 · Cited by 30
“It is estimated that at least $28 billion is wasted on irreproducible preclinical science annually in the United States alone (Freedman et al., 2015).”
BackgroundCobey, K., Fehlmann, C., Christ-Franco, M., Ayala, A., Sikora, L., Rice, D., Xu, C.-C., Ioannidis, J., Lalu, M., Ménard, A., et al. (2023) · eLife · doi:10.7554/elife.78518 · Cited by 12
“This contributes to research waste (Nasser et al., 2017; Ioannidis et al., 2014; Freedman et al., 2015).”
Ashraf, M. R., Melemenidis, S., Liu, K., Grilj, V., Jansen, J., Velasquez, B., Connell, L., Schulz, J. B., Bailat, C. J., Libed, A., et al. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.03.017 · Cited by 24
Hollyer, T. R., Bordoni, L., Kousholt, B. S., van Luijk, J. A. K. R., Ritskes-Hoitinga, M. J., & Østergaard, L. (2018) · Journal of Neurochemistry · doi:10.1111/jnc.14633 · Cited by 25
Reproducible and transparent research practices in published neurology research
Rauh, S., Torgerson, T., Johnson, A. L., Pollard, J., Tritz, D. J., & Vassar, M. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-020-0091-5 · Cited by 18
Lightweight Distributed Provenance Model for Complex Real–world Environments
Wittner, R., Mascia, C., Gallo, M., Frexia, F., Müller, H., Plass, M., Geiger, J., & Holub, P. (2022) · Scientific Data · doi:10.1038/s41597-022-01537-6 · Cited by 29
“The significant impact of flawed research results on health, economics and political decisions has been frequently stated 8–11 .”
BackgroundTesting an active intervention to deter researchers’ use of questionable research practices
Bruton, S. V., Brown, M., Sacco, D. F., & DIDLAKE, R. H. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0085-3 · Cited by 27
Degen, P. M., & Medo, M. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011630 · Cited by 28
“Although the broader replication crisis in science [20–22, 44 , 45 ] includes numerous human • Significant DEGs from one small cohort are unlikely to be significant in another small cohort (low replicability), unless it is known that the population is very homogeneous (e.g. cell cultures).”
Marino, M. J. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.12.011 · Cited by 27
Intestinal stem cells and intestinal organoids
Zhao, Q., Guan, J., & Wang, X. (2020) · Journal of genetics and genomics/Journal of Genetics and Genomics · doi:10.1016/j.jgg.2020.06.005 · Cited by 23
Kruesi, L., Burstein, F., & Tanner, K. (2020) · Journal of Knowledge Management · doi:10.1108/jkm-05-2020-0370 · Cited by 23
Opportunities and Risks for Research Biobanks in the COVID-19 Era and Beyond
Simeon‐Dubach, D., & Henderson, M. K. (2020) · Biopreservation and Biobanking · doi:10.1089/bio.2020.0079 · Cited by 23
“Very rarely do these ‘‘biobanks’’ implement the majority of guidelines for modern biobanks,(1,2,5) and they may experience the corresponding negative consequences for quality,(6) utilization, and reproducibility of research results.(10) Biobanks that apply robust business management, as well as scientific best practic…”
BackgroundEthical use of animal models in musculoskeletal research
Allen, M. J., Hankenson, K. D., Goodrich, L. R., Boivin, G. P., & von Rechenberg, B. (2016) · Journal of Orthopaedic Research® · doi:10.1002/jor.23485 · Cited by 23
“Importantly, the introduction of ARRIVE represents an important first step toward reducing the problem of irreproducibility that plagues science in general but animal models in particular.(12,13)”
BackgroundRufiange, M., Rousseau‐Blass, F., & Pang, D. (2019) · Veterinary Record Open · doi:10.1136/vetreco-2018-000322 · Cited by 26
Ramirez, F. D., Jung, R. G., Motazedian, P., Perry‐Nguyen, D., Santo, P. D., MacDonald, Z., Clancy, A. A., Labinaz, A., Promislow, S., Simard, T., et al. (2019) · Stroke · doi:10.1161/strokeaha.119.026564 · Cited by 24
Bikard, M. A., Fernandez‐Mateo, I., & Mogra, R. (2025) · Administrative Science Quarterly · doi:10.1177/00018392251331957 · Cited by 26
Biobank Quality Management in the BBMRI.be Network
Linsen, L., T’Joen, V., Straeten, C. V. D., Landuyt, K. V., Marbaix, É., Bekaert, S., & Ectors, N. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00141 · Cited by 22
“Different categories of errors contribute to this irreproducibility, with biological reagents and reference materials having the biggest impact (2).”
BackgroundMethodSteinfath, M., Vogl, S., Violet, N., Schwarz, F., Mielke, H., Selhorst, T., Greiner, M., & Schönfelder, G. (2018) · PLoS ONE · doi:10.1371/journal.pone.0202762 · Cited by 26
Walters, C., Harter, Z. J., Wayant, C., Vo, N., Warren, M. H., Chronister, J., Tritz, D. J., & Vassar, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033962 · Cited by 20
Miedel, M. T., Varmazyad, M., Xia, M., Brooks, M. M., Gavlock, D. C., Reese, C. E., Behari, J., Soto–Gutiérrez, A., Gough, A., Taylor, D. L., et al. (2025) · Cell Reports Methods · doi:10.1016/j.crmeth.2025.101028 · Cited by 24
“20,21 27 While a number of approaches are being implemented to address some of these issues to improve quality control, 23,27 having access to detailed study data and metadata along with a standardized way of assessing the assay/model reproducibility is critical.”
Genetically Defined Strains in Drug Development and Toxicity Testing
Festing, M. F. W. (2016) · Methods in molecular biology · doi:10.1007/978-1-4939-3661-8_1 · Cited by 25
Metadata
- Authors
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S.
- Affiliations
- Global Biological Standards Institute (US)
- Boston University (US)
- Corresponding author
- Freedman, Leonard P.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-06-09
- Volume
- 13
- Issue
- 6
- Pages
- e1002165
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4461318
- OpenAlex
- W585341442
- Cited by
- 1,055
- References
- 35
- Citations by year
- 2026: 48 · 2025: 45 · 2024: 69 · 2023: 92 · 2022: 93 · 2021: 96 · 2020: 142 · 2019: 116 · 2018: 109 · 2017: 93 · 2016: 103 · 2015: 49
- FWCI
- 188.58
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- OpenBU (Boston University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Keywords
- reproducibility, preclinical research, research costs, research waste, reproducibility framework
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.