The Economics of Reproducibility in Preclinical Research
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible-in the United States alone. We outline a framework for solutions and a plan for long-term improvements in reproducibility rates that will help to accelerate the discovery of life-saving therapies and cures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Studies reporting the prevalence of irreproducibility. Source: Begley and Ellis [6], Prinz et al. [7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al. [9].](https://figures.cochat.ai/paper-figures/a30bd9bd65dc4bedb5db19b413eb814f/the-economics-of-reproducibility-in-preclinical-research-figure-1.jpg)
![Fig 2 Estimated US preclinical research spend and categories of errors that contribute to irreproducibility. Note that the percentage value of error for each category is the midpoint of the high and low prevalence estimates for that category divided (weighted) by the sum of all midpoint error rates (see S1 Dataset). Source: Chakma et al. [18] and the American Association for the Advancement of Science (AAAS) [19].](https://figures.cochat.ai/paper-figures/207a0a58df5c4620a819d7aa1f4aa960/the-economics-of-reproducibility-in-preclinical-research-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Besides, reliably determining actual translational success rates is unmanageable as long as the current status of reporting of preclinical research leaves room for improvement [89], and non-reproducibility is such a critical issue in both animal [1] and human [90] studies.”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of biomedical research, the majority of incorrect results are estimated to derive from the incorrect use of material standards and experimental reagents [2, 3].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The S&B tool therefore directly addresses the larger problem of material reagents and standards representing the major source of incorrect published results from pre-clinical research [3, 4].”
From Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current drug development is handicapped by high attrition rates; many molecules that were promising during preclinical development fail during subsequent clinical testing [1].”
From Animal to human translation: a systematic scoping review of reported concordance rates (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [16]
Experimental — accuracy not yet measured
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Governing the Commons: The Evolution of Institutions for Collective Action
Field, B. C., & Ostrom, E. (1992) · Land Economics · doi:10.2307/3146384 · Cited by 18,859
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“[7], Vasilevsky [8], Hartshorne and Schachner [5], and Glasziou et al.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · cited 1×
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · cited 1×
“Much has been written about the alarming number of preclinical studies that were later found to be irreproducible [1,2].”
BackgroundBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Partial Identification of Probability Distributions
(2003) · Springer Series in Statistics · doi:10.1007/b97478 · Cited by 698
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Research works printed but not linked: 1
- CF Manski, 2003, Springer Series in Statistics, 178
Cited by
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Moving Forward: Challenges and Directions for Psychopathological Network Theory and Methodology
Fried, E. I., & Cramer, A. O. J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617705892 · Cited by 938
Genetic and transcriptional evolution alters cancer cell line drug response
Ben-David, U., Siranosian, B., Ha, G., Tang, H., Oren, Y., Hinohara, K., Strathdee, C. A., Dempster, J., Lyons, N. J., Burns, R., et al. (2018) · Nature · doi:10.1038/s41586-018-0409-3 · Cited by 931
Zhou, N., & Bao, J. (2020) · Database · doi:10.1093/database/baaa021 · Cited by 788
“We acknowledge that this assumption may introduce some errors because unreliable scientific claims might exist in original studies, especially in life sciences research which is beset by low reproducibility rates (16).”
BackgroundMuthukrishna, M., & Henrich, J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0522-1 · Cited by 638
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“…more reliable, including wider sharing and reanalysis of code, data, and research materials [2,18]; valuing replications and reanalyses [2,5,6,19,20]; changes in statistical approaches with regards to power [21,22] and how evidence is assessed [23]; interactive and more transparent ways of…”
BackgroundMethodInfluentialCollins, B. C., Hunter, C. L., Liu, Y., Schilling, B., Rosenberger, G., Bader, S. L., Chan, D. W., Gibson, B. W., Gingras, A.-C., Held, J. M., et al. (2017) · Nature Communications · doi:10.1038/s41467-017-00249-5 · Cited by 576
Yu, Z., Guindani, M., Grieco, S. F., Chen, L., Holmes, T. C., & Xu, X. (2022) · Neuron · doi:10.1016/j.neuron.2021.10.030 · Cited by 563
Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition)
Cossarizza, A., Chang, H., Radbruch, A., Abrignani, S., Addo, R., Akdis, M., Andrä, I., Andreata, F., Annunziato, F., Arranz, E., et al. (2021) · European Journal of Immunology · doi:10.1002/eji.202170126 · Cited by 531
Increasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462
Poynton, F. E., Bright, S. A., Blasco, S., Williams, D. C., Kelly, J. M., & Gunnlaugsson, T. (2017) · Chemical Society Reviews · doi:10.1039/c7cs00680b · Cited by 421
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
“Furthermore, they found sample (United States vs. international) and setting (online vs. in-lab) differences for nearly one-third (10/32) of their moderation tests, seven of which were among the largest effects (i.e., anchoring, allowed–forbidden).”
BackgroundConsidering sex as a biological variable in preclinical research
Miller, L. R., Marks, C., Becker, J. B., Hurn, P. D., Chen, W., Woodruff, T., McCarthy, M. M., Sohrabji, F., Schiebinger, L., Wetherington, C. L., et al. (2017) · The FASEB Journal · doi:10.1096/fj.201600781r · Cited by 391
“However, investigators have often been unable to reproduce promising preclinical research results, with a recent report (19) estimating that roughly53%of irreproduciblepreclinical results were causedby root factors categorized as studydesign and data analysis and reporting, which resulted in considerable loss of econo…”
BackgroundReproducibility of animal research in light of biological variation
Voelkl, B., Altman, N. S., Forsman, A., Forstmeier, W., Gurevitch, J., Jaric, I., Karp, N. A., Kas, M. J., Schielzeth, H., Van de Casteele, T., et al. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0313-3 · Cited by 390
Zushin, P.-J. H., Mukherjee, S., & Wu, J. C. (2023) · Journal of Clinical Investigation · doi:10.1172/jci175824 · Cited by 347
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346
“[2,3]) has triggered reflection and meta-research about the ways in which this transformation process is susceptible to confusion and corruption.”
BackgroundUsing prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
“J Exp Psychol Learn Mem Cogn 34(3):478–494.”
BackgroundInfluentialOpportunities and challenges in the collection and analysis of digital phenotyping data
Onnela, J.-P. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0771-3 · Cited by 319
“A comparison of fi ve analyses of reproducibility in preclinical research reported prevalence of irreproducibility to vary from 51 to 89% [35].”
BackgroundAnimal to human translation: a systematic scoping review of reported concordance rates
Leenaars, C. H. C., Kouwenaar, C., Stafleu, F. R., Bleich, A., Ritskes-Hoitinga, M., De Vries, R. B. M., & Meijboom, F. L. B. (2019) · Journal of Translational Medicine · doi:10.1186/s12967-019-1976-2 · Cited by 314
Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities
Bligaard, T., Bullock, R. M., Campbell, C. T., Chen, J. G., Gates, B. C., Gorte, R. J., Jones, C. W., Jones, W. D., Kitchin, J. R., & Scott, S. L. (2016) · ACS Catalysis · doi:10.1021/acscatal.6b00183 · Cited by 285
Metal–organic frameworks for biological applications
Abánades Lázaro, I., Chen, X., Ding, M., Eskandari, A., Fairen-Jimenez, D., Giménez-Marqués, M., Gref, R., Lin, W., Luo, T., & Forgan, R. S. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00320-8 · Cited by 252
“Reproducibility has long been recognized as a major issue in preclinical research, impacting technological development and inducing significant economic cost 225 ; such concerns are particularly prevalent in drug delivery research 226,227 .”
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
Reproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262
Considering aspects of the 3Rs principles within experimental animal biology
Sneddon, L. U., Halsey, L. G., & Bury, N. R. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.147058 · Cited by 254
“Many authors have discussed how to combat irreproducibility (Freedman et al., 2015; Ioannidis et al., 2015; McNutt, 2014; Nosek et al., 2015; Woolston, 2014; http://validation.scienceexchange.com/ #/reproducibility-initiative).”
BackgroundDisease model discovery from 3,328 gene knockouts by The International Mouse Phenotyping Consortium
Meehan, T. F., Conte, N., West, D. B., Jacobsen, J. O., Mason, J., Warren, J., Chen, C.-K., Tudose, I., Relac, M., Matthews, P., et al. (2017) · Nature Genetics · doi:10.1038/ng.3901 · Cited by 243
“This is in line with previous reports describing reproducibility of biomedical models 16 .”
Rafi, Z., & Greenland, S. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01105-9 · Cited by 243
“Webelieve thatneglect of theseproblems is amajor contributor to the current crisis of statistics in science [4–9].”
Assessing and mitigating batch effects in large-scale omics studies
Yu, Y., Mai, Y., Zheng, Y., & Shi, L. (2024) · Genome Biology · doi:10.1186/s13059-024-03401-9 · Cited by 202
“Irreproducibility caused by batch effects can also result in rejected papers, discredited research findings, and financial losses [43].”
BackgroundIsothermal titration calorimetry
Bastos, M., Abian, O., Johnson, C. M., Ferreira-da-Silva, F., Vega, S., Jimenez-Alesanco, A., Ortega-Alarcon, D., & Velazquez-Campoy, A. (2023) · Nature Reviews Methods Primers · doi:10.1038/s43586-023-00199-x · Cited by 212
On determining sample size in experiments involving laboratory animals
Festing, M. F. (2018) · Laboratory Animals · doi:10.1177/0023677217738268 · Cited by 213
“One estimate puts the cost at $28 billion being wasted per annum in the United States alone.(4) The causes of this irreproducibility crisis have not been fully identified.”
BackgroundAutomation in the Life Science Research Laboratory
Holland, I., & Davies, J. A. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.571777 · Cited by 200
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation
Hunter, P. (2017) · The EMBO Reports · doi:10.15252/embr.201744876 · Cited by 201
“One of the largest meta-analyses concluded that low levels of reproducibility, at best around 50% of all preclinical biomedical research, were delaying lifesaving therapies, increasing pressure on research budgets and raising costs of drug development [1].”
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
Cohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., Gaignard, A., Hinsen, K., Larmande, P., Bras, Y. L., Lemoine, F., et al. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.01.012 · Cited by 181
“Over the recent years, many authors have drawn attention to the rise of purely computational experiments which are not reproducible [2, 3, 4, 5].”
BackgroundPoeker, S. A., Geirnaert, A., Berchtold, L., Greppi, A., Krych, L., Steinert, R. E., de Wouters, T., & Lacroix, C. (2018) · Scientific Reports · doi:10.1038/s41598-018-22438-y · Cited by 176
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175
Leung, V., Rousseau-Blass, F., Beauchamp, G., & Pang, D. S. J. (2018) · PLOS ONE · doi:10.1371/journal.pone.0197882 · Cited by 172
The ghosts of HeLa: How cell line misidentification contaminates the scientific literature
Horbach, S. P. J. M., & Halffman, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186281 · Cited by 167
“potentially severe and costly [17], with grants, patents and even drug trials based on misidentified cells [18].”
BackgroundA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161
Issues with data and analyses: Errors, underlying themes, and potential solutions
Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708279115 · Cited by 157
“One estimate calculated this waste to be half of all preclinical research, amounting to an estimated $28 billion going to irreproducible research in the United States alone in 2015 (79).”
BackgroundOxygenation in Cell Culture: Critical Parameters for Reproducibility Are Routinely Not Reported
Al-Ani, A., Toms, D., Kondro, D., Thundathil, J., Yu, Y., & Ungrin, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204269 · Cited by 152
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 167
Reproducible and replicable pain research: a critical review
Lee, H., Lamb, S. E., Bagg, M. K., Toomey, E., Cashin, A. G., & Moseley, G. L. (2018) · Pain · doi:10.1097/j.pain.0000000000001254 · Cited by 150
“It has been estimated that non-reproducible preclinical research consumes $28 billion/year (USD) [16] and that 85% of biomedical research resources are wasted on biased research [35].”
BackgroundOverlooked Benefits of using Polyclonal Antibodies
Ascoli, C. A., & Aggeler, B. (2018) · BioTechniques · doi:10.2144/btn-2018-0065 · Cited by 142
“As the potential economic impact of low reproducibility rates in life science research were assessed [9], thought leaders and key stakeholders assembled to begin to formulate approaches to resolve the dilemma, first by proposing ‘pillars’ for application-specific antibody validation [10] and subsequently by establishi…”
BackgroundRamirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138
“The suggestion that methodologically robust studies are given equal consideration as studies at greater risk of bias by cardiovascular researchers is troubling as it undermines the reputed “self-correcting” tenet of science and increases the risk of pursuing unfruitful avenues of research.(2,5,44,48) Cogent analyses a…”
Provencher, S., Archer, S. L., Ramirez, F. D., Hibbert, B. M., Paulin, R., Boucherat, O., Lacasse, Y., & Bonnet, S. (2018) · Circulation Research · doi:10.1161/circresaha.117.312579 · Cited by 143
“Such large-scale attrition of investigational drugs is potentially harmful to individuals in trials and consumes scarce human and material resources.(14,17,18) The failure to test new classes of PAH therapy in humans relates, in part, to choices by Pharma, related to their willingness to take risks (including risks of…”
Backgroundvon Kortzfleisch, V. T., Karp, N. A., Palme, R., Kaiser, S., Sachser, N., & Richter, S. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-73503-4 · Cited by 154
An Analysis of Mesenchymal Stem Cell-Derived Extracellular Vesicles for Preclinical Use
Tieu, A., Lalu, M. M., Slobodian, M., Gnyra, C., Fergusson, D. A., Montroy, J. R., Burger, D. E., Stewart, D. J., & Allan, D. (2020) · ACS Nano · doi:10.1021/acsnano.0c01363 · Cited by 153
FOSTERING INTEGRITY IN RESEARCH
Arrison, T., & Nerem, R. M. (2017) · Special publications · doi:10.1002/9781119067825.ch2 · Cited by 136
“Past a certain point, efforts to eliminate all of the possible factors that cause irreproducibility would be prohibitively expensive (Freedman et al., 2015).”
InfluentialLucanic, M., Plummer, W. T., Chen, E., Harke, J., Foulger, A., Onken, B., Coleman-Hulbert, A. L., Dumas, K. J., Guo, S., Johnson, E., et al. (2017) · Nature Communications · doi:10.1038/ncomms14256 · Cited by 144
High Throughput Methods in the Synthesis, Characterization, and Optimization of Porous Materials
Clayson, I. G., Hewitt, D., Hutereau, M., Pope, T., & Slater, B. (2020) · Advanced Materials · doi:10.1002/adma.202002780 · Cited by 143
Bikard, M. A., & Marx, M. (2019) · Management Science · doi:10.1287/mnsc.2019.3385 · Cited by 140
“The rate of replicability of published academic studies is estimated to be between 11% and 49% (Begley and Ellis 2012, Hartshorne and Schachner 2012, Freedman et al. 2015).”
Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Jarvis, M. F., & Williams, M. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2015.12.001 · Cited by 129
ISSCR standards for the use of human stem cells in basic research
Ludwig, T. E., Andrews, P. W., Barbaric, I., Benvenisty, N., Bhattacharyya, A., Crook, J. M., Daheron, L. M., Draper, J. S., Healy, L. E., Huch, M., et al. (2023) · Stem Cell Reports · doi:10.1016/j.stemcr.2023.08.003 · Cited by 126
“Misidentification or cross-contamination of materials can occur at all stages of work with cell lines, including during their derivation or isolation, culture, storage, or distribution, and its prevalence is well documented in the literature (American Type Culture Collection Standards Development Organization Workgrou…”
BackgroundA survey of early-career researchers in Australia
Christian, K., Johnstone, C., Larkins, J., Wright, W., & Doran, M. R. (2021) · eLife · doi:10.7554/elife.60613 · Cited by 140
The NIH Somatic Cell Genome Editing program
Saha, K., Sontheimer, E. J., Brooks, P. J., Dwinell, M. R., Gersbach, C. A., Liu, D. R., Murray, S. A., Tsai, S. Q., Wilson, R. C., Anderson, D. G., et al. (2021) · Nature · doi:10.1038/s41586-021-03191-1 · Cited by 139
Rosso, M., Wirz, R., Loretan, A. V., Sutter, N., da Cunha, C. T. P., Jarić, I., Würbel, H., & Voelkl, B. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104928 · Cited by 140
“…obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. o Relevant literature: (Baker, 2015; Begley and Ioannidis, 2015; Freedman et al., 2015; Roth and Cox, 2015; Baker, 2016; Miyakawa, 2020; Smith and Lilley, 2019) 1.”
Bias in the reporting of sex and age in biomedical research on mouse models
Flórez-Vargas, O., Brass, A., Karystianis, G., Bramhall, M., Stevens, R., Cruickshank, S., & Nenadic, G. (2016) · eLife · doi:10.7554/elife.13615 · Cited by 124
Kara, N., Stukalin, Y., & Einat, H. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.003 · Cited by 133
“Low reproducibility rates are not unique to neuropsychiatric diseases models and a recent study suggests that within life science research the cumulative (total) prevalence of irreproducible preclinical research exceeds 50% of published results (Freedman et al., 2015).”
ResultPredictive validity in drug discovery: what it is, why it matters and how to improve it
Scannell, J. W., Bosley, J., Hickman, J. A., Dawson, G. R., Truebel, H., Ferreira, G. S., Richards, D. B., & Treherne, J. M. (2022) · Nature Reviews Drug Discovery · doi:10.1038/s41573-022-00552-x · Cited by 142
Bauer, M., Strom, M., Hammond, D., & Shigdar, S. L. (2019) · Molecules · doi:10.3390/molecules24234377 · Cited by 131
“Reagent variations account for an estimated 36% of total irreproducibility [8] of biological assays, with antibodies representing the most ubiquitously utilised group of reagents [3].”
BackgroundThe Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118
“It has been estimated that over half of preclinical research is irreproducible (see [2]).”
BackgroundReplication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 145
“It seems expedient with regards to the evolution of knowledge to critically examine new insights that could change the discourse in a field (Loasby, 2002; Popper, 1959; Freedman et al., 2015).”
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
Quality Issues of Research Antibodies
Weller, M. G. (2016) · Analytical Chemistry Insights · doi:10.4137/aci.s31614 · Cited by 131
Yu, Y., Zhang, N., Mai, Y., Ren, L., Chen, Q., Cao, Z., Chen, Q., Liu, Y., Hou, W., Yang, J., et al. (2023) · Genome biology · doi:10.1186/s13059-023-03047-z · Cited by 139
“On the other hand, systematic variations including batch effects have become one of the major causes of irreproduc-ibility [21, 22].”
BackgroundP-Values: Misunderstood and Misused
Vidgen, B., & Yasseri, T. (2016) · Frontiers in Physics · doi:10.3389/fphy.2016.00006 · Cited by 143
“This simulator, which analyses whether Republicans or Democrats being in office affects the US economy, shows how tests can be manipulated to produce statistically significant results supporting either parties.”
BackgroundLow availability of code in ecology: A call for urgent action
Čulina, A., van den Berg, I., Evans, S. R., & Sánchez‐Tójar, A. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000763 · Cited by 137
Rhon, D. I., Fritz, J. M., Kerns, R. D., McGeary, D. D., Coleman, B. C., Farrokhi, S., Burgess, D. J., Goertz, C., Taylor, S. L., & Hoffmann, T. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01640-7 · Cited by 117
“Research that cannot be replicated leads to waste and setbacks in medical advancement [4].”
BackgroundEnhancing Reuse of Data and Biological Material in Medical Research: From FAIR to FAIR-Health
Holub, P., Kohlmayer, F., Praßer, F., Mayrhofer, M. T., Schlünder, I., Martin, G. M., Casati, S., Koumakis, L., Wutte, A., Kozera, Ł., et al. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2017.0110 · Cited by 113
“, omics data and imaging data); (5) expertise in various fields related to longterm preservation and analyses of biological material and data, ethical/legal expertise; and (6) additional services related to biobanks as infrastructure (e.”
Does health informatics have a replication crisis?
Coiera, E. W., Ammenwerth, E., Georgiou, A., & Magrabi, F. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy028 · Cited by 109
“Indeed, one recent study suggests that “irreproducible preclinical research exceeds 50%” of all studies.(3) The poor reproducibility of experimental findings in any discipline can be due to problems with experimental design, statistical errors, small sample sizes, outcome switching,(4) selective reporting of significa…”
BackgroundBowen, A., & Casadevall, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504955112 · Cited by 104
“Recently, it was estimated that more than $28 billion is spent each year in the United States on irreproducible preclinical research and that the prevalence of these studies in the literature exceeds 50% (8).”
BackgroundDraeger, E., Sawant, A. R., Johnstone, C. D., Koger, B., Becker, S. J., Vujašković, Ž., Jackson, I., & Poirier, Y. P. (2019) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2019.06.2545 · Cited by 111
Analytical techniques for multiplex analysis of protein biomarkers
van Gool, A. J., Corrales, F. J., Čolović, M. B., Krstic, D. Z., Oliver‐Martos, B., Martínez-Cáceres, E. M., Jakaša, I., Gajski, G., Brun, V., Kyriacou, K. C., et al. (2020) · Expert Review of Proteomics · doi:10.1080/14789450.2020.1763174 · Cited by 119
“In addition, the need for stringent quality control of assay reagents to avoid research irreproducibility [4] is increased when multiple proteins are targeted in one assay.”
BackgroundCait, J., Cait, A., Scott, R. W., Winder, C. B., & Mason, G. J. (2022) · BMC Biology · doi:10.1186/s12915-021-01184-0 · Cited by 133
Reproducibility2020: Progress and priorities
Freedman, L. P., Venugopalan, G., & Wisman, R. (2017) · F1000Research · doi:10.12688/f1000research.11334.1 · Cited by 102
“[3] Based on these findings, GBSI completed an economic study in 2015 and estimated that the prevalence of irreproducible preclinical research exceeds 50% with associated annual costs of approximately $28B in the United States alone.[4]”
BackgroundSelf-citationLausch, A., Borg, E., Bumberger, J., Dietrich, P., Heurich, M., Huth, A., Jung, A., Klenke, R., Knapp, S., Mollenhauer, H., et al. (2018) · Remote Sensing · doi:10.3390/rs10071120 · Cited by 118
“Due to errors in research design and realization, together with the wrong development of previous data science in recording, saving, processing, and linking research data, approximately 50% (corresponding to some US$28B/year) of all research data and information can no longer be reproduced [177].”
BackgroundEstimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 119
Bridging Translation by Improving Preclinical Study Design in AKI
deCaestecker, M. P., Humphreys, B. D., Liu, K. D., Fissell, W. H., Cerdá, J., Nolin, T. D., Askenazi, D. J., Mour, G. K., Harrell, F. E., Pullen, N., et al. (2015) · Journal of the American Society of Nephrology · doi:10.1681/asn.2015070832 · Cited by 99
Metaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F. M., Chee, Y. E., Wintle, B. C., Burgman, M. A., Mccarthy, M. A., & Gordon, A. (2016) · BioScience · doi:10.1093/biosci/biw159 · Cited by 122
Haslacher, H., Gerner, M. C., Hofer, P., Jurkowitsch, A., Hainfellner, J. A., Kain, R., Wagner, O. F., & Perkmann, T. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2018.0032 · Cited by 96
“It is increasingly recognized that poor reproducibility of results in biomedical research is to a relevant degree caused by low quality materials, leading to an annual loss of *$28 billion in the United States alone.(1) Consequently, numerous biobanks have emerged to address the need for biomaterial and data produced…”
BackgroundReporting of lipidomics data should be standardized
Liebisch, G., Ekroos, K., Hermansson, M., & Ejsing, C. S. (2017) · Biochimica et Biophysica Acta (BBA) - Molecular and Cell Biology of Lipids · doi:10.1016/j.bbalip.2017.02.013 · Cited by 94
Hippo pathway mediates resistance to cytotoxic drugs
Gujral, T. S., & Kirschner, M. W. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1703096114 · Cited by 91
“Discrepancy in experimental findings is often blamed on sloppiness and has been excoriated as part of an epidemic of irreproducible scientific experiments (51).”
BackgroundHaaker, J., Maren, S., Andreatta, M., Merz, C. J., Richter, J., Richter, S. H., Drexler, S. M., Lange, M. D., Jüngling, K., Nees, F., et al. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.09.020 · Cited by 97
Levelling the Translational Gap for Animal to Human Efficacy Data
Ferreira, G. S., Veening-Griffioen, D. H., Boon, W., Moors, E. H. M., & van Meer, P. J. K. (2020) · Animals · doi:10.3390/ani10071199 · Cited by 103
“Nonetheless, the estimated cost of irreproducibility (~30 billion USD per year in the US alone) likely dwarfs the cost for the broad implementation of these strategies [44].”
BackgroundStandard Operating Procedure to Optimize Resazurin-Based Viability Assays
Petiti, J., Revel, L., & Divieto, C. (2024) · Biosensors · doi:10.3390/bios14040156 · Cited by 99
“Notably, despite the use of common estimators, the pharmacologic drug response showed a high discordance between the studies [11,12].”
BackgroundInfluentialBest practices in nutrition science to earn and keep the public's trust
Garza, C., Stover, P. J., Ohlhorst, S. D., Field, M. S., Steinbrook, R., Rowe, S. B., Woteki, C. E., & Campbell, E. G. (2019) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqy337 · Cited by 97
“A metainterpretation aimed at determining how to increase the public’s trust in the food safety system following retail food safety outbreaks in the United States, Canada, Australia, New Zealand, and Great Britain concluded that “provision of transparent scientific information to inform the public may support public t…”
BackgroundInfluentialGenerative adversarial networks for financial trading strategies fine-tuning and combination
Koshiyama, A., Firoozye, N. B., & Treleaven, P. C. (2020) · Quantitative Finance · doi:10.1080/14697688.2020.1790635 · Cited by 125
“ng cannot be understated, and innumerable references highlight the issues with phacking which has been an issue for considerable periods but making headlines more recently (see e.g., [35], [26], [7], [13]), and [24]) although it may not always be willful ([16]), and is pernicious in nance (see for instance, [44]). 2.1…”
BackgroundInfluentialOpen science, communal culture, and women’s participation in the movement to improve science
Murphy, M. C., Mejia, A. F., Mejia, J., Yan, X., Cheryan, S., Dasgupta, N., Destin, M., Fryberg, S. A., García, J. A., Haines, E. L., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1921320117 · Cited by 95
Fini, R., Perkmann, M., & Roß, J.-M. (2021) · Organization Science · doi:10.1287/orsc.2021.1455 · Cited by 107
“Third, entrepreneurship may provide an incentive to produce higher quality science, since nonreplicable research is of little value for commercialization (Freedman et al. 2015).”
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14325 · Cited by 88
A discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 106
“Researchers were concerned about the implication of systematic replication failures, which include a weakened knowledge base, higher public distrust in science, and funding cuts (11–13).”
The consequences of recurrent genetic and epigenetic variants in human pluripotent stem cells
Andrews, P. W., Barbaric, I., Benvenisty, N., Draper, J. S., Ludwig, T. E., Merkle, F. T., Sato, Y., Spits, C., Stacey, G. N., Wang, H., et al. (2022) · Cell stem cell · doi:10.1016/j.stem.2022.11.006 · Cited by 87
Pre-Clinical Testing of Therapies for Traumatic Brain Injury
DeWitt, D. S., Hawkins, B. E., Dixon, C. E., Kochanek, P. M., Armstead, W. M., Bass, C. R., Bramlett, H. M., Büki, A., Dietrich, W. D., Ferguson, A. R., et al. (2018) · Journal of Neurotrauma · doi:10.1089/neu.2018.5778 · Cited by 92
“Estimates of the numbers of irreproducible pre-clinical studies have ranged from *40% to as high as 80%.(66)”
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Sakolish, C., Weber, E. J., Kelly, E. J., Himmelfarb, J., Mouneimne, R., Grimm, F. A., House, J. S., Wade, T. L., Han, A., Chiu, W. A., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-33099-2 · Cited by 88
Design of experiments (DOE)—history, concepts, and relevance to in vitro culture
Niedz, R. P., & Evens, T. J. (2016) · In Vitro Cellular & Developmental Biology - Plant · doi:10.1007/s11627-016-9786-1 · Cited by 99
“Difficulty repeating published research results is a well-recognized issue by the commercial plant tissue culture community and is a major problem in biomedical (Collins and Tabak 2014; Begley and Ioannidis 2015; Freedman et al. 2015) and psychology research (Aarts et al. 2015).”
BackgroundBest practices in cell culture: an overview
Baust, J. M., Buehring, G. C., Campbell, L. H., Elmore, E. L., Harbell, J. W., Nims, R. W., Price, P. J., Reid, Y. A., & Simione, F. P. (2017) · In Vitro Cellular & Developmental Biology - Animal · doi:10.1007/s11626-017-0177-7 · Cited by 83
The power of forecasts to advance ecological theory
Lewis, A. S. L., Rollinson, C. R., Allyn, A., Ashander, J., Brodie, S. J., Brookson, C. B., Collins, E. L., Dietze, M. C., Gallinat, A. S., Juvigny‐Khenafou, N. P. D., et al. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13955 · Cited by 87
“Creating specific, quantitative predictions that can be tested with incoming data is particularly important in the face of the ongoing reproducibility crisis across scientific disciplines (e.g. Freedman et al., 2015 ; Open Science Collaboration, 2015 ).”
Metadata
- Authors
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S.
- Affiliations
- Global Biological Standards Institute (US)
- Boston University (US)
- Corresponding author
- Freedman, Leonard P.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-06-09
- Volume
- 13
- Issue
- 6
- Pages
- e1002165
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4461318
- OpenAlex
- W585341442
- Cited by
- 1,055
- References
- 35
- Citations by year
- 2026: 48 · 2025: 45 · 2024: 69 · 2023: 92 · 2022: 93 · 2021: 96 · 2020: 142 · 2019: 116 · 2018: 109 · 2017: 93 · 2016: 103 · 2015: 49
- FWCI
- 188.58
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-04-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- OpenBU (Boston University) (submittedVersion, cc-by)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Keywords
- reproducibility, preclinical research, research costs, research waste, reproducibility framework
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.