CoChat
Sign Up
Cited by 279Pharmaceutical Economics and PolicyComputational Drug Discovery MethodsStatistical Methods in Clinical Trials

When Quality Beats Quantity: Decision Theory, Drug Discovery, and the Reproducibility Crisis

Open access
Scannell, Jack W., Bosley, Jim·2016·PLOS ONE
DOI 10.1371/journal.pone.0147215PMC PMC4749240
Open this paper in CoChat — free accountOpen PDF

Abstract

A striking contrast runs through the last 60 years of biopharmaceutical discovery, research, and development. Huge scientific and technological gains should have increased the quality of academic science and raised industrial R&D efficiency. However, academia faces a "reproducibility crisis"; inflation-adjusted industrial R&D costs per novel drug increased nearly 100 fold between 1950 and 2010; and drugs are more likely to fail in clinical development today than in the 1970s. The contrast is explicable only if powerful headwinds reversed the gains and/or if many "gains" have proved illusory. However, discussions of reproducibility and R&D productivity rarely address this point explicitly. The main objectives of the primary research in this paper are: (a) to provide quantitatively and historically plausible explanations of the contrast; and (b) identify factors to which R&D efficiency is sensitive. We present a quantitative decision-theoretic model of the R&D process. The model represents therapeutic candidates (e.g., putative drug targets, molecules in a screening library, etc.) within a "measurement space", with candidates' positions determined by their performance on a variety of assays (e.g., binding affinity, toxicity, in vivo efficacy, etc.) whose results correlate to a greater or lesser degree. We apply decision rules to segment the space, and assess the probability of correct R&D decisions. We find that when searching for rare positives (e.g., candidates that will successfully complete clinical development), changes in the predictive validity of screening and disease models that many people working in drug discovery would regard as small and/or unknowable (i.e., an 0.1 absolute change in correlation coefficient between model output and clinical outcomes in man) can offset large (e.g., 10 fold, even 100 fold) changes in models' brute-force efficiency. We also show how validity and reproducibility correlate across a population of simulated screening and disease models. We hypothesize that screening and disease models with high predictive validity are more likely to yield good answers and good treatments, so tend to render themselves and their diseases academically and commercially redundant. Perhaps there has also been too much enthusiasm for reductionist molecular models which have insufficient predictive validity. Thus we hypothesize that the average predictive validity of the stock of academically and industrially "interesting" screening and disease models has declined over time, with even small falls able to offset large gains in scientific knowledge and brute-force efficiency. The rate of creation of valid screening and disease models may be the major constraint on R&D productivity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0147215

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conventional preclinical models, including computational, traditional cell culture, and animal models, often miss drug toxicities and have limited predictive validity, so failure is common and harm is realized only in clinical trials or after market entry.

    “Conventional preclinical models often miss drug toxicities, meaning the harm these drugs pose to humans is only realized in clinical trials or when they make it to market.”

    From Performance assessment and economic analysis of a human Liver-Chip for predictive toxicology (2022) · cites this paper as [4]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 81

  1. [1]Scannell J, Blanckley A, Boldon H, Warrington B. Diagnosing the decline in pharmaceutical R&D efficiency. Nat Rev Drug Discov. 2012; 11: p. 191– 200. 10.1038/nrd3681 22378269 · DOI · Paper page
  2. [2]Hogan JC. Combinatorial chemistry in drug discovery. Nat Biotechnol. 1997; 15: p. 328– 330. 9094132
  3. [3]Geysen HM, Schoenen F, Wagner D, Wagner R. Combinatorial compound libraries for drug discovery: an ongoing challenge. Nat Rev Drug Discov. 2003; 2: p. 222– 230. 12612648
  4. [4]Nature Biotechnology. Combinatorial chemistry. Nat Biotechnol. 2000; 18 supplement: p. IT50– IT52. 11221719
  5. [5]Dolle RE. Historical overview of chemical library design In Zhou JZ, editor. Chemical Library Design (Methods in Molecular Biology 685).: Springer Science; 2011 p. 3– 25. 20981516
  6. [6]Sanger F. Sequences, sequences, and sequences. Annu Rev Biochem. 1988; 57: p. 1– 28. 2460023
  7. [11]Mayr LM, Fuerst P. The future of high-throughput screening. J. Biomol. Screen. 2008; 13: p. 443– 448. 10.1177/1087057108319644 18660458 · DOI
  8. [12]Brown A, Casadevall A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc Natl Acad Sci U S A. 2015. PMC4568675 26283360
  9. [16]Perrin S. Preclinical research: Make mouse studies work. Nature. 2014; 507: p. 423– 425. 24678540
  10. [17]DiMasi J. Success rates for new drugs entering clinical testing in the United States. Clin Pharmacol Ther. 1995; 58: p. 1– 14. 7628176
  11. [18]Hay M, Thomas J, Craighead J, Economides C, Rosenthal J. Clinical development success rates for investigational drugs. Nat Biotechnol. 2014; 32: p. 40– 51. 10.1038/nbt.2786 24406927 · DOI · Paper page
  12. [19]Booth B, Zemmil R. Prospects for productivity. Nat Rev Drug Discov. 2004; 3: p. 451– 456. 15136792
  13. [20]Munos B. Lessons from 60 years of pharmaceutical innovation. Nat Rev Drug Discov. 2010;: p. 959– 968. 19949401
  14. [21]Paul S, Mytelka D, Dunwiddie C, Persinger C, Munos B, Lindborg SSA. How to improve R&D productivity: the pharmaceutical industry's grand challenge. Nat Rev Drug Discov. 2010; 9: p. 203– 214. 10.1038/nrd3078 20168317 · DOI · Paper page
  15. [22]Berndt E, Nass D, Kleinrock M, Aitken M. Decline in economic returns from new drugs raises questions about sustaining innovation. Health Aff (Millwood). 2015; 34: p. 245– 252. 25646104
  16. [23]Scannell J, Hinds S, Evans R. Financial returns on R&D: Looking back at history, looking forward to adaptive licensing. Rev Recent Clin Trials. 2015; 10: p. 28– 43. 25925881
  17. [24]Le Fanu J. The Rise and Fall of Modern Medicine. London: Little Brown; 1999. 10465212
  18. [25]Horrobin D. Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat Rev Drug Discov. 2003; 2: p. 151– 154. 12563306
  19. [26]Horrobin D. Realism in drug discovery—could Cassandra be right. Nat Biotechnol. 2011; 19: p. 1099– 1100. 11731767
  20. [29]Colquhoun D. An investigation of the false discovery rate and the misinterpretation of P values. R Soc Open Sci. 2014; 1: p. 140216 10.1098/rsos.140216 26064558 PMC4448847 · DOI · Paper page
  21. [30]Edwards W, Miles R, von Winterfeldt D. Advances in Decision Analysis: From Foundations to Applications. New York: Cambridge Unversity Press; 2007.
  22. [31]Peterson M. An Introduction to Decision Theory. Cambridge: Cambridge University Press; 2009.
  23. [32]Spiegelhalter D, Abrams K, Myles J. Bayesian Approaches to Clinical Trials and Health-care Evaluation: John Wiley and Sons; 2004.
  24. [35]Senn S. Statistical Issues in Drug Development. 2nd ed. Chichester: John Wiley & Sons, Ltd; 2007.
  25. [36]Willan A, Pinto E. The value of information and optimal clinical trial design. Stat Med. 2005; 24: p. 1791– 1806. 15806619
  26. [37]Bacchetti P. Current sample size conventions: Flaws, harms, and alternatives. BMC Med. 2010; 8: p. 17 10.1186/1741-7015-8-17 20307281 PMC2856520 · DOI
  27. [38]Bacchetti P, McCulloch C, Segal M. Simple, defensible sample sizes based on cost efficiency. Biometrics. 2008; 64(2): p. 577– 585. 10.1111/j.1541-0420.2008.01004_1.x 18482055 PMC2769573 · DOI
  28. [39]Simon R. Simple, defensible sample sizes based on cost efficiency—Discussions. Biometrics. 2008; 64(2): p. 589– 591. PMC2769573 18482055
  29. [40]Mueller P, Rosner G. Simple, defensible sample sizes based on cost efficiency—Discussions. Biometrics. 2008; 64(2): p. 587– 589. PMC2769573 18482055
  30. [41]Hanley J, Shapiro S. Simple, defensible sample sizes based on cost efficiency—Discussions. Biometrics. 2008; 64(2): p. 586– 587. PMC2769573 18482055
  31. [42]Bacchetti P, M CE, Segal M. Simple, defensible sample sizes based on cost efficiency—Rejoinder. Biometrics. 2008; 64(2): p. 592– 594. PMC2769573 18482055
  32. [43]Bacchetti P, Deeks S, McCune J. Breaking free of sample size dogma to perform innovative translational research. Sci Transl Med. 2011; 3: p. 87ps24 10.1126/scitranslmed.3001628 21677197 PMC3134305 · DOI
  33. [44]Detsky A. Using cost-effectiveness analysis to improve the efficiency of allocating funds to clinical trials. Stat Med. 1990; 9: p. 173– 184. 2111932
  34. [45]Berry D. Bayesian clinical trials. Nat Rev Drug Discov. 2006; 5: p. 26– 36. 16485344
  35. [47]Leach A, Gillet V. An Introduction to Chemoinformatics. Dordrecht: Springer; 2005.
  36. [48]Ajay A, Walters W, Murcko M. Can we learn to distinguish between 'drug-like' and 'nondrug-like' molecules. J Med Chem. 1998; 41: p. 3314– 3324. 9719583
  37. [50]Zhang J, Thomas D, Chung Y, Oldenburg K. A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J Biomol Screen. 1999; 4: p. 67– 73. 10838414
  38. [51]Pickett S, McLay I, Clark D. Enhancing the hit-to-lead properties of lead optimisation libraries. J Chem Inf Comput Sci. 2000; 40: p. 263– 272. 10761127
  39. [52]Lipinski CA, Lombardo F, Dominy BW, Feeney PJ. Experimental and computational approaches to estimate solubility and permiability in drug discovery and development settings. Adv Drug Deliv Rev. 2001: p. 3– 26. 11259830
  40. [53]Walters W, Namchuk M. Designing screens: How to make your hits a hit. Nat Rev Drug Discov. 2003; 2: p. 259– 266. 12669025
  41. [54]Bender A, Bojanic D, Davies J, Crisman T, Mikhailov D, Scheiber J, et al Which aspects of HTS are empirically correlated with downstream success? Curr Opin Drug Discov Devel. 2008; 11(3). 18428086
  42. [55]Langer T, Hoffman R, Bryant S, Lesur B. Hit finding: Towards 'smarter' approaches. Curr Opin Pharmacol. 2009; 9: p. 589– 593. 10.1016/j.coph.2009.06.001 19576852 · DOI
  43. [56]Bickerton G, P GV, Besnard J, Muresan S, Hopkins A. Quantifying the chemical beauty of drugs. Nature Chem. 2012; 4: p. 90– 98. 22270643 PMC3524573
  44. [57]Cumming J, Davis A, Muresan S, Haeberlein M, Chen H. Chemical predictive modelling to improve compound quality. Nat Rev Drug Discov. 2013; 12: p. 948– 462. 24287782
  45. [58]Hopkins A, Kesuru G, Leeson P, Rees D, Reynolds C. The role of ligand efficiency metrics in drug discovery. Nat Rev Drug Discov. 2014; 13: p. 105– 121. 10.1038/nrd4163 24481311 · DOI · Paper page
  46. [59]Lendrem D, Lendrem B, Peck R, Senn S, Day S, Isaacs J. Progression-seeking bias and rational optimism in research and development. Nat Rev Drug Discov. 2015; 14: p. 219– 221. 10.1038/nrd4320-c1 25656595 · DOI
  47. [60]Huang R, Wallqvist A, Covell D. Assessment of in vitro and in vivo activities in the National Cancer Institute's anticancer screen with respect to chemical structure, target specificity, and mechanism of action. J Med Chem. 2006; 49: p. 1964– 1979. 16539384
  48. [61]Wilner P. The validity of animal models in depression. Psychopharmacology (Berl). 1984; 83: p. 1– 16. 6429692
  49. [63]Gart J, Buck A. Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests. Am J Epidemiol. 1966; 83: p. 593– 602. 5932703
  50. [64]Howard R. The foundations of decision analysis revisited In Edwards W, Miles R, von Winterfeldt D. Advances in Decision Analysis: From Foundations to Applications. New York: Cambridge University Press; 2007 p. 32– 56.
  51. [66]Metcalf B, Dillon S. Target Validation in Drug Discovery: Academic Press; 2006.
  52. [67]Ioannidis J, Khoury M. Improving validation practices in "Omics" research. Science. 2011; 334: p. 1230– 1232. 10.1126/science.1211811 22144616 PMC5624327 · DOI
  53. [68]Coma I, Clark L, Diez E, Harper G, Herranz J, Hofmann G, et al Process validation and screen reproducibility in high-throughput screening. J Biomol Screen. 2009; 14: p. 66– 76. 10.1177/1087057108326664 19171922 · DOI
  54. [70]Henderson V, Kimmelman J, Fergusson D, Grimshaw J, Hackam D. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 2013; 7: p. e1001489. PMC3720257 23935460
  55. [71]Belzung C, Lemoine M. Criteria for validity for animal models of psychiatric disorders: focus on anxiety disorders and depression. Biol Mood Anxiety Disord. 2011; 1: p. 9 10.1186/2045-5380-1-9 22738250 PMC3384226 · DOI
  56. [73]Beck B. Model evaluation and performance In El-Shaarawi A, Piegorsch W. Encyclopedia of Environmetrics. Chichester: John Wiley & Sons, Ltd; 2002 p. 1275– 1279.
  57. [74]Lewis K. Plaforms for antibiotic discovery. Nature Rev Drug Discov. 2013; 12: p. 371– 387. 23629505
  58. [75]Dranchak P, MacArthur R, Guha R, Zuercher W, Drewry D, Auld D, et al Profile of the GSK published protein kinase inhibitor set across ATP-dependent and -independent luciferases: Implications for reporter-gene assays. PLoS One. 2013; 8: p. e57888 10.1371/journal.pone.0057888 23505445 PMC3591448 · DOI
  59. [76]Pedró-Rosa L, Buckner F, Ranade R, Eberhart C, Madoux F, Gillespie J, et al Identification of potent inhibitors of the Trypanosoma brucei methionyl-tRNA synthetase inhibitors via high throughput orthogonal screening. J Mol Screen. 2014; 20: p. 122– 130. PMC4378865 25163684
  60. [78]Pammolli F, Magazzini L, Riccaboni M. The productivity crisis in pharmaceutical R&D. Nature Rev Drug Discov. 2011; 10: p. 428– 438. 21629293
  61. [79]Horrobin DF. Innovation in the pharmaceutical industry. J R Soc Med. 2000; 93: p. 341– 345. 10928019 PMC1298056
  62. [81]Sams-Dodd F. Target-based drug discovery: Is something wrong? Drug Discov Today. 2005; 10: p. 139– 147. 15718163
  63. [82]Silver L. Challenges of antibacterial discovery. Clin Microbiol Rev. 2011; 24: p. 71– 109. 10.1128/CMR.00030-10 21233508 PMC3021209 · DOI
  64. [83]Joyner M, Prendergast F. Chasing Mendel: five questions for personalized medicine. J. Physiol. 2014; 592: p. 2381– 2388. 10.1113/jphysiol.2014.272336 24882820 PMC4048096 · DOI
  65. [84]Payne D, Gwynn M, Holmes D, Pompliano D. Drugs for bad bugs: confronting the challenges of antibacterial discovery. Nat Rev Drug Discov. 2007; 6: p. 29– 40. 17159923
  66. [85]Sams-Dodd F. Is poor research the cause of the declining productivity of the pharmaceutical industry? An industry in need of a paradigm shift. Drug Discov Today. 2013; 18: p. 211– 217. 10.1016/j.drudis.2012.10.010 23131208 · DOI
  67. [86]Goldstein D. Common genetic variation and human traits. N Engl J Med. 2009; 360: p. 1696– 1698. 10.1056/NEJMp0806284 19369660 · DOI
  68. [87]C Do, Francke U, Eriksson N. Comparison of family history and SNPs for predicting risk of complex disease. PLoS Genetics. 2012; 8: p. e1002973 10.1371/journal.pgen.1002973 23071447 PMC3469463 · DOI
  69. [88]Healy D. The Antidepressant Era: Harvard University Press; 1997.
  70. [89]Healy D. The Creation of Psychopharmacology. Cambridge, MA: New Directions Publishing Corporation; 2002.
  71. [90]DeMonaco HJ, Ali A, Von Hippel E. The major role of clinicians in the discovery of off-label drug therapies. Pharmacotherapy. 2006; 26: p. 323– 332. 16503712
  72. [93]Collins F, Tabak L. Policy: NIH plans to enhance reproducibility. Nature. 2014; 505: p. 612– 613. 24482835 PMC4058759
  73. [95]Young N, Ioannidis J, Al-Unaydli O. Why current publication practices may distort science. PLoS Med. 2008; 5: p. e201 10.1371/journal.pmed.0050201 18844432 PMC2561077 · DOI
  74. [97]Benner S, Bains W, Seager S. Models and standards of proof in cross-disciplinary science: The case of arsenic DNA. Astrobiology. 2013; 13: p. 510– 513. 10.1089/ast.2012.0954 23634974 · DOI · Paper page
  75. [98]McArthur R, Borsini F, editors. Animal and Translational Models for CNS Drug Discovery. Volume 1: Psychiatric Disorders Amsterdam: Elsevier; 2008.
  76. [102]Eder J, Sedrani R, Wiesmann C. The discovery of first-in-class drugs: Origins and evolution. Nat Rev Drug Discov. 2014 July. 25033734
  77. [103]Brunke S, Quintin L, Kasper L, Jacobsen I, Richter M, Hiller EST, et al Of mice, flies—and men? Comparing fungal infection models for large-scale screening efforts. Dis Model Mech. 2015; 8: p. 473– 486. 10.1242/dmm.019901 25786415 PMC4415897 · DOI
  78. [104]Wong C, Cheng K, Rigas B. Preclinical predictors of anticancer drug efficacy: critical assessment with emphasis on whether nanomolar potency should be required of candidate agents. J Pharmacol Exp Ther. 2012; 341: p. 572– 578. 10.1124/jpet.112.191957 22448039 PMC3362883 · DOI
  79. [105]Holbeck S, Collins J, Doroshow J. Analysis of Food and Drug Administration-approved anticancer agents in the NCI60 panel of human tumour cell lines. Mol Cancer Ther.; 9: p. 1451– 1460. 10.1158/1535-7163.MCT-10-0106 20442306 PMC2868078 · DOI
  80. [106]Petrone P, Simms B, Nigsch F, Lounkine E, Kutchukian P, Cornett A, et al Rethinking molecular similarity: comparing compounds on the basis of biological activity. ACS Chem Biol. 2012; 7: p. 1399– 1409. 10.1021/cb3001028 22594495 · DOI
  81. [107]Macleod M, Lawson McLean A, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, et al Risk of bias in reports of in vivo research: A focus for improvement. PLOS Biol. 2015; 13(10): p. e1002273 10.1371/journal.pbio.1002273 26460723 PMC4603955 · DOI · Paper page

Cited by

Sorted by citations

Authors
Scannell, J. W., & Bosley, J.
Affiliations
University of Oxford (GB)
Corresponding author
  • Scannell, Jack W.
  • Bosley, Jim
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2016-02-10
Volume
11
Issue
2
Pages
e0147215
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0147215
PMC
PMC4749240
OpenAlex
W2267520918
Cited by
279
References
15
Citations by year
2026: 19 · 2025: 16 · 2024: 25 · 2023: 39 · 2022: 18 · 2021: 43 · 2020: 19 · 2019: 24 · 2018: 24 · 2017: 34 · 2016: 18
FWCI
64.87
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Topics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Pharmaceutical Economics and Policy
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
replication crisis, drug discovery, R&D productivity, predictive validity, screening model, disease models, decision-theoretic model, research and development costs, measurement space, industrial research
MeSH terms
Biopharmaceutics, Cost-Benefit Analysis, Decision Theory, Efficiency, False Positive Reactions, Humans, Models, Theoretical, Quality Control, Research, Reproducibility of Results, Drug Discovery, High-Throughput Screening Assays

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review