Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 901–1,000 of 2,889
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
Smith, M. T., Lau, T., Wallace, V. C. J., Wyse, B. D., & Rice, A. S. C. (2014) · Behavioural Pharmacology · doi:10.1097/fbp.0000000000000025 · Cited by 33
Unvalidated antibodies and misleading results
Rimm, D. L., Schalper, K. A., & Pusztai, L. (2014) · Breast Cancer Research and Treatment · doi:10.1007/s10549-014-3061-0 · Cited by 29
“Poorly validated antibodies may provide misleading information and contribute to the much larger issue of significant non-reproducibility of published research results [2, 7].”
60 YEARS OF NEUROENDOCRINOLOGY: Biology of human craniopharyngioma: lessons from mouse models
Martínez-Barberá, J. P. (2015) · Journal of Endocrinology · doi:10.1530/joe-15-0145 · Cited by 30
Enhancing the Efficacy of Radiation Therapy: Premises, Promises, and Practicality
Coleman, C. N., Lawrence, T. S., & Kirsch, D. G. (2014) · Journal of Clinical Oncology · doi:10.1200/jco.2014.57.3865 · Cited by 28
“The problem of reproducibility of preclinical studies from academic laboratories has received much needed attention.(35,36) On the basis of that information and the so-called null trials workshop, the Radiation Research Program is completing a detailed study of the methodology used in preclinical radiation modifier st…”
MethodS-100 Negative Granular Cell Tumor of the Oral Cavity
Solomon, L. W., & Vélez, I. (2015) · Head and Neck Pathology · doi:10.1007/s12105-015-0673-6 · Cited by 28
“Antibodies from different laboratories can have dramatically different results [46].”
ResultSocietal challenges of precision medicine: Bringing order to chaos
Salgado, R., Moore, H. M., Martens, J. W. M., Lively, T., Malik, S. M., McDermott, U., Michiels, S., Moscow, J. A., Tejpar, S., McKee, T. C., et al. (2017) · European Journal of Cancer · doi:10.1016/j.ejca.2017.07.028 · Cited by 29
“The lack of such standardised tissue collection processes for target validation and drug development may be one of the contributory factors explaining why not all of academic landmark studies in this area can be reproduced [27].”
BackgroundSun, Q., Welsh, K. J., Bruns, D. E., Sacks, D. B., & Zhao, Z. (2019) · Clinical Chemistry · doi:10.1373/clinchem.2019.309575 · Cited by 29
“Many initial clinical studies show promising results, but different or even contradictory findings are seen in subsequent studies (7, 8).”
MethodResultGrabowski, M., Cymborowski, M., Porebski, P. J., Osinski, T., Shabalin, I. G., Cooper, D. R., & Minor, W. (2019) · Structural Dynamics · doi:10.1063/1.5128672 · Cited by 28
Schüchner, S., Behm, C., Mudrak, I., & Ogris, E. (2020) · Science Signaling · doi:10.1126/scisignal.aax9730 · Cited by 29
Critical Analysis of Non-Thermal Plasma-Driven Modulation of Immune Cells from Clinical Perspective
Smolková, B., Frtús, A., Uzhytchak, M., Lunová, M., Kubinová, Š., Dejneka, A., & Lunov, O. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21176226 · Cited by 31
“Specifically, in many cases, biomedical studies take shortcuts around the used methodology, resulting in devastating consequences [153,154].”
BackgroundYao, F., Yan, C., Zhang, Y., Shen, L., Zhou, D., & Ni, J. (2020) · Journal of Proteomics · doi:10.1016/j.jprot.2020.103991 · Cited by 33
Improving energy research practices: guidance for transparency, reproducibility and quality
Huebner, G., Fell, M., & Watson, N. (2021) · Buildings and Cities · doi:10.5334/bc.67 · Cited by 22
Functionalized polymeric patch for localized oxaliplatin delivery to treat gastric cancer
Bandi, S. P., & Venuganti, V. V. K. (2021) · Materials Science and Engineering C · doi:10.1016/j.msec.2021.112302 · Cited by 28
GGM Knockoff Filter: False Discovery Rate Control for Gaussian Graphical Models
Li, J., & Maathuis, M. H. (2021) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12430 · Cited by 46
“To obtain a reliable outcome and alleviate reproducibility issues (see, e.g. Baker, 2016; Begley & Ellis, 2012; Ioannidis, 2005), it is important to control the number of false positive edges.”
BackgroundHamilton, D. G., Page, M. J., Finch, S. J., Everitt, S., & Fidler, F. M. (2022) · BMC Medicine · doi:10.1186/s12916-022-02644-2 · Cited by 51
“In modern medical research, such concerns have sparked several influential research initiatives in pre-clinical medicine and cancer biology which have greatly reshaped our understanding of the extent and causes of irreproducible research [44–48] — an issue which is of particular interest to the medical research indust…”
BackgroundPardhi, E. R., Yadav, R., Chaurasiya, A., Madan, J., Guru, S. K., Singh, S. B., & Mehra, N. K. (2023) · Life Sciences · doi:10.1016/j.lfs.2023.121771 · Cited by 30
The methodological quality of animal research in critical care: the public face of science
Bara, M., & Joffe, A. R. (2014) · Annals of Intensive Care · doi:10.1186/s13613-014-0026-8 · Cited by 31
“Given the generally poor translation rate of AR to human medicine [6-8,13-18,27,55,56] (e.”
Maniadis, Z., Tufano, F., & List, J. A. (2015) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0193-230620150000018008 · Cited by 33
“…undertaking the replication of eighteen prominently published studies in the field of ‘psychology and economics’.14 It has also been proposed that the editors themselves should make direct replication a necessary prerequisite for publication (Nosek, Spies and Motyl, 2012; Begley and Ellis, 2012).”
MethodBorries, C., Sandel, A. A., Koenig, A., Fernández‐Duque, E., Kamilar, J. M., Amoroso, C. R., Barton, R. A., Bray, J., Fiore, A. D., Gilby, I. C., et al. (2016) · Evolutionary Anthropology Issues News and Reviews · doi:10.1002/evan.21502 · Cited by 30
Barnes, R. O., Shea, K. E., & Watson, P. H. (2016) · Biopreservation and Biobanking · doi:10.1089/bio.2016.0021 · Cited by 29
Volunteer Science: An Online Laboratory for Experiments in Social Psychology
Radford, J. P., Pilny, A., Reichelmann, A. V., Keegan, B., Welles, B. F., Hoye, J., Ognyanova, K., Meleis, W. M., & Lazer, D. M. J. (2016) · Social Psychology Quarterly · doi:10.1177/0190272516675866 · Cited by 39
“Recommendations for addressing the replication crisis involve increasing sample sizes, sharing data and study materials, and performing independent verification (Begley and Ellis 2012; Ioannidis 2005; Pasher and Wagenmakers 2012).”
BackgroundStatistical contributions to bioinformatics: Design, modelling, structure learning and integration
Morris, J. S., & Baladandayuthapani, V. (2017) · Statistical Modelling · doi:10.1177/1471082x17698255 · Cited by 32
“The field of multi-platform genomics has been plagued with lack of reproducibility (Ioannidis et al., 2009; Begley and Ellis, 2012), much of which can be explained by two primary factors: (a) sensitivity of the technologies to varying experimental leading to poorly documented and sometimes flawed…”
BackgroundPhenotypic Screening in C. elegans as a Tool for the Discovery of New Geroprotective Drugs
Bulterijs, S., & Braeckman, B. P. (2020) · Pharmaceuticals · doi:10.3390/ph13080164 · Cited by 33
Bloomfield, R. J., Rennekamp, K. M., & Steenhoven, B. A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3118687 · Cited by 28
Wyndham, J. M., Weisenberg, N., McCarty, C. A., Goldman, G. T., Brown, M., & Borenstein, J. (2022) · doi:10.1126/aaas.ade8203 · Cited by 32
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28
“Amgen researchers declared that they had been unable to reproduce the findings in 47 of 53 “landmark” cancer papers (Begley and Ellis 2012; Baker 2016b)”
BackgroundA New Account of Replication in the Experimental Life Sciences
Guttinger, S. (2019) · Philosophy of Science · doi:10.1086/703555 · Cited by 27
Dichtl, S., Posch, W., & Wilflingseder, D. (2024) · European Journal of Immunology · doi:10.1002/eji.202250356 · Cited by 39
True, L. D. (2014) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-013-1531-0 · Cited by 27
“However, at least 50 % of the data in publications reporting the types of preclinical studies on which tissuebased biomarker assays is based are not reproducible [22].”
Rocha, T. B.-M., Hutz, M. H., Salatino‐Oliveira, A., Genro, J. P., Polanczyk, G. V., Sato, J. R., Wehrmeister, F. C., Barros, F. C., Menezes, A. M. B., Rohde, L. A., et al. (2015) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2015.14070896 · Cited by 29
Cramond, F., Irvine, C., Liao, J., Howells, D., Sena, E., Currie, G., & Macleod, M. (2016) · Scientometrics · doi:10.1007/s11192-016-1964-8 · Cited by 27
“Poor replication of in vitro molecular and cellular biology studies has also been reported (Begley and Ellis 2012; Prinz et al. 2011) and linked in part to poor descriptions of the experimental and analytical details.”
BackgroundMapping ethical and social aspects of cancer biomarkers
Blanchard, A. (2016) · New Biotechnology · doi:10.1016/j.nbt.2016.06.1458 · Cited by 27
“Both groups benefit from improved understanding of patients’ concerns” [21].”
BackgroundResultInfluentialWu, J., Qu, Y., Shi, K., et al. (2018) · Chinese Chemical Letters · doi:10.1016/j.cclet.2018.10.004 · Cited by 27
Kessler, M. D., Bateman, N. W., Conrads, T. P., Maxwell, G. L., Hotopp, J. C. D., & O’Connor, T. D. (2019) · Cancer · doi:10.1002/cncr.32020 · Cited by 29
Centralized scientific communities are less likely to generate replicable results
Danchev, V., Rzhetsky, A., & Evans, J. A. (2019) · eLife · doi:10.7554/elife.43094 · Cited by 27
Six Laws of Open Source Drug Discovery
Todd, M. H. (2019) · ChemMedChem · doi:10.1002/cmdc.201900565 · Cited by 30
Should We Strive to Make Science Bias-Free? A Philosophical Assessment of the Reproducibility Crisis
Hudson, R. G. S. (2021) · Journal for General Philosophy of Science · doi:10.1007/s10838-020-09548-w · Cited by 33
(2017) · eLife · doi:10.7554/elife.23693 · Cited by 27
“In recent years, however, the validity of published findings in a number of areas of scientific research, including cancer research, have been called into question (Begley and Ellis, 2012; Baker, 2016).”
BackgroundKousholt, B. S., Præstegaard, K. F., Stone, J., Thomsen, A. F., Johansen, T. T., Ritskes-Hoitinga, M. J., & Wegener, G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275962 · Cited by 29
Innovating Aging: Promises and Pitfalls on the Road to Life Extension
Vijg, J., & de Grey, A. D. N. J. (2014) · Gerontology · doi:10.1159/000357670 · Cited by 35
Porru, M., Artuso, S., Salvati, E., Bianco, A., Franceschin, M., Diodoro, M. G., Passeri, D., Orlandi, A., Savorani, F., D’Incalci, M., et al. (2015) · Molecular Cancer Therapeutics · doi:10.1158/1535-7163.mct-15-0253 · Cited by 27
“Despite many antineoplastic compounds showed favorable tumor responses in preclinical studies, more than 95% of these therapeutics failed to confirm efficacy in clinical trials and many factors are responsible for this high failure rate (23, 24).”
BackgroundCould the Quality of Published Ecotoxicological Research Be Better?
Harris, C. A., & Sumpter, J. P. (2015) · Environmental Science & Technology · doi:10.1021/acs.est.5b01465 · Cited by 27
The role of anticitrullinated protein antibodies in the early stages of rheumatoid arthritis
Dekkers, J., Toes, R. E. M., Huizinga, T. W. J., & van der Woude, D. (2016) · Current Opinion in Rheumatology · doi:10.1097/bor.0000000000000277 · Cited by 31
“It is interesting that no other positive data have been reported allowing that these two positive papers are similar as the many nonreplicated preclinical papers in other fields such as oncology [60].”
BackgroundCancer drug discovery: recent innovative approaches to tumor modeling
Lovitt, C. J., Shelper, T., & Avery, V. M. (2016) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2016.1214562 · Cited by 31
“discovery assays is increasing the diversity of cell lines used in experiments to be more representative of the patient population [11].”
BackgroundIrreproducible biology research costs put at $28 billion per year
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.17711 · Cited by 29
Hunniford, V. T., Grudniewicz, A., Fergusson, D. A., Montroy, J., Grigor, E., Lansdell, C., Lalu, M. M., & On behalf of The Canadian Critical Care Translational Biology Group (2023) · eLife · doi:10.7554/elife.76300 · Cited by 26
“…the quality of the design, conduct, and reporting of preclinical studies may lead to a reduction in research waste ( Chalmers et al., 2014 ; Ioannidis et al., 2014 ), as well as increase their utility in informing the development of novel therapies ( Begley and Ellis, 2012 ; Langley et al., 2017 ).”
Bioinformatic approaches to augment study of epithelial-to-mesenchymal transition in lung cancer
Beck, T. N., Chikwem, A. J., Solanki-Patel, N., & Golemis, E. A. (2014) · Physiological Genomics · doi:10.1152/physiolgenomics.00062.2014 · Cited by 29
“The most likely explanation for the inconsistent drug response data is the use of slightly different assays and experimental protocols, unfortunately a common issue in highthroughput studies (24).”
BackgroundEstrogen receptor alpha gene amplification in breast cancer: 25 years of debate
Holst, F. (2016) · World Journal of Clinical Oncology · doi:10.5306/wjco.v7.i2.160 · Cited by 27
Reproducible and transparent research practices in published neurology research
Rauh, S., Torgerson, T., Johnson, A. L., Pollard, J., Tritz, D. J., & Vassar, M. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-020-0091-5 · Cited by 18
Statistical Significance Testing for Natural Language Processing
Dror, R., Peled-Cohen, L., Shlomov, S., & Reichart, R. (2020) · Synthesis lectures on human language technologies · doi:10.2200/s00994ed1v01y202002hlt045 · Cited by 75
Storey, J., Gobbetti, T., Olzinski, A. R., & Berridge, B. R. (2021) · ILAR Journal · doi:10.1093/ilar/ilac004 · Cited by 31
Construction of multiple concentration gradients for single-cell level drug screening
Shen, S., Zhang, F., Zhang, Y., Li, Y., Niu, Y., Pang, L., & Wang, J. (2023) · Microsystems & Nanoengineering · doi:10.1038/s41378-023-00516-0 · Cited by 28
Scientific misconduct: the dark side of science
Carafoli, E. (2015) · RENDICONTI LINCEI · doi:10.1007/s12210-015-0415-4 · Cited by 38
“Hence the strong temptation not to include suspicious data and/or manipulate them’’ (Begley and Ellis 2012).”
BackgroundResultOptimizing drug development in oncology by clinical trial simulation: Why and how?
Gal, J., Milano, G. A., Ferrero, J.-M., Saâda-Bouzid, E., Viotti, J., Chabaud, S., Gougis, P., Tourneau, C. L., Schiappa, R., Paquet, A. C., et al. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx055 · Cited by 34
The Pursuit of Therapeutic Biomarkers with High-Throughput Cancer Cell Drug Screens
Williams, S. P., & McDermott, U. (2017) · Cell chemical biology · doi:10.1016/j.chembiol.2017.06.011 · Cited by 30
Rosell, A., Moser, B., Hisada, Y., Chinthapatla, R., Lian, G., Yang, Y., Flick, M. J., & Mackman, N. (2020) · Research and Practice in Thrombosis and Haemostasis · doi:10.1002/rth2.12363 · Cited by 26
Hsu, C.-W., Huang, R., Khuc, T., Shou, D., Bullock, J., Grooby, S., Griffin, S., Zou, C., Little, A. S., Astley, H. M., et al. (2016) · Oncotarget · doi:10.18632/oncotarget.6995 · Cited by 29
Scientific rigor through videogames
Treuille, A., & Das, R. (2014) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2014.08.005 · Cited by 22
Menke, J., Eckmann, P., Ozyurt, I. B., Roelandse, M., Anderson, N., Grethe, J., Gamst, A., & Bandrowski, A. (2022) · Journal of Medical Internet Research · doi:10.2196/37324 · Cited by 25
“However, over the last decade, numerous reports on research irreproducibility have shed light on a lingering problem, one that is proving to be both troublesome and costly [1-5].”
BackgroundResultImproving the Drug Development Process: More Not Less Randomized Trials
Djulbegović, B., Hozo, I., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2013.283742 · Cited by 32
Epithelial cancers in the post-genomic era: should we reconsider our lifestyle?
Holly, J. M. P., Zeng, L., & Perks, C. M. (2013) · Cancer and Metastasis Reviews · doi:10.1007/s10555-013-9445-5 · Cited by 31
“The variability, lack of rigour and lack of reproducibility of preclinical cancer research was highlighted by a recent report from the oncology laboratory of a prominent biotechnology firm where they tried to reproduce the findings from 53 landmark publications and found that the findings of only 6 of these studies co…”
BackgroundPreclinical data reproducibility for R&D—the challenge for neuroscience
Steckler, T. (2015) · Psychopharmacology · doi:10.1007/s00213-014-3836-3 · Cited by 25
“A more recent analysis from Amgen (Begley and Ellis 2012) suggested that those non-reproducible studies may have an even stronger impact on the field than those that can be reproduced, possibly because the more remarkable and exciting findings are reported in higher impact journals.”
BackgroundQuantifying differences in cell line population dynamics using CellPD
Juarez, E. F., Lau, R., Friedman, S. H., Ghaffarizadeh, A., JONCKHEERE, E. A., Agus, D. B., Mumenthaler, S. M., & Macklin, P. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0337-5 · Cited by 25
Karpen, S. C. (2017) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe6570 · Cited by 25
“Amgen was only able to replicate six of its 53 landmark cancer studies.(8) Bayer could only replicate 25% of the 67 studies that it attempted.”
BackgroundYosten, G. L. C., Adams, J. C., Bennett, C. N., Bunnett, N. W., Scheman, R., Sigmund, C. D., Yates, B. J., Zucker, I. H., & Samson, W. K. (2018) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.00274.2018 · Cited by 25
Simon, S. M., Meldrum, H. M., Ndung’u, E., & Ledley, F. D. (2018) · CBE—Life Sciences Education · doi:10.1187/cbe.17-03-0057 · Cited by 26
Measurement in biology is methodized by theory
Montévil, M. (2019) · Biology & Philosophy · doi:10.1007/s10539-019-9687-x · Cited by 28
Variation in diagnostic NAFLD/NASH read-outs in paired liver samples from rodent models
Jensen, V. S., Tveden‐Nyborg, P., Zacho-Rasmussen, C., Quaade, M. L., Ipsen, D. H., Hvid, H., Fledelius, C., Wulff, E. M., & Lykkesfeldt, J. (2019) · Journal of Pharmacological and Toxicological Methods · doi:10.1016/j.vascn.2019.106651 · Cited by 28
“…preclinical studies within several fields of research has emphasized the need for increased attention as to how to reduce variation and bias, and improve the validity of reported findings (Baker, 2016; Begley & Ellis, 2012; Goodman, Fanelli, & Ioannidis, 2016; Prinz, Schlange, & Asadullah, 2011).”
BackgroundHarnessing data science to advance radiation oncology
Vogelius, I. R., Petersen, J., & Bentzen, S. M. (2020) · Molecular Oncology · doi:10.1002/1878-0261.12685 · Cited by 29
“…research chain has limitations that are well described in the literature, including the reproducibility crisis and, equally important but less appreciated, the risk of falsely rejecting a good target prior to entering the human testing phase (Bau-mann et al. , 2001; Begley and Ellis, 2012).”
BackgroundMandal, R. K., Denny, J. E., Waide, M. L., Li, Q., Bhutiani, N., Anderson, C. D., Baby, B. V., Jala, V. R., Egilmez, N. K., & Schmidt, N. W. (2020) · BMC Biology · doi:10.1186/s12915-020-00810-7 · Cited by 29
Pulverer, B. (2015) · The EMBO Journal · doi:10.15252/embj.201570090 · Cited by 24
Hylander, B. L., Repasky, E. A., & Sexton, S. (2022) · Animals · doi:10.3390/ani12030371 · Cited by 30
“Two studies which drew attention to this issue were by Bayer [2] and Amgen [10].”
BackgroundShang, Z., Li, D., Chen, J., Wang, R., Wang, M., Zhang, B., Wang, X., & Wanyan, P. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.855309 · Cited by 28
“In conclusion, in the field of experimental research, it is necessary to take measures to promote data sharing and develop policies to encourage and require journals to publish studies with negative or neutral results in order to avoid the “file drawer” effect and reduce the impact of publication bias (51).”
BackgroundImproving the predictive value of interventional animal models data
Zeiss, C. J. (2014) · Drug Discovery Today · doi:10.1016/j.drudis.2014.10.015 · Cited by 25
Flaws in animal studies exploring statins and impact on meta‐analysis
Moja, L., Pecoraro, V. L., Ciccolallo, L., Dall’Olmo, L., Virgili, G., & Garattini, S. K. (2014) · European Journal of Clinical Investigation · doi:10.1111/eci.12264 · Cited by 26
Carp, J. (2013) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-013-0188-0 · Cited by 24
Brücher, B. L. D. M., Li, Y., Schnabel, P. A., Däumer, M., Wallace, T. J., Kube, R., Zilberstein, B., Steele, S. R., Voskuil, J. L. A., & Jamall, I. S. (2016) · Clinical and Translational Medicine · doi:10.1186/s40169-016-0093-6 · Cited by 25
“However, we need to remind ourselves that less than 20 % of highly ranked so-called “landmark” publications have been shown to be irreproducible although these are among the most cited references [158].”
BackgroundGetting personal with the “reproducibility crisis”: interviews in the animal research community
Fitzpatrick, B. G., Koustova, E., & Wang, Y. (2018) · Lab Animal · doi:10.1038/s41684-018-0088-6 · Cited by 31
Immune Relevant and Immune Deficient Mice: Options and Opportunities in Translational Research
Radaelli, E., Santagostino, S. F., Sellers, R. S., & Brayton, C. F. (2018) · ILAR Journal · doi:10.1093/ilar/ily026 · Cited by 28
Organoid in colorectal cancer: progress and challenges
Ji, D., & Wu, A. (2020) · Chinese Medical Journal · doi:10.1097/cm9.0000000000000882 · Cited by 28
Retractions of scientific publications: responsibility and accountability
Katavić, V. (2014) · Biochemia Medica · doi:10.11613/bm.2014.024 · Cited by 31
“To test/confirm published results a group of researchers tried to replicate the results from 53 published “landmark” studies in biomedicine, and managed to confirm only 11% (25).”
ResultMacleod, M. R., & The NPQIP Collaborative group (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/187245 · Cited by 25
Examining the predictive validity of NIH peer review scores
Lindner, M. D., & Nakamura, R. K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126938 · Cited by 24
Data publication consensus and controversies
Kratz, J. E., & Strasser, C. A. (2014) · F1000Research · doi:10.12688/f1000research.3979.1 · Cited by 25
Extracting physics of life at the molecular level: A review of single-molecule data analyses
Colomb, W., & Sarkar, S. K. (2015) · Physics of Life Reviews · doi:10.1016/j.plrev.2015.01.017 · Cited by 28
Measures of biosimilarity in monoclonal antibodies in oncology: the case of bevacizumab
Ebbers, H. C., van Meer, P. J. K., Moors, E. H. M., Mantel-Teeuwisse, A. K., Leufkens, H. G. M., & Schellekens, H. (2013) · Drug Discovery Today · doi:10.1016/j.drudis.2013.05.004 · Cited by 26
Barriers to advancing nanotechnology to better improve and translate nanomedicines
Wang, Y., & Grainger, D. W. (2014) · Frontiers of Chemical Science and Engineering · doi:10.1007/s11705-014-1442-x · Cited by 25
Birds of a Feather: Applied Behavior Analysis and Quality of Life
Gambrill, E. D. (2012) · Research on Social Work Practice · doi:10.1177/1049731512465775 · Cited by 29
“Seventy percent of cancer studies cannot be replicated (Begley & Ellis, 2012).”
BackgroundGlial cells as drug targets: What does it take?
Möller, T., & Boddeke, H. W. G. M. (2016) · Glia · doi:10.1002/glia.22993 · Cited by 25
Walters, C., Harter, Z. J., Wayant, C., Vo, N., Warren, M. H., Chronister, J., Tritz, D. J., & Vassar, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033962 · Cited by 20
The biology and engineered modeling strategies of cancer-nerve crosstalk
Gregory, E., Dugan, R., David, G., & Song, Y. H. (2020) · Biochimica et Biophysica Acta (BBA) - Reviews on Cancer · doi:10.1016/j.bbcan.2020.188406 · Cited by 23
A community-led initiative for training in reproducible research
Auer, S., Haelterman, N. A. T., Weissgerber, T. L., Erlich, J. C., Susilaradeya, D., Julkowska, M., Gazda, M. A., Schwessinger, B., Jadavji, N. M., Reproducibility for Everyone Team, et al. (2021) · eLife · doi:10.7554/elife.64719 · Cited by 28
Are business and management journals anti-replication? An analysis of editorial policies
Tipu, S. A. A., & Ryan, J. C. (2021) · Management Research Review · doi:10.1108/mrr-01-2021-0050 · Cited by 26
What do cellular responses to acidity tell us about cancer?
Błaszczak, W., & Swietach, P. (2021) · Cancer and Metastasis Reviews · doi:10.1007/s10555-021-10005-3 · Cited by 31
Zhang, C., Deng, R., Zhang, G., He, X., Chen, H., Chen, B., Wan, L., & Kang, X. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.847444 · Cited by 28
Reproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 43
Singh, D., Deosarkar, S. P., Cadogan, E. B., Flemington, V., Bray, A., Zhang, J., Reiserer, R. S., Schaffer, D. K., Gerken, G. B., Britt, C. M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001624 · Cited by 26
von Hippel, P. T. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211072525 · Cited by 30
“Eighty-nine percent of “landmark” findings in preclinical cancer research (Begley & Ellis, 2012), 32% of highly cited clinical trials in medicine (Ioannidis, 2005), and 60% of experiments published in top psychology journals (Open Science Collaboration, 2015) have failed to produce similar results…”
ResultRuthenium Complexes as Potential Cancer Cell Growth Inhibitors for Targeted Chemotherapy
Katheria, S. (2022) · ChemistrySelect · doi:10.1002/slct.202201645 · Cited by 26
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.