Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Radiomics: Images Are More than Pictures, They Are Data
Gillies, R. J., Kinahan, P. E., & Hricak, H. (2016) · Radiology · doi:10.1148/radiol.2015151169 · Cited by 8,421
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
Reproducible brain-wide association studies require thousands of individuals
Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Hendrickson, T. J., et al. (2022) · Nature · doi:10.1038/s41586-022-04492-9 · Cited by 1,928
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
“…of flawed methodology and applications in other fields typically prompts calls for improvement rather than demolition of the field (e.g., papers noting serious problems with pre-clinical cancer studies have not called for an end to pre-cancer clinical research but rather for its improvement 18 ).”
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
How to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
“…several large pharmaceutical companies revealed that their efforts to replicate exciting preclinical findings from published academic studies in cancer biology were only rarely verifying the original results (Begley & Ellis, 2012; see also Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
“Likewise, Begley and Ellis (2012) reported an effort by Amgen to replicate 53 landmark studies of basic research for cancer treatment.”
BackgroundEV-TRACK: transparent reporting and centralizing knowledge in extracellular vesicle research
EV-TRACK Consortium, Van Deun, J., Mestdagh, P., Agostinis, P., Akay, Ö., Anand, S., Anckaert, J., Martinez, Z. A., Baetens, T., Beghein, E., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4185 · Cited by 1,284
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232
Tumour heterogeneity in the clinic
Bedard, P. L., Hansen, A. R., Ratain, M. J., & Siu, L. L. (2013) · Nature · doi:10.1038/nature12627 · Cited by 1,211
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Self-citationHou, K., Xue, C., & Zhang, L. (2020) · The Review of Financial Studies · doi:10.1093/rfs/hhy131 · Cited by 989
The Smart Drug Delivery System and Its Clinical Potential
Liu, D., Yang, F., Xiong, F., & Gu, N. (2016) · Theranostics · doi:10.7150/thno.14858 · Cited by 1,066
“Though there are tremendous amount of publications about advanced DDSs, little has been published on the relevance of safety and efficacy evaluation of smart DDSs in animals to be predictive clinical effect in humans [8, 193].”
BackgroundThe economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055
“Source: Begley and Ellis [6], Prinz et al.”
BackgroundAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
“And indeed, recent results from biomedical and cancer research suggest that replication rates are lower than 50%, with some as low as 11% (Begley & Ellis, 2012; Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
ResultThe ARRIVE guidelines 2019: updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703181 · Cited by 1,036
Non-coding RNAs as drug targets
Matsui, M., & Corey, D. R. (2017) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2016.117 · Cited by 1,032
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,005
“Scientists at the biotechnology firm Amgen attempted to replicate 53 landmark studies in hematology and oncology, but were able to confirm the original findings in only six cases, implying an apparent failure rate of 89% (Begley & Ellis, 2012).”
BackgroundMachine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
Moving Forward: Challenges and Directions for Psychopathological Network Theory and Methodology
Fried, E. I., & Cramer, A. O. J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617705892 · Cited by 938
“In cancer research, an investigation showed that only 6 out of 53 landmark papers could be replicated (Begley & Ellis, 2012), and a recent study concluded that the irreproducibility of preclinical research in the life sciences surpasses 50%, leading to about US$28,000,000,000 spent on research in…”
BackgroundOpportunities and challenges in phenotypic drug discovery: an industry perspective
Moffat, J. G., Vincent, F., Lee, J. A., Eder, J., & Prunotto, M. (2017) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2017.111 · Cited by 939
Principles for Valid Histopathologic Scoring in Research
Gibson-Corley, K. N., Olivier, A. K., & Meyerholz, D. K. (2013) · Veterinary Pathology · doi:10.1177/0300985813485099 · Cited by 936
Structure and dynamics of molecular networks: A novel paradigm of drug discovery
Csermely, P., Korcsmáros, T., Kiss, H. J. M., London, G., & Nussinov, R. (2013) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2013.01.016 · Cited by 892
“Currently we often lack these high-quality datasets especially on system dynamics (Henney & Superti-Furga, 2008; Prinz et al., 2011; Begley & Ellis, 2012; Landis et al., 2012).”
BackgroundResultInfluentialIncreasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880
“The importance of replication and reproducibility has recently been exemplified through studies showing that scientific papers commonly leave out experimental details essential for reproduction [5], studies showing difficulties with replicating published experimental results [6], an increase in retracted papers [7], a…”
BackgroundThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
“Publication bias can have dire consequences, as illustrated recently by clear failures to replicate medical findings (Begley & Ellis, 2012; Prinz, Schlange & Asadullah, 2011).”
BackgroundReproducibility crisis: Blame it on the antibodies
Baker, M. (2015) · Nature · doi:10.1038/521274a · Cited by 830
Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2019.01.012 · Cited by 813
“Overall, it is clear that the consequences of bad statistics can be dramatic – for example when wrong cures for cancer are identified at the pre-clinical stage of research, and are then passed on to the clinical trial phase (Begley and Ellis, 2012).”
BackgroundImmPort: disseminating data to the public for the future of immunology
Bhattacharya, S., Andorf, S., Gomes, L., Dunn, P., Schaefer, H., Pontius, J., Berger, P., Desborough, V., Smith, T., Campbell, J., et al. (2014) · Immunologic Research · doi:10.1007/s12026-014-8516-1 · Cited by 806
“Begley and Ellis [16] noted in Nature on their work at Amgen to try to reproduce 53 landmark studies, with only 11 % of those studies able to be reproduced.”
BackgroundHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
“Pharmaceutical industry teams have championed reproducibility in preclinical research [56,57] because they depend on pre-clinical academic investigations accurately pinpointing useful drug targets.”
BackgroundDeep learning in medical imaging and radiation therapy
Sahiner, B., Pezeshk, A., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., Summers, R. M., & Giger, M. L. (2019) · Medical Physics · doi:10.1002/mp.13264 · Cited by 740
“A 2012 study of 53 landmark papers in basic cancer research was able to replicate the original results of just six of these studies.(370) Moreover, a study reviewing radiomics using texture features, that is, “conventional” radiomics, for the prediction of survival, found that all of the results of nine published stud…”
ResultIs the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
“Writing in Nature, Begley and Ellis (2012) described the firsthand experience of the Amgen Corporation over a 10-year period in attempting to build drug development programs upon 53 “landmark” published studies in preclinical (basic) cancer research.”
BackgroundDrug discovery and development: Role of basic biological research
Mohs, R. C., & Greig, N. H. (2017) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1016/j.trci.2017.10.005 · Cited by 673
Preclinical Mouse Cancer Models: A Maze of Opportunities and Challenges
Day, C.-P., Merlino, G., & Van Dyke, T. (2015) · Cell · doi:10.1016/j.cell.2015.08.068 · Cited by 676
Preclinical research: Make mouse studies work
Perrin, S. (2014) · Nature · doi:10.1038/507423a · Cited by 668
Current hurdles to the translation of nanomedicines from bench to the clinic
Đorđević, S., Gonzalez, M. M., Conejos-Sánchez, I., Carreira, B., Pozzi, S., Acúrcio, R. C., Satchi-Fainaro, R., Florindo, H. F., & Vicent, M. J. (2022) · Drug Delivery and Translational Research · doi:10.1007/s13346-021-01024-2 · Cited by 657
Reproducibility: Standardize antibodies used in research
Bradbury, A., & Plückthun, A. (2015) · Nature · doi:10.1038/518027a · Cited by 647
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
“In a similar investigation, Begley and Ellis (2012) reported a meager 11% replication rate.”
BackgroundFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
“The concern over replication exists in many research domains, including advertising (Madden et al., 1995), biology (Begley & Ellis, 2012; Powell, 2007), economics (Anderson, Greene, McCullough, & Vinod, 2005; Dewald, Thursby, & Anderson, 1986; Kane, 1984), library sciences (Winters, 1996),…”
ResultInfluentialAnimal models of human disease: challenges in enabling translation
McGonigle, P., & Ruggeri, B. (2014) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.08.006 · Cited by 610
The ARRIVE guidelines 2.0: updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · The Journal of Physiology · doi:10.1113/jp280389 · Cited by 593
“…point, where poor reporting impacts on the development of therapeutics and irreproducible findings can spawn an entire field of research, or trigger clinical studies, subjecting patients to interventions unlikely to be effective (Scott et al. 2008, Begley & Ellis, 2012, Begley & Ioannidis, 2015).”
BackgroundFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
PREPARE: guidelines for planning animal research and testing
Smith, A. J., Clutton, R. E., Lilley, E., Hansen, K. E. A., & Brattelid, T. (2018) · Laboratory Animals · doi:10.1177/0023677217724823 · Cited by 587
Collins, B. C., Hunter, C. L., Liu, Y., Schilling, B., Rosenberger, G., Bader, S. L., Chan, D. W., Gibson, B. W., Gingras, A.-C., Held, J. M., et al. (2017) · Nature Communications · doi:10.1038/s41467-017-00249-5 · Cited by 576
A brief history of nearly EV‐erything – The rise and rise of extracellular vesicles
Couch, Y., Buzàs, E. I., Di Vizio, D., Gho, Y. S., Harrison, P., Hill, A. F., Lötvall, J., Raposo, G., Stahl, P. D., Théry, C., et al. (2021) · Journal of Extracellular Vesicles · doi:10.1002/jev2.12144 · Cited by 566
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
“…from diverse fields suggests that when efforts are made to repeat or reproduce published research, the repeatability and reproducibility is dismal (Begley & Ellis, 2012; Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Hothorn & Leisch, 2011; Ioannidis et al., 2009; Prinz, Schlange, & Asadullah,…”
BackgroundProtein kinase C, an elusive therapeutic target?
Mochly-Rosen, D., Das, K., & Grimes, K. V. (2012) · Nature Reviews Drug Discovery · doi:10.1038/nrd3871 · Cited by 563
Power Analysis for Parameter Estimation in Structural Equation Modeling: A Discussion and Tutorial
Wang, Y. A., & Rhemtulla, M. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920918253 · Cited by 544
Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition)
Cossarizza, A., Chang, H., Radbruch, A., Abrignani, S., Addo, R., Akdis, M., Andrä, I., Andreata, F., Annunziato, F., Arranz, E., et al. (2021) · European Journal of Immunology · doi:10.1002/eji.202170126 · Cited by 531
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
“Subsequent studies of attempts to replicate medical findings yielded similarly gloomy results (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011).”
Phenotypic screening in cancer drug discovery — past, present and future
Moffat, J. G., Rudolph, J., & Bailey, D. (2014) · Nature Reviews Drug Discovery · doi:10.1038/nrd4366 · Cited by 507
Meta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMC Veterinary Research · doi:10.1186/s12917-020-02451-y · Cited by 496
Tumor Organoids as a Pre-clinical Cancer Model for Drug Discovery
Weeber, F., Ooft, S. N., Dijkstra, K. K., & Voest, E. E. (2017) · Cell Chemical Biology · doi:10.1016/j.chembiol.2017.06.012 · Cited by 486
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
From Big Data to Precision Medicine
Hulsen, T., Jamuar, S. S., Moody, A. R., Karnes, J. H., Varga, O., Hedensted, S., Spreafico, R., Hafler, D. A., & McKinney, E. F. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00034 · Cited by 459
“However, it has been reported that results from a concerning proportion of even the highest ranking scientific articles may not be reproduced [∼11% only were (30)].”
Pre-registration in social psychology—A discussion and suggested template
van 't Veer, A. E., & Giner-Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 439
“…2003; Turner, Matthews, Linardatos, Tell, & Rosenthal, 2008), and of low replication rates of published medical research in registered clinical trials (e.g., Begley & Ellis, 2012; Mullane & Williams, 2013; Prinz, Schlange & Asadullah, 2011), led to calls for greater openness in registration.”
BackgroundAssessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441
“Three recent studies reported attempts to replicate published findings in preclinical medicine [69–71].”
ResultMindfulness Meditation and Psychopathology
Wielgosz, J., Goldberg, S. B., Kral, T. R. A., Dunne, J. D., & Davidson, R. J. (2019) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093423 · Cited by 435
“For example, concerns about replicability and publication bias (CoronadoMontoya et al. 2016) are relevant across psychological (DeCoster et al. 2015) and medical research (Begley & Ellis 2012).”
BackgroundSzucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453
AI for life: Trends in artificial intelligence for biotechnology
Holzinger, A., Keiblinger, K., Holub, P., Zatloukal, K., & Müller, H. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.02.001 · Cited by 438
Big Data and Large Sample Size: A Cautionary Note on the Potential for Bias
Kaplan, R. M., Chambers, D. A., & Glasgow, R. E. (2014) · Clinical and Translational Science · doi:10.1111/cts.12178 · Cited by 421
Poynton, F. E., Bright, S. A., Blasco, S., Williams, D. C., Kelly, J. M., & Gunnlaugsson, T. (2017) · Chemical Society Reviews · doi:10.1039/c7cs00680b · Cited by 421
Limits to Personalized Cancer Medicine
Tannock, I. F., & Hickman, J. A. (2016) · New England Journal of Medicine · doi:10.1056/nejmsb1607705 · Cited by 415
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
“And, in cancer biology and related fields, two drug companies (Bayer and Amgen) reported failures to replicate findings from promising studies that could have led to new therapies (Prinz et al., 2011; Begley and Ellis, 2012).”
BackgroundMisinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Reproducibility of animal research in light of biological variation
Voelkl, B., Altman, N. S., Forsman, A., Forstmeier, W., Gurevitch, J., Jaric, I., Karp, N. A., Kas, M. J., Schielzeth, H., Van de Casteele, T., et al. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0313-3 · Cited by 390
Noble, S., Spann, M. N., Tokoglu, F., Shen, X., Constable, R. T., & Scheinost, D. (2017) · Cerebral Cortex · doi:10.1093/cercor/bhx230 · Cited by 377
ITK: enabling reproducible research and open science
McCormick, M., Liu, X., Jomier, J., Marion, C., & Ibanez, L. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00013 · Cited by 376
“The importance of restoring the practice of reproducibility verification in scientific research has resurged in recent years in light of worrisome findings of inconsistency, lack of quality and even fraud in what were otherwise considered to be high-quality publications (Begley and Ellis, 2012).”
Sala, E., Mema, E., Himoto, Y., Veeraraghavan, H., Brenton, J. D., Snyder, A., Weigelt, B., & Vargas, H. A. (2017) · Clinical Radiology · doi:10.1016/j.crad.2016.09.013 · Cited by 369
“Furthermore, when scientists at Amgen (Thousand Oaks, CA, USA) tried to replicate 53 landmark studies in the basic science of cancer, they were able to reproduce the original results of just six (41).”
BackgroundKeshari, K. R., & Wilson, D. M. (2014) · Chem. Soc. Rev. · doi:10.1039/c3cs60124b · Cited by 358
Little Fish, Big Data: Zebrafish as a Model for Cardiovascular and Metabolic Disease
Gut, P., Reischauer, S., Stainier, D. Y. R., & Arnaout, R. (2017) · Physiological Reviews · doi:10.1152/physrev.00038.2016 · Cited by 333
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialAnimal Models of Hypertension: A Scientific Statement From the American Heart Association
Lerman, L. O., Kurtz, T. W., Touyz, R. M., Ellison, D. H., Chade, A. R., Crowley, S. D., Mattson, D. L., Mullins, J. J., Osborn, J., Eirin, A., et al. (2019) · Hypertension · doi:10.1161/hyp.0000000000000090 · Cited by 329
Opportunities and challenges in the collection and analysis of digital phenotyping data
Onnela, J.-P. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0771-3 · Cited by 319
Next-generation sequencing data interpretation: enhancing reproducibility and accessibility
Nekrutenko, A., & Taylor, J. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3305 · Cited by 318
A checklist for maximizing reproducibility of ecological niche models
Feng, X., Park, D. S., Walker, C., Peterson, A. T., Merow, C., & Papeş, M. (2019) · Nature Ecology & Evolution · doi:10.1038/s41559-019-0972-5 · Cited by 323
Implementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
Liposomes used as a vaccine adjuvant-delivery system: From basics to clinical immunization
Wang, N., Chen, M., & Wang, T. (2019) · Journal of Controlled Release · doi:10.1016/j.jconrel.2019.04.025 · Cited by 308
“Fifth, the prevalence of low reproducibility of the currently published outcomes, especially, in drug development field [168], resulted from the inappropriate designs with little practical rationality combined with the positive result-oriented publishing bias and the misconception of “research for publication” [169],…”
BackgroundHenderson, V. C., Kimmelman, J., Fergusson, D., Grimshaw, J. M., & Hackam, D. G. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001489 · Cited by 302
“Various commentators have reported problems such as difficulty in replicating preclinical studies [5,6], publication bias [7], and the prevalence of methodological practices that result in threats to validity [8].”
BackgroundClinical trial design in the era of precision medicine
Fountzilas, E., Tsimberidou, A. M., Vo, H. H., & Kurzrock, R. (2022) · Genome Medicine · doi:10.1186/s13073-022-01102-1 · Cited by 305
“The success rate of randomized clinical oncology trials is ~38% [28, 29].”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.