Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 1,401–1,500 of 2,889
Scharm, M., & Waltemath, D. (2016) · F1000Research · doi:10.12688/f1000research.9379.1 · Cited by 12
“Repeated mentions of problems with replication and reproducibility 2–4 led to new standards, tools, and methods for the transfer of reproducible simulation studies 5–9 .”
MethodSchönbrodt, F. D., Maier, M., Heene, M., & Bühner, M. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000386 · Cited by 12
Dietrich, G., Krebs, J., Liman, L., Fette, G., Ertl, M., Kaspar, M., Störk, S., & Puppe, F. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0729-0 · Cited by 12
Hengst, J. A., Dick, T. E., Smith, C. D., & Yun, J. K. (2020) · Cancer Biology & Therapy · doi:10.1080/15384047.2020.1798696 · Cited by 12
“Consistent with previous reports,43-45 DMS (1),37 SKI-II (2),15,37 PF-543 (3) and FTY720 (7),38 all induced degradation of SphK1 relative to vehicle treated controls (Figure 2(b)).”
BackgroundInfluentialPassaro, A. P., Aydin, O., Saif, M. T. A., & Stice, S. L. (2021) · Scientific Reports · doi:10.1038/s41598-021-88675-w · Cited by 13
Lu, H. S., & Daugherty, A. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.856102 · Cited by 11
McGill, M. P., & Threadgill, D. W. (2023) · Journal of Clinical Investigation · doi:10.1172/jci173750 · Cited by 12
Advancing reproducibility can ease the ‘hard truths’ of synthetic biology
Lux, M. W., Strychalski, E. A., & Vora, G. J. (2023) · Synthetic Biology · doi:10.1093/synbio/ysad014 · Cited by 19
Challenges of Open Data in Medical Research
Floca, R. O. (2013) · Springer eBooks · doi:10.1007/978-3-319-00026-8_22 · Cited by 12
When Are Results Too Good to Be True?
Churchill, G. A. (2014) · Genetics · doi:10.1534/genetics.114.169912 · Cited by 11
Williams, R. (2015) · Circulation Research · doi:10.1161/circresaha.115.307532 · Cited by 13
BIO Comes Into the Cold to Adopt BRISQ
Simeon‐Dubach, D., & Moore, H. M. (2014) · Biopreservation and Biobanking · doi:10.1089/bio.2014.1241 · Cited by 11
Baldassarro, V. A., Dolci, L. S., Mangano, C., Giardino, L., Gualandi, C., Focarete, M. L., & Calzà, L. (2016) · BioResearch open access · doi:10.1089/biores.2016.0025 · Cited by 11
Maugeri‐Saccà, M., Barba, M., Vici, P., Pizzuti, L., Sergi, D., Catenaro, T., Lauro, L. D., Mottolese, M., Santini, D., Milella, M. S., et al. (2016) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2016.08.004 · Cited by 13
Reproducibility in Computational Science: A Case Study: Randomness of the Digits of Pi
Bailey, D. H., Borwein, J. M., Brent, R. P., et al. (2016) · Experimental Mathematics · doi:10.1080/10586458.2016.1163755 · Cited by 11
Modern Prometheus: editing the human genome with Crispr-Cas9
Kozubek, J. (2016) · CERN Document Server (European Organization for Nuclear Research) · Cited by 11
The unique status of first-in-human studies: strengthening the social value requirement
Habets, M. G. J. L., van Delden, J. J. M., & Bredenoord, A. L. (2016) · Drug Discovery Today · doi:10.1016/j.drudis.2016.11.016 · Cited by 16
Campbell, H., & Gustafson, P. (2019) · The American Statistician · doi:10.1080/00031305.2018.1555101 · Cited by 14
“…2017), recent large-scale efforts to reproduce published results in a number of different fields (economics, Camerer et al. 2016; psychology, OpenScienceCollaboration 2015; oncology, Begley and Ellis 2012), have also raised concerns about the reliability and reproducibility of published science.”
ResultData Sharing in Neurosurgery and Neurology Journals
Johnson, J. N., Hanson, K. A., Jones, C. A., Grandhi, R., Guerrero, J. R., & Rodriguez, J. S. (2018) · Cureus · doi:10.7759/cureus.2680 · Cited by 12
Wasted Money in United States Biomedical and Agricultural Animal Research
Keen, J. (2019) · Animal Experimentation: Working Towards a Paradigm Change · doi:10.1163/9789004391192_011 · Cited by 11
Lalu, M. M., Leung, G. J., Dong, Y., Montroy, J. R., Butler, C., Auer, R. C., & Fergusson, D. A. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-029475 · Cited by 11
A Reckless Guide to P-values: Local Evidence, Global Errors
Lew, M. J. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_286 · Cited by 17
“…would indeed help to minimise the number of times that pharmacologists make fools of themselves by reducing the number of false positive results in pharmacological journals and, consequently, reduce the number of faulty leads that fail to translate into a therapeutic [Begley and Ellis, 2012].”
BackgroundMeasurement practices in hallucinations research
Smailes, D., Alderson‐Day, B., Hazell, C. M., Wright, A. C., & Moseley, P. (2021) · Cognitive Neuropsychiatry · doi:10.1080/13546805.2021.1999224 · Cited by 14
“Over the past decade, psychology (like many other disciplines, such as oncology; Begley & Ellis, 2012) has faced a so-called ‘ replication crisis ’ , where many fi ndings have been shown to not be replicable (see Nosek et al., 2015).”
BackgroundIoannidis, J. P. A. (2019) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-019-09621-y · Cited by 10
The scientific method: pillar and pitfall of cancer research
Tu, S., Bilen, M. A., & Tannir, N. M. (2014) · Cancer Medicine · doi:10.1002/cam4.248 · Cited by 11
“Begley and Ellis reminded us that only 11% of our “landmark” research studies are reproducible [1].”
BackgroundAssessing the validity and reproducibility of genome-scale predictions
Sugden, L. A., Tackett, M., Savva, Y. A., Thompson, W. A., & Lawrence, C. E. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt508 · Cited by 11
“, 2010), oncology (Begley and Ellis, 2012), neuroscience (Button et al.”
BackgroundInfluentialAssessing the reproducibility of discriminant function analyses
Andrew, R. L., Albert, A., Renaut, S., Rennison, D. J., Bock, D. G., & Vines, T. (2015) · PeerJ · doi:10.7717/peerj.1137 · Cited by 14
“Submitted 13 February 2015 Accepted 8 July 2015 Published 4 August 2015 Corresponding author Tim Vines, vines@zoology.ubc.ca Academic editor Claus Wilke Additional Information and Declarations can be found on page 16 DOI 10.7717/peerj.1137 Copyright 2015 Andrew et al. Distributed under Creative Commons CC-BY 3.0 OPEN…”
MethodInfluentialBiological Anthropology in 2018: Grounded in Theory, Questioning Contexts, Embracing Innovation
Martin, M. A. (2019) · American Anthropologist · doi:10.1111/aman.13233 · Cited by 13
“As quantitative social scientists whose research interests and methods often overlap with psychology, medicine, and public health, biological anthropologists are likely aware of the growing “replicability crisis” in these fields (e.g., Glenn Begley and Ellis 2012; Open Science Collaboration 2015).”
BackgroundAuthentication of experimental materials: A remedy for the reproducibility crisis?
Li, F., Hu, J., Xie, K., & He, T. (2015) · Genes & Diseases · doi:10.1016/j.gendis.2015.07.001 · Cited by 11
“According to two reports by Bayer and Amgen published in 2011 and 2012, 64e89% of the socalled “landmark” results could not be reproduced in their pre-clinical validation experiments.(1,2) One plausible explanation for this out of proportion irreproducibility is related to the intricacy of the scientific experiments,…”
ResultIntegrative Data Analysis for Research in Developmental Psychopathology
Mun, E., Jiao, Y., & Xie, M. (2016) · Wiley eBooks · doi:10.1002/9781119125556.devpsy123 · Cited by 11
“These reports pointed out that findings from single studies are often not replicable, and are subject to publication bias and afflicted by low power and high false discovery rates (e.g., Begley & Ellis, 2012; Ioannidis, 2005).”
BackgroundRevisiting the Hierarchical Multiscale LSTMPreprint
Kádár, Á., Côté, M.-A., Chrupała, G., & Alishahi, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03595 · Cited by 13
“A reproduction study by the Open Science Collaboration (2015) estimates that only 40% of research in psychology is reproducible, while Begley and Ellis (2012) end up confirming only 11% of preclinical cancer studies.”
BackgroundLucas, F., Gil‐Pulido, J., LaMacchia, J., Preffer, F., Wallace, P. K., & Lopez, P. (2020) · Cytometry Part A · doi:10.1002/cyto.a.23940 · Cited by 10
The learning curve of murine subretinal injection among clinically trained ophthalmic surgeons
Huang, P., Narendran, S., Pereira, F., Fukuda, S., Nagasaka, Y., Apicella, I., Yerramothu, P., Marion, K. M., Cai, X., Sadda, S. R., et al. (2022) · Translational Vision Science & Technology · doi:10.1167/tvst.11.3.13 · Cited by 10
Recommendations to improve use and reporting of statistics in animal experiments
Rowe, A. (2022) · Laboratory Animals · doi:10.1177/00236772221140669 · Cited by 15
“The international biotech company Amgen found only six out of 53 (11%) landmark preclinical cancer studies published in high impact factor journals over a 10-year period were reproducible.(15) Landmark was defined as something completely new, such as fresh approaches to targeting cancers or alternative clinical uses f…”
BackgroundTranslational Research: From Biological Discovery to Public Benefit (or Not)
Emmert‐Buck, M. R. (2014) · Advances in Biology · doi:10.1155/2014/278789 · Cited by 12
“But the applied science phase becomes evenmore difficult now due to this explosion of new information [147, 148].”
BackgroundEvaluation of Candidate Biomarkers of Type 1 Diabetes via the Core for Assay Validation
Speake, C., & Odegard, J. (2015) · Biomarker Insights · doi:10.4137/bmi.s29697 · Cited by 10
Reproducibility in experimentation – the implications for regulatory toxicology
Berry, S. C. (2014) · Toxicology Research · doi:10.1039/c4tx00069b · Cited by 10
Fisher, D., & Parisis, N. (2015) · EMBO Reports · doi:10.15252/embr.201541256 · Cited by 12
“Yet, one could argue that this is not much different to the current situation anyway, where “publish or perish” encourages hasty publishing and where many studies published in high-impact journals turn out to be irreproducible [10].”
Academic–Pharma Drug Discovery Alliances: Seeking Ways to Eliminate the Valley of Death
Hammonds, T. R. (2015) · Future Medicinal Chemistry · doi:10.4155/fmc.15.111 · Cited by 10
Making the scientific literature fail-safe
Casadevall, A., & Fang, F. C. (2018) · Journal of Clinical Investigation · doi:10.1172/jci123884 · Cited by 10
Dailey, L. A. (2018) · Journal of Interdisciplinary Nanomedicine · doi:10.1002/jin2.52 · Cited by 10
“%) were deemed sufficiently validated for further preclinical development (Begley & Ellis, 2012).”
BackgroundPredictors of Success of Phase II Pediatric Oncology Clinical Trials
Franshaw, L., Tsoli, M., Byrne, J. A., Mayoh, C., Sivarajasingam, S., Norris, M. D., Marshall, G. M., & Ziegler, D. S. (2019) · The Oncologist · doi:10.1634/theoncologist.2017-0666 · Cited by 11
Ferreira, G. S., Veening-Griffioen, D. H., Boon, W., Hooijmans, C. R., Moors, E. H. M., Schellekens, H., & van Meer, P. J. K. (2020) · European Journal of Pharmacology · doi:10.1016/j.ejphar.2020.173153 · Cited by 12
“As evidenced in our risk of bias assessment – and corroborated by previous literature – animal research is still poorly reported, and often essential details regarding the used methodology are not reported (Bebarta et al., 2003; Begley and Ellis, 2012; Hooijmans et al., 2019).”
ResultAnimal Study Translation: The Other Reproducibility Challenge
Berridge, B. R. (2021) · ILAR Journal · doi:10.1093/ilar/ilac005 · Cited by 13
Halloran, J. P., Nohouji, N. A., Hafez, M. A., Besier, T. F., Chokhandre, S. K., Elmasry, S. S., Hume, D. R., Imhauser, C. W., Rooks, N. B., Schneider, M., et al. (2022) · Journal of Orthopaedic Research® · doi:10.1002/jor.25358 · Cited by 10
Čoma, M., Manning, J. C., Kaltner, H., & Gál, P. (2023) · Expert Opinion on Therapeutic Targets · doi:10.1080/14728222.2023.2175318 · Cited by 11
Improving reproducibility and reusability in the Journal of Cheminformatics
Hoyt, C. T., Zdrazil, B., Guha, R., Jeliazkova, N., Martínez‐Mayorga, K., & Nittinger, E. (2023) · Journal of Cheminformatics · doi:10.1186/s13321-023-00730-y · Cited by 14
Multifactorial processes to slowing the biological clock: Insights from a comparative approach
Sanchez, J. R., Milton, S., Corbit, K. C., & Buffenstein, R. (2015) · Experimental Gerontology · doi:10.1016/j.exger.2015.08.017 · Cited by 10
“2014) and the numerous clinical failures are well documented (Begley and Ellis 2012; Benatar 2007; Cavanaugh et. al.”
BackgroundRedrawing the frontiers in the age of post-publication review
Galbraith, D. W. (2015) · Frontiers in Genetics · doi:10.3389/fgene.2015.00198 · Cited by 12
“Furthermore, studies found to be non-reproducible may be cited by secondary publications at higher rates than those found to be reproducible ( Begley and Ellis, 2012).”
BackgroundResultIs There a Role for Publication Consultants and How Should Their Contribution be Recognized?
Kendall, G., Yee, A. S. V., & McCollum, B. (2015) · Science and Engineering Ethics · doi:10.1007/s11948-015-9710-9 · Cited by 10
“Begley and Ellis (2012) have recently suggested that pre-clinical standards need to be raised in cancer research.”
BackgroundQuality assurance mechanisms for the unregulated research environment
Riedl, D. H., & Dunn, M. K. (2013) · Trends in biotechnology · doi:10.1016/j.tibtech.2013.06.007 · Cited by 10
Poser, S., Chenoweth, J. G., Colantuoni, C., Masjkur, J. R., Chrousos, G. P., Bornstein, S. R., McKay, R. D. G., & Androutsellis‐Theotokis, A. (2015) · Stem Cells Translational Medicine · doi:10.5966/sctm.2015-0105 · Cited by 10
Science Between Trust and Control: Non-Reproducibility in Scholarly Publishing
Franzen, M. (2016) · John Wiley & Sons, Inc. eBooks · doi:10.1002/9781118865064.ch22 · Cited by 11
Immune System Model Calibration by Genetic Algorithm
Presbitero, A., Krzhizhanovskaya, V. V., Mancini, E., et al. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.11.020 · Cited by 10
“Science advances by building on top of published results, hence corroboration is only necessary especially now that a mere 11% of published research is reproducible [5].”
Bongard, R. D., Lepley, M. A., Thakur, K., Talipov, M. R., Nayak, J., Jones, R. A., Bohl, C., Sweeney, N. L., Ramchandran, R., Rathore, R., et al. (2017) · BMC Biochemistry · doi:10.1186/s12858-017-0083-3 · Cited by 10
Williams, M. J., Mullane, K., & Curtis, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00005-7 · Cited by 13
To Grasp Cognition in Action, Combine Behavioral Experiments with Protocol Analysis
Reypens, C., & Levine, S. S. (2017) · Emerald Group Publishing Limited eBooks · doi:10.1108/s2397-52102017006 · Cited by 13
Scherz, P. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108593694 · Cited by 12
Plavén‐Sigray, P., Matheson, G. J., Gustavsson, P. J., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193770 · Cited by 10
The role of metadata in reproducible computational research
Leipzig, J. N., Nüst, D., Hoyt, C. T., Ram, K., & Greenberg, J. (2021) · Patterns · doi:10.1016/j.patter.2021.100322 · Cited by 197
“Two events in the life sciences stand as watershed moments in this crisis: the publication of manipulated and falsified predictive cancer therapeutic signatures by a biomedical researcher at Duke and subsequent forensic investigation by Keith Baggerly and David Coombes,(14) and a review by scientists at Amgen who coul…”
BackgroundKourany, J. A. (2020) · Canadian Journal of Philosophy · doi:10.1017/can.2020.34 · Cited by 14
“For example, in 2012 it was reported in Nature that scientists at the biotechnology company Amgen had attempted to replicate fifty-three “landmark” cancer studies, but only six of the fifty-three attempts were successful (11 percent) (Begley and Ellis 2012).”
BackgroundKavanagh, C. M., & Kapitány, R. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.38713 · Cited by 11
The Potential of Notebooks for Scientific Publication, Reproducibility and Dissemination
Rowe, F., Maier, G., Arribas‐Bel, D., & Rey, S. J. (2020) · REGION · doi:10.18335/region.v7i3.357 · Cited by 10
Is the replication crisis a base-rate fallacy?
Autzen, B. (2021) · Metamedicine · doi:10.1007/s11017-022-09561-8 · Cited by 9
“1 3 To illustrate, consider a study by Glenn Begley and Lee Ellis, in which scientists at the biotechnology firm Amgen tried to replicate 53 high-impact findings in pre-clinical cancer research but could confirm the findings of only 6 studies (11%) [2].”
BackgroundCurrent Status and Future Opportunities in Modeling Clinical Characteristics of Multiple Sclerosis
Liu, J., Kelly, E., & Bielekova, B. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.884089 · Cited by 11
“It is estimated that between 51% and 89% of the published literature in biomedical sciences is not reproducible ( 15, 23, 24) and poor study design, based on small sample sizes (11, 13) and the failure to prevent bias (25–27) is the major contributor to this reproducibility crisis.”
BackgroundHow reproducible are the measurements of leaf fluctuating asymmetry?
Kozlov, M. V. (2015) · PeerJ · doi:10.7717/peerj.1027 · Cited by 10
“The current findings on low reproducibility of biomedical results (Prinz, Schlange & Asadullah, 2011; Begley & Ellis, 2012) urged the two prestigious journals, Nature and Science, to modify their guidelines in order to improve quality control in biomedical research (McNutt, 2014; Nature editorial,…”
BackgroundWorkman, P., & Collins, I. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-396521-9.00001-2 · Cited by 10
ImmPort: Shared research data for bioinformatics and immunology
Dunn, P., Thomson, E., Campbell, J. P., Smith, T., Desborough, V., Wiser, J. A., Schaefer, H., Bhattacharya, S., Butte, A. J., Andorf, S., et al. (2015) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2015.7359752 · Cited by 11
Die Sicherung der Nachhaltigkeit von Biobanken
Kirsten, R., & Hummel, M. (2016) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-015-2302-7 · Cited by 9
Enabling Stem Cell Characterization from Large Microscopy Images
Bajcsy, P., Vandecreme, A., Amelot, J., et al. (2016) · Computer · doi:10.1109/mc.2016.191 · Cited by 10
“This approach is less than ideal, and has led to the following scientific problems in stem cells [2], very low reproducibility of published work [3], and inadequate conclusions on biological variability.”
BackgroundHandKAchip - Hands-free killing assay on a chip
Lee, K. S., Lee, L. E., & Levine, E. (2016) · Scientific Reports · doi:10.1038/srep35862 · Cited by 11
van der Bent, M. L., Filho, O. P. D. S., van Luijk, J. A. K. R., Brock, R. E., & Wansink, D. G. (2018) · Scientific Reports · doi:10.1038/s41598-018-22316-7 · Cited by 11
Le biais de confirmation en recherche
Larivée, S. J., Sénéchal, C., St-Onge, Z., & Sauvé, M.-R. (2019) · Revue de psychoéducation · doi:10.7202/1060013ar · Cited by 37
Soft windowing application to improve analysis of high-throughput phenotyping data
Haselimashhadi, H., Mason, J., Muñoz‐Fuentes, V., López, F., Babalola, K., Acar, E. F., Kumar, V., White, J. K., Flenniken, A. M., King, R., et al. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz744 · Cited by 10
A Skeptical Eye on PsiPreprint
Wagenmakers, E., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. J. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/syq2h · Cited by 10
“Prominent failures to replicate include preclinical findings in cancer biology (e.g., Begley & Ellis, 2012), findings on social priming (e.g., Donnellan, Lucas, & Cesario, in press; Doyen, Klein, Pichon, & Cleeremans, 2012; Harris, Coburn, Rohrer, & Pashler, 2013; LeBel & Wilbur, in press; LeBel &…”
BackgroundThe Somatic Mutation Working Group of the SEQC-II Consortium (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/626440 · Cited by 12
The Australian academic STEMM workplace post-COVID: a picture of disarrayPreprint
Christian, K., Larkins, J., & Doran, M. R. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.06.519378 · Cited by 12
“Their team attempted to replicate 53 “landmark” cancer studies [34].”
BackgroundFacilitating transparency in spinal cord injury studies using data standards and ontologies
Lemmon, V., Abeyruwan, S., Visser, U., & Bixby, J. (2014) · Neural Regeneration Research · doi:10.4103/1673-5374.125322 · Cited by 9
“Similarly, scientists at Abbot could reproduce only 11% of landmark studies (Begley et al., 2012).”
Reproducibility failure in biomedical research: problems and solutions
Haven, T. L., & Ioannidis, J. P. A. (2026) · Annual Review of Medicine · doi:10.1146/annurev-med-050124-050859 · Cited by 9
Kuske, R. R., & Young, S. S. (2014) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2013.12.002 · Cited by 9
From significance testing to estimation and Open Science: How esci can help
Calin-Jageman, R. J., & Cumming, G. (2024) · International Journal of Psychology · doi:10.1002/ijop.13132 · Cited by 16
Ahuis, T. P., Smyk, M. K., Laloux, C., Aulehner, K., Bray, J. G., Waldron, A., Miljanovic, N., Seiffert, I., Song, D., Boulanger, B., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309521 · Cited by 10
“Currently, a replicability crisis is ongoing in many scientific disciplines, including biomedical and preclinical research [1–7].”
Mapping nine decades of research integrity studies (1935–2024): A scientometric analysis
Tang, G. (2025) · Accountability in Research · doi:10.1080/08989621.2025.2470860 · Cited by 16
“…and their increasing causes (Grieneisen and Zhang 2012; Steen, Casadevall, and Fang 2013; Van Noorden 2011), different features of the RRI concept (Owen, Macnaghten, and Stilgoe 2012), and the improvement of research culture, publication practices, and incentive structures (Begley and Lee 2012).”
Rethinking the development of Ebola treatments
Gupta, R. K. P. (2014) · The Lancet Global Health · doi:10.1016/s2214-109x(14)70304-3 · Cited by 9
Khan, I., Fraser, A. N., Bray, M., Smith, P. J., White, N. S., Carpenter, A. E., & Errington, R. J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu554 · Cited by 9
“A major contributing factor to irreproducibility is difficulty in interpreting complex experimental methodologies and designs from written text and in assessing variations among different experiments.”
BackgroundDenner, J., & Graham, M. L. (2015) · Xenotransplantation · doi:10.1111/xen.12169 · Cited by 9
Hengartner, M. P. (2017) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2017.00003 · Cited by 10
“Original studies subjecting psychological and medical research to close scrutiny have indeed provided ample evidence that many, and sometimes even a majority, of published findings are irreproducible, false positive, or severely inflated (Ioannidis, 2005b; Turner et al., 2008; Munafo et al., 2009; Bakker et al., 2012;…”
BackgroundKerpel‐Fronius, S., Becker, S., Barrett, J., Brun, J., Carlesi, R. M., Chan, A., Collia, L. F., Dubois, D. J., Kleist, P., Koski, G., et al. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.00843 · Cited by 11
“It is therefore very disappointing that from 53 landmark studies published in prestigious journals only 6% reported sufficiently robust data to drive reliably human medicines development programs (Begley and Ellis, 2012).”
BackgroundBest practices in statistical computing
Sanchez, R., Griffin, B. A., Pane, J., & McCaffrey, D. F. (2021) · Statistics in Medicine · doi:10.1002/sim.9169 · Cited by 10
From viability to cell death: Claims with insufficient evidence in high-impact cell culture studies
Özkaya, A. B., & Geyik, C. (2022) · PLoS ONE · doi:10.1371/journal.pone.0250754 · Cited by 11
Variation and Covariation in Large-Scale Replication Projects: An Evaluation of Replicability
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2054816 · Cited by 13
“…rates of only about one-quarter for pharmaceutical development studies (Prinz, Schlange, and Asadullah 2011), one-tenth for cancer research studies (Begley and Ellis 2012), one-third for psychological research studies (Open Science Collaboration 2015), six-tenths for economics research studies…”
BackgroundMartone, M. E., & Nakamura, R. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.6650ce2b · Cited by 10
A toolkit for open and pluralistic conservation science
Burgman, M. A., Chiaravalloti, R. M., Fidler, F. M., Huan, Y., McBride, M. F., Marcoci, A., Norman, J., Vercammen, A., Wintle, B. C., & Yu, Y. (2022) · Conservation Letters · doi:10.1111/conl.12919 · Cited by 12
“The latter has contributed to what is known as the “reproducibility crisis,” which refers to the failure to successfully replicatemany scientific findings across disciplines from preclinical medicine to economics (e.g., Begley & Ellis, 2012; Open Science Collaboration, 2015; Camerer et al, 2016).”
BackgroundClinical development of new drugs for adults and children with cancer, 2010-2020
Arfè, A., Narang, C., DuBois, S. G., Reaman, G. H., & Bourgeois, F. T. (2023) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djad082 · Cited by 10
Developing a Toolbox of Antibodies Validated for Array Tomography-Based Imaging of Brain Synapses
Micheva, K. D., Gong, B., Collman, F., Weinberg, R. J., Smith, S. J., Trimmer, J. S., & Murray, K. D. (2023) · eNeuro · doi:10.1523/eneuro.0290-23.2023 · Cited by 9
“The lack of highly validated Abs for research is a widely-recognized problem (Begley and Ellis, 2012; Baker, 2015a,b) that has forced laboratories to employ extensive in-house Ab testing before their use (Bordeaux et al., 2010; Micheva et al., 2010a).”
BackgroundCook, R. J., & Lawless, J. F. (2023) · The Journal of Rheumatology · doi:10.3899/jrheum.2023-0499 · Cited by 10
Wei, D., Sun, J., Luo, Z., Zhang, G., Liu, Y., Zhang, H., Xie, Z., Gu, Z., & Tao, W. A. (2024) · Analytical Chemistry · doi:10.1021/acs.analchem.3c04464 · Cited by 9
The erosion of research integrity: the need for culture change
Ellis, L. M. (2015) · The Lancet Oncology · doi:10.1016/s1470-2045(15)00085-6 · Cited by 10
Self-citationNovel or reproducible: That is the question
Hagen, K. G. T. (2016) · Glycobiology · doi:10.1093/glycob/cww036 · Cited by 12
Ethics of Cancer Gene Transfer Clinical Research
Kimmelman, J. (2015) · Methods in Molecular Biology · doi:10.1007/978-1-4939-2727-2_15 · Cited by 2
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.