Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 701–800 of 2,889
Data Submission and Quality in Microarray-Based MicroRNA Profiling
Witwer, K. W. (2013) · Clinical Chemistry · doi:10.1373/clinchem.2012.193813 · Cited by 43
Diaz, Z., Aguilar‐Mahecha, A., Paquet, É. R., Basik, M., Orain, M., Camlioglu, E., Constantin, A. M., Benlimame, N., Bachvarov, D. R., Jannot, G., et al. (2013) · Modern Pathology · doi:10.1038/modpathol.2013.81 · Cited by 43
Reproducibility of histopathological findings in experimental pathology of the mouse: a sorry tail
Ward, J. M., Schofield, P. N., & Sundberg, J. P. (2017) · Lab Animal · doi:10.1038/laban.1214 · Cited by 45
Analyzing crowdsourced ratings of speech-based take-over requests for automated driving
Bazilinskyy, P., & de Winter, J. (2017) · Applied Ergonomics · doi:10.1016/j.apergo.2017.05.001 · Cited by 53
“Small samples are a prime cause of poor replicability, a message that has now transpired to many fi elds, including medicine (Arrowsmith, 2011; Begley and Ellis, 2012; Freedman et al., 2015), economics (Ioannidis and Doucouliagos, 2013), and neuro-science (Button et al., 2013).”
BackgroundThe Energy-Growth Nexus: History, Development, and New Challenges
Hajko, V., Sebri, M., Al‐Saidi, M., & Balsalobre‐Lorente, D. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812746-9.00001-8 · Cited by 56
Porcine Models of Pancreatic Cancer
Bailey, K. L., & Carlson, M. A. (2019) · Frontiers in Oncology · doi:10.3389/fonc.2019.00144 · Cited by 54
Does Chemical Engineering Research Have a Reproducibility Problem?
Han, R., Walton, K. S., & Sholl, D. S. (2019) · Annual Review of Chemical and Biomolecular Engineering · doi:10.1146/annurev-chembioeng-060718-030323 · Cited by 47
Shokoohmand, A., Ren, J., Baldwin, J., Atack, A., Shafiee, A., Theodoropoulos, C., Wille, M., Tran, P. A., Bray, L. J., Smith, D. S., et al. (2019) · Biomaterials · doi:10.1016/j.biomaterials.2019.119402 · Cited by 44
High heterogeneity undermines generalization of differential expression results in RNA-Seq analysis
Cui, W., Xue, H., Wei, L., Jin, J., Tian, X., & Wang, Q. (2021) · Human Genomics · doi:10.1186/s40246-021-00308-5 · Cited by 49
“known scientific findings from cancer biology were confirmed in only 6 cases out of 53 [15, 16].”
BackgroundDo multiple experimenters improve the reproducibility of animal studies?
von Kortzfleisch, V. T., Ambrée, O., Karp, N. A., Meyer, N., Novak, J., Palme, R., Rosso, M., Touma, C., Würbel, H., Kaiser, S., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001564 · Cited by 50
“Based on such replication studies, the prevalence for irreproducible results was estimated to range between 50% and 90% and the majority of scientists in the life sciences are convinced that this “reproducibility crisis” needs to be addressed [1,5].”
BackgroundA meta-analysis of threats to valid clinical inference in preclinical research of sunitinib
Henderson, V. C., Demko, N., Hakala, A., MacKinnon, N., Federico, C. A., Fergusson, D., & Kimmelman, J. (2015) · eLife · doi:10.7554/elife.08351 · Cited by 42
Starokozhko, V., Kallio, M. J., Howell, Å. K., Salmi, A. M., Andrew-Nielsen, G., Goldammer, M., Burggraf, M., Löbker, W., Böhmer, A., Agricola, E., et al. (2020) · Drug Discovery Today · doi:10.1016/j.drudis.2020.10.017 · Cited by 45
Rice, A. S. C., Morland, R., Huang, W., Currie, G. L., Sena, E. S., & Macleod, M. R. (2013) · Scandinavian Journal of Pain · doi:10.1016/j.sjpain.2013.02.002 · Cited by 42
Ringer, L., Sirajuddin, P., Tricoli, L., Waye, S. E., Choudhry, M. U., Parasido, E., Sivakumar, A., Heckler, M. M., Naeem, A., Abdelgawad, I. A., et al. (2014) · Oncotarget · doi:10.18632/oncotarget.2528 · Cited by 42
“The lack of clinically relevant cell lines has had a major impact on effective therapeutic development [47, 48].”
Opinion versus evidence for the need to move away from animal testing
Härtung, T. (2017) · ALTEX · doi:10.14573/altex.1703291 · Cited by 54
“But these exciting results are often shaky – the only 10-25% reproducibility of important scientific papers published by the pharmaceutical industry is alarming (Begley and Ellis, 2012; Prinz et al., 2011).”
BackgroundAccelerating Biomedical Discoveries through Rigor and Transparency
Hewitt, J., Brown, L. L., Murphy, S. J., Grieder, F. B., & Silberberg, S. D. (2017) · ILAR Journal · doi:10.1093/ilar/ilx011 · Cited by 55
Tavares, A. B. M. L. A., Neto, J. X., Fulco, U. L., & Albuquerque, E. L. (2018) · Scientific Reports · doi:10.1038/s41598-018-20325-0 · Cited by 44
Effect size and statistical power in the rodent fear conditioning literature – A systematic review
Carneiro, C. F. D., Moulin, T. C., Macleod, M. R., & Amaral, O. B. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196258 · Cited by 54
“In light of the low reproducibility of published studies in various fields of biomedical research (Scott et al., 2008; Prinz et al., 2011; Begley and Ellis, 2012) which is thought by many to be a consequence of low statistical power and excessive reliance on significance tests (Ioannidis, 2005; Higginson and Munafò, 2…”
BackgroundQuestionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 52
Cucchi, D. G. J., Groen, R. W. J., Janssen, J. J. W. M., & Cloos, J. J. (2020) · Drug Resistance Updates · doi:10.1016/j.drup.2020.100730 · Cited by 47
Rodriguez, K. M., Wu, W., Alebrahim, T., Cao, Y., Freeman, B. D., Harrigan, D. J., Jhalaria, M., Kratochvil, A. M., Kumar, S. K., Lee, W. H., et al. (2022) · Journal of Membrane Science · doi:10.1016/j.memsci.2022.120746 · Cited by 48
Open data, open review and open dialogue in making social sciences plausiblePreprint
Vuong, Q. (2021) · doi:10.31219/osf.io/da9zs · Cited by 42
Fergusson, D. A., Avey, M. T., Barron, C. C., Bocock, M., Biefer, K., Boet, S., Bourque, S. L., Conic, I., Chen, K., Dong, Y., et al. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215221 · Cited by 40
“Poorly designed, executed, and reported preclinical studies have contributed to the replication crisis, and increase waste of research funding, laboratory animals, and personnel time [8, 9].”
BackgroundLinking Brain Age Gap to Mental and Physical Health in the Berlin Aging Study II
Jawinski, P., Markett, S. A., Drewelies, J., Düzel, S., Demuth, I., Steinhagen‐Thiessen, E., Wagner, G. G., Gerstorf, D., Lindenberger, U., Gaser, C., et al. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.791222 · Cited by 46
Animal Models of Choroidal Neovascularization: A Systematic Review
Fabian-Jessing, B. K., Jakobsen, T. S., Jensen, E. G., Alsing, S., Hansen, S., Aagaard, L., Askou, A. L., Bek, T., & Corydon, T. J. (2022) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.63.9.11 · Cited by 44
Artificial intelligence as a tool in drug discovery and development
Kokudeva, M., Vichev, M., Naseva, E., Miteva, D., & Velikova, T. V. (2024) · World Journal of Experimental Medicine · doi:10.5493/wjem.v14.i3.96042 · Cited by 78
Guidelines for manuscript submission in the peer-reviewed pharmacological literature
Mullane, K., Enna, S. J., Piette, J. G., & Williams, M. J. (2015) · Biochemical Pharmacology · doi:10.1016/j.bcp.2015.06.023 · Cited by 44
González-Beltrán, A., Li, P., Zhao, J., Avila-Garcia, M. S., Roos, M., Thompson, M., van der Horst, E., Kaliyaperumal, R., Luo, R., Lee, T.-L., et al. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127612 · Cited by 41
Genetically Engineered Mouse Models for Drug Development and Preclinical Trials
Lee, H. (2014) · Biomolecules & Therapeutics · doi:10.4062/biomolther.2014.074 · Cited by 50
“Therefore, technological advances that allow for easy and effi cient target identifi cation and validation steps early in the drug discovery process are essential for reducing both the attrition rate and the cost of launching a new drug (Begley and Ellis, 2012).”
BackgroundHow many times should we screen a chemical library to discover an anticancer drug?
López‐Lázaro, M. (2014) · Drug Discovery Today · doi:10.1016/j.drudis.2014.12.006 · Cited by 46
Citation Sentiment Analysis in Clinical Trial Papers.
Xu, J., Zhang, Y., Wu, Y., Wang, J., Dong, X., & Xu, H. (2015) · PubMed · Cited by 49
Open innovation: Towards sharing of data, models and workflows
Conrado, D. J., Karlsson, M. O., Romero, K., Sarr, C., & Wilkins, J. J. (2017) · European Journal of Pharmaceutical Sciences · doi:10.1016/j.ejps.2017.06.035 · Cited by 41
“…of scientific areas ranging from target identification to psychology have recently demonstrated an unexpectedly high lack of reproducibility (Begley and Ellis, 2012; Ebrahim et al., 2014; Open Science Collaboration, 2015; Prinz et al., 2011; Wang et al., 2016), leading to a lack of trust in…”
BackgroundTorres, L., & Tiersch, T. R. (2018) · Journal of the World Aquaculture Society · doi:10.1111/jwas.12541 · Cited by 41
Donovan, K. M., Leidinger, M., McQuillen, L. P., Goeken, J. A., Hogan, C. M., Harwani, S. C., Flaherty, H. A., & Meyerholz, D. K. (2018) · Comparative Medicine · doi:10.30802/aalas-cm-18-000017 · Cited by 41
Case Reports of Pre-clinical Replication Studies in Metabolism and Diabetes
von Herrath, M., Pagni, P. P., Grove, K., Christoffersson, G., Tang-Christensen, M., Karlsen, A. E., & Petersen, J. S. (2019) · Cell Metabolism · doi:10.1016/j.cmet.2019.02.004 · Cited by 39
Alavijeh, A. A., Barati, M. A., Barati, M., & Dehkordi, H. A. (2019) · Advanced Pharmaceutical Bulletin · doi:10.15171/apb.2019.043 · Cited by 45
Detecting academic fraud using Benford law: The case of Professor James Hunton
Horton, J., Kumar, D. K., & Wood, A. (2020) · Research Policy · doi:10.1016/j.respol.2020.104084 · Cited by 58
van der Naald, M., Wenker, S., Doevendans, P. A., Wever, K. E., & Chamuleau, S. A. J. (2020) · BMJ Open Science · doi:10.1136/bmjos-2019-100051 · Cited by 40
PERFICT: A Re‐imagined foundation for predictive ecology
McIntire, E. J. B., Chubaty, A. M., Cumming, S. G., Andison, D., Barros, C., Boisvenue, C., Haché, S., Luo, Y., Micheletti, T., & Stewart, F. E. C. (2022) · Ecology Letters · doi:10.1111/ele.13994 · Cited by 50
MicroRNA biomarkers of type 2 diabetes: evidence synthesis from meta-analyses and pathway modelling
Zhu, H., & Leung, S.-W. (2022) · Diabetologia · doi:10.1007/s00125-022-05809-z · Cited by 44
“Recent studies indicate that seminal findings from academic laboratories can be reproduced only 11–50% of the time [47, 48].”
BackgroundAntunes, B. A., & Hill, D. R. C. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100655 · Cited by 78
“(Begley and Ellis, 2012) raised a reproducibility concern for cancer research.”
BackgroundHollingshead, M. G., Stockwin, L. H., Alcoser, S. Y., Newton, D. L., Orsburn, B. C., Bonomi, C. A., Borgel, S. D., Divelbiss, R., Dougherty, K. M., Hager, E. J., et al. (2014) · BMC Genomics · doi:10.1186/1471-2164-15-393 · Cited by 40
“GO:0005201 extracellular matrix structural constituent 3 experimental data since the methods for generating test tumors can impact the study profoundly [38].”
BackgroundMadsen, D. H., & Bugge, T. H. (2015) · The Journal of Cell Biology · doi:10.1083/jcb.201501034 · Cited by 41
“Encouragingly, however, the heightened awareness of problems with reproducibility of preclinical cancer research (Prinz et al., 2011; Begley and Ellis, 2012; Bradbury and Plückthun, 2015) has triggered new initiatives involving representatives from major funding agencies, academic researchers, reviewers, journal edito…”
BackgroundMiller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 43
“In the medical context, the high costs of FPs are clear (e.g., Begley & Ellis, 2012).”
BackgroundLayer-by-layer approach for a uniformed fabrication of a cell patterned vessel-like construct
Wilkens, C. A., Rivet, C. J., Akentjew, T. L., Alverio, J., Khoury, M., & Acevedo, J. P. (2016) · Biofabrication · doi:10.1088/1758-5090/9/1/015001 · Cited by 42
Göçeri, E., Goksel, B., Elder, J. B., et al. (2016) · International Journal for Numerical Methods in Biomedical Engineering · doi:10.1002/cnm.2862 · Cited by 40
“p1⁄41 u p ð Þ Bi ð Þ−u p ð Þ Bi ð Þ 2 : (5)”
BackgroundThe statistical crisis in science: how is it relevant to clinical neuropsychology?
Gelman, A., & Geurts, H. M. (2017) · The Clinical Neuropsychologist · doi:10.1080/13854046.2016.1277557 · Cited by 45
“Recent theoretical and experimental results have cast doubt on what previously had been believed to be solid findings in various fields (e.g. see Baker, 2016; Begley & Ellis, 2012; Button et al., 2013; Dumas-Mallet, Button, Boraud, Munafo, & Gonon, 2016; Ioannidis & Panagiotou, 2011).”
ResultOudega, M., Hao, P., Shang, J., Haggerty, A. E., Wang, Z., Sun, J. H., Liebl, D. J., Shi, Y., Cheng, L., Duan, H., et al. (2018) · Experimental Neurology · doi:10.1016/j.expneurol.2018.11.003 · Cited by 43
“The general consensus is that basic and translational biomedical research suffers from a lack in robustness with low success rates in replication studies reported in different scientific areas (Prinz et al., 2011; Begley and Ellis, 2012), including SCI and repair (Steward et al., 2012).”
Liu, Y., Blackburn, H. D., Taylor, S. S., & Tiersch, T. R. (2019) · Theriogenology · doi:10.1016/j.theriogenology.2019.05.020 · Cited by 45
Liu, Y., Althoff, T., & Heer, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376533 · Cited by 65
Acute alcohol response phenotype in heavy social drinkers is robust and reproducible
Roche, D. J. O., Palmeri, M. D., & King, A. C. (2014) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.12280 · Cited by 39
“…failures to replicate in the medical field, with only 11% of high-impact findings in preclinical cancer research successfully replicated, there has been a renewed focus on the importance of reproducibility of novel and influential results in various fields of science (Begley and Ellis, 2012).”
BackgroundSherman, E. J., & Emmer, B. T. (2021) · Scientific Reports · doi:10.1038/s41598-021-95308-9 · Cited by 41
Four principles for improved statistical ecology
Popovic, G. C., Mason, T. J., Drobniak, S. M., Marques, T. A., Potts, J., Joo, R., Altwegg, R., Burns, C. C. I., Mccarthy, M. A., Johnston, A., et al. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14270 · Cited by 64
“Failure to follow good practices has led to low reproducibility of findings in many fields (Begley & Ellis, 2012 ; Camerer et al., 2018 ; Open Science Collaboration, 2015 ).”
BackgroundMullane, K., & Williams, M. J. (2015) · Biochemical Pharmacology · doi:10.1016/j.bcp.2015.07.002 · Cited by 42
“The current level of concern on the issue of reproducibility is generally considered to have been kick started by Ioannidis’ seminal paper, “Why Most Published Research Findings Are False” [12] and further elaborated in the commentaries by Begley and others [1,2,3] and by initiatives from the NIH [13,14].”
BackgroundCone, A. C., Ambrosi, C., Scemes, E., Martone, M. E., & Sosinsky, G. E. (2013) · Frontiers in Pharmacology · doi:10.3389/fphar.2013.00006 · Cited by 39
“A recent editorial in Nature highlighted significant reproducibility issues in clinical drug trials because of insufficient or selective reporting (Begley and Ellis, 2012).”
BackgroundSome properties of p-curves, with an application to gradual publication bias.
Ulrich, R., & Miller, J. O. (2017) · Psychological Methods · doi:10.1037/met0000125 · Cited by 38
Animal models of cardiac arrest: A systematic review of bias and reporting
Fabian-Jessing, B. K., Vallentin, M. F., Secher, N., Hansen, F. B., Dezfulian, C., Granfeldt, A., & Andersen, L. W. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.01.047 · Cited by 45
Gholaminejad, A., Tehrani, H. A., & Fesharaki, M. G. (2018) · Biomarkers · doi:10.1080/1354750x.2018.1488275 · Cited by 39
“On the other hand, a survey on data reproducibility of biomedical science concludes that the formative result from academic laboratories could only be reproduced 11–50% of the time (Begley and Ellis 2012, Prinz et al. 2011).”
BackgroundA visual tool for defining reproducibility and replicability
Patil, P., Peng, R. D., & Leek, J. T. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0629-z · Cited by 38
Applied Biophysical Methods in Fragment-Based Drug Discovery
Coyle, J., & Walser, R. (2020) · SLAS DISCOVERY · doi:10.1177/2472555220916168 · Cited by 50
Aquarium: open-source laboratory software for design, execution and data management
Vrana, J. D., de Lange, O., Yang, Y., Newman, G., Saleem, A., Miller, A., Cordray, C., Halabiya, S. F., Parks, M., Lopez, E., et al. (2021) · Synthetic Biology · doi:10.1093/synbio/ysab006 · Cited by 44
“A consequence of less structured approaches is that many experiments cannot be repeated by different researchers (2, 3).”
BackgroundFabrication of PNIPAm-based thermoresponsive hydrogel microwell arrays for tumor spheroid formation
Dhamecha, D. L., Le, D., Chakravarty, T., Perera, K. D. C., Dutta, A., & Menon, J. U. (2021) · Materials Science and Engineering C · doi:10.1016/j.msec.2021.112100 · Cited by 43
Pyridine Moiety: An Insight into Recent Advances in the Treatment of Cancer
Sahu, R., Mishra, R., Kumar, R., Salahuddin, S., Majee, C., Mazumder, A., & Kumar, A. (2021) · Mini-Reviews in Medicinal Chemistry · doi:10.2174/1389557521666210614162031 · Cited by 39
Controlled One-on-One Encounters between Immune Cells and Microbes Reveal Mechanisms of Phagocytosis
Heinrich, V. (2015) · Biophysical Journal · doi:10.1016/j.bpj.2015.06.042 · Cited by 44
“The risks of translating such insights into medical applications cause growing concern for clinicians, patients, and entrepreneurs (4,5).”
BackgroundFrequentist and Bayesian inference: A conceptual primer
van Zyl, C. J. J. (2018) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2018.06.004 · Cited by 45
Role of Environment and Experimenter in Reproducibility of Behavioral Studies With Laboratory Mice
Nigri, M., Åhlgren, J., Wolfer, D. P., & Võikar, V. (2022) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2022.835444 · Cited by 48
Hunt, C. A., Kennedy, R., Kim, S. H. J., & Ropella, G. E. P. (2013) · WIREs Systems Biology and Medicine · doi:10.1002/wsbm.1222 · Cited by 40
Hakkarainen, T. W., Dellinger, P. E., Evans, H. L., Farjah, F., Farrokhi, E. T., Steele, S. R., Thirlby, R. C., & Flum, D. R. (2013) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2013.11.018 · Cited by 44
“Two recent pharmaceutical studies demonstrated reproducibility rates of published data ranging from 11% to 25%.(12,13) Reasons why the results from the much larger-scale Swenson and coauthors(7) study and our current SCOAP analysis may not support the findings of the RCT by Darouiche include differential application o…”
ResultIntroduction to the Special Section on Advancing Our Methods and Practices
Ledgerwood, A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529448 · Cited by 41
“…within our field was situated within a broader context of similar issues emerging across a range of scientific disciplines, from cancer research to genetics to neuroscience (e.g., Begley & Ellis, 2012; Button et al., 2013; Fanelli, 2012; Ioannidis, 2005; Moonesinghe, Khoury, & Janssens, 2007).”
BackgroundSafeguarding Structural Data Repositories against Bad Apples
Minor, W., Dauter, Z., Helliwell, J. R., Jaskólski, M., & Wlodawer, A. (2016) · Structure · doi:10.1016/j.str.2015.12.010 · Cited by 38
“For example, some pharmaceutical companies have revealed that they can independently verify fewer than 25% of preclinical cancer studies published in major journals (Begley and Ellis, 2012).”
BackgroundBates, M., Berliner, A. J., Lachoff, J., et al. (2016) · ACS Synthetic Biology · doi:10.1021/acssynbio.6b00108 · Cited by 38
Henderson, C. M., Bollinger, J. G., Becker, J. O., Wallace, J. M., Laha, T. J., MacCoss, M. J., & Hoofnagle, A. N. (2017) · PROTEOMICS - CLINICAL APPLICATIONS · doi:10.1002/prca.201600103 · Cited by 37
“Often, the results of these studies can not be reproduced in the laboratories seeking to use these protein targets to diagnose and monitor human diseases and this has been attributed, in part, to inadequately evaluated assays [44].”
BackgroundThe failure of Energy-Economy Nexus: A meta-analysis of 104 studies
Hajko, V. (2017) · Energy · doi:10.1016/j.energy.2017.02.095 · Cited by 40
Physics of human cooperation: experimental evidence and theoretical models
Sánchez, Á. (2018) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/aaa388 · Cited by 47
Chuan-Peng, H., Kong, X., Er, I.-J., Al, E., & Peng, K. (2018) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2018.00951 · Cited by 56
Human-Robot Interaction in Groups: Methodological and Research Practices
Oliveira, R., Arriaga, P., & Paiva, A. (2021) · Multimodal Technologies and Interaction · doi:10.3390/mti5100059 · Cited by 51
A living document: reincarnating the research article
Shanahan, D. R. (2015) · Trials · doi:10.1186/s13063-015-0666-5 · Cited by 42
Chemical biology approaches to target validation in cancer
Blagg, J., & Workman, P. (2014) · Current Opinion in Pharmacology · doi:10.1016/j.coph.2014.07.007 · Cited by 41
Data Management in the Modern Structural Biology and Biomedical Research Environment
Zimmerman, M. D., Grabowski, M., Domagalski, M. J., MacLean, E., Chruszcz, M., & Minor, W. (2014) · Methods in molecular biology · doi:10.1007/978-1-4939-0354-2_1 · Cited by 40
“Reliable information about the relationships between functional and structural data could potentially save millions of dollars in the drug discovery process [1].”
BackgroundOverflow in science and its implications for trust
Siebert, S., Machesky, L. M., & Insall, R. H. (2015) · eLife · doi:10.7554/elife.10825 · Cited by 34
“Particular themes include the low reproducibility of published findings in certain fields (Begley and Ellis, 2012; Nature Biotechnology, 2012; Button et al., 2013; Fishburn, 2014; Nuffield Council on Bioethics, 2014), ineffective peer review processes (Eisen, 2011), increasing rates of retractions…”
BackgroundDesigning, conducting, and reporting reproducible animal experiments
Wilson, E., Ramage, F. J., Wever, K. E., Sena, E. S., Macleod, M. R., & Currie, G. L. (2023) · Journal of Endocrinology · doi:10.1530/joe-22-0330 · Cited by 44
“Beyond financial and resource concerns, the persistence of non-reproducible findings in the literature may mislead the planning of future research (Begley & Ellis 2012), generating further downstream research waste.”
BackgroundFrench, A., Bravery, C. A., Smith, J., Chandra, A., Archibald, P., Gold, J. D., Artzi, N., Kim, H., Barker, R., Meissner, A., et al. (2015) · Stem Cells Translational Medicine · doi:10.5966/sctm.2014-0233 · Cited by 36
THE INNOVATIVE MEDICINES INITIATIVE: A PUBLIC PRIVATE PARTNERSHIP MODEL TO FOSTER DRUG DISCOVERY
Vaudano, E. (2013) · Computational and Structural Biotechnology Journal · doi:10.5936/csbj.201303017 · Cited by 43
Hurdles in Basic Science Translation
Perry, C. J., & Lawrence, A. J. (2017) · Frontiers in Pharmacology · doi:10.3389/fphar.2017.00478 · Cited by 39
“One of the points that was particularly concerning in Amgen’s study is that many of the findings that could not be reproduced had formed the foundation of ongoing programs of research without any initial attempt to replicate the original finding – suggesting that there was a great deal of ongoing research based on som…”
BackgroundResultInfluentialResazurin Live Cell Assay: Setup and Fine-Tuning for Reliable Cytotoxicity Results
Rodríguez‐Corrales, J. Á., & Josan, J. S. (2017) · Methods in molecular biology · doi:10.1007/978-1-4939-7201-2_14 · Cited by 37
Levi-Setti, P. E., Cirillo, F., Smeraldi, A., Morenghi, E., Mulazzani, G., & Albani, E. (2017) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-017-1092-2 · Cited by 36
How measurement science can improve confidence in research results
Plant, A. L., Becker, C. A., Hanisch, R. J., Boisvert, R. F., Possolo, A. M., & Elliott, J. T. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2004299 · Cited by 35
Autio, J. A., Zhu, Q., Li, X., Glasser, M. F., Schwiedrzik, C. M., Fair, D. A., Zimmermann, J., Yacoub, E. S., Menon, R. S., Essen, D. C. V., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118082 · Cited by 41
The fall and rise of pharmacology – (Re-)defining the discipline?
Winquist, R. J., Mullane, K., & Williams, M. J. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.09.011 · Cited by 44
Embedding data provenance into the Learning Health System to facilitate reproducible research
Ćurčin, V. (2016) · Learning Health Systems · doi:10.1002/lrh2.10019 · Cited by 44
Branco, A., Cohen, K. B., Vossen, P. T. J. M., Ide, N., & Calzolari, N. (2017) · Language Resources and Evaluation · doi:10.1007/s10579-017-9380-0 · Cited by 35
“…been shown that for some published results with major impact, replication of published results is difficult or impossible (e.g. Prinz et al. 2011; Begley and Ellis 2012; Fokkens et al. 2013; Anderson et al. 2015), and that exaggerated and false claims, sometimes with fabricated data and fake…”
ResultDorrigiv, D., Simeone, K., Communal, L., Kendall‐Dupont, J., St‐Georges‐Robillard, A., Péant, B., Carmona, E., Mes‐Masson, A., & Gervais, T. (2021) · Cancers · doi:10.3390/cancers13164208 · Cited by 43
“informed consent from patients, whereas patient tumor specimens are often readily available from surgery or biopsies with minimal or no additional risk for the patient [55].”
BackgroundMartin, A. K. (2022) · PhilPapers (PhilPapers Foundation) · Cited by 35
Luxenhofer, R. (2015) · Nanomedicine · doi:10.2217/nnm.15.139 · Cited by 42
Leukemia inhibitory factor increases glucose uptake in mouse skeletal muscle
Brandt, N., O'Neill, H. M., Kleinert, M., Schjerling, P., Vernet, E., Steinberg, G. R., Richter, E. A., & Jørgensen, S. B. (2015) · American Journal of Physiology-Endocrinology and Metabolism · doi:10.1152/ajpendo.00313.2014 · Cited by 35
Poste, G. H., Compton, C. C., & Barker, A. D. (2014) · Expert Review of Molecular Diagnostics · doi:10.1586/14737159.2015.974561 · Cited by 36
Can you trust your animal study data?
Peers, I. S., South, M. C., Ceuppens, P. R., Bright, J. D., & Pilling, E. A. (2014) · Nature Reviews Drug Discovery · doi:10.1038/nrd4090-c1 · Cited by 35
An Introduction to Replication Research in Gifted Education: Shiny and New Is Not the Same as Useful
Makel, M. C., & Plucker, J. A. (2015) · Gifted Child Quarterly · doi:10.1177/0016986215578747 · Cited by 32
“Other researchers have found even lower replication success rates in medical research (e.g., Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011).”
ResultSaxena, D., Spino, M., Tricta, F., Connelly, J., Cracchiolo, B. M., Hanauske, A.-R., Gandolfi, D. D., Mathews, M. B., Karn, J., Holland, B. K., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154842 · Cited by 39
“Furthermore, preclinical models for novel effects and targets are unreliable predictors for achievable biological impact and have been implicated in the high attrition rate of clinical trials [47,48].”
BackgroundA Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 40
“In the field of cancer biology, Begley and Ellis (2012) attempted to replicate a set of 59 “landmark” studies in the field.”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.