Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 401–500 of 2,889
Zingarelli, B., Coopersmith, C. M., Drechsler, S., Efron, P. A., Marshall, J. C., Moldawer, L. L., Wiersinga, W. J., Xiao, X., Osuchowski, M. F., & Thiemermann, C. (2018) · Shock · doi:10.1097/shk.0000000000001243 · Cited by 95
“Failure to randomize and/or blind were key factors behind the inability to reproduce landmark preclinical findings in anticancer, cardiovascular, and amyloid lateral sclerosis therapies (54, 55).”
BackgroundOrganoid Models for Precision Cancer Immunotherapy
Sun, C., Lan, H., Fang, X., Yang, X., & Jin, K. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.770465 · Cited by 107
Rodríguez-Molares, A., Rindal, O. M. H., Bernard, O., et al. (2017) · IEEE International Ultrasonics Symposium · doi:10.1109/ultsym.2017.8092389 · Cited by 84
“In 2012, a study published in Nature [3] found that 47 out of 53 medical research papers on cancer research were irreproducible.”
BackgroundPaller, C. J., Bradbury, P. A., Ivy, S. P., Seymour, L., LoRusso, P. M., Baker, L., Rubinstein, L., Huang, E., Collyar, D., Groshen, S., et al. (2014) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-14-0521 · Cited by 84
“Publication of negative results through journals such as the Journal of Negative Results in BioMedicine (32, 33) CCR Reviews © 2014 American Association for Cancer Research 4a.”
BackgroundSelf-citationTowards structured sharing of raw and derived neuroimaging data across existing resources
Keator, D. B., Helmer, K., Steffener, J., Turner, J. A., Van Erp, T. G. M., Gadde, S., Ashish, N., Burns, G. A., & Nichols, B. N. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.094 · Cited by 79
“…neuroimaging data, meta-data, and complete provenance data along with research publications is also likely to accelerate scientific discovery through earlier detection of non-replicable research findings; a topic that has recently received a lot of attention (Begley and Ellis, 2012 ; Yong, 2012).”
BackgroundInfluentialSystems biology strategies to study lipidomes in health and disease
Hyötyläinen, T., & Orešič, M. (2014) · Progress in Lipid Research · doi:10.1016/j.plipres.2014.06.001 · Cited by 89
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Do Pluripotent Stem Cells Exist in Adult Mice as Very Small Embryonic Stem Cells?
Miyanishi, M., Mori, Y., Seita, J., Chen, J. Y., Karten, S., Chan, C. K. F., Nakauchi, H., & Weissman, I. L. (2013) · Stem Cell Reports · doi:10.1016/j.stemcr.2013.07.001 · Cited by 85
Nagasundaram, N., Yapp, E. K. Y., Le, N. Q. K., Kamaraj, B., Al‐Subaie, A. M., & Yeh, H. (2019) · BioMed Research International · doi:10.1155/2019/8427042 · Cited by 97
Church, D., Stapleton, P., Vasudevan, A., & O'Keefe, T. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.951451 · Cited by 82
“An analysis in the journal Nature characterized this “a shocking result” (Begley and Ellis, 2012).”
BackgroundA Chemical Genetics Approach for the Functional Assessment of Novel Cancer Genes
Zhou, Q., Derti, A., Ruddy, D., Rakiec, D., Kao, I., Lira, M., Gibaja, V., Chan, H., Yang, Y., Min, J., et al. (2015) · Cancer Research · doi:10.1158/0008-5472.can-14-2930 · Cited by 82
“In fact, it is likely that the propensity of RNAibased methods to yield false-positive results significantly contributes to the recently raised concerns around the lack of reproducibility in cancer biology publications (6, 7).”
BackgroundAnimal models of Huntington's disease for translation to the clinic: Best practices
Menalled, L., & Brunner, D. (2014) · Movement Disorders · doi:10.1002/mds.26006 · Cited by 85
“Lack of Replicability: A Complex Problem Several publications have pointed out the serious lack of replication of preclinical studies and failure to translate to the clinic.(38-40) Two of the most important reasons behind the failure of translation of apparent positive results from preclinical trials in clinical setti…”
BackgroundAndrade, E., Bento, A. F., Cavalli, J., de Oliveira, S. K. F., Freitas, C. S., Marcon, R., Schwanke, R. C., Siqueira, J. M., & Calixto, J. Β. (2016) · Brazilian Journal of Medical and Biological Research · doi:10.1590/1414-431x20165644 · Cited by 99
“Studies conducted by researchers from Amgen and Bayer Healthcare showed a very low rate of reproducibility (89 and 78%, respectively) of articles published in international journals (71,72).”
BackgroundPerspectives on In Vitro to In Vivo Extrapolations
Härtung, T. (2017) · Applied In Vitro Toxicology · doi:10.1089/aivt.2016.0026 · Cited by 100
Some Methodological Deficiencies in Empirical Research Articles in Accounting
Dyckman, T. R., & Zeff, S. A. (2014) · Accounting Horizons · doi:10.2308/acch-50818 · Cited by 81
“10 Documentation of these two statements involving even some of the more famous researchers in the natural sciences can be found in Broad and Wade (1982), Hartmann and Moers (1999 and 2003), and Bailey, Hasselback, and Karcher (2001).”
BackgroundInfluentialGood Laboratory Practice for optimization research
Kendall, G., Bai, R., Błażewicz, J., Causmaecker, P. D., Gendreau, M., John, R. I., Li, J., McCollum, B., Pesch, E., Qu, R., et al. (2015) · Journal of the Operational Research Society · doi:10.1057/jors.2015.77 · Cited by 95
“Begley and Ellis (2012) argue that standards need to be raised in pre-clinical trials for cancer research.”
BackgroundInterpretations and methods: Towards a more effectively self-correcting social psychology
Jussim, L. J., Crawford, J. T., Anglin, S. M., Stevens, S. T., & Duarte, J. L. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.003 · Cited by 89
“…to the integrity of scientific research has recently come to the fore across the sciences, including questionable research practices, failures to replicate, publication biases, and political biases (Begley & Ellis, 2012; Duarte et al., 2015; Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011).”
BackgroundWe Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness
Muñoz, J., & Young, C. (2018) · Sociological Methodology · doi:10.1177/0081175018777988 · Cited by 84
“For example, in biomedicine, private industry labs have reported that 65 percent to 89 percent of “landmark” publications are based on nonreplicable false positive findings (Begley and Ellis 2012; Prinz et al. 2011).”
BackgroundTranslational Research in Cardiovascular Repair
Chamuleau, S. A. J., van der Naald, M., Climent, A. M., Kraaijeveld, A. O., Wever, K., Duncker, D. J., Fernández‐Avilés, F., & Bolli, R. (2018) · Circulation Research · doi:10.1161/circresaha.117.311565 · Cited by 78
“19 When investigated with computer simulations, a research claim is more likely to be false than true.”
Integrating imaging and omics data: A review
Antonelli, L., Guarracino, M. R., Maddalena, L., & Sangiovanni, M. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.04.032 · Cited by 79
Lemmon, V., Ferguson, A. R., Popovich, P. G., Xu, X., Snow, D. M., Igarashi, M., Beattie, C. E., & Bixby, J. L. (2014) · Journal of Neurotrauma · doi:10.1089/neu.2014.3400 · Cited by 81
“4,5 Similarly, a project funded by the National Institute of Neurological Disorders and Stroke (NINDS) resulted in a high failure rate in replicating published pre-clinical results for treatment of spinal cord injury (SCI).”
Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.0549 · Cited by 78
The possibility and desirability of replication in the humanities
Peels, R., & Bouter, L. M. (2018) · Humanities and Social Sciences Communications · doi:10.1057/s41599-018-0149-x · Cited by 93
Pogrow, S. (2019) · The American Statistician · doi:10.1080/00031305.2018.1549101 · Cited by 79
“Small effect sizes also appear to be a reason why so much of influential published experimental research cannot be replicated in psychology (Open Science Collaboration 2015), and oncology (Begley and Ellis 2012), and why laboratory research results often do not predict actual outcomes in psychiatry (Kraemer 2016).”
ResultScientific conclusions need not be accurate, justified, or believed by their authors
Dang, H., & Bright, L. K. (2021) · Synthese · doi:10.1007/s11229-021-03158-9 · Cited by 90
Murine Models of Inflammatory Bowel Disease (IBD): Challenges of Modeling Human Disease
DeVoss, J., & Diehl, L. J. (2013) · Toxicologic Pathology · doi:10.1177/0192623313509729 · Cited by 89
“However, recent publications have highlighted several major pharmaceutical companies’ struggles in attempting to reproduce published literature (Arrowsmith 2011; Prinz, Schlange, and Asadullah 2011; Begley and Ellis 2012).”
Festing, M. F. W. (2014) · ILAR Journal · doi:10.1093/ilar/ilu045 · Cited by 78
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
“However, numerous reports over the past few years lament the inability of others to replicate published scientific results (Begley & Ellis 2012; Jasny et al. 2011; Nat.”
BackgroundResultIs there a role for statistics in artificial intelligence?
Friedrich, S., Antes, G., Behr, S., Binder, H., Brannath, W., Dumpert, F., Ickstadt, K., Kestler, H. A., Lederer, J., Leitgöb, H., et al. (2021) · Advances in Data Analysis and Classification · doi:10.1007/s11634-021-00455-6 · Cited by 115
From Repeatability to Reproducibility and Corroboration
Feitelson, D. G. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723875 · Cited by 94
Confidence crisis of results in biomechanics research
Knudson, D. V. (2017) · Sports Biomechanics · doi:10.1080/14763141.2016.1246603 · Cited by 87
“…and robustness of experimental results in a variety of scientific fields due to bias from errors in experimental design and statistical analysis (Begley & Ellis, 2012; Buchanan & Lohse, 2016; Button et al., 2013; Hoekstra, Morey, Rouder, & Wagenmakers, 2014; Ioannidis, 2005, 2012; Pashler &…”
Backgroundhttps://www.altex.org/index.php/altex/article/view/1339
Krebs, A., Waldmann, T., Wilks, M. F., van Vugt-Lussenburg, B. M. A., van der Burg, B., Terron, A., Steger-Hartmann, T., Ruegg, J., Rovida, C., Pedersen, E., et al. (2019) · ALTEX · doi:10.14573/altex.1909271 · Cited by 77
Translating Mouse Models: Immune Variation and Efficacy Testing
Sellers, R. S. (2017) · Toxicologic Pathology · doi:10.1177/0192623316675767 · Cited by 77
“This sentiment seems to be increasingly common as drugs that are successful preclinically fail in clinical trials (Begley and Ellis 2012; Prinz, Schlange, and Asadullah 2011; Mak, Evaniew, and Ghert 2014).”
BackgroundInfluentialStrech, D., & Dirnagl, U. (2019) · BMJ Open Science · doi:10.1136/bmjos-2018-000048 · Cited by 74
Frost, A. D. S., & Ledgerwood, A. (2020) · Journal of Pacific Rim Psychology · doi:10.1017/prp.2020.7 · Cited by 76
Auspurg, K., & Brüderl, J. (2021) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231211024421 · Cited by 101
“Results were quite discomforting in that the audits reported an unexpectedly low replication rate (e.g., see Begley and Ellis 2012; Camerer et al. 2018; Christensen and Miguel 2018; Open Science Collaboration 2015).”
ResultChaulagain, B., Gothwal, A., Lamptey, R. N. L., Trivedi, R., Mahanta, A. K., Layek, B., & Singh, J. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24032710 · Cited by 75
The five pillars of computational reproducibility: bioinformatics and beyond
Ziemann, M., Poulain, P., & Bora, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad375 · Cited by 121
Bandrowski, A. E., & Martone, M. E. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.030 · Cited by 75
“Reproducibility in science and the scientific literature has been a hot topic these last few years, especially with Begley’s explosive report showing that Amgen, with all of its resources, could not reproduce many of the top cancer studies (Begley and Ellis, 2012).”
BackgroundPeer review: The current landscape and future trends
Jubb, M. (2016) · Learned Publishing · doi:10.1002/leap.1008 · Cited by 75
“…research, nor has it ensured that published research is reproducible by independent researchers (although there is much debate about replicability, especially in health and life sciences: e.g. Begley & Ellis, 2012; Mobley et al., 2013; Nature, 2014; Freedman et al., 2015; Jump, 2015a and 2015b).”
BackgroundIs Most Published Research Really False?
Leek, J. T., & Jager, L. R. (2016) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-060116-054104 · Cited by 90
A reversible haploid mouse embryonic stem cell biobank resource for functional genomics
Elling, U., Wimmer, R. A., Leibbrandt, A., Burkard, T., Michlits, G., Leopoldi, A., Micheler, T., Abdeen, D., Zhuk, S., Aspalter, I. M., et al. (2017) · Nature · doi:10.1038/nature24027 · Cited by 75
Boba: Authoring and Visualizing Multiverse Analyses
Liu, Y., Kale, A., Althoff, T., & Heer, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028985 · Cited by 73
Baltussen, G., Swinkels, L. A. P., & van Vliet, P. (2019) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.06.030 · Cited by 92
Life Science’s Average Publishable Unit (APU) Has Increased over the Past Two Decades
Cordero, R. J. B., Leon-Rodriguez, C. M. D., Alvarado-Torres, J. K., Rodriguez, A. R., & Casadevall, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0156983 · Cited by 74
“In recent years, the quality of the scientific literature has been questioned by an increasing number of retractions [6,7], the finding that most retractions are due to misconduct [7,53] and the realization that many published studies are not reproducible [3,4].”
BackgroundPickett, J. T., & Roche, S. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-017-9886-2 · Cited by 82
“Even beyond fraud, several highly publicized, large-scale replication efforts have reported low success rates, replicating only a minority of published findings (Begley and Ellis 2012; Open Science Collaboration 2015).”
Hendriks, F., Kienhues, D., & Bromme, R. (2020) · Public Understanding of Science · doi:10.1177/0963662520902383 · Cited by 73
Leach, S. M., Gibbings, S. L., Tewari, A., Atif, S. M., Vestal, B. E., Danhorn, T., Janssen, W. J., Wager, T. D., & Jakubzick, C. (2020) · Cell Reports · doi:10.1016/j.celrep.2020.108337 · Cited by 79
“In spite of these encouraging translational examples, numerous failures in drug development across multiple diseases make it clear that rodent-to-human translation remains an important challenge (Begley and Ellis, 2012).”
BackgroundKlages‐Mundt, A., Harz, D., Gudgeon, L., et al. (2020) · doi:10.1145/3419614.3423261 · Cited by 74
Authorization of Animal Experiments Is Based on Confidence Rather than Evidence of Scientific Rigor
Vogt, L., Reichlin, T. S., Nathues, C., & Würbel, H. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000598 · Cited by 73
“For example, in a study on microarray gene expression, only 8 out of 18 studies could be reproduced [2]; Prinz and colleagues [3] found large inconsistencies (65%) between published and in-house data in the fields of oncology, women’s health, and cardiovascular diseases; oncologists from Amgen could confirm only 6 out…”
ResultStriving for transparent and credible research: practical guidelines for behavioral ecologists
Ihle, M., Winney, I. S., Krystalli, A., & Croucher, M. (2017) · Behavioral Ecology · doi:10.1093/beheco/arx003 · Cited by 75
“Reports of researchers being unable to reproduce results within and between labs and the disproportionate attention that the irreproducible studies receive highlight several problems with how we conduct research (Prinz et al. 2011; Begley and Ellis 2012; Open Science Collaboration 2015).”
ResultReproducibility of science: Fraud, impact factors and carelessness
Eisner, D. A. (2018) · Journal of Molecular and Cellular Cardiology · doi:10.1016/j.yjmcc.2017.10.009 · Cited by 73
“Similarly, Amgen scientists could only confirm findings in 11% of “landmark” studies [4].”
ResultEvolution and impact of high content imaging.
Way, G. P., Sailem, H., Shave, S., Kasprowicz, R., & Carragher, N. O. (2023) · SLAS Discovery · doi:10.1016/j.slasd.2023.08.009 · Cited by 67
“our era where experiments fail to replicate more often than not [148], we must continue to report as much metadata as possible, using standardized identifiers [149].”
The researchers’ view of scientific rigor—survey on the conduct and reporting of in vivo research
Reichlin, T. S., Vogt, L., & Würbel, H. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165999 · Cited by 72
Imaging methods are vastly underreported in biomedical research
Marqués, G., Pengo, T., & Sanders, M. A. (2020) · eLife · doi:10.7554/elife.55133 · Cited by 77
Increasing the reproducibility of fluid biomarker studies in neurodegenerative studies
Mattsson-Carlgren, N., Palmqvist, S., Blennow, K., & Hansson, O. (2020) · Nature Communications · doi:10.1038/s41467-020-19957-6 · Cited by 72
“biomarker research 18,19 .”
Computational Methodologies for Real-Space Structural Refinement of Large Macromolecular Complexes
Goh, B. C., Hadden‐Perilla, J. A., Bernardi, R. C., Singharoy, A., McGreevy, R., Rudack, T., Cassidy, C. K., & Schulten, K. J. (2016) · Annual Review of Biophysics · doi:10.1146/annurev-biophys-062215-011113 · Cited by 75
“Reproducibility is a topic of intense debate in the biomedical and life sciences community (15, 52).”
BackgroundSystematic heterogenization for better reproducibility in animal experimentation
Richter, S. H. (2017) · Lab Animal · doi:10.1038/laban.1330 · Cited by 78
Developing New Immunosuppression for the Next Generation of Transplant Recipients: The Path Forward
Stegall, M. D., Morris, R. E., Alloway, R. R., & Mannon, R. B. (2016) · American Journal of Transplantation · doi:10.1111/ajt.13582 · Cited by 73
Graduate Biomedical Science Education Needs a New Philosophy
Bosch, G., & Casadevall, A. (2017) · mBio · doi:10.1128/mbio.01539-17 · Cited by 87
Björndahl, L., Barratt, C. L. R., Mortimer, D., Agarwal, A., Aitken, R. J., Alvarez, J. G., Aneck-Hahn, N., Arver, S., Baldi, E., Bassas, L., et al. (2022) · Human Reproduction · doi:10.1093/humrep/deac189 · Cited by 71
“Ten years ago, Begley and Ellis proposed a series of recommendations to improve the reliability of studies in preclinical cancer research (Begley and Ellis, 2012) that helped to initiate a series of developments to address and improve reproducibility.”
BackgroundPreclinical efficacy studies in investigator brochures: do they enable risk–benefit assessment?
Wieschowski, S., Chin, W. W. L., Federico, C., Sievers, S., Kimmelman, J., & Strech, D. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004879 · Cited by 70
Brown, N., Cambruzzi, J., Cox, P. J., Davies, M., Dunbar, J., Plumbley, D., Sellwood, M. A., Sim, A., Williams-Jones, B. I., Zwierzyna, M., et al. (2018) · Progress in medicinal chemistry · doi:10.1016/bs.pmch.2017.12.003 · Cited by 83
Ferreira, G. S., Veening-Griffioen, D. H., Boon, W., Moors, E. H. M., Wied, C. C. G., Schellekens, H., & van Meer, P. J. K. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218014 · Cited by 85
Perspective: Dimensions of the scientific method
Voit, E. O. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007279 · Cited by 79
Common Misconceptions about Data Analysis and Statistics
Motulsky, H. J. (2014) · The Journal of Pharmacology and Experimental Therapeutics · doi:10.1124/jpet.114.219170 · Cited by 68
Smetana, K., André, S., Kaltner, H., Kopitz, J., & Gabius, H.-J. (2013) · Expert Opinion on Therapeutic Targets · doi:10.1517/14728222.2013.750651 · Cited by 69
“In order to avoid disappointment in clinical testing, due to ‘suboptimal preclinical validation’ [117] or ‘overly optimistic assumptions’ [118], the target and the setting must be chosen critically.”
BackgroundMilette, K., Roseman, M., & Thombs, B. D. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2010.09.015 · Cited by 71
“Furthermore, robust evidence shows that the results of a significant proportion of preclinical studies cannot be replicated [31–35].”
BackgroundOptimizing sampling approaches along ecological gradients
Schweiger, A., Irl, S. D. H., Steinbauer, M. J., Dengler, J., & Beierkuhnlein, C. (2015) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12495 · Cited by 84
“In most cases, the reason for this was low statistical power of the studies caused by small sample size and/or a small number of replicates (Bakker, van Dijk & Wicherts 2012; Begley & Ellis 2012; Button et al. 2013).”
BackgroundScientists’ Reputations Are Based on Getting It Right, Not Being Right
Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002460 · Cited by 67
Rigorous Science: a How-To Guide
Casadevall, A., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01902-16 · Cited by 110
“Rigor has gained new prominence amid concerns about a lack of reproducibility in important studies (2, 3), an epidemic of retractions due to misconduct (4), and the discovery that the published literature is riddled with problematic images (5).”
BackgroundDistinct roles of theta and alpha oscillations in the involuntary capture of goal-directed attention
Harris, A. M., Dux, P. E., Jones, C. N., & Mattingley, J. B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.008 · Cited by 73
“…results in several scientific fields, including cognitive science (Open Science Collaboration 2015; Szucs & Ioannidis 2016), economics (Camerer et al. 2016), and medicine (Begley & Ellis, 2012; Prinz et al. 2011), this replication provided additional confidence in the veracity of our conclusions.”
A new ecosystem for evidence synthesis
Nakagawa, S., Dunn, A. G., Lagisz, M., Bannach‐Brown, A., Grames, E. M., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Westgate, M. J., Arnold, P. A., et al. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-1153-2 · Cited by 69
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 66
In search of preclinical robustness
Peers, I. S., Ceuppens, P. R., & Harbron, C. G. (2012) · Nature Reviews Drug Discovery · doi:10.1038/nrd3849 · Cited by 68
Clayton, A., Buschmann, D., Byrd, J. B., Carter, D. R. F., Cheng, L., Compton, C., Daaboul, G., Devitt, A., Falcon‐Perez, J. M., Gardiner, C., et al. (2018) · Journal of Extracellular Vesicles · doi:10.1080/20013078.2018.1473707 · Cited by 68
“Selections took account of gender, career stage balance and representation from all geographical chapters, and the workshop was structured from bench to bedside [1]: Pre-analytical variables, biobanking and vesicle isolation [2], discovery of EV biomarkers [3], systems/assays to detect EV-markers”
MethodResultCutaneous Papillomaviruses and Non-melanoma Skin Cancer: Causal Agents or Innocent Bystanders?
Hasche, D., Vinzón, S. E., & Rösl, F. (2018) · Frontiers in Microbiology · doi:10.3389/fmicb.2018.00874 · Cited by 76
“Since there is a strong social pressure on basic research to bring reproducible laboratory results into the clinic (Begley and Ellis, 2012), there are indeed many recent initiatives to improve this transfer by validating the relevance and impact of in vitro and in vivo models with respect of their translational applic…”
BackgroundGholaminejad, A., Tehrani, H. A., & Fesharaki, M. G. (2018) · Journal of Nephrology · doi:10.1007/s40620-018-0511-5 · Cited by 69
Refining animal research: The Animal Study Registry
Bert, B., Heinl, C., Chmielewska, J., Schwarz, F., Grune, B., Hensel, A., Greiner, M., & Schönfelder, G. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000463 · Cited by 67
Common misconceptions about data analysis and statistics
Motulsky, H. J. (2014) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-014-1037-6 · Cited by 71
“This is a form of multiple comparisons (Berry 2007).”
BackgroundGomar, J. J., Conejero‐Goldberg, C., Davies, P., Goldberg, T. E., & Initiative, A. D. N. (2014) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2013.11.009 · Cited by 67
“Given the often undervalued but widespread phenomenon of failure to replicate findings in published biomedical research [7,8], we believe that confirming the validity of a model of prediction for the transition from MCI to AD is of great value and it contributes to the clarification of the processes implicated in tran…”
López‐Lázaro, M. (2015) · Oncoscience · doi:10.18632/oncoscience.132 · Cited by 80
“Researchers blame these failures on the complexity of the disease and on the limitations of the preclinical tools we require for our research [9].”
BackgroundInfluentialAndy’s Algorithms: new automated digital image analysis pipelines for FIJI
Law, A. M. K., Yin, J. X. M., Castillo, L., Young, A. I. J., Piggin, C., Rogers, S., Caldon, C. E., Burgess, A., Millar, E. K. A., O’Toole, S. A., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-15885-6 · Cited by 67
Leichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
“Psychology is not alone with this worrying result — many other disciplines such as neuroscience (Button et al., 2013), economics (Ioannidis et al., 2017) or cancer research (Begley and Ellis, 2012) have also found problems of replicability.”
BackgroundImpacted Science: Impact Is Not Importance
Casadevall, A., & Fang, F. C. (2015) · mBio · doi:10.1128/mbio.01593-15 · Cited by 74
“Even when there is no obvious misconduct, there is a concern that many studies in highly cited journals are not reproducible (23).”
BackgroundUnderstanding experiments and research practices for reproducibility: an exploratory study
Samuel, S., & König‐Ries, B. (2021) · PeerJ · doi:10.7717/peerj.11140 · Cited by 76
“Many studies have also been attempted to check the reproducibility of published results by replicating studies ( Ioannidis et al., 2009 ; Prinz, Schlange & Asadullah, 2011 ; Begley & Ellis, 2012 ; Pimentel et al., 2019 ; Raff, 2019 ).”
BackgroundResultInfluentialDickey, D. D., & Giangrande, P. H. (2015) · Methods · doi:10.1016/j.ymeth.2015.11.020 · Cited by 66
“Even more concerning is the idea that the results of 47 out of 53 highimpact preclinical studies were unable to be replicated largely due to inadequately characterized antibodies [82, 83].”
BackgroundNAFLD Preclinical Models: More than a Handful, Less of a Concern?
Oligschlaeger, Y., & Shiri‐Sverdlov, R. (2020) · Biomedicines · doi:10.3390/biomedicines8020028 · Cited by 74
Miedema, F. F. (2021) · Springer eBooks · doi:10.1007/978-94-024-2115-6 · Cited by 136
Reproducibility of Fluorescent Expression from Engineered Biological Constructs in E. coli
Beal, J., Haddock-Angelli, T., Gershater, M., de Mora, K., Lizarazo, M., Hollenhorst, J., Rettberg, R., & Contributors, I. I. S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150182 · Cited by 65
Project INTEGRATE: An integrative study of brief alcohol interventions for college students.
Mun, E., The Project INTEGRATE Team, TORRE, J. D. L., Atkins, D. C., WHITE, H. R., Ray, A. E., Kim, S. Y., Jiao, Y., Clarke, N., Huo, Y., et al. (2014) · Psychology of Addictive Behaviors · doi:10.1037/adb0000047 · Cited by 63
“In addition, published findings in the biomedical, as well as psychological, research fields have poor replicability (Begley & Ellis, 2012; Ioannidis, 2005; Nosek, Spies, & Motyl, 2012).”
BackgroundBello, N. M., Kramer, M., Tempelman, R. J., Stroup, W. W., St-Pierre, N. R., Craig, B. Α., Young, L. J., & Gbur, E. E. (2016) · Journal of Dairy Science · doi:10.3168/jds.2016-11516 · Cited by 66
“…is likely to have implications on the seemingly pervasive issue of “research irreproducibility” across scientific disciplines that has been raised in multiple editorials published over the last several years (Ioannidis, 2005; Begley and Ellis, 2012; Nuzzo, 2015; Open Science Collaboration, 2015).”
BackgroundThe challenges with the validation of research antibodies
Voskuil, J. L. A. (2017) · F1000Research · doi:10.12688/f1000research.10851.1 · Cited by 66
Mogil, J. S. (2017) · Lab Animal · doi:10.1038/laban.1223 · Cited by 78
Nanomedicine: is It Lost in translation?
Greish, K. F., Mathur, A., Bakhiet, M., & Taurin, S. (2018) · Therapeutic Delivery · doi:10.4155/tde-2017-0118 · Cited by 71
Crisis or self-correction: Rethinking media narratives about the well-being of science
Jamieson, K. H. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708276114 · Cited by 78
“The authors add: “There are many examples of outstanding research that has been rapidly and reliably translated into clinical benefit” (42).”
BackgroundInfluentialCheng, H., Fan, X., Wang, X., Ye, E., Loh, X. J., Li, Z., & Wu, Y.-L. (2018) · Biomacromolecules · doi:10.1021/acs.biomac.7b01693 · Cited by 64
Why (and how) we should publish negative data
Nimpf, S., & Keays, D. A. (2019) · EMBO Reports · doi:10.15252/embr.201949775 · Cited by 72
Pitoulis, F. G., Watson, S. A., Perbellini, F., & Terracciano, C. M. (2019) · Cardiovascular Research · doi:10.1093/cvr/cvz341 · Cited by 74
A Proposal Regarding Reporting of In Vitro Testing Results
Smith, M. A., & Houghton, P. J. (2013) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-13-0043 · Cited by 69
“On the one hand, translational research conducted by academic researchers and evaluated by pharmaceutical companies has been found to often lack reproducibility, and claimed conclusions have not been supported by further independent testing (1, 2).”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.