Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 801–900 of 2,889
Brinkman, R. R. (2019) · Cytometry Part A · doi:10.1002/cyto.a.23883 · Cited by 37
“It is especially important for clinical studies due to what has been deemed a reproducibility crisis in medicine, as only 11% of a series of preclinical cancer studies could be replicated (1).”
BackgroundCampbell, A., Mohl, J. E., Gutiérrez, D. A., Varela-Ramírez, A., & Boland, T. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00082 · Cited by 36
“This method is a critical step in preclinical studies (Swinney and Anthony, 2011; Begley and Ellis, 2012; Peng et al., 2016).”
MethodScheidecker, B., Shinohara, M., Sugimoto, M., Danoy, M., Nishikawa, M., & Sakai, Y. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00524 · Cited by 36
Martin, A. K. (2022) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s0963180121000499 · Cited by 47
Translational paradigms in pharmacology and drug discovery
Mullane, K., Winquist, R. J., & Williams, M. J. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.10.019 · Cited by 35
“replicability of many studies [35,36].”
BackgroundInfluentialBlack and White as Valence Cues: A Large-Scale Replication Effort of
Meier, B. P., Fetterman, A. K., & Robinson, M. D. (2015) · Social Psychology · doi:10.1027/1864-9335/a000236 · Cited by 35
Lessons Learned from Radiation Oncology Clinical Trials
Liu, F., participants, F. T. W., Okunieff, P. G., Bernhard, E. J., Stone, H. B., Yoo, S. S., Coleman, C. N., Vikram, B., Brown, M., Buatti, J. M., et al. (2013) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-13-1116 · Cited by 35
“The lack of data reproducibility is a major problem for drug development, wherein two thirds of these studies have significant inconsistencies (69, 70).”
The empirical march: Making science better at self-correction.
Makel, M. C. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035803 · Cited by 39
“…evidence supporting our conclusions and recommendations, our fields may not be able to withstand the type of scandals that have besieged the medical or social psychological research communities when highly cited research fails to replicate (e.g., Begley & Ellis, 2012; Pashler & Wagenmakers, 2012).”
BackgroundWang, Z., & Ma’ayan, A. (2016) · F1000Research · doi:10.12688/f1000research.9110.1 · Cited by 33
Transparent reporting for reproducible science
Weissgerber, T. L., Garovic, V. D., Winham, S. J., Milic, N. M., & Prager, E. M. (2016) · Journal of Neuroscience Research · doi:10.1002/jnr.23785 · Cited by 33
“A recent paper highlighted low statistical power and small sample sizes as major obstacles to reproducibility in the neuro-sciences (Button et al. 2013).”
BackgroundQiu, Y., Wu, C., Jiang, J., Hao, Y., Zhao, Y., Xu, J., Yu, T., & Ji, P. (2016) · Materials Science and Engineering C · doi:10.1016/j.msec.2016.10.081 · Cited by 35
An, G. C. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0497-0 · Cited by 40
“In fact, this sensitivity to initial/experimental conditions is exactly how the Crisis of Reproducibility came to be recognized, as meticulously followed experimental procedures intended to replicate prior results were unable to do so (Begley and Ellis 2012).”
ResultKampa, M., Schick, A., Sebastian, A., Wessa, M., Tüscher, O., Kalisch, R., & Yuen, K. S. L. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116223 · Cited by 34
“Even lower replication rates of 11% have been reported in preclinical cancer research (Begley and Ellis, 2012).”
BackgroundThe role of replication in psychological science
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00329-2 · Cited by 49
Ekkekakis, P., Swinton, P. A., & Tiller, N. B. (2023) · Sports Medicine · doi:10.1007/s40279-023-01880-7 · Cited by 40
Greek, R., & Hansen, L. A. (2013) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2013.06.002 · Cited by 36
How Academia and the Pharmaceutical Industry Can Work Together
Rosenblatt, M. (2013) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201209-075ps · Cited by 34
Deun, J. V., & Hendrix, A. (2017) · Journal of Extracellular Vesicles · doi:10.1080/20013078.2017.1379835 · Cited by 36
“Failure to follow these principles can result in data that are difficult to interpret and reproduce, as has been reported for other fields in life science.[9] The MISEV guidelines as established by the ISEV board were an important first step in establishing standards for EV research.”
Tecnoscienza. Italian Journal of Science and Technology Studies
Pelizza, A. (2018) · Cited by 35
Cell Surface and Membrane Engineering: Emerging Technologies and Applications
Saeui, C. T., Mathew, M. P., Liu, L., Urias, E., & Yarema, K. J. (2015) · Journal of Functional Biomaterials · doi:10.3390/jfb6020454 · Cited by 38
Do Corresponding Authors Take Responsibility for Their Work? A Covert Survey
Teunis, T., Nota, S. P. F. T., & Schwab, J. H. (2014) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-014-3868-3 · Cited by 35
“After publication of a manuscript, authors should remain available for access to raw data and data clarification, especially because of the importance of individual participant data for meta-analyses [8, 11], the low reproducibility of basic science results [1, 7, 13], and the encountered difficulties when implementin…”
BackgroundResultData gaps limit the translational potential of preclinical research
Kleiman, R. J., & Ehlers, M. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aac9888 · Cited by 31
Examining Data Repository Guidelines for Qualitative Data Sharing
Antes, A. L., Walsh, H. A., Strait, M., Hudson‐Vitale, C., & DuBois, J. M. (2017) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264617744121 · Cited by 46
“First, sharing data supports transparency in research—a growing concern among the scientific community in light of concerns about data reproducibility (Begley & Ellis, 2012; Nosek et al., 2015; Nuzzo, 2015).”
BackgroundRedox-Responsive Micellar Nanoparticles from Glycosaminoglycans for CD44 Targeted Drug Delivery
Carvalho, A. M., Teixeira, R., Novoa-Carballal, R., Pires, R. A., Reis, R. L., & Pashkuleva, I. (2018) · Biomacromolecules · doi:10.1021/acs.biomac.8b00561 · Cited by 34
Reproducibility of antimicrobial test methods
Parker, A. E., Hamilton, M. A., & Goeres, D. M. (2018) · Scientific Reports · doi:10.1038/s41598-018-30282-3 · Cited by 33
Real life outcomes vs. clinical trial results
Zarbin, M. A. (2019) · Journal of Ophthalmic and Vision Research · doi:10.4103/jovr.jovr_279_18 · Cited by 34
“[9] Unfortunately, both in clinical science[10,11] and in basic science,[12,13] poor reproducibility of experimental results is an ongoing and important problem.”
BackgroundRecent development of gene therapy for pancreatic cancer using non-viral nanovectors
Liu, Y., Wu, W., Wang, Y., Han, S., Yuan, Y., Huang, J., Shuai, X., & Peng, Z. M. (2021) · Biomaterials Science · doi:10.1039/d1bm00748c · Cited by 36
Acute effects of alcohol on social and personal decision making
Karlsson, H., Persson, E., Perini, I., Yngve, A., Heilig, M. A., & Tinghög, G. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01218-9 · Cited by 39
CNS Target Identification and Validation: Avoiding the Valley of Death or Naive Optimism?
Hutson, P. H., Clark, J. A., & Cross, A. J. (2016) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010716-104624 · Cited by 40
Risk of bias reporting in the recent animal focal cerebral ischaemia literature
Bahor, Z., Liao, J., Macleod, M., Bannach-Brown, A., McCann, S., Wever, K., Thomas, J., Ottavi, T., Howells, D., Rice, A., et al. (2017) · Clinical science · doi:10.1042/cs20160722 · Cited by 34
Kappenman, E. S., & Keil, A. (2016) · Psychophysiology · doi:10.1111/psyp.12787 · Cited by 36
“…results (Ioannidis, 2005), underpowered studies (Button et al., 2013), and most recently by large-scale studies reporting failures to replicate published findings in fields such as psychology (Open Science Collaboration, 2015), medicine (Begley & Ellis, 2012), and neuroscience (Steckler, 2015).”
ResultEstablishment of genetically diverse patient-derived xenografts of colorectal cancer.
Burgenske, D. M., Monsma, D. J., Dylewski, D., Scott, S. B., Sayfie, A. D., Kim, D. G., Luchtefeld, M. A., Martin, K. R., Stephenson, P. L., III, Hostetter, G. H., et al. (2014) · Europe PMC (PubMed Central) · Cited by 37
Finding the Path to Biobank Sustainability Through Sound Business Planning
Henderson, M. K., Simeon‐Dubach, D., & Albert, M. (2015) · Biopreservation and Biobanking · doi:10.1089/bio.2015.29039.mh · Cited by 33
“The size and the cost of irreproducibility is enormous: up to 80–90% of pre-clinical research cannot be reproduced irrespective of the impact factor of the journal in which these articles have been published.(2,3) A recent article”
Torres, L., Liu, Y., Guitreau, A. M., et al. (2017) · Zebrafish · doi:10.1089/zeb.2017.1426 · Cited by 32
The international MAQC Society launches to enhance reproducibility of high-throughput technologies
Shi, L. M., Kusko, R. L., Wolfinger, R. D., Haibe‐Kains, B., Fischer, M., Sansone, S., Mason, C. E., Furlanello, C., Jones, W. D., Ning, B., et al. (2017) · Nature Biotechnology · doi:10.1038/nbt.4029 · Cited by 31
TOXsIgN: a cross-species repository for toxicogenomic signatures
Darde, T., Gaudriault, P., Béranger, R., Clément, L., Caillarec-Joly, A., Sallou, O., Bonvallot, N., Chevrier, C., Mazaud‐Guittot, S., Jégou, B., et al. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty040 · Cited by 37
Issues in Reproducible Simulation Research
Fitzpatrick, B. G. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0496-1 · Cited by 36
Exact replication: Foundation of science or game of chance?
Piper, S. K., Grittner, U., Rex, A., Riedel, N., Fischer, F. H., Nadon, R., Siegerink, B., & Dirnagl, U. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000188 · Cited by 36
Assessments of Research Competencies for Clinical Investigators: A Systematic Review
Ianni, P. A., Samuels, E. M., Eakin, B. L., Perorazio, T. E., & Ellingrod, V. L. (2019) · Evaluation & the Health Professions · doi:10.1177/0163278719896392 · Cited by 42
“Email: pianni@med.umich.edu are often compromised by methodological errors (Ioannidis et al., 2014), a lack of replicability and reproducibility (Begley & Ellis, 2012; Drucker, 2016; Peers, Ceuppens, & Harbron, 2012; Prinz, Schlange, & Asadullah, 2011), and statistical errors (Garcia-Berthou &…”
BackgroundSilva, C. J. P., Liebsch, G., Meier, R. J., Gutbrod, M. S., Balmayor, E. R., & van Griensven, M. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00595 · Cited by 37
“…as one of the most underappreciated problems of basic and translational research (Place et al., 2017; Ast and Mootha, 2019) potentially having a direct impact not only on the experimental results but in their reproducibility (Begley and Ellis, 2012; Begley and Ioannidis John, 2015; Karp, 2018).”
BackgroundForrest, W. F., Alicke, B., Mayba, O., Osinska, M., Jakubczak, M., Piątkowski, P., Choniawko, L., Starr, A., & Gould, S. E. (2020) · Cancer Research · doi:10.1158/0008-5472.can-20-0342 · Cited by 38
“Improvements to the statistical framework can simultaneously enhance decisionmaking (10), reduce the numbers of animals required (11), and support larger translational efforts by enabling summary results from each study to be compared through meta-analysis (12).”
BackgroundRodgers, P., & Collings, A. (2021) · eLife · doi:10.7554/elife.75830 · Cited by 34
Wieschowski, S., Silva, D. S., & Strech, D. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000391 · Cited by 32
Irreproducible text-book “knowledge”: The effects of color bands on zebra finch fitness
Wang, D., Forstmeier, W., Ihle, M., Khadraoui, M., Jerónimo, S., Martin, K., & Kempenaers, B. (2018) · Evolution · doi:10.1111/evo.13459 · Cited by 35
“However, current scientific practice is often far from reaching that ideal state (Begley and Ellis 2012; Collaboration 2015; Freedman et al. 2015; Baker 2016; Kousta et al. 2016; Forstmeier et al. 2017; Ihle et al. 2017).”
BackgroundProtzko, J., & Schooler, J. W. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch6 · Cited by 40
Antibody characterization is critical to enhance reproducibility in biomedical research
Kahn, R. A., Virk, H., Laflamme, C., Houston, D. W., Polinski, N. K., Meijers, R., Levey, A. I., Saper, C. B., Errington, T. M., Turn, R. E., et al. (2024) · eLife · doi:10.7554/elife.100211 · Cited by 32
Perspectives on organ-on-a-chip technology for natural products evaluation
Wang, X., Miao, Y., Zhao, X.-M., Liu, X., Hu, Y.-W., & Deng, D. (2024) · Food & medicine homology. · doi:10.26599/fmh.2024.9420013 · Cited by 43
Reporting research antibody use: how to increase experimental reproducibility
Helsby, M. A., Fenn, J. R., & Chalmers, A. D. (2013) · F1000Research · doi:10.12688/f1000research.2-153.v2 · Cited by 29
Sánchez‐Pagés, S., Rodríguez-Ruiz, C., & Turiégano, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112157 · Cited by 34
Naim, I., Song, Y. C., Liu, Q., Huang, L., Kautz, H., Luo, J., & Gildea, D. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1017 · Cited by 33
“However, these instructions are often incomplete, and do not spell out implicit assumptions and knowledge, causing the results to be difficult to reproduce (Begley and Ellis, 2012).”
BackgroundMellman, I., Hubbard-Lucey, V. M., Tontonoz, M. J., Kalos, M. D., Chen, D. S., Allison, J. P., Drake, C. G., Levitsky, H., Lonberg, N., van der Burg, S. H., et al. (2016) · Cancer Immunology Research · doi:10.1158/2326-6066.cir-16-0045 · Cited by 31
“This low success rate is due, in part, to a large number of drugs with suboptimal preclinical validation moving through the pipeline (11).”
BackgroundUlrich, R., Erdfelder, E., Deutsch, R., Strauss, B. M., Brüggemann, A., Hannover, B., Tuschen‐Caffier, B., Kirschbaum, C., Blickle, G., Møller, J., et al. (2016) · Psychologische Rundschau · doi:10.1026/0033-3042/a000296 · Cited by 37
Recent advancement in pyrrolidine moiety for the management of cancer: A review
Sahu, R., Shah, K., Malviya, R., Paliwal, D., Sagar, S. R., Singh, S. K., & Prajapati, B. G. (2024) · Results in Chemistry · doi:10.1016/j.rechem.2023.101301 · Cited by 31
Dissemination Bias in Systematic Reviews of Animal Research: A Systematic Review
Mueller, K. F., Briel, M., Strech, D., Meerpohl, J. J., Lang, B., Motschall, E., Gloy, V., Lamontagne, F., & Bassler, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0116016 · Cited by 32
“Since the number of systematic reviews of preclinical research is growing, also the problem of dissemination bias in systematic reviews of preclinical research is getting more important [15–17].”
BackgroundPrimary 4T1 tumor resection provides critical “window of opportunity” for immunotherapy
Ghochikyan, A., Davtyan, A., Hovakimyan, A., Davtyan, H., Poghosyan, A., Bagaev, A., Ataullakhanov, R. I., Nelson, E. L., & Agadjanyan, M. G. (2013) · Clinical & Experimental Metastasis · doi:10.1007/s10585-013-9619-0 · Cited by 34
“One major factor leading to inability to successfully translate preclinical animal model work into the clinical arena is the limited number of tumor cell lines recapitulating human disease, including the spontaneous development of metastases, some of which are poorly characterized, or subjected to extensive genetic dr…”
BackgroundEarly Phase Clinical Trial Designs – State of Play and Adapting for the Future
Harrington, J., Hernandez-Guerrero, T. C., & Basu, B. (2017) · Clinical Oncology · doi:10.1016/j.clon.2017.10.005 · Cited by 30
Hylander, B. L., Eng, J. W. ‐., & Repasky, E. A. (2017) · Advances in experimental medicine and biology · doi:10.1007/978-3-319-67577-0_12 · Cited by 35
“In two major studies by drug companies, Bayer [122] and Amgen [125] investigated the reproducibility of preclinical experiments and found that less than 25% and 11%, respectively, of the studies were able to be duplicated.”
BackgroundComparative oncology approach to drug repurposing in osteosarcoma
Parrales, A., McDonald, P. R., Ottomeyer, M., Roy, A., Shoenen, F. J., Broward, M. A., Bruns, T., Thamm, D. H., Weir, S. J., Neville, K. A., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194224 · Cited by 31
Mohammed, J. S., Karim, N. A. A., Eliaser, E. M., Waziri, P. M., Ahmed, H., Barau, M. M., Kong, L., & Razis, A. F. A. (2018) · Nutrients · doi:10.3390/nu10091174 · Cited by 33
“However, studies have shown that the major worrying factor in the drug development process and discovery is the water insolubility of potential anti-cancer agents due to the presence of bulky hydrophobic moieties in their structures [71,72], which reflect difficulties in biotransformation with eventual therapeutic cha…”
BackgroundEthics and science: an introduction
(2013) · Choice Reviews Online · doi:10.5860/choice.50-4958 · Cited by 31
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
Quality Assurance in Biobanking for Pre-Clinical Research
Simeon‐Dubach, D., Zeisberger, S. M., & Hoerstrup, S. P. (2016) · Transfusion Medicine and Hemotherapy · doi:10.1159/000448254 · Cited by 33
Writing a scientific paper—A brief guide for new investigators
Vitse, C. L., & Poland, G. A. (2016) · Vaccine · doi:10.1016/j.vaccine.2016.11.091 · Cited by 37
“The results were surprising: scientific findings were confirmed in only 11% of the 53 studies they chose to review [25].”
BackgroundResultInfluentialPlant studies on fluctuating asymmetry in Russia: Mythology and methodology
Kozlov, M. V. (2017) · Russian Journal of Ecology · doi:10.1134/s1067413617010106 · Cited by 30
“The recently discovered low reproducibility of the results of biomedical research (Prinz et al., 2011; Begley and Ellis, 2012) has prompted many journals, including Nature and Science, to strengthen their requirements for the methodological parts of the manuscripts, paying special attention to…”
BackgroundCurran, T. (2018) · Open Biology · doi:10.1098/rsob.180098 · Cited by 28
Experimental replications in animal trials
Frommlet, F., & Heinze, G. (2020) · Laboratory Animals · doi:10.1177/0023677220907617 · Cited by 36
“The warnings of statisticians about the consequences of the commonly applied procedures of study design and data analysis have a long history,(5,6) but it seems only recently that the research community has become widely aware that there actually is a severe problem and that action has to be taken to improve the quali…”
BackgroundJohnson, M. E., Chen, A., Faeder, J. R., Henning, P., Moraru, I. I., Meier‐Schellersheim, M., Murphy, R. F., Prüstel, T., Theriot, J. A., & Uhrmacher, A. M. (2020) · Molecular Biology of the Cell · doi:10.1091/mbc.e20-08-0530 · Cited by 40
A reaction norm perspective on reproducibility
Voelkl, B., & Würbel, H. (2021) · Theory in Biosciences · doi:10.1007/s12064-021-00340-y · Cited by 37
Doctor Ex Machina: A Critical Assessment of the Use of Artificial Intelligence in Health Care
Svensson, A. M., & Jotterand, F. (2021) · The Journal of Medicine and Philosophy A Forum for Bioethics and Philosophy of Medicine · doi:10.1093/jmp/jhab036 · Cited by 41
Howard, M. O., & Garland, E. L. (2015) · Journal of the Society for Social Work and Research · doi:10.1086/681099 · Cited by 30
Lasota, J. P., Xi, L., Coates, T., et al. (2013) · Modern Pathology · doi:10.1038/modpathol.2013.89 · Cited by 30
Dasari, A. N., Phan, A. T., Gupta, S. K., Rashid, A., Yeung, S. J., Hess, K. R., Chen, H., Tarco, E., Chen, H., Wei, C., et al. (2015) · Endocrine Related Cancer · doi:10.1530/erc-15-0002 · Cited by 31
“…meaningful conclusions. iv) Finally, these discrepancies between preclinical and clinical results may reinforce the inherent limitations of the reproducibility and translation of preclinical data (Begley & Ellis 2012), especially in a tumor type such as NET, which lacks robust preclinical models.”
ResultSfanos, K. S., Yegnasubramanian, S., Nelson, W. G., Lotan, T. L., Kulaç, İ., Hicks, J. L., Zheng, Q., Bieberich, C. J., Haffner, M. C., & Marzo, A. M. D. (2018) · Asian journal of urology · doi:10.1016/j.ajur.2018.11.006 · Cited by 30
Explaining divergent observations regarding osteocalcin/GPRC6A endocrine signaling
Pi, M., Nishimoto, S. K., & Darryl Quarles, L. (2021) · Endocrinology · doi:10.1210/endocr/bqab011 · Cited by 29
“It is estimated that 70% of researchers fail to confirm another scientist’s experiments (59, 60).”
BackgroundEvaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M. (2017) · doi:10.31219/osf.io/5y8w7 · Cited by 29
How significant are your data? The need for a culture shift
Michel, M. C. (2014) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-014-1044-7 · Cited by 29
Robustness and reproducibility for AI learning in biomedical sciences: RENOIR
Barberis, A., Aerts, H., & Buffa, F. (2024) · Scientific Reports · doi:10.1038/s41598-024-51381-4 · Cited by 27
Singh, S. T., Pan, C., Wood, R. W., Yeh, C., Yeh, S., Sha, K., Krolewski, J. J., & Nastiuk, K. L. (2015) · BMC Urology · doi:10.1186/s12894-015-0091-9 · Cited by 28
“Specifically, the apparent efficacy in murine models is frequently not replicable in human trials [7, 8].”
BackgroundNonparametric Predictive Inference for Reproducibility of Basic Nonparametric Tests
Coolen, F. P. A., & Binhimd, S. M. S. (2013) · Journal of Statistical Theory and Practice · doi:10.1080/15598608.2013.819792 · Cited by 29
Tumor Heterogeneity and Personalized Cancer Medicine: Are we Being Outnumbered?
Cirkel, G. A., Hooijdonk, C. G. G., Koudijs, M. J., Willems, S. M., & Voest, E. E. (2014) · Future Oncology · doi:10.2217/fon.13.214 · Cited by 33
Biosafety Data as Confidential Business Information
Nielsen, K. M. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001499 · Cited by 33
Intersection of Systematic Review Methodology with the NIHReproducibility Initiative
Thayer, K. A., Wolfe, M. S., Rooney, A. A., Boyles, A. L., Bucher, J. R., & Birnbaum, L. S. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1408671 · Cited by 29
“Consistent with this opinion are reports that drug develop ment has been hin dered and many clinical trials wasted based on published findings from pre clinical studies that with further effort could not be reproduced (Begley and Ellis 2012).”
BackgroundKenakin, T., Bylund, D. B., Toews, M. L., Mullane, K., Winquist, R. J., & Williams, M. J. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.10.024 · Cited by 28
Langhof, H., Chin, W. W. L., Wieschowski, S., Federico, C. A., Kimmelman, J., & Strech, D. (2018) · British Journal of Pharmacology · doi:10.1111/bph.14485 · Cited by 34
“Regulatory guidelines could strongly support these activities (Begley and Ellis, 2012).”
BackgroundGuide to the statistical analysis plan
Yuan, I., Topjian, A., Kurth, C. D., Kirschen, M. P., Ward, C. G., Zhang, B., & Mensinger, J. L. (2019) · Pediatric Anesthesia · doi:10.1111/pan.13576 · Cited by 31
Yang, Y., Yang, H. H., Tang, B., Wu, A. M. L., Flanders, K. C., Moshkovich, N., Weinberg, D. S., Welsh, M. A., Weng, J., Ochoa, H. J., et al. (2020) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-19-2370 · Cited by 29
“However, there has been emerging awareness recently of limitations of the preclinical research endeavor in generating information that translates usefully into clinical practice (22, 23).”
BackgroundLalu, M. M., Montroy, J. R., Begley, C. G., Bubela, T. M., Hunniford, V. T., Ripsman, D., Wesch, N. L., Kimmelman, J., Macleod, M. R., Moher, D., et al. (2020) · F1000Research · doi:10.12688/f1000research.23663.2 · Cited by 35
Self-citationMisspecification and unreliable interpretations in psychology and social science.
Vowels, M. J. (2021) · Psychological Methods · doi:10.1037/met0000429 · Cited by 49
Karp, N. A., Pearl, E. J., Stringer, E. J., Barkus, C., Ulrichsen, J. C., & du Sert, N. P. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001873 · Cited by 36
Leithner, C., Füchtemeier, M., Jorks, D., Mueller, S. G., Dirnagl, U., & Royl, G. (2015) · Stroke · doi:10.1161/strokeaha.114.007832 · Cited by 27
Peer review versus editorial review and their role in innovative science
Steinhäuser, G., Adlassnig, W., Risch, J. A., Anderlini, S., Arguriou, P., Armendariz, A. Z., Bains, W., Baker, C. V., Barnes, M., Barnett, J., et al. (2012) · Metamedicine · doi:10.1007/s11017-012-9233-1 · Cited by 40
“Recent studies by world-class pharmaceutical companies that have failed to repeat high-profile peer reviewed research suggest that an astonishing number of such studies cannot be reproduced [11, 12].”
BackgroundKnopp, K. L., Stenfors, C., Baastrup, C., Bannon, A. W., Calvo, M., Caspani, O., Currie, G. L., Finnerup, N. B., Huang, W., Kennedy, J. D., et al. (2015) · Scandinavian Journal of Pain · doi:10.1016/j.sjpain.2015.01.006 · Cited by 32
The Application of Aptamers for Immunohistochemistry
Bauer, M., MacDonald, J. P., Henri, J., Duan, W., & Shigdar, S. L. (2016) · Nucleic Acid Therapeutics · doi:10.1089/nat.2015.0569 · Cited by 30
“Inconsistency in antibody purity and specificity has been implicated, at least in part, in the lack of reproducibility in 47 out of 53 landmark preclinical cancer trials controversially reported by Begley and Ellis in 2012 [12,13].”
Viscum album (L.) in experimental animal tumors: A meta-analysis
Bonamin, L. V., de Carvalho, A. C., & Waisse, S. (2017) · Experimental and Therapeutic Medicine · doi:10.3892/etm.2017.4372 · Cited by 30
“One hypothesis that could explain such a decrease in the number of studies is the trend towards reducing animal protocols in cancer research (34) due to their poor translational features, but this remains to be confirmed.”
A Postgenomic Perspective on Molecular Cytogenetics
Heng, H. H., Horne, S. D., Chaudhry, S. R., Regan, S. M., Liu, G., Abdallah, B. Y., & Ye, C. J. (2017) · Current Genomics · doi:10.2174/1389202918666170717145716 · Cited by 31
Yarborough, M. A., Bredenoord, A. L., D’Abramo, F., Joyce, N. C., Kimmelman, J., Ogbogu, U., Sena, E. S., Strech, D., & Dirnagl, U. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2006343 · Cited by 30
Cassotta, M., Pistollato, F., & Battino, M. (2019) · ALTEX · doi:10.14573/altex.1910011 · Cited by 39
Riehm, K. E., Azar, M., & Thombs, B. D. (2015) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2015.04.010 · Cited by 28
Open Science CollaborationPreprint
Nosek, B. A., Spies, J. R., Cohn, M., Bartmess, E. N., Lakens, D., Holman, D., Cohoon, J., Lewis, M., Gordon-McKeon, S., & IJzerman, H. R. (2012) · OSF Preprints (OSF Preprints) · Cited by 29
Pre-Registration in Social Psychology—a Discussion and Suggested TemplatePreprint
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4frms · Cited by 32
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.