Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 501–600 of 2,889
The development of high-content screening (HCS) technology and its importance to drug discovery
Fraietta, I., & Gasparri, F. (2016) · Expert Opinion on Drug Discovery · doi:10.1517/17460441.2016.1165203 · Cited by 76
“poor target validation,[21] 2014 saw the highest number of FDA approvals of new molecular entities in 18 years, and”
BackgroundOpen science and modified funding lotteries can impede the natural selection of bad science
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.190194 · Cited by 66
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 71
“For example, 11% of landmark preclinical cancer trials (Begley & Ellis, 2012) and 35% of pharmacology studies were found to replicate (Prinz, Schlange & Asadullah, 2011).”
BackgroundReplicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
Dror, R., Baumer, G., Bogomolov, M., & Reichart, R. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00074 · Cited by 63
“…to as the cornerstone of science (Moonesinghe et al., 2007), replicability analysis is of predominant importance in many scientific fields including psychology (Collaboration, 2012), genomics (Heller et al., 2014), economics (Herndon et al., 2014) and medicine (Begley and Ellis, 2012), among others.”
BackgroundWhiteaker, J. R., Zhao, L., Yan, P., Ivey, R. G., Voytovich, U. J., Moore, H. D., Lin, C., & Paulovich, A. G. (2015) · Molecular & Cellular Proteomics · doi:10.1074/mcp.o115.050351 · Cited by 62
Expert curation in UniProtKB: a case study on dealing with conflicting and erroneous data
Poux, S., Magrane, M., Arighi, C. N., Bridge, A., O'Donovan, C., Laiho, K., & The UniProt Consortium (2014) · Database · doi:10.1093/database/bau016 · Cited by 62
The Critique of Empirical Social Science: New Policies atManagement and Organization Review
Lewin, A. Y., Chiu, C., Fey, C. F., Levine, S. S., McDermott, G., Murmann, J. P., & Tsang, E. W. K. (2016) · Management and Organization Review · doi:10.1017/mor.2016.43 · Cited by 68
“© 2016 The International Association for Chinese Management Research”
BackgroundInfluentialThe Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science
Freese, J., & Peterson, D. (2018) · Sociological Theory · doi:10.1177/0735275118794987 · Cited by 69
“This includes findings of poor rates of successful replication in drug target (Prinz, Schlange, and Asadullah 2011) and cancer (Begley and Ellis 2012) research and influential papers raising alarms about neuroscience (Button et al. 2013) and medical genetics (Greene et al. 2009).”
BackgroundParadis, E. G., Pinilla, L. T., Holder, B. P., Abed, Y., Boivin, G., & Beauchemin, C. A. A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0126115 · Cited by 61
In Vivo Tomographic Imaging of Deep-Seated Cancer Using Fluorescence Lifetime Contrast
Rice, W. L., Shcherbakova, D. M., Verkhusha, V. V., & Kumar, A. T. N. (2015) · Cancer Research · doi:10.1158/0008-5472.can-14-3001 · Cited by 68
“The technique presented here should, therefore, allow precise localization of early-stage metastasis across various organ systems and the efficient screening of cancer therapies (34).”
MethodImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Brain and Behavior · doi:10.1002/brb3.1141 · Cited by 60
Hossain, M., Habib, I., Singha, K., & Kumar, A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23172 · Cited by 84
Reprogramming of human cancer cells to pluripotency for models of cancer progression
Kim, J., & Zaret, K. S. (2015) · The EMBO Journal · doi:10.15252/embj.201490736 · Cited by 67
“Cancer therapeutics that work in mouse models can fail in clinical human trials (Ledford, 2011; Begley & Ellis, 2012).”
PhenStat: A Tool Kit for Standardized Analysis of High Throughput Phenotypic Data
Kurbatova, N., Mason, J. C., Morgan, H., Meehan, T. F., & Karp, N. A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131274 · Cited by 64
“Applying the appropriate statistical analysis is a challenge in assessing biological data [28 – 30] and is an area of active research for high throughput phenotyping [10,20].”
MethodAnimal Models in Translational Research: Rosetta Stone or Stumbling Block?
Bolker, J. A. (2017) · BioEssays · doi:10.1002/bies.201700089 · Cited by 75
Why the Reward Structure of Science Makes Reproducibility Problems Inevitable
Heesen, R. (2018) · The Journal of Philosophy · doi:10.5840/jphil20181151239 · Cited by 70
“Two studies by pharmaceutical companies reported that they could reproduce less than a quarter of results in cancer biology (Prinz et al. 2011, Begley and Ellis 2012).”
BackgroundNCTN Assessment on Current Applications of Radiomics in Oncology
Nie, K., Al‐Hallaq, H., Li, X. A., Benedict, S., Sohn, J. W., Moran, J. M., Fan, Y., Huang, M., Knopp, M. V., Michalski, J. M., et al. (2019) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2019.01.087 · Cited by 64
Effects of weaning age and housing conditions on phenotypic differences in mice
Bailoo, J. D., Voelkl, B., Varholick, J. A., Novak, J., Murphy, E., Rosso, M., Palme, R., & Würbel, H. (2020) · Scientific Reports · doi:10.1038/s41598-020-68549-3 · Cited by 73
Wu, V. S., Kanaya, N., Lo, C., Mortimer, J. E., & Chen, S. (2015) · The Journal of Steroid Biochemistry and Molecular Biology · doi:10.1016/j.jsbmb.2015.05.005 · Cited by 75
Guidelines for experiments using antisense oligonucleotides and double-stranded RNAs
Gagnon, K. T., & Corey, D. R. (2019) · Nucleic Acid Therapeutics · doi:10.1089/nat.2018.0772 · Cited by 59
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
Imagining the “open” university: Sharing scholarship to improve research and education
McKiernan, E. C. (2017) · PLoS Biology · doi:10.1371/journal.pbio.1002614 · Cited by 68
“In recent years, large-scale projects in the fields of psychology [78] and cancer biology [79, 80] have 273 a empted to reproduce key findings and found a low rate of reproducibility.”
BackgroundUser-friendly workflows for catchment modelling: Towards reproducible SWAT+ model studies
Chawanda, C. J., George, C., Thiery, W., Griensven, A. V., Tech, J., Arnold, J., & Srinivasan, R. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104812 · Cited by 45
Ioannidis, J. P. A. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt036 · Cited by 60
Flow Cytometry: Impact on Early Drug Discovery
Edwards, B. S., & Sklar, L. A. (2015) · SLAS DISCOVERY · doi:10.1177/1087057115578273 · Cited by 66
Genes, Environments, and Sex Differences in Alcohol Research
Salvatore, J. E., Cho, S. B., & Dick, D. M. (2017) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.2017.78.494 · Cited by 65
“RECENT CONCERNS REGARDING the reproducibilityof science have prompted growing interest in what investigators can do to increase confidence in findings that will ultimately inform preventive interventions to improve health (Begley & Ellis, 2012; Open Science Collaboration, 2015).”
BackgroundEvaluation of the selectivity and sensitivity of isoform- and mutation-specific RAS antibodies
Waters, A. M., Ozkan-Dagliyan, I., Vaseva, A. V., Fer, N., Strathern, L. A., Hobbs, G. A., Tessier-Cloutier, B., Gillette, W. K., Bagni, R., Whiteley, G. R., et al. (2017) · Science Signaling · doi:10.1126/scisignal.aao3332 · Cited by 61
Biomechanical regulation of drug sensitivity in an engineered model of human tumor
Marturano-Kruik, A., Villasante, A., Yaeger, K. A., Ambati, S. R., Chramiec, A. G., Raimondi, M. T., & Vunjak‐Novakovic, G. (2017) · Biomaterials · doi:10.1016/j.biomaterials.2017.10.020 · Cited by 67
Scientific productivity: An exploratory study of metrics and incentives
Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195321 · Cited by 68
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Synthesis of Findings, Current Investigations, and Future Directions: Operation Brain Trauma Therapy
Kochanek, P. M., Bramlett, H. M., Shear, D. A., Dixon, C. E., Mondello, S., Dietrich, W. D., Hayes, R. L., Wang, K., Poloyac, S. M., Empey, P. E., et al. (2015) · Journal of Neurotrauma · doi:10.1089/neu.2015.4133 · Cited by 60
“This was brought to light in a landmark article by Begley and Ellis 21 and Begley and Ioan-nidis 22 that called into question pre-clinical research in the field of cancer therapy development when the Hematology and Oncology Department at the biotechnology firm Amgen tried to confirm published findings from 53 articles tha…”
Microchip-based single-cell functional proteomics for biomedical applications
Lu, Y., Liu, Y., Wei, W., & Shi, Q. (2017) · Lab on a Chip · doi:10.1039/c7lc00037e · Cited by 62
Self-correction in science: Meta-analysis, bias and social structure
Bruner, J., & Holman, B. (2019) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2019.02.001 · Cited by 68
Tutty, M. A., Movia, D., & Prina‐Mello, A. (2022) · Drug Delivery and Translational Research · doi:10.1007/s13346-022-01147-0 · Cited by 75
“However, rodent models also show fundamental differences in anatomy, physiology and immune response [88-90].”
BackgroundZhang, H., Liu, X., Warden, C. D., Huang, Y., Loera, S., Xue, L., Zhang, S., Chu, P., Zheng, S., & Yen, Y. (2014) · BMC Cancer · doi:10.1186/1471-2407-14-664 · Cited by 60
“A recent investigation demonstrated that the reproducibility of current preclinical cancer researches was only 11% [52].”
BackgroundBringing together the academic drug discovery community
Slusher, B. S., Conn, P. J., Frye, S. V., Glicksman, M. A., & Arkin, M. R. (2013) · Nature Reviews Drug Discovery · doi:10.1038/nrd4155 · Cited by 61
Button, K. S., Bal, L., Clark, A., & Shipley, T. (2016) · BMC Psychology · doi:10.1186/s40359-016-0167-7 · Cited by 58
“The evidence of unreliable findings across biomedical and social sciences is compelling [2, 15, 20, 36, 42].”
Fecher, B., Friesike, S., Hebing, M., & Linek, S. B. (2017) · Humanities and Social Sciences Communications · doi:10.1057/palcomms.2017.51 · Cited by 71
“A study published in Nature reported the failure to replicate significant experiments in cancer research in 47 out of 53 cases (Begley and Ellis, 2012).”
BackgroundQuantity and Reporting Quality of Kidney Research
Chatzimanouil, M. K. T., Wilkens, L., & Anders, H.-J. (2019) · Journal of the American Society of Nephrology · doi:10.1681/asn.2018050515 · Cited by 60
Integrative Pharmacogenomics Analysis of Patient-Derived Xenografts
Mer, A. S., Ba-Alawi, W., Smirnov, P., Wang, Y. X., Brew, B., Ortmann, J., Tsao, M.-S., Cescon, D. W., Goldenberg, A., & Haibe-Kains, B. (2019) · Cancer Research · doi:10.1158/0008-5472.can-19-0349 · Cited by 59
Redford, K. H., Brooks, T. M., MacFarlane, N., & Adams, J. S. (2019) · doi:10.2305/iucn.ch.2019.05.en · Cited by 78
Be positive about negatives–recommendations for the publication of negative (or null) results
Bespalov, A., Steckler, T., & Skolnick, P. (2019) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2019.10.007 · Cited by 59
Hoffmann, S., Schönbrodt, F. D., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.201925 · Cited by 85
“In particular, a series of attempts to reproduce the results of published research findings in different disciplines found that these replication efforts produced much weaker evidence than the original study [4, 5, 6, 7].”
ResultInternational Guidelines for Veterinary Tumor Pathology: A Call to Action
Meuten, D. J., Moore, F. M., Donovan, T., Bertram, C., Klopfleisch, R., Foster, R. A., Smedley, R. C., Dark, M. J., Milovancev, M., Stromberg, P. C., et al. (2021) · Veterinary Pathology · doi:10.1177/03009858211013712 · Cited by 82
Rigor or Mortis: Best Practices for Preclinical Research in Neuroscience
Steward, O., & Balice‐Gordon, R. J. (2014) · Neuron · doi:10.1016/j.neuron.2014.10.042 · Cited by 63
“For example, one study in oncology research reports failure to replicate 90% of the tested papers (Begley and Ellis, 2012).”
Serendipity in Cancer Drug Discovery: Rational or Coincidence?
Prasad, S., Gupta, S. C., & Aggarwal, B. B. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2016.03.004 · Cited by 64
Science is not a signal detection problem
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914237117 · Cited by 56
A horizon scan of global conservation issues for 2015
Sutherland, W. J., Clout, M. N., Depledge, M. H., Dicks, L. V., Dinsdale, J., Entwistle, A. C., Fleishman, E., GIBBONS, D. W., Keim, B., Lickorish, F. A., et al. (2014) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2014.11.002 · Cited by 95
“For example, 47 of 53 landmark pre-clinical trials in cancer research could not be replicated (Begley & Ellis 2012).”
BackgroundCollins, B. C., Hunter, C. L., Liu, Y., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/074567 · Cited by 59
Stone, H. B., Bernhard, E. J., Coleman, C. N., Deye, J. A., Capala, J., Mitchell, J. B., & Brown, J. M. (2016) · Translational Oncology · doi:10.1016/j.tranon.2016.01.002 · Cited by 64
“[2,51–54]).”
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 59
“3%) landmark studies on the field of hematology and oncology were confirmed in replication studies [4].”
Reproducible image handling and analysis
Miura, K., & Nørrelykke, S. F. (2021) · The EMBO Journal · doi:10.15252/embj.2020105889 · Cited by 65
“Only six out of 53 could be reproduced (Begley & Ellis, 2012 ). each ROI.”
BackgroundAre we witnessing the decline effect in the Type D personality literature? What can be learned?
Coyne, J. C., & de Voogd, J. N. (2012) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2012.09.016 · Cited by 66
“Across fields, there have been a number of suggestions for changes in editorial practices that are intended to make the occurrence of decline effects less common [61–65].”
BackgroundMade in Academia: The Effect of Institutional Origin on Inventors’ Attention to Science
Bikard, M. A. (2018) · Organization Science · doi:10.1287/orsc.2018.1206 · Cited by 72
Thomenius, M. J., Totman, J., Harvey, D. M., Mitchell, L. H., Riera, T. V., Cosmopoulos, K., Grassian, A. R., Klaus, C. R., Foley, M., Admirand, E. A., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197372 · Cited by 64
“Although estimates have been attempted [44, 45], it is unknown how much of the literature in cancer biology will ultimately be effected by undiscovered artifacts associated with RNAi technology.”
From sociology of quantification to ethics of quantification
Saltelli, A., & Fiore, M. D. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00557-0 · Cited by 67
“A sectors hit by the reproducibility crisis, and the first where the alarm was sounded (Begley and Ellis, 2012), is that of biomedical research (Harris, 2017; Ioannidis, 2005, 2016), already affected by a collapse in trust (Goldacre, 2012).”
BackgroundExploiting metabolic vulnerabilities for personalized therapy in acute myeloid leukemia
Stuani, L., Sabatier, M., & Sarry, J.-E. (2019) · BMC Biology · doi:10.1186/s12915-019-0670-4 · Cited by 56
Cassotta, M., Forbes-Hernández, T. Y., Iglesias, R. C., Ruíz, R., Elexpuru-Zabaleta, M., Giampieri, F., & Battino, M. A. (2020) · Nutrients · doi:10.3390/nu12061827 · Cited by 62
Agent-Based Modeling in Systems Pharmacology
Cosgrove, J., Butler, J., Alden, K., Read, M. N., Kumar, V., Cucurull-Sanchez, L., Timmis, J., & Coles, M. C. (2015) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12018 · Cited by 68
RIPOSTE: a framework for improving the design and analysis of laboratory-based research
Masca, N. G., Hensor, E. M., Cornelius, V. R., Buffa, F. M., Marriott, H. M., Eales, J. M., Messenger, M. P., Anderson, A. E., Boot, C., Bunce, C., et al. (2015) · eLife · doi:10.7554/elife.05519 · Cited by 56
“Lack of reproducibility (or ‘irreproducibility’) is an acknowledged problem within biomedicine that has recently been gaining increased attention (Begley and Ellis, 2012; Begley, 2013; The Economist, 2013; Macleod et al., 2014).”
BackgroundIs Participation in Cancer Phase I Trials Really Therapeutic?
Kimmelman, J. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2016.67.9902 · Cited by 65
Genotypic variability enhances the reproducibility of an ecological study
Milcu, A., Freitas, R. P., Ellison, A. M., Blouin, M., Scheu, S., Freschet, G. T., Rose, L., Barot, S., Cesarz, S., Eisenhauer, N., et al. (2018) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0434-x · Cited by 62
Overview of precision oncology trials: challenges and opportunities
Fountzilas, E., & Tsimberidou, A. M. (2018) · Expert Review of Clinical Pharmacology · doi:10.1080/17512433.2018.1504677 · Cited by 64
Festing, M. F. W. (2020) · Scientific Reports · doi:10.1038/s41598-020-74538-3 · Cited by 66
The use and misuse of statistical methodologies in pharmacology research
Marino, M. J. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.05.017 · Cited by 61
“There has been growing concern that much of what is published in both the preclinical and clinical literature is misleading, resulting in the creation of a ‘‘house of cards’’ that undermines the core values of the biomedical research enterprise through key findings that cannot be replicated [1,2].”
BackgroundRETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
RetractedThe reproducibility of biomedical research: Sleepers awake!
Bustin, S. A. (2014) · Biomolecular Detection and Quantification · doi:10.1016/j.bdq.2015.01.002 · Cited by 56
Llovera, G., & Liesz, A. (2016) · Journal of Neurochemistry · doi:10.1111/jnc.13516 · Cited by 56
Skaga, E., Skaga, I. Ø., Grieg, Z., Sandberg, C. J., Langmoen, I. A., & Vik-Mo, E. O. (2019) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-019-02920-4 · Cited by 56
Duncan, R. (2017) · Journal of drug targeting · doi:10.1080/1061186x.2017.1358729 · Cited by 59
“Begley and Ellis [128] discuss the problem in relation to preclinical cancer research.”
BackgroundEnzastaurin: A lesson in drug development
Bourhill, T., Narendran, A., & Johnston, R. N. (2017) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2017.02.003 · Cited by 55
Fostering responsible research practices is a shared responsibility of multiple stakeholders
Bouter, L. M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.016 · Cited by 53
A creative destruction approach to replication: Implicit work and sex morality across cultures
Tierney, W., Hardy, J., Ebersole, C. R., Viganola, D., Clemente, E. G., Gordon, M., Hoogeveen, S., Haaf, J., Dreber, A., Johannesson, M., et al. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104060 · Cited by 51
“…movement to reexamine published findings suggests replication rates of 36% in psychology (Open Science Collaboration, 2015), 11–25% in biomedicine (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011), 61% in experimental economics (Camerer et al., 2016), 70% in experimental philosophy…”
Randomization Does Not Help Much, Comparability Does
Saint-Mont, U. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132102 · Cited by 55
Everitt, J. I. (2014) · Toxicologic Pathology · doi:10.1177/0192623314555162 · Cited by 65
“One experience that illustrates this concern is a report from the hematology/oncology department at Amgen, where industry scientists chose 53 ‘‘landmark’’ publications, many in top-tier journals, and were only able to confirm the findings completely in 6 of the publications (Begley and Ellis 2012).”
BackgroundQuality really matters: the need to improve specimen quality in biomedical research
Simeon‐Dubach, D., Burt, A. D., & Hall, P. A. (2012) · The Journal of Pathology · doi:10.1002/path.4117 · Cited by 55
“Therefore findings of low reproducibility of published scientific data as reported recently by two different groups are perhaps not surprising [6,7].”
BackgroundIntegrated genetic and pharmacologic interrogation of rare cancers
Hong, A. L., Tseng, Y., Cowley, G. S., Jonas, O. H., Cheah, J. H., Kynnap, B. D., Doshi, M. B., Oh, C., Meyer, S. C., Church, A. J., et al. (2016) · Nature Communications · doi:10.1038/ncomms11987 · Cited by 59
Ten Basic Rules of Antibody Validation
Weller, M. G. (2018) · Analytical Chemistry Insights · doi:10.1177/1177390118757462 · Cited by 62
Precision medicine review: rare driver mutations and their biophysical classification
Nussinov, R., Jang, H., Tsai, C., & Cheng, F. (2019) · Biophysical Reviews · doi:10.1007/s12551-018-0496-2 · Cited by 57
Page, M., Moher, D., Fidler, F., Higgins, J., Brennan, S., Haddaway, N. R., Hamilton, D., Kanukula, R., Karunananthan, S., Maxwell, L., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 39
Getting to reproducible antibodies: the rationale for sequenced recombinant characterized reagents
Bradbury, A. M., & Plückthun, A. (2015) · Protein Engineering Design and Selection · doi:10.1093/protein/gzv051 · Cited by 56
“Even though antibodies are widely considered to be an important contributor to the irreproducibility problem plaguing biomedical research (Begley and Ellis, 2012; Begley, 2013; Bradbury and Plückthun, 2015a,b), it may be challenging to convince scientists to switch from the animal-derived…”
BackgroundSzűcs, D., & Ioannidis, J. (2019) · bioRxiv · doi:10.1101/809715 · Cited by 414
Moving forward: challenges and directions for psychopathological network theory and methodologyPreprint
Fried, E. I., & Cramer, A. O. J. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mh3cf · Cited by 63
Identification of stable senescence‐associated reference genes
Hernandez‐Segura, A., Rubingh, R., & Demaria, M. (2019) · Aging Cell · doi:10.1111/acel.12911 · Cited by 54
“It is important that the field makes a shift toward better laboratory practices, particularly in times in which reproducibility of reports has been questioned (Baker & Dolgin, 2017; Begley & Ellis, 2012; eLife, 2017; Gutierrez, Mauriat, Pelloux, Bellini, & Van Wuytswinkel, 2008).”
BackgroundSingh, V. P., Pratap, K., Sinha, J., Desiraju, B. K., Bahal, D., & Kukreti, R. (2016) · International Journal of Immunopathology and Pharmacology · doi:10.1177/0394632016671728 · Cited by 63
“Data analysis Data extracted are analyzed by using the following statistical methods: (1) Choice effect size measure; (2) Calculation of an effect size for each comparison; (3) Choice of model: random and fixed effects model; (4) Calculation of a summary effect size; (5) Calculation the heterogeneity and if so, which…”
BackgroundMethodThe Antibody Society’s antibody validation webinar series
Voskuil, J. L. A., Bandrowski, A., Begley, C. G., Bradbury, A. R. M., Chalmers, A. D., Gomes, A. V., Hardcastle, T., Lund-Johansen, F., Plückthun, A., Roncador, G., et al. (2020) · mAbs · doi:10.1080/19420862.2020.1794421 · Cited by 54
Self-citationPrediction Interval: What to Expect When You’re Expecting … A Replication
Spence, J. R., & Stanley, D. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0162874 · Cited by 63
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
Opportunistic biases: Their origins, effects, and an integrated solution.
DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015) · American Psychologist · doi:10.1037/a0039191 · Cited by 51
“As the recent publicity of biased results in multiple fields (Begley & Ellis, 2012) suggests, if we are not able to reduce opportunistic biases in the practice of science, we risk losing continued scientific progress along with all of its practical benefits.”
BackgroundCan the behavioral sciences self-correct? A social epistemic study
Romero, F. (2016) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2016.10.002 · Cited by 57
Andersson, A., Remnestål, J., Nellgård, B., Vunk, H., Kotol, D., Edfors, F., Uhlén, M., Schwenk, J. M., Ilag, L. L., Zetterberg, H., et al. (2019) · Clinica Chimica Acta · doi:10.1016/j.cca.2019.03.243 · Cited by 53
“Despite a general agreement that assays in preclinical research do not need to follow the same standards as assays used in clinical applications, it is increasingly evident that the success of biomarker discovery depends on validated analytical assays [21,23,24].”
Gendoo, D. M. A. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.01.010 · Cited by 53
“Despite advancements, poor model characterization still affects development of anticancer treatments, and attributes to only a small percent of potential drugs passing FDA approval at the clinical stage [62– 64].”
BackgroundSustainability in drug discovery
Wynendaele, E., Furman, C., Wielgomas, B., Larsson, P., Hak, E., Block, T. R., Calenbergh, S. V., Willand, N., Markuszewski, M. J., Odell, L. R., et al. (2021) · Medicine in Drug Discovery · doi:10.1016/j.medidd.2021.100107 · Cited by 55
Preclinical models for drug discovery for metastatic disease
Wakefield, L. M., Agarwal, S., & Tanner, K. (2023) · Cell · doi:10.1016/j.cell.2023.02.026 · Cited by 61
Duration of psychological therapy: Relation to recovery and improvement rates in UK routine practice
Stiles, W. B., Barkham, M., & Wheeler, S. (2015) · British Journal of Psychiatry · doi:10.1192/bjp.bp.114.145565 · Cited by 53
van Meer, P. J. K., Graham, M. L., & Schuurman, H. (2015) · European Journal of Pharmacology · doi:10.1016/j.ejphar.2015.02.055 · Cited by 62
Common misconceptions about data analysis and statistics
Motulsky, H. J. (2014) · British Journal of Pharmacology · doi:10.1111/bph.12884 · Cited by 68
“, 2011), and investigators at Amgen were able to reproduce only 6 of 53 studies in basic cancer biology despite often cooperating with the original investigators (Begley and Ellis, 2012).”
Harnessing QbD, Programming Languages, and Automation for Reproducible Biology
Sadowski, M. I., Grant, C., & Fell, T. S. (2015) · Trends in biotechnology · doi:10.1016/j.tibtech.2015.11.006 · Cited by 60
“There may be other signs of impending problems: several recent studies have reported low rates of reproducibility across several areas of science including drug discovery [14], psychology [15], synthetic biology [16], medicine [17,18], and cancer research [19].”
BackgroundPal, D., Blair, H., Elder, A., Dormon, K., Rennie, K., Coleman, D. J. L., Weiland, J., Rankin, K. S., Filby, A. J., Heidenreich, O. T., et al. (2016) · Leukemia · doi:10.1038/leu.2016.79 · Cited by 63
Approaches to Evaluate Lung Inflammation in Translational Research
Meyerholz, D. K., Sieren, J. C., Beck, A. P., & Flaherty, H. A. (2017) · Veterinary Pathology · doi:10.1177/0300985817726117 · Cited by 59
“Recently, scientific reproducibility has emerged as a key element for improving the quality of scientific studies and data.(2,8,23) How do we develop this level of scientific rigor? Casadevall and Fang(24) suggested 5 fundamental elements for scientific rigor: redundancy in experimental design, sound statistical desig…”
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.