Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 101–200 of 2,889
Protein purification and analysis: next generation Western blotting techniques
Mishra, M., Tiwari, S., & Gomes, A. V. (2017) · Expert Review of Proteomics · doi:10.1080/14789450.2017.1388167 · Cited by 301
“More in-depth studies showed that only 6 of 53 published preclinical studies could be replicated [34].”
BackgroundHow scientists fool themselves – and how they can stop
Nuzzo, R. (2015) · Nature · doi:10.1038/526182a · Cited by 296
No raw data, no science: another possible source of the reproducibility crisis
Miyakawa, T. (2020) · Molecular Brain · doi:10.1186/s13041-020-0552-2 · Cited by 268
Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation
Fourches, D., Muratov, E., & Tropsha, A. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00129 · Cited by 294
“Begley and Ellis 13 discussed a similar analysis performed at Amgen, yielding an even lower rate of reproducibility of 11%.”
BackgroundResearch waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers
Glasziou, P., & Chalmers, I. (2018) · BMJ · doi:10.1136/bmj.k4645 · Cited by 294
The 3Rs in research: a contemporary approach to replacement, reduction and refinement
MacArthur Clark, J. (2018) · British Journal of Nutrition · doi:10.1017/s0007114517002227 · Cited by 290
“Significant concern has been expressed in recent years about reproducibility with a review by Amgen able to confirm findings in only six out of fifty-three (11%) ‘landmark’ pre-clinical cancer trials((21)).”
BackgroundReplication in Psychological Science
Lindsay, D. S. (2015) · Psychological Science · doi:10.1177/0956797615616374 · Cited by 280
Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities
Bligaard, T., Bullock, R. M., Campbell, C. T., Chen, J. G., Gates, B. C., Gorte, R. J., Jones, C. W., Jones, W. D., Kitchin, J. R., & Scott, S. L. (2016) · ACS Catalysis · doi:10.1021/acscatal.6b00183 · Cited by 285
The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications
Bik, E. M., Casadevall, A., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.00809-16 · Cited by 287
“that only 6 could be reproduced (9), and other pharmaceutical”
BackgroundSoderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
“Confidence in the applied usefulness of the publication process has taken a hit, as clinical trials have demonstrated unusually low success rates when trying to reproduce academic results (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011; Turner, Matthews, et al., 2008).”
ResultPublication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
When Quality Beats Quantity: Decision Theory, Drug Discovery, and the Reproducibility Crisis
Scannell, J. W., & Bosley, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0147215 · Cited by 279
“However, in contrast [12], many results derived with today’s powerful tools appear irreproducible[13] [14][15] [16]; today’s drug candidates are more likely to fail in clinical trials than those in the 1970s [17] [18]; R&D costs per drug approved roughly doubled every ~9 years between 1950 and 2010 [19] [20] [1], with…”
ResultA machine learning Automated Recommendation Tool for synthetic biology
Radivojević, T., Costello, Z., Workman, K., & Garcia Martin, H. (2020) · Nature Communications · doi:10.1038/s41467-020-18008-4 · Cited by 277
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMJ Open Science · doi:10.1136/bmjos-2020-100115 · Cited by 273
Basic & Clinical Pharmacology & Toxicology Policy for Experimental and Clinical studies
Tveden‐Nyborg, P., Bergmann, T. K., & Lykkesfeldt, J. (2018) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.13059 · Cited by 271
Vasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271
“Unique antibody identification required at least one of the following: (1) an identifier resolving to a universal registry/database identifier such as the Antibody Registry (www. antibodyregistry.org) or eagle-i repository (http://www.eagle-i.net), or a vendor name and catalog number for resolving to a single offering…”
BackgroundInfluentialZhu, H., & Leung, S. W. (2015) · Diabetologia · doi:10.1007/s00125-015-3510-2 · Cited by 271
Murine Hind Limb Long Bone Dissection and Bone Marrow Isolation
Amend, S. R., Valkenburg, K. C., & Pienta, K. J. (2016) · Journal of Visualized Experiments · doi:10.3791/53936 · Cited by 267
Evidence-based Guidelines for Controlling pH in Mammalian Live-Cell Culture Systems
Michl, J., Park, K. C., & Swietach, P. (2019) · Communications Biology · doi:10.1038/s42003-019-0393-7 · Cited by 267
Extracellular matrix composition of connective tissues: a systematic review and meta-analysis
McKee, T. J., Perlman, G., Morris, M., & Komarova, S. V. (2019) · Scientific Reports · doi:10.1038/s41598-019-46896-0 · Cited by 266
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
What is wrong with evidence based policy, and how can it be improved?
Saltelli, A., & Giampietro, M. (2017) · Futures · doi:10.1016/j.futures.2016.11.012 · Cited by 264
Reproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262
Inherited Retinal Degenerations: Current Landscape and Knowledge Gaps
Duncan, J. L., Pierce, E. A., Laster, A. M., Daiger, S. P., Birch, D. G., Ash, J. D., Iannaccone, A., Flannery, J. G., Sahel, J. A., Zack, D. J., et al. (2018) · Translational Vision Science & Technology · doi:10.1167/tvst.7.4.6 · Cited by 257
Preserving Statistical Validity in Adaptive Data Analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. L. (2015) · Proceedings of the forty-seventh annual ACM symposium on Theory of Computing · doi:10.1145/2746539.2746580 · Cited by 248
“As an instance of this problem, we propose and investigate the question of estimating the expectations of m adaptively chosen functions on an unknown distribution given n random samples.”
MethodOrgan-on-a-chip meets artificial intelligence in drug evaluation
Deng, S., Li, C., Cao, J., Cui, Z., Du, J., Fu, Z., Yang, H., & Chen, P. (2023) · Theranostics · doi:10.7150/thno.87266 · Cited by 260
Llovera, G., Hofmann, K., Roth, S., Salas-Pérdomo, A., Ferrer-Ferrer, M., Perego, C., Zanier, E. R., Mamrak, U., Rex, A., Party, H., et al. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aaa9853 · Cited by 253
Disease model discovery from 3,328 gene knockouts by The International Mouse Phenotyping Consortium
Meehan, T. F., Conte, N., West, D. B., Jacobsen, J. O., Mason, J., Warren, J., Chen, C.-K., Tudose, I., Relac, M., Matthews, P., et al. (2017) · Nature Genetics · doi:10.1038/ng.3901 · Cited by 243
Design of spherically structured 3D in vitro tumor models -Advances and prospects
Ferreira, L. P., Gaspar, V. M., & Mano, J. F. (2018) · Acta Biomaterialia · doi:10.1016/j.actbio.2018.05.034 · Cited by 249
Matheson, G. J. (2019) · PeerJ · doi:10.7717/peerj.6918 · Cited by 244
Embracing failure: What the Phase III progesterone studies can teach about TBI clinical trials
Stein, D. G. (2015) · Brain Injury · doi:10.3109/02699052.2015.1065344 · Cited by 249
“problem in the field of stroke, while Begley and Ellis [9] make the same claim for pre-clinical cancer research.”
BackgroundModel Uncertainty and Robustness: A Computational Framework for Multimodel Analysis
Young, C., & Holsteen, K. (2017) · Sociological Methods & Research · doi:10.1177/0049124115610347 · Cited by 239
Applicability Domain for QSAR Models: Where Theory Meets Reality
Gadaleta, D., Mangiatordi, G. F., Catto, M., Carotti, A., & Nicolotti, O. (2016) · International Journal of Quantitative Structure-Property Relationships · doi:10.4018/ijqspr.2016010102 · Cited by 240
Integrins as Therapeutic Targets: Successes and Cancers
Raab-Westphal, S., Marshall, J., & Goodman, S. (2017) · Cancers · doi:10.3390/cancers9090110 · Cited by 241
“The limited translatability of such cell-line derived tumor models to human disease has been extensively discussed, and is one of the many contributory factors to the lack of reproducibility of preclinical cancer studies [215].”
BackgroundGit can facilitate greater reproducibility and increased transparency in science
Ram, K. (2013) · Source Code for Biology and Medicine · doi:10.1186/1751-0473-8-7 · Cited by 230
Lazic, S. E., & Essioux, L. (2013) · BMC Neuroscience · doi:10.1186/1471-2202-14-37 · Cited by 237
“Given the above concerns, it is not surprising that the pharmaceutical industry has difficulty reproducing many published results [36-39].”
ResultHutton, C., Wagener, T., Freer, J., Han, D., Duffy, C., & Arheimer, B. (2016) · Water Resources Research · doi:10.1002/2016wr019285 · Cited by 233
Microbiota and reproducibility of rodent models
Franklin, C. L., & Ericsson, A. C. (2017) · Lab Animal · doi:10.1038/laban.1222 · Cited by 232
Artificial Intelligence for Drug Discovery: Are We There Yet?
Hasselgren, C., & Oprea, T. I. (2024) · Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-040323-040828 · Cited by 236
“Bayer (171) and Amgen (172) have reported low reproducibility rates (33% and 11%, respectively) of high-impact publications, and many biomedical publications are false (173).”
BackgroundSix red flags for suspect work
Begley, C. G. (2013) · Nature · doi:10.1038/497433a · Cited by 230
Self-citationBon-EV: an improved multiple testing procedure for controlling false discovery rates
Li, D., Xie, Z., Zand, M., Fogg, T., & Dye, T. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1414-x · Cited by 230
“In recent years, irreproducibility of results in biomedical research has drawn increasing attention in the popular press and academia [17, 18].”
BackgroundImproving target assessment in biomedical research: the GOT-IT recommendations
Emmerich, C. H., Gamboa, L. M., Hofmann, M. C. J., Bonin-Andresen, M., Arbach, O., Schendel, P., Gerlach, B., Hempel, K., Bespalov, A., Dirnagl, U., et al. (2021) · Nature Reviews Drug Discovery · doi:10.1038/s41573-020-0087-3 · Cited by 226
Modelling science trustworthiness under publish or perish pressure
Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. A. (2018) · Royal Society Open Science · doi:10.1098/rsos.171511 · Cited by 225
“Across much of experimental science from psychology [10] to biomedical science [11–13] and cancer research [14], there is concern over an apparent reproducibility crisis.”
BackgroundDielectrophoresis for Biomedical Sciences Applications: A Review
Abd Rahman, N., Ibrahim, F., & Yafouz, B. (2017) · Sensors · doi:10.3390/s17030449 · Cited by 228
“Medical sciences are an integrated multidisciplinary field of study that has mainly focused on healthcare diagnostics [48] [49] [50] .”
BackgroundPfefferle, A. D., Herschkowitz, J. I., Usary, J., Harrell, J. C., Spike, B. T., Adams, J. R., Torres-Arzayus, M. I., Brown, M., Egan, S. E., Wahl, G. M., et al. (2013) · Genome Biology · doi:10.1186/gb-2013-14-11-r125 · Cited by 224
“This low success rate reflects not only the difficulty of developing anticancer therapeutics, but also identifies flaws in preclinical testing methodology for selecting the most appropriate cancer patient subset for early clinical testing [5,6].”
Earth, D. O. (2014) · International Linear Collider · Cited by 223
Garner, J. P. (2014) · ILAR Journal · doi:10.1093/ilar/ilu047 · Cited by 221
Genomic evolution of cancer models: perils and opportunities
Ben-David, U., Beroukhim, R., & Golub, T. R. (2019) · Nature Reviews Cancer · doi:10.1038/s41568-018-0095-3 · Cited by 222
Holzinger, A., Dehmer, M., & Jurisica, I. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s6-i1 · Cited by 219
“Often it is not possible to verify and to replicate experiments, which is the case for example in classical non-computing experimental sciences [126].”
BackgroundTatem, K. S., Quinn, J. L., Phadke, A., Yu, Q., Gordish-Dressman, H., & Nagaraju, K. (2014) · Journal of Visualized Experiments · doi:10.3791/51785 · Cited by 219
Paik, D. T., Chandy, M., & Wu, J. C. (2020) · Pharmacological Reviews · doi:10.1124/pr.116.013003 · Cited by 216
“…al., 2019; Wu et al., 2019a), dilated cardiomyopathy (Sun et al., 2012; Wu et al., 2015; Lee et al., 2019a), Brugada Syndrome (Liang et al., 2016; Belbachir et al., 2019), left ventricular noncompaction (Kodo et al., 2016), LEOPARD syndrome (Carvajal-Vergara et al., 2010), and arrhythmogenic…”
BackgroundMachine Learning for Chemical Reactivity: The Importance of Failed Experiments
Strieth‐Kalthoff, F., Sandfort, F., Kühnemund, M., Schäfer, F. R., Kuchen, H., & Glorius, F. (2022) · Angewandte Chemie International Edition · doi:10.1002/anie.202204647 · Cited by 217
Poor Translatability of Biomedical Research Using Animals — A Narrative Review
Marshall, L. J., Bailey, J., Cassotta, M., Herrmann, K., & Pistollato, F. (2023) · Alternatives to Laboratory Animals · doi:10.1177/02611929231157756 · Cited by 219
Tumour-on-a-chip: microfluidic models of tumour morphology, growth and microenvironment
Tsai, H.-F., Trubelja, A., Shen, A. Q., & Bao, G. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0137 · Cited by 214
Preclinical research in Rett syndrome: setting the foundation for translational success
Katz, D. M., Berger-Sweeney, J. E., Eubanks, J. H., Justice, M. J., Neul, J. L., Pozzo-Miller, L., Blue, M. E., Christian, D., Crawley, J. N., Giustetto, M., et al. (2012) · Disease Models & Mechanisms · doi:10.1242/dmm.011007 · Cited by 206
“…in animal models of amyotrophic lateral sclerosis (ALS) (Scott et al., 2008), women’s health and cardiovascular disease (Prinz et al., 2011), cancer (Begley and Ellis, 2012; Prinz et al., 2011) or spinal cord injury (Steward et al., 2012), the authors of follow-up studies were unable to replicate…”
BackgroundMullane, K., & Williams, M. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2012.11.014 · Cited by 213
Prestigious Science Journals Struggle to Reach Even Average Reliability
Brembs, B. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00037 · Cited by 213
“However, in studies where several journals were covered, the highest ranking journals did not stand out with a particularly high reproducibility, suggesting it may be only average in these journals (Scott et al., 2008; Prinz et al., 2011; Begley and Ellis, 2012).”
BackgroundPublication bias in meta‐analyses from the Cochrane Database of Systematic Reviews
Kicinski, M., Springate, D. A., & Kontopantelis, E. (2015) · Statistics in Medicine · doi:10.1002/sim.6525 · Cited by 210
Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions
Higginson, A. D., & Munafò, M. R. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000995 · Cited by 192
Modelling diabetic nephropathy in mice
Azushima, K., Gurley, S. B., & Coffman, T. M. (2018) · Nature Reviews Nephrology · doi:10.1038/nrneph.2017.142 · Cited by 209
On determining sample size in experiments involving laboratory animals
Festing, M. F. (2018) · Laboratory Animals · doi:10.1177/0023677217738268 · Cited by 213
Dried Blood Spots for Global Health Diagnostics and Surveillance: Opportunities and Challenges
Lim, M. D. (2018) · The American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.17-0889 · Cited by 210
Future cancer research priorities in the USA: a Lancet Oncology Commission
Jaffee, E. M., Dang, C. V., Agus, D. B., Alexander, B. M., Anderson, K. C., Ashworth, A., Barker, A. D., Bastani, R., Bhatia, S., Bluestone, J. A., et al. (2017) · The Lancet Oncology · doi:10.1016/s1470-2045(17)30698-8 · Cited by 208
Choose and Use Your Chemical Probe Wisely to Explore Cancer Biology
Blagg, J., & Workman, P. (2017) · Cancer Cell · doi:10.1016/j.ccell.2017.06.005 · Cited by 203
“Another valuable technique is the engineering of a functional target to interact with chemical probes not recognized by the wild-type protein (Bishop et al., 2000; Baud et al., 2014).”
BackgroundAre university rankings useful to improve research? A systematic review
Vernon, M. M., Balas, E. A., & Momani, S. (2018) · PLOS ONE · doi:10.1371/journal.pone.0193762 · Cited by 201
Developing predictive assays: The phenotypic screening “rule of 3”
Vincent, F., Loria, P., Pregel, M., Stanton, R., Kitching, L., Nocka, K., Doyonnas, R., Steppan, C., Gilbert, A., Schroeter, T., et al. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab1201 · Cited by 199
Look back in anger – what clinical studies tell us about preclinical work
Hartung, T. (2013) · ALTEX · doi:10.14573/altex.2013.3.275 · Cited by 200
“Similarly, Begley and ellis (2012) from Amgen in Nature “Raise standards for preclinical cancer research … Fifty-three papers were deemed ‘landmark’ studies …scientific findings 2018.””
Is everything we eat associated with cancer? A systematic cookbook review
Schoenfeld, J. D., & Ioannidis, J. P. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.047142 · Cited by 198
“Studies may spuriously highlight results that barely achieve statistical significance (15, 23) or report effect estimates that either are overblown (24, 25) or cannot be replicated in other studies (24, 26, 27).”
BackgroundReliability of the chronic mild stress model of depression: A user survey
Willner, P. (2017) · Neurobiology of Stress · doi:10.1016/j.ynstr.2016.08.001 · Cited by 198
“For example, failure rates of 75e90% were found for attempts within the pharmaceutical industry to replicate preclinical studies (Prinz et al., 2011; Begley and Ellis, 2012); similarly, a failure rate of around two thirds was reported for attempts to replicate experiments published in high ranking…”
BackgroundDo the Apoe −/− and Ldlr −/– Mice Yield the Same Insight on Atherogenesis?
Getz, G. S., & Reardon, C. A. (2016) · Arteriosclerosis, Thrombosis, and Vascular Biology · doi:10.1161/atvbaha.116.306874 · Cited by 198
“itive or false-negative results, which, if present, would compound the interpretation of the differences discussed below.(49)”
Transparency and Reproducibility of Observational Cohort Studies Using Large Healthcare Databases.
Wang, S., Verpillat, P., Rassen, J., Patrick, A., Garry, E., & Bartels, D. (2016) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.329 · Cited by 197
“The issue of nonreproducibility of preclinical studies was recently highlighted in a number of science and nature articles, in which separate attempts by pharmaceutical, medical, or other research groups were able to reproduce only 11–45% of published preclinical studies, often after significant efforts by reproducing…”
BackgroundGene annotation bias impedes biomedical research
Haynes, W. A., Tomczak, A., & Khatri, P. (2018) · Scientific Reports · doi:10.1038/s41598-018-19333-x · Cited by 192
de Groot, A. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 188
Progress in using systematic reviews of animal studies to improve translational research
Hooijmans, C. R., & Ritskes-Hoitinga, M. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001482 · Cited by 195
“In 2012, Begley and Ellis called for opportunities to present negative data, and suggested that journal editors should initiate this cultural change [36].”
BackgroundThe multifactorial role of the 3Rs in shifting the harm-benefit analysis in animal models of disease
Graham, M. L., & Prescott, M. J. (2015) · European Journal of Pharmacology · doi:10.1016/j.ejphar.2015.03.040 · Cited by 195
“For example, in 2012 scientists from Amgen and the University of Texas M.D. Anderson Cancer Center reported in Nature their attempts to confirm findings from 53 ‘landmark’ papers in the preclinical cancer field (Begley and Ellis, 2012).”
BackgroundDirnagl, U., & Endres, M. (2014) · Stroke · doi:10.1161/strokeaha.113.004075 · Cited by 192
Tomczak, A., Mortensen, J. M., Winnenburg, R., Liu, C., Alessi, D. T., Swamy, V., Vallania, F., Lofgren, S., Haynes, W., Shah, N. H., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-23395-2 · Cited by 190
Toward best practice in cancer mutation detection with whole-genome and whole-exome sequencing
Xiao, W., Ren, L., Chen, Z., Fang, L. T., Zhao, Y., Lack, J., Guan, M., Zhu, B., Jaeger, E., Kerrigan, L., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00994-5 · Cited by 190
How Reproducible Are Isotherm Measurements in Metal–Organic Frameworks?
Park, J., Howe, J. D., & Sholl, D. S. (2017) · Chemistry of Materials · doi:10.1021/acs.chemmater.7b04287 · Cited by 186
What is science’s crisis really about?
Saltelli, A., & Funtowicz, S. (2017) · Futures · doi:10.1016/j.futures.2017.05.010 · Cited by 187
“The crisis leaves no field untouched, from preclinical medical research (Begley & Ellis, 2012; Begley, 2013) to clinical research (Ioannidis, 2016).”
BackgroundScientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
“…articles have been published since 1965 (Pan et al., 2016)—and evidence that many published findings are less robust than initially thought (Begley & Ellis, 2012; Errington et al., 2014; Open Science Collaboration, 2015; Prinz, Schlange, & Asadullah, 2011), researchers must determine how…”
BackgroundTowards Reproducible Brain-Wide Association Studies
Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Feczko, E., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.21.257758 · Cited by 181
Bergh, D. D., Sharp, B. M., Aguinis, H., & Li, M. (2017) · Strategic Organization · doi:10.1177/1476127017701076 · Cited by 182
“For example, scholars report an inability to replicate the findings of seminal studies (e.g. Begley and Ellis, 2012; Ioannidis, 2012), leading some to suggest a possible credibility crisis may exist in science (e.g. Baker, 2012; Carpenter, 2012; Cortina et al., 2017).”
ResultGil, Y., David, C. H., Demir, I., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 180
“Although it is recommended that code is shared as it is written [Barnes, 2010], there are a few best practices for preparing code for publication.”
BackgroundMethodCohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., Gaignard, A., Hinsen, K., Larmande, P., Bras, Y. L., Lemoine, F., et al. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.01.012 · Cited by 181
“Reproducibility has obviously been studied in science in larger contexts than computational reproducibility, in particular where wet experiments are involved, and especially in the context of preclinical studies ( e.g. , [25, 26, 27, 28], see [29] for a review on this topic).”
BackgroundWang, S. V., Schneeweiss, S., Berger, M. L., Brown, J., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O., Mullins, C. D., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4295 · Cited by 182
“There has been an increasing movement towards “open science”, an umbrella term that covers study registration, data sharing, public protocols and more detailed, transparent reporting.(19-28) To address these issues in the field of healthcare database research, a Joint Task Force between the International Society for P…”
BackgroundLazic, S. E., & Essioux, L. (2016) · Cited by 182
Applications of organoids for cancer biology and precision medicine
Lo, Y.-H., Karlsson, K., & Kuo, C. J. (2020) · Nature Cancer · doi:10.1038/s43018-020-0102-y · Cited by 182
Micro-dissected tumor tissues on chip: an ex vivo method for drug testing and personalized therapy
Astolfi, M., Péant, B., Lateef, M. A., Rousset, N., Kendall-Dupont, J., Carmona, E., Monet, F., Saad, F., Provencher, D., Mes-Masson, A.-M., et al. (2016) · Lab on a Chip · doi:10.1039/c5lc01108f · Cited by 182
Tan, V. Y., Lewis, S. J., Adams, J. C., & Martin, R. M. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-52 · Cited by 179
“Discussion The importance of demonstrating the reproducibility of biomarker studies to validate their utility for the clinic is increasingly well recognized [67].”
BackgroundCommon pitfalls in preclinical cancer target validation
Kaelin, W. G. (2017) · Nature Reviews Cancer · doi:10.1038/nrc.2017.32 · Cited by 175
Emerging tumor spheroids technologies for 3D in vitro cancer modeling
Rodrigues, T., Kundu, B., Silva-Correia, J., Kundu, S. C., Oliveira, J. M., Reis, R. L., & Correlo, V. M. (2018) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2017.10.018 · Cited by 178
Single cell–resolution western blotting
Kang, C.-C., Yamauchi, K. A., Vlassakis, J., Sinkala, E., Duncombe, T. A., & Herr, A. E. (2016) · Nature Protocols · doi:10.1038/nprot.2016.089 · Cited by 178
Wu, Y., & Xie, L. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.030 · Cited by 178
“Here, we introduce two major approaches: multi-modal supervised learning and knowledge graph.”
Metabolomics and Multi-Omics Integration: A Survey of Computational Methods and Resources
Eicher, T., Kinnebrew, G., Patt, A., Spencer, K., Ying, K., Ma, Q., Machiraju, R., & Mathé, E. A. (2020) · Metabolites · doi:10.3390/metabo10050202 · Cited by 177
Imaging preclinical tumour models: improving translational power
de Jong, M., Essers, J., & van Weerden, W. M. (2014) · Nature Reviews Cancer · doi:10.1038/nrc3751 · Cited by 176
Granger, N., Blamires, H., Franklin, R. J. M., & Jeffery, N. D. (2012) · Brain · doi:10.1093/brain/aws268 · Cited by 174
“The potential of these factors to obstruct effective translation has recently received increasing attention in other fields of medicine (Lowenstein and Castro, 2009; Begley and Ellis, 2012) but can be difficult to discern using conventional laboratory models.”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.