Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 1,201–1,300 of 2,889
The Problem of Biased Data and Potential Solutions for Health and Environmental Assessments
Suter, G. W., & Cormier, S. M. (2014) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.2014.974499 · Cited by 17
Rewarding high-power replication research
Wagenmakers, E., & Forstmann, B. U. (2013) · Cortex · doi:10.1016/j.cortex.2013.09.010 · Cited by 16
“…are reproducible is presently unknown, but results from cancer biology are cause for concern: pharmaceutical companies recently revealed that their efforts to replicate published preclinical findings succeed only rarely (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011;Wadman, 2013).”
Zimmer, A. S., & Steeg, P. S. (2014) · Journal of Molecular Medicine · doi:10.1007/s00109-014-1226-2 · Cited by 19
Lost in Translation: What is Stopping Inhaled Nanomedicines From Realizing Their Potential?
Kumar, A., Dailey, L. A., & Forbes, B. (2014) · Therapeutic Delivery · doi:10.4155/tde.14.47 · Cited by 19
What is the (p-) value of the P-value?
Gale, R. P., Hochhaus, A., & Zhang, M.-J. (2016) · Leukemia · doi:10.1038/leu.2016.193 · Cited by 17
Pasterkamp, G., Hoefer, I. E., & Prakken, B. J. (2016) · Nature Biotechnology · doi:10.1038/nbt.3691 · Cited by 16
From Famine to Feast: Developing Early-Phase Combination Immunotherapy Trials Wisely
Day, D., Monjazeb, A. M., Sharon, E., Ivy, S. P., Rubin, E. H., Rosner, G. L., & Butler, M. O. (2017) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-16-3064 · Cited by 17
“Currently, there are substantial limitations to nonclinical studies, including suboptimal reproducibility (74, 75), publication bias, and insufficiently characterized combination index (defined as a quantitative measure of combination drug effects; ref.”
BackgroundCoronavirus in context: Scite.ai tracks positive and negative citations for COVID-19 literature
Khamsi, R. (2020) · Nature · doi:10.1038/d41586-020-01324-6 · Cited by 16
Yarborough, M. A. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-106014 · Cited by 19
Becceneri, A. B., Fuzer, A. M., Plutin, A. M., Batista, A. A., Lelièvre, S. A., & Cominetti, M. R. (2020) · Inorganic Chemistry Frontiers · doi:10.1039/d0qi00502a · Cited by 17
Statistical reproducibility for pairwise t-tests in pharmaceutical research
Simkus, A., Coolen, F. P. A., Coolen‐Maturi, T., Karp, N. A., & Bendtsen, C. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211041765 · Cited by 23
“We determine the bounds L and R, based on the sample, as follows: L = x(1) −maxi(x(i) − x(i−1)) and R = x(n) +maxi(x(i) − x(i−1)), where i = 2, 3, .”
BackgroundMethodIs preclinical research in cancer biology reproducible enough?
Kane, P. B., & Kimmelman, J. (2021) · eLife · doi:10.7554/elife.67527 · Cited by 22
Satyamitra, M. M., Perez-Horta, Z., DiCarlo, A. L., Cassatt, D. R., Rios, C. I., Price, P. W., & Taliaferro, L. P. (2022) · Radiation Research · doi:10.1667/rade-21-00198.1 · Cited by 16
CURRENT BEST PRACTICES AND BIOBANKING RECOMMENDATIONS
Doludin, Y. V., Borisova, A. L., Pokrovskaya, M. S., Stefanyuk, O. V., Sivakova, O. V., Botsoeva, S. A., Meshkov, A. N., Metelskaya, V. A., & Drapkina, O. M. (2019) · Russian Clinical Laboratory Diagnostics · doi:10.18821/0869-2084-2019-64-12-769-776 · Cited by 16
Smith, A. J., Lambert, P. C., & Rutherford, M. J. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01654-1 · Cited by 15
Biobank Classification in an Australian Setting
Rush, A., Christiansen, J. H., Farrell, J. P., Goode, S. M., Scott, R. J., Spring, K. J., & Byrne, J. A. (2015) · Biopreservation and Biobanking · doi:10.1089/bio.2015.0007 · Cited by 16
“It is now well recognized that high, uniform biospecimen quality is integral for translational research, 34,35 and conversely, that in-adequate biospecimen quality can generate spurious results, waste already limited resources, and impede research progress.”
Fry, D. (2014) · ILAR Journal · doi:10.1093/ilar/ilu031 · Cited by 18
“…of positive outcomes (Bebarta et al. 2003), poor correspondence between animal experiments and clinical findings (Perel et al. 2006), the variation in reported effects (Scott et al. 2008), and difficulty in repeating published studies in-house (Begley and Ellis 2012; Prinz et al. 2011).”
ResultRepresentation of probabilistic scientific knowledge
Soldatova, L., Rzhetsky, A., Grave, K. D., & King, R. D. (2013) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-4-s1-s7 · Cited by 15
Scuderi, L. A., Weissmann, G. S., Hartley, A. J., Yang, X., & Lancaster, N. (2017) · Aeolian Research · doi:10.1016/j.aeolia.2017.05.004 · Cited by 18
The Ethics of Scientific Publishing: Black, White, and “Fifty Shades of Gray”
Zietman, A. L. (2017) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2017.06.009 · Cited by 17
“Begley and Ellis in a survey of more than 1500 investigators, found that whereas those working in physics and engineering had reasonable confidence in the work published in their fields, the vast majority of those in medicine believe that more than half of published results are simply not reproducible (2).”
BackgroundHarmonization of blood-based indicators of iron status: making the hard work matter
Hoofnagle, A. N. (2017) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.117.155895 · Cited by 18
“The translation of clinical research relies on reproducible, interpretable results (1).”
MethodApplication of Small Epigenetic Modulators in Pediatric Medulloblastoma
Zwergel, C., Romanelli, A., Stazi, G., Besharat, Z. M., Catanzaro, G., Tafani, M., Valente, S., & Mai, A. (2018) · Frontiers in Pediatrics · doi:10.3389/fped.2018.00370 · Cited by 16
“The failure of translating preclinical results into clinical success has been extensively discussed (20).”
BackgroundUnderstanding of researcher behavior is required to improve data reliability
Wass, M. N., Ray, L., & Michaelis, M. (2019) · GigaScience · doi:10.1093/gigascience/giz017 · Cited by 22
Insights into Criteria for Statistical Significance from Signal Detection Analysis
Witt, J. K. (2019) · Meta-Psychology · doi:10.15626/mp.2018.871 · Cited by 18
“This threshold has recently been challenged due to numerous failures to replicate findings published in top journals (Begley & Ellis, 2012; Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundInconsistency of findings due to low power: A structural MRI study of bilingualism
Munson, B. A., & Hernández, A. E. (2019) · Brain and Language · doi:10.1016/j.bandl.2019.104642 · Cited by 15
Bajgar, A., Saloň, I., Krejčová, G., Doležal, T., Jindra, M., & Štĕpánek, F. (2019) · Biomaterials Science · doi:10.1039/c9bm00539k · Cited by 16
CRITICAL REVIEW IN PHARMACEUTICAL SCIENCES
publisher, P. E. (2014) · ADVANCED SCIENCES INDEX · Cited by 16
Maximizing the Reproducibility of Your ResearchPreprint
Bosco, F. A., Carp, J., Field, J. G., IJzerman, H. R., Lewis, M., Munafo, M. R., Nosek, B. A., Prenoveau, J. M., Spies, J. R., & Giner‐Sorolla, R. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/k9mn3_v1 · Cited by 17
Personalized Pain Medicine: Pipe Dream or Reality?
Bruehl, S. (2015) · Anesthesiology · doi:10.1097/aln.0000000000000638 · Cited by 14
Better reporting for better research: a checklist for reproducibility
Kenall, A., Edmunds, S., Goodman, L., Bal, L., Flintoft, L., Shanahan, D. R., & Shipley, T. (2015) · GigaScience · doi:10.1186/s13742-015-0071-8 · Cited by 15
The Best Model of a Cat Is Several Cats
Gustafsson, C., & Vallverdú, J. (2016) · Trends in biotechnology · doi:10.1016/j.tibtech.2015.12.006 · Cited by 18
Linsen, L., Landuyt, K. V., & Ectors, N. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2019.00309 · Cited by 15
Quality of interventional animal experiments in Chinese journals: compliance with ARRIVE guidelines
Zhao, B. Q., Jiang, Y., Zhang, T., Shang, Z., Zhang, W., Hu, K., Chen, F., Mei, F., Gao, Q., Zhao, L., et al. (2020) · BMC Veterinary Research · doi:10.1186/s12917-020-02664-1 · Cited by 24
Testing the reproducibility and robustness of the cancer biology literature by robot
Roper, K., Abdel‐Rehim, A., Hubbard, S. A., Carpenter, M. R., Rzhetsky, A., Soldatova, L., & King, R. D. (2022) · Journal of The Royal Society Interface · doi:10.1098/rsif.2021.0821 · Cited by 23
Understanding the importance of quality control and quality assurance in preclinical PET/CT imaging
McDougald, W., & Mannheim, J. G. (2022) · EJNMMI Physics · doi:10.1186/s40658-022-00503-w · Cited by 22
Targeted immunotherapy and nanomedicine for rhabdomyosarcoma: The way of the future
Morel, V. J., Rössler, J. K., & Bernasconi, M. (2024) · Medicinal Research Reviews · doi:10.1002/med.22059 · Cited by 20
Examining the replicability of online experiments selected by a decision market
Holzmeister, F., Johannesson, M., Camerer, C. F., Chen, Y., Ho, T., Hoogeveen, S., Huber, J., Imai, N., Imai, T., Jin, L. J., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02062-9 · Cited by 26
Vitamin E and the Healing of Bone Fracture: The Current State of Evidence
Borhanuddin, B., Fozi, N. F. M., & Mohamed, I. N. (2012) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2012/684510 · Cited by 17
Huang, M. C., Ma, Q., Liu, X., Li, B., & Ma, H. (2015) · Analytical Chemistry · doi:10.1021/acs.analchem.5b00694 · Cited by 15
Navigating systems ideas for health practice: Towards a common learning device
Reynolds, M., Sarriot, E., Swanson, R. C., & Rusoja, E. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12872 · Cited by 16
“PhD (2), Robert Chad Swanson DO(3), and Evan Rusoja MD/PhD (4)”
BackgroundKharasch, E. D. (2018) · Anesthesiology · doi:10.1097/aln.0000000000002122 · Cited by 16
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results
Coker, B., Rudin, C. D., & King, G. (2021) · Management Science · doi:10.1287/mnsc.2020.3818 · Cited by 42
“Amgen replicated 11% of 53 scientific findings (Begley and Ellis 2012).”
BackgroundPublications Are Not the Finish Line: Focusing on Societal Rather Than Publication Impact
Kools, F. R. W., Mirali, S., Holst, S., Nijhof, S. L., Cavalli, G., & Grandner, M. A. (2018) · Frontiers in Medicine · doi:10.3389/fmed.2018.00314 · Cited by 21
Preconditioning against renal ischaemia reperfusion injury: the failure to translate to the clinic
O'Kane, D. B., Baldwin, G. S., Bolton, D. M., Ischia, J. J., & Patel, O. (2019) · Journal of Nephrology · doi:10.1007/s40620-019-00582-6 · Cited by 16
Leipold, S., Oderbolz, C., Greber, M., & Jäncke, L. (2019) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2018.12.016 · Cited by 13
Confounds in “Failed” Replications
Bressan, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01884 · Cited by 18
Cellular Metrology: Scoping for a Value Proposition in Extra- and Intracellular Measurements
Faruqui, N., Kummrow, A., Fu, B., Divieto, C., Rojas-Cornejo, F., Kisulu, F., Cavalcante, J. J. V., Wang, J., Campbell, J. J., Martins, J. L., et al. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00456 · Cited by 16
Machery, É. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00354-9 · Cited by 24
Pyridine Moiety: Recent Advances in Cancer Treatment
Sahu, R., Mishra, R., Kumar, R., Majee, C., Salahuddin, S., MAZUMDER, A., & Kumar, A. (2021) · Indian Journal of Pharmaceutical Sciences · doi:10.36468/pharmaceutical-sciences.763 · Cited by 17
How (not) to measure replication
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00377-2 · Cited by 19
Evans, N., Marušić, A., Foeger, N., Löfstrôm, E., van Hoof, M., Vrijhoef-Welten, S., Inguaggiato, G., Dierickx, K., Bouter, L. M., & Widdershoven, G. A. M. (2021) · Research Ideas and Outcomes · doi:10.3897/rio.7.e68258 · Cited by 13
Optimizing Research Output: How Can Psychological Research Methods Be Improved?
Miller, J. O., & Ulrich, R. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-094927 · Cited by 21
How failure to falsify in high-volume science contributes to the replication crisis
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2022) · eLife · doi:10.7554/elife.78830 · Cited by 25
Basic Statistics in Cell Biology
Vaux, D. L. (2014) · Annual Review of Cell and Developmental Biology · doi:10.1146/annurev-cellbio-100913-013303 · Cited by 15
Challenges in Applying the Results of Clinical Trials to Clinical Practice
Zarbin, M. A. (2016) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2016.1266 · Cited by 16
Mesenchymal Stem Cells Relevance in Multicellular Bioengineered 3D In Vitro Tumor Models
Ferreira, L. P., Gaspar, V. M., Henrique, R., Jerónimo, C., & Mano, J. F. (2017) · Biotechnology Journal · doi:10.1002/biot.201700079 · Cited by 15
“The bioengineering of such models must be based on similarities to human biology and disease specific features, so as to ensure higher predictability of preclinical research and exclude non-adequate anti-cancer therapeutic candidates prior to human clinical trials [3,13].”
The Academic–Industrial Complexity: Failure to Launch
Levin, L. A., & Béhar‐Cohen, F. (2017) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2017.10.003 · Cited by 15
“Bayer found that only 20–25% of academic findings could be reproduced by their in-house team [3], and Amgen found only 11% [4].”
BackgroundThe Elements of Translational Chemical Biology
Copeland, R. A., & Boriack‐Sjodin, P. A. (2017) · Cell chemical biology · doi:10.1016/j.chembiol.2017.11.003 · Cited by 15
Of mice and men: long-term safety of assisted reproduction treatments
Sharpe, R. M. (2018) · Human Reproduction · doi:10.1093/humrep/dey071 · Cited by 17
Ebuehi, O. A. T., Anams, C., Gbenle, O. D., & Ajagun‐Ogunleye, O. M. (2019) · Journal of Food Biochemistry · doi:10.1111/jfbc.12767 · Cited by 23
“One of the most devastating health threatening disease of this era is cancer, whose clinical research attempts have experienced fail‐ ures over time (Begley & Ellis, 2012).”
BackgroundAn assessment of transparency and reproducibility‐related research practices in otolaryngology
Johnson, A. L., Torgerson, T., Skinner, M., Hamilton, T., Tritz, D. J., & Vassar, M. (2019) · medRxiv · doi:10.1002/lary.28322 · Cited by 8
“Our findings align with previous literature in various fields of medicine.(15,22,23) Many of these irreproducible studies lack specific methodology, resultant data, or access to materials needed to conduct the investigation.”
ResultSivakova, O. V., Pokrovskaya, M. S., Metelskaya, V. A., Limonova, A. S., Meshkov, A. N., & Drapkina, O. M. (2019) · Russian Journal of Preventive Medicine · doi:10.17116/profmed20192206295 · Cited by 15
The Life Sciences Translational Challenge: The European Perspective
Barker, R., & Scannell, J. W. (2014) · Therapeutic Innovation & Regulatory Science · doi:10.1177/2168479014561340 · Cited by 18
“Estimates of the proportion of irreproducible science vary, but on some measures, more than 70% of results published in high-quality biomedical journals cannot be replicated in industrial laboratories.(36,37) There are also some striking disease-specific examples.”
BackgroundWhen numbers fail: do researchers agree on operationalization of published research?
Haucke, M. N., Hoekstra, R., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.191354 · Cited by 30
“However, the replicability of scientific findings remains low in a range of disciplines, such as psychology [5] and biomedicine [6].”
BackgroundSharing the Recipe: Reproducibility and Replicability in Research Across Disciplines
Rahal, R.-M., Hamann, H., Brohmer, H., & Pethig, F. (2022) · Research Ideas and Outcomes · doi:10.3897/rio.8.e89980 · Cited by 19
Evaluating Research Transparency and Openness in Communication Sciences and Disorders Journals
Schroeder, S. R., Gaeta, L., Amin, M. E., Chow, J. C., & Borders, J. C. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00330 · Cited by 17
Gutiérrez, A. C. Q., Lindegger, D. J., Heravi, A. T., Stojanov, T., Sykora, M. D., Elayan, S., Mooney, S. J., Naslund, J. A., Fadda, M., & Gruebner, O. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021473 · Cited by 18
Hunniford, V. T., Kühler, R., Wolf, B., Keppeler, D., Strenzke, N., & Moser, T. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1105562 · Cited by 16
“It is well-documented that the translation from preclinical experiments to clinical investigation is wrought with high rates of failure, significant costs, and large timelines; which presents a contentious issue in the scientific community (Begley and Ellis, 2012; Landis et al., 2012; Chalmers et al., 2014; Ioannidis…”
BackgroundDose prescription and reporting in stereotactic body radiotherapy: A multi-institutional study
Das, I. J., Yadav, P., Andersen, A. D., Chen, Z. J., Huang, L., Langer, M., Lee, C., Li, L., Popple, R. A., Rice, R. K., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.109571 · Cited by 16
Tempest in a teacup: An analysis of p-Hacking in organizational research
Gupta, A., & Bosco, F. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281938 · Cited by 19
Working towards a consensus for antibody validation
Reiss, P. D., Min, D., & Leung, M. Y. (2014) · F1000Research · doi:10.12688/f1000research.5668.1 · Cited by 15
CYP2D6 Genotype and Adjuvant Tamoxifen
Berry, D. A. (2014) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2014.96 · Cited by 14
Changing computational research. The challenges ahead
Neylon, C., Aerts, J. A., Brown, C. T., Coles, S. J., Hatton, L., Daniel Lemire, A., Millman, J., Murray‐Rust, P., Pérez, F., Saunders, N. F. W., et al. (2012) · Source Code for Biology and Medicine · doi:10.1186/1751-0473-7-2 · Cited by 15
“There have been great examples of replicability [1,2] in research communication, and examples of horrifying failure of reproducibility (as described in [3]) with serious questions being raised on the ability of our current system of research communication to guarantee, or even encourage, that published research be rep…”
BackgroundMisleading mouse studies waste medical resources
Hayden, E. C. (2014) · Nature · doi:10.1038/nature.2014.14938 · Cited by 15
Daugherty, A., Hegele, R. A., Mackman, N., Rader, D. J., Schmidt, A. M., & Weber, C. (2016) · Arteriosclerosis Thrombosis and Vascular Biology · doi:10.1161/atvbaha.116.307906 · Cited by 15
Metascience: Reproducibility blues
Munafo, M. R. (2017) · Nature · doi:10.1038/543619a · Cited by 14
In vitro models of the human microbiota and microbiome
McDonald, J. A. K. (2017) · Emerging Topics in Life Sciences · doi:10.1042/etls20170045 · Cited by 23
Towards Quantifying Uncertainty in Data Analysis & Exploration.
Chung, Y., Servan-Schreiber, S., Zgraggen, E., & Kraska, T. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 15
Recent advances in understanding the circadian clock in renal physiology
Crislip, G. R., Masten, S., & Gumz, M. L. (2018) · Current Opinion in Physiology · doi:10.1016/j.cophys.2018.06.002 · Cited by 15
Bio-Curation for Cellular Signalling: The KAMI Project
Harmer, R., Cornec, Y.-S. L., Légaré, S., & Oshurko, E. (2019) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2019.2906164 · Cited by 14
AKT1 and genetic vulnerability to bipolar disorder
Millischer, V., Matheson, G. J., Martinsson, L., Ek, I. R., Schalling, M., Lavebratt, C., & Backlund, L. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112677 · Cited by 18
Powell, D. R., Revelli, J., Doree, D. D., DaCosta, C. M., Desai, U., Shadoan, M. K., Rodriguez, L., Mullens, M., Yang, Q. M., Ding, Z., et al. (2021) · Diabetes Metabolic Syndrome and Obesity · doi:10.2147/dmso.s322083 · Cited by 16
The frequent insignificance of a “significant” p ‐value
McGiffin, D. C., Cumming, G., & Myles, P. S. (2021) · Journal of Cardiac Surgery · doi:10.1111/jocs.15960 · Cited by 15
(2013) · Nature Chemical Biology · doi:10.1038/nchembio.1269 · Cited by 14
Box, W. J., Webb, M. T., & Galizia, M. (2022) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.2c01414 · Cited by 14
Bolon, B. N., Francke, S., Rae, J. M. C., Polack, E., Regan, K. S., McInnes, E. F., Young, J. K., Keane, K. A., Perry, R., Romeike, A., et al. (2022) · Toxicologic Pathology · doi:10.1177/01926233221135563 · Cited by 15
Lebendiker, M., Danieli, T., & de Marco, A. (2014) · BMC Research Notes · doi:10.1186/1756-0500-7-585 · Cited by 12
Forwood, S., Ahern, A. L., Hollands, G. J., Fletcher, P. C., & Marteau, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0071475 · Cited by 18
“Alongside the merits of understanding calorie estimates, this study echoes some of the issues of replications in scientific research in general [4,14], and more recently in psychological research in particular [5,15].”
BackgroundHow difficult is the validation of clinical biomarkers?
Voskuil, J. L. A. (2015) · F1000Research · doi:10.12688/f1000research.6395.1 · Cited by 14
Communication in Drug Development: “Translating” Scientific Discovery
Settleman, J., & Cohen, R. L. (2016) · Cell · doi:10.1016/j.cell.2016.02.050 · Cited by 14
“In a revealing study, Amgen scientists described their efforts to replicate results from dozens of landmark cancer research publications (Begley and Ellis, 2012).”
Drug discovery in focal and segmental glomerulosclerosis
Pullen, N., & Fornoni, A. (2016) · Kidney International · doi:10.1016/j.kint.2015.12.058 · Cited by 14
Grogan, T., & Elashoff, D. A. (2016) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2016.1230216 · Cited by 14
Singh, S., Srivastava, A., Srivastava, P., Dhuriya, Y. K., Pandey, A., Kumar, D., & Rajpurohit, C. S. (2016) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2016.00072 · Cited by 16
“Preclinical studies often do not efficiently translate to the clinic and the clinical trial failures have been reported time and again (Prinz et al., 2011; Begley and Ellis, 2012).”
BackgroundMetabolic systems biology: a brief primer
Edwards, L. M. (2016) · The Journal of Physiology · doi:10.1113/jp272275 · Cited by 18
“Given recent data highlighting poor reproducibility in the scientific literature in general (Begley & Ellis, 2012), perhaps this should be the default setting for us all.”
BackgroundMind the Gap: Big Data vs. Interoperability and Reproducibility of Science
Craglia, M., & Nativi, S. (2018) · Springer eBooks · doi:10.1007/978-3-319-65633-5_6 · Cited by 14
“For example, Begley and Ellis (2013) reported that 47 out of 53 seminal publications in hematology and oncology could not be reproduced.”
BackgroundRhetorical Citizenship and the Science of Science Communication
Fahnestock, J. (2019) · Argumentation · doi:10.1007/s10503-019-09499-7 · Cited by 17
Physics approaches to the spatial distribution of immune cells in tumors
Yu, C. C., Wortman, J. C., He, T.-F., Solomon, S. T., Zhang, B., Rosario, A., Wang, R., Tu, T. Y., Schmolze, D. D., Yuan, Y., et al. (2020) · Reports on Progress in Physics · doi:10.1088/1361-6633/abcd7b · Cited by 18
Comparative Medicine: An Inclusive Crossover Discipline.
Macy, J. D., & Horváth, T. L. (2017) · PubMed Central · Cited by 17
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2018) · Cancer Reports · doi:10.1002/cnr2.1150 · Cited by 21
“These factors and others might contribute to between 50% and 90% of the published papers being irreproducible (Begley & Ellis, 2012; Glasziou et al., 2014; Hartshorne & Schachner, 2012; Kilkenny et al., 2009; Macleod et al., 2015, 2014; Moher & Altman, 2015; van der Worp & Macleod, 2011).”
Saxena, K., & Konopleva, M. Y. (2020) · Expert Opinion on Investigational Drugs · doi:10.1080/13543784.2020.1804856 · Cited by 14
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.