Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 601–700 of 2,889
Bello, N. M., & Renter, D. G. (2018) · Journal of Dairy Science · doi:10.3168/jds.2017-13978 · Cited by 68
Antibody validation: a view from the mountains
Taussig, M. J., Fonseca, C., & Trimmer, J. S. (2018) · New Biotechnology · doi:10.1016/j.nbt.2018.08.002 · Cited by 53
“antibodies in research can also be seen as part of a larger drive to enhance rigour, reliability and transparency of biomedical research in general [12-18].”
Replicability and replication in the humanities
Peels, R. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-018-0060-4 · Cited by 80
First results from psychology’s largest reproducibility test
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.17433 · Cited by 54
George, C. H., Stanford, S. C., Alexander, S., Cirino, G., Docherty, J. R., Giembycz, M. A., Hoyer, D., Insel, P. A., Izzo, A. A., Ji, Y. H., et al. (2017) · British Journal of Pharmacology · doi:10.1111/bph.13925 · Cited by 52
“This has added further fuel to the fire that was first stoked up by the findings from Bayer (Prinz et al., 2011) and Amgen (Begley and Ellis, 2012) and supported by many others (for e.g. see McGrath and Lilley, 2015; Liu et al., 2016; Ortuno et al., 2016; Wang et al., 2016).”
BackgroundManaging risks in drug discovery: reproducibility of published findings
Kannt, A., & Wieland, T. (2016) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-016-1216-8 · Cited by 57
“Moreover, more credit should be given for teaching and mentoring (Begley and Ellis 2012)”
BackgroundInfluentialMolecular Diagnostics in Pathology: Time for a Next-Generation Pathologist?
Fassan, M. (2018) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2017-0269-ra · Cited by 57
A Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51
“In the early 2010s, Begley and Ellis [8] attempted to fully replicate 53 highly cited cancer trial studies but only achieved a success rate of 11%.”
BackgroundThe Role of Replication in Engineering Education Research
Benson, L., & Borrego, M. J. (2015) · Journal of Engineering Education · doi:10.1002/jee.20082 · Cited by 54
Critical Appraisal of Studies Using Laboratory Animal Models
O'Connor, A. M. C., & Sargeant, J. M. (2014) · ILAR Journal · doi:10.1093/ilar/ilu038 · Cited by 59
Morrell, W. C., Birkel, G. W., Forrer, M., López, T. D.-F., Backman, T. W. H., Dussault, M., Petzold, C. J., Baidoo, E. E. K., Costello, Z., Ando, D., et al. (2017) · ACS Synthetic Biology · doi:10.1021/acssynbio.7b00204 · Cited by 57
Parrish, M. C., Tan, Y. J., Grimes, K., & Mochly‐Rosen, D. (2018) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010818-021625 · Cited by 69
Kennedy, R., Kuvshinov, D., Sdrolia, A., Kuvshinova, E., Hilton, K., Crank, S., Beavis, A. W., Green, V., & Greenman, J. (2019) · Scientific Reports · doi:10.1038/s41598-019-42745-2 · Cited by 59
AVOCADO: Visualization of Workflow–Derived Data Provenance for Reproducible Biomedical Research
Stitz, H., Luger, S., Streit, M., & Gehlenborg, N. (2016) · Computer Graphics Forum · doi:10.1111/cgf.12924 · Cited by 61
“The failure to reproduce the results of a large number of such studies has raised major concerns in the biomedical research community and triggered several efforts to address this issue [BE12, HG13,BI15,Buc15].”
Güttinger, S. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-019-0269-1 · Cited by 84
Interneuron Dysfunction and Inhibitory Deficits in Autism and Fragile X Syndrome
Nomura, T. (2021) · Cells · doi:10.3390/cells10102610 · Cited by 56
van der Vet, P. E., & Nijveen, H. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0008-5 · Cited by 50
Do Policy Statements on Media Effects Faithfully Represent the Science?
Elson, M., Ferguson, C. J., Gregerson, M., Hogg, J. L., Ivory, J., Klisanin, D., Markey, P. M., Nichols, D., Siddiqui, S., & Wilson, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918811301 · Cited by 49
“Concerns along these lines are certainly not limited to behavioral research, and examples can be found in other fields, such as nutrition (Singal, 2017) and cancer (Begley & Ellis, 2016) research.”
BackgroundASTM international workshop on standards and measurements for tissue engineering scaffolds
Simon, C. G., Yaszemski, M. J., Ratcliffe, A., Tomlins, P., Luginbuehl, R., & Tesk, J. A. (2015) · Journal of Biomedical Materials Research Part B: Applied Biomaterials · doi:10.1002/jbm.b.33286 · Cited by 50
Biospecimens, biomarkers, and burgeoning data: the imperative for more rigorous research standards
Poste, G. H. (2012) · Trends in Molecular Medicine · doi:10.1016/j.molmed.2012.09.003 · Cited by 51
Unraveling the Complexities of Life Sciences Data
Higdon, R., Haynes, W. A., Stanberry, L. I., Stewart, E., Yandl, G., Howard, C., Broomall, W., Kolker, N., & Kolker, E. (2012) · Big Data · doi:10.1089/big.2012.1505 · Cited by 50
Multiplicities in Cancer Research: Ubiquitous and Necessary Evils
Berry, D. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs301 · Cited by 50
The Design and Statistical Analysis of Animal Experiments: Introduction to this Issue
Festing, M. F. W., & Nevalainen, T. O. (2014) · ILAR Journal · doi:10.1093/ilar/ilu046 · Cited by 53
A Guide to Reproducibility in Preclinical Research
Samsa, G., & Samsa, L. A. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002351 · Cited by 50
Kochanek, P. M., Dixon, C. E., Mondello, S., Wang, K., Lafrenaye, A. D., Bramlett, H. M., Dietrich, W. D., Hayes, R. L., Shear, D. A., Gilsdorf, J. S., et al. (2018) · Frontiers in Neurology · doi:10.3389/fneur.2018.00640 · Cited by 53
Guilt by association: How scientific misconduct harms prior collaborators
Hussinger, K., & Pellens, M. (2019) · Research Policy · doi:10.1016/j.respol.2018.01.012 · Cited by 49
Kirouac, D. C., Cicali, B., & Schmidt, S. (2019) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12390 · Cited by 46
Faggion, C. M., Huivin, R., Aranda, L., Pandis, N., & Alarcon, M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.02.011 · Cited by 49
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49
“In a similar effort, researchers at Amgen could not reproduce 47 of 53 landmark oncological findings for potential drug targets [10].”
ResultThe Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P., & Carneiro, C. F. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 49
“…reproducibility of published results in certain areas of biomedical research were initially raised by theoretical models (Ioannidis, 2005a), systematic reviews of the existing literature (Ioannidis, 2005b) and alarm calls by the pharmaceutical industry (Begley and Ellis, 2012; Prinz et al., 2011).”
BackgroundWyatt, T. D. (2020) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0262 · Cited by 56
“Some non-reproducible pre-clinical papers have generated a whole field of study, with hundreds of secondary publications that expanded on ideas in the original study but which did not test its fundamental basis [43], in an echo perhaps of ‘putative human pheromones’.”
BackgroundGreen, S. B. (2015) · BMC Medical Ethics · doi:10.1186/s12910-015-0043-7 · Cited by 53
“We need a system that will facilitate a transparent discovery process that frequently and consistently leads to significant patient benefit' [37].”
BackgroundUse of Humanized Mice to Study the Pathogenesis of Autoimmune and Inflammatory Diseases
Koboziev, I., Jones‐Hall, Y., Valentine, J. F., Webb, C. R., Furr, K. L., & Grisham, M. B. (2015) · Inflammatory Bowel Diseases · doi:10.1097/mib.0000000000000446 · Cited by 57
Donner, D. G., Kiriazis, H., Du, X., Marwick, T. H., & McMullen, J. R. (2018) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00157.2018 · Cited by 50
“Issues of reproducibility in preclinical research are of increasing ethical and economic concern, particularly in some fields including cancer and cardiovascular disease, following reports of rates as low as 11 25% (1, 2, 23).”
BackgroundInfluentialChhin, C. S., Taylor, K. A., & Wei, W. S. (2018) · Educational Researcher · doi:10.3102/0013189x18788047 · Cited by 49
Giner‐Sorolla, R. (2018) · European Review of Social Psychology · doi:10.1080/10463283.2018.1542902 · Cited by 56
Vodovotz, Y., Xia, A. Y., Read, E. L., Bassaganya-Riera, J., Hafler, D. A., Sontag, E. D., Wang, J., Tsang, J. S., Day, J., Kleinstein, S. H., et al. (2016) · Trends in Immunology · doi:10.1016/j.it.2016.11.006 · Cited by 56
“The lack of rigor and reproducibility of experimental studies is an area of widespread concern in both preclinical and clinical research [76,77].”
Packaging data analytical work reproducibly using R (and friends)Preprint
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · doi:10.7287/peerj.preprints.3192v1 · Cited by 50
Gold-Nanoparticle Hybrid Nanostructures for Multimodal Cancer Therapy
Ali, A. A., Abuwatfa, W. H., Al-Sayah, M. H., & Husseini, G. A. (2022) · Nanomaterials · doi:10.3390/nano12203706 · Cited by 55
Liu, Y., Gill, E., & Huang, Y. Y. S. (2017) · Future Science OA · doi:10.4155/fsoa-2016-0084 · Cited by 51
On the low reproducibility of cancer studies
Wen, H., Wang, H., He, X., & Wu, C. (2018) · National Science Review · doi:10.1093/nsr/nwy021 · Cited by 47
“It is curious that the RP:CB project (4, 6) together with the earlier reports (3) are met with near total silence.”
BackgroundInfluentialExperiments done in Black-6 mice: what does it mean?
Åhlgren, J., & Võikar, V. (2019) · Lab Animal · doi:10.1038/s41684-019-0288-8 · Cited by 53
Gavert, N. S., Zwang, Y., Weiser, R., Greenberg, O., Halperin, S., Jacobi, O., Mallel, G., Sandler, O., Berger, A. J., Stossel, E., et al. (2022) · Nature Cancer · doi:10.1038/s43018-021-00325-2 · Cited by 48
The Importance of Biobanking in Cancer Research
Castillo-Pelayo, T., Babinszky, S., LeBlanc, J., & Watson, P. H. (2015) · Biopreservation and Biobanking · doi:10.1089/bio.2014.0061 · Cited by 48
“Then in between these short-term measures and overall outcomes sits the issue of reproducibility, 25–27 This potential barrier to effective transformation of short-term productivity to long-term impact has not been systematically assessed.”
Meuten, D. J., & Meuten, T. (2016) · doi:10.1002/9781119181200.ch15 · Cited by 48
Optimized design and analysis of preclinical intervention studies in vivo
Laajala, T. D., Jumppanen, M., Huhtaniemi, R., Fey, V., Kaur, A., Knuuttila, M., Aho, E., Oksala, R., Westermarck, J., Mäkelä, S., et al. (2016) · Scientific Reports · doi:10.1038/srep30723 · Cited by 48
“1038/srep30723 (2): Each individual j is connected to other G-1 matched individuals; (3): Each individual i has G-1 other individuals that are matched to the individual i; (4): If individual i is matched to individual j, then individual j is also matched to individual i (no single direction relationships allowed); (5)…”
Replication in computing education research: researcher attitudes and experiences
Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A. (2016) · Proceedings of the 16th Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999554 · Cited by 48
“, biology [7], health care [19], political science [15], and computational science [37]).”
BackgroundAcquisition, Analysis, and Sharing of Data in 2015 and Beyond: A Survey of the Landscape
Antman, E. M., Benjamin, E. J., Harrington, R. A., Houser, S. R., Peterson, E. D., Bauman, M. A., Brown, N., Bufalino, V. J., Califf, R. M., Creager, M. A., et al. (2015) · Journal of the American Heart Association · doi:10.1161/jaha.115.002810 · Cited by 50
“Discouragingly, these studies demonstrated that only 11% to 22% of the preclinical studies could be replicated.(71,72) As a consequence of the concern about reproducibility, there is an NIH initiative to modify data acquisition and sharing in the preclinical arena.”
BackgroundGenome editing in large animal models
Maynard, L. H., Humbert, O., Peterson, C. W., & Kiem, H. (2021) · Molecular Therapy · doi:10.1016/j.ymthe.2021.09.026 · Cited by 51
Ethical Considerations for Phase I Trials in Oncology
Bittlinger, M., Bicer, S., Peppercorn, J. M., & Kimmelman, J. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.21.02125 · Cited by 57
Can Drug Repurposing Accelerate Precision Oncology?
Schipper, L. J., Zeverijn, L. J., Garnett, M. J., & Voest, E. E. (2022) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0612 · Cited by 46
Cell therapy in muscular dystrophies: many promises in mice and dogs, few facts in patients
Skuk, D., & Tremblay, J. P. (2015) · Expert Opinion on Biological Therapy · doi:10.1517/14712598.2015.1057564 · Cited by 49
“Moreover, preclinical studies must be very robust to withstand the challenges and rigors of clinical trials [112].”
BackgroundRimm, D. L. (2014) · Nature Methods · doi:10.1038/nmeth.2896 · Cited by 49
Survey statistics of automated segmentations applied to optical imaging of mammalian cells
Bajcsy, P., Cardone, A., Chalfoun, J., Halter, M. W., Juba, D., Kociołek, M., Majurski, M. P., Peskin, A. P., Simon, C. G., Simon, M., et al. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0762-2 · Cited by 53
“The irreproducibility of biological studies has recently be highlighted [27, 28].”
BackgroundKelly, J., & Knottenbelt, W. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.00962 · Cited by 51
“A similar study in cancer biology found that only 6 of the results in 53 highprofile papers could be replicated [8], [10].”
ResultKosmidou, C., Efstathiou, N. E., Hoang, M. V., Notomi, S., Konstantinou, E. K., Hirano, M., Takahashi, K., Maidana, D. E., Tsoka, P., Young, L., et al. (2018) · Scientific Reports · doi:10.1038/s41598-017-17634-1 · Cited by 47
Tamarov, K. P., Näkki, S., Xu, W., & Lehto, V. (2018) · Journal of Materials Chemistry B · doi:10.1039/c8tb00462e · Cited by 54
Human augmentation of ecosystems: objectives for food production and science by 2045
Funabashi, M. (2018) · npj Science of Food · doi:10.1038/s41538-018-0026-4 · Cited by 57
Power to the People: Power, Negative Results and Sample Size
Gaskill, B. N., & Garner, J. P. (2019) · Journal of the American Association for Laboratory Animal Science · doi:10.30802/aalas-jaalas-19-000042 · Cited by 54
Bloomfield, R. J., Rennekamp, K. M., & Steenhoven, B. A. (2018) · Journal of Accounting Research · doi:10.1111/1475-679x.12208 · Cited by 54
Biotech giant publishes failures to confirm high-profile science
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.19269 · Cited by 47
A Framework for Improving the Quality of Research in the Biological Sciences
Casadevall, A., Ellis, L. M., Davies, E. W., McFall-Ngai, M., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01256-16 · Cited by 46
Self-citationKennedy, J. J., Yan, P., Zhao, L., Ivey, R. G., Voytovich, U. J., Moore, H. D., Lin, C., Pogosova‐Agadjanyan, E. L., Stirewalt, D. L., Reding, K. W., et al. (2015) · Molecular & Cellular Proteomics · doi:10.1074/mcp.o115.054940 · Cited by 50
Computational Reproducibility in The Wildlife Society's Flagship Journals
ArchMiller, A. A., Johnson, A. D., Nolan, J., Edwards, M., Elliott, L. H., Ferguson, J. M., Iannarilli, F., Vélez, J., Vitense, K., Johnson, D. H., et al. (2020) · Journal of Wildlife Management · doi:10.1002/jwmg.21855 · Cited by 45
“Within the last 10 years, researchers have reported glaring issues in replication in several scienti fi c fi elds, including cancer biology (Begley and Ellis 2012), experimental psychology (Open Science Framework 2015), economics (Camerer et al. 2016), and animal behavior (Wang et al. 2018).”
BackgroundQuantum Chemistry Simulation of Biological Molecules
Albuquerque, E. L., Fulco, U. L., Caetano, E. W. S., & Freire, V. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108774956 · Cited by 54
Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0069 · Cited by 61
Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis
Nelson, N. C., Ichikawa, K., Chung, J., & Malik, M. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254090 · Cited by 54
Uhlig, C., Krause, H., Koch, T., de Abreu, M. G., & Spieth, P. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134205 · Cited by 48
“The problem of insufficient reporting of data contributes to a lack of reproducibility, as seen in recent publications on cancer research [55,56].”
BackgroundTime to do something about reproducibility
Morrison, S. J. (2014) · eLife · doi:10.7554/elife.03981 · Cited by 44
“Several analyses of certain subsets of cancer studies have concluded that most were not reproducible (Ioannidis et al., 2009; Prinz et al., 2011; Begley and Ellis, 2012).”
BackgroundMa, X., Hui, H., Jin, Y., Dong, D., Liang, X., Yang, X., Tan, K., Dai, Z., Cheng, Z., & Tian, J. (2016) · Biomaterials · doi:10.1016/j.biomaterials.2016.02.007 · Cited by 46
Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy
Flier, J. S. (2016) · Molecular Metabolism · doi:10.1016/j.molmet.2016.11.006 · Cited by 49
Evaluating the credibility of histopathology data in environmental endocrine toxicity studies
Wolf, J. C., & Maack, G. (2016) · Environmental Toxicology and Chemistry · doi:10.1002/etc.3695 · Cited by 49
Wieschowski, S., Biernot, S., Deutsch, S., Glage, S., Bleich, A., Tolba, R., & Strech, D. (2019) · PLOS ONE · doi:10.1371/journal.pone.0223758 · Cited by 44
“Despite the methodological limitations of the existing assessments of publication rates, opinions regarding publication bias in animal research play an important role in current debates on, for example, the “reproducibility crisis” [8, 9] and “failure rates in clinical research” [10, 11].”
BackgroundBiodegradable metals for bone fracture repair in animal models: a systematic review
Zhang, J., Shang, Z., Jiang, Y., Zhang, K., Li, X., Ma, M., Li, Y., & Ma, B. (2020) · Regenerative Biomaterials · doi:10.1093/rb/rbaa047 · Cited by 47
“Therefore, in the field of experimental research, it is necessary to take measures to promote data sharing and encourage journals to publish studies with negative or neutral results to avoid the ‘filedrawer problem’ and reduce the impact of publication bias on their results [73].”
BackgroundSynthetic biology: from hype to impact
Gardner, T. S. (2013) · Trends in biotechnology · doi:10.1016/j.tibtech.2013.01.018 · Cited by 46
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
Quantifying research waste in ecology
Purgar, M., Klanjšček, T., & Čulina, A. (2022) · Nature Ecology & Evolution · doi:10.1038/s41559-022-01820-0 · Cited by 54
Modulation of the cancer cell transcriptome by culture media formulations and cell density
Kim, S. W., KIM, S.-J., Langley, R. R., & Fidler, I. J. (2015) · International Journal of Oncology · doi:10.3892/ijo.2015.2930 · Cited by 52
Yu, J., Silva, J., & Califano, A. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv556 · Cited by 45
Sins, J. W. R., Mager, D. J., Davis, S. C. A. T., Biemond, B. J., & Fijnvandraat, K. (2017) · Blood Advances · doi:10.1182/bloodadvances.2017007211 · Cited by 46
The antibody horror show: an introductory guide for the perplexed
Goodman, S. L. (2018) · New Biotechnology · doi:10.1016/j.nbt.2018.01.006 · Cited by 48
Kay, K., Dolcy, K., Bies, R. R., & Shah, D. K. (2019) · The AAPS Journal · doi:10.1208/s12248-019-0302-5 · Cited by 50
History and Trends of 3D Bioprinting
Thayer, P., Martinez, H., & Gatenholm, E. (2020) · Methods in molecular biology · doi:10.1007/978-1-0716-0520-2_1 · Cited by 56
Azad, T. D., Ehresman, J., Ahmed, A. K., Staartjes, V. E., Lubelski, D., Stienen, M. N., Veeravagu, A., & Ratliff, J. K. (2021) · The Spine Journal · doi:10.1016/j.spinee.2020.10.006 · Cited by 45
Strong Inference in Mathematical Modeling: A Method for Robust Science in the Twenty-First Century
Ganusov, V. V. (2016) · Frontiers in Microbiology · doi:10.3389/fmicb.2016.01131 · Cited by 49
“The need for more robust ways of doing science, including mathematical modeling, is well recognized (Begley and Ellis, 2012; Fang and Casadevall, 2012).”
BackgroundResultPublish and Who Should Perish: You or Science?
Kun, Á. (2018) · Publications · doi:10.3390/publications6020018 · Cited by 44
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44
“The currently available empirical estimates on the topic suggest that less than half (ranging from 49% down to 11%) of scientific results are reproducible (Prinz et al., 2011; Begley and Ellis, 2012; Freedman et al., 2015, 2017).”
BackgroundThe Biology of Malignant Mesothelioma and the Relevance of Preclinical Models
Blanquart, C., Jaurand, M., & Jean, D. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.00388 · Cited by 46
Metascience as a Scientific Social Movement
Peterson, D., & Panofsky, A. L. (2023) · Minerva · doi:10.1007/s11024-023-09490-3 · Cited by 119
Ghislat, G., Hernández-Hernández, S., Piyawajanusorn, C., & Ballester, P. J. (2024) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2024.2403639 · Cited by 61
“This requires identifying mutually beneficial collaborations rather than service from one expert to another, contributing to drive project direction rather than arriving only in time for post-mortem analyses[105], and learning the basics of the other disciplines to enable fluent…”
BackgroundYao, F., Zhang, C., Du, W., Liu, C., & Xu, Y. (2015) · PLoS ONE · doi:10.1371/journal.pone.0138213 · Cited by 50
“However, markers so developed have had only limited applications since tissue-based gene-expression data are generally not available for most patients [11, 12].”
BackgroundToward Precision Healthcare: Context and Mathematical Challenges
Colijn, C., Jones, N. S., Johnston, I. G., Yaliraki, S. N., & Barahona, M. (2017) · Frontiers in Physiology · doi:10.3389/fphys.2017.00136 · Cited by 53
“The translational impact of this trend is starkly illustrated by recent failures to reproduce landmark cancer studies and low success rates in clinical trials (Prinz et al., 2011; Begley and Ellis, 2012).”
ResultOperation Brain Trauma Therapy: 2016 Update
Kochanek, P. M., Bramlett, H. M., Dixon, C. E., Dietrich, W. D., Mondello, S., Wang, K., Hayes, R. L., Lafrenaye, A. D., Povlishock, J. T., Tortella, F. C., et al. (2018) · Military Medicine · doi:10.1093/milmed/usx184 · Cited by 44
Pietschnig, J., Siegel, M., Eder, J. S. N., & Gittler, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02874 · Cited by 56
Boulbes, D. R., Costello, T., Baggerly, K., Fan, F., Wang, R., Bhattacharya, R., Ye, X., & Ellis, L. M. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-18-0227 · Cited by 41
“Respondent characteristics With our eligibility criteria of (1) being a graduate student or postdoctoral fellow and (2) performing bench science, 467 of our total 576 respondents were deemed eligible.”
BackgroundSelf-citationThe role of scoping reviews in reducing research waste
Khalil, H., Peters, M. D. J., McInerney, P. A., Godfrey, C. M., Alexander, L., Evans, C., Pieper, D., Moraes, E. B., Tricco, A. C., Munn, Z., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.09.012 · Cited by 43
Unsupervised Alignment of Natural Language Instructions with Video Segments
Naim, I., Song, Y., Liu, Q., Kautz, H., Luo, J., & Gildea, D. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8939 · Cited by 48
“A recent study reported that many major results in cancer biology are not reproducible (Begley and Ellis 2012).”
BackgroundEdwards, S. R., Hamlin, A. S., Marks, N., Coulson, E. J., & Smith, M. T. (2014) · Clinical and Experimental Pharmacology and Physiology · doi:10.1111/1440-1681.12277 · Cited by 48
The Impact of Preclinical Irreproducibility on Drug Development
Freedman, L., & Gibson, M. (2015) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.9 · Cited by 43
Predictive modelling of gene expression from transcriptional regulatory elements
Budden, D., Hurley, D., & Crampin, E. J. (2014) · Briefings in Bioinformatics · doi:10.1093/bib/bbu034 · Cited by 41
“This also increases the difficulty of reproducing research, highlighted by a recent study by Begley et al. finding that 47 of 53 cancer-related research papers contained irreproducible results [72].”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.