Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 301–400 of 2,889
Transient expression of human antibodies in mammalian cells
Vazquez-Lombardi, R., Névoltris, D., Luthra, A., Schofield, P., Zimmermann, C., & Christ, D. (2017) · Nature Protocols · doi:10.1038/nprot.2017.126 · Cited by 123
Moore, A., Wozniak, M., Yousef, A., Barnes, C. C., Cha, D., Courchesne, E., & Pierce, K. (2018) · Molecular Autism · doi:10.1186/s13229-018-0202-z · Cited by 130
“Debate and controversy regarding the replication of findings from the biological sciences and psychology have been common in recent years [51, 52].”
BackgroundBustin, S., & Nolan, T. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12801 · Cited by 113
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.001 · Cited by 108
“For example, an initiative by Bayer Healthcare to replicate 67 pre-clinical studies led to a reproducibility rate of 20–25% (Prinz et al., 2011), and researchers at Amgen were only able to replicate 6 of 53 influential cancer biology studies (Begley & Ellis, 2012).”
Ma'ayan, A. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0391 · Cited by 111
Fong, E. L. S., Wan, X., Yang, J., Morgado, M., Mikos, A. G., Harrington, D. A., Navone, N. M., & Farach-Carson, M. C. (2016) · Biomaterials · doi:10.1016/j.biomaterials.2015.10.059 · Cited by 111
“We also evaluated whether [1] standard molecular biology tools (such as gene knockdown) can be employed to manipulate cellecell interactions for mechanistic investigations and [2] the effect of dovitinib in vivo [17] can be at least in part, reflected in this in vitro PCa PDX model.”
Nakagawa, S., & Parker, T. H. (2015) · BMC Biology · doi:10.1186/s12915-015-0196-3 · Cited by 110
Is once enough?: on the extent and content of replications in human-computer interaction
Hornbæk, K., Sander, S. S., Bargas-Avila, J. A., & Grue Simonsen, J. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557004 · Cited by 107
“A recent special section in the journal Science called replications the “scientific gold standard” [21]; Bazerman [2] wrote that “in replication the private chimera becomes the communal fact”.”
BackgroundIs There a Free Lunch in Inference?
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. (2016) · Topics in Cognitive Science · doi:10.1111/tops.12214 · Cited by 103
Kabza, M., Karolak, J. A., Rydzanicz, M., Szcześniak, M. W., Nowak-Malczewska, D. M., Ginter-Matuszewska, B., Polakowski, P., Płoski, R., Szaflik, J. P., & Gajęcka, M. (2017) · European Journal of Human Genetics · doi:10.1038/ejhg.2017.4 · Cited by 121
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3018 · Cited by 127
“There has been an increasing movement towards “open science”, an umbrella term that covers study registration, data sharing, public protocols and more detailed, transparent reporting.(19-28) To address these issues in the field of healthcare database research, a Joint Task Force between the International Society for P…”
McGreivy, N., & Hakim, A. H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00897-5 · Cited by 269
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k4bgq · Cited by 115
AI-enabled image fraud in scientific publications
Gu, J., Wang, X., Li, C., Zhao, J., Fu, W., Liang, G., & Qiu, J. (2022) · Patterns · doi:10.1016/j.patter.2022.100511 · Cited by 103
“15,16 The current forensic methods for scientific image fraud rely on unnatural repetitions found through visual inspection 11 or intrinsic irregularities visualized through forensic tools.”
MethodLabala, S., Jose, A., Chawla, S. R., Khan, M. S., Bhatnagar, S., Kulkarni, O. P., & Venuganti, V. V. K. (2017) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2017.03.087 · Cited by 110
Packaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. (2018) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 108
“Long considered an axiom of science, the reproducibility of scientific research has recently come under scrutiny after some highly publicized failures to reproduce biomedical studies (Begley and Ellis 2012) and psychological experiments (Open Science Collaboration et al. 2015).”
BackgroundTowards Another Scientific Revolution
Bartling, S. H., & Friesike, S. (2013) · Springer eBooks · doi:10.1007/978-3-319-00026-8_1 · Cited by 128
Introducing Therioepistemology: the study of how knowledge is gained from animal research
Garner, J. P., Gaskill, B. N., Weber, E. M., Dallaire, J. A., & Pritchett‐Corning, K. R. (2017) · Lab Animal · doi:10.1038/laban.1224 · Cited by 115
Electrophysiological Analysis of Brain Organoids: Current Approaches and Advancements
Passaro, A. P., & Stice, S. L. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.622137 · Cited by 129
Zhang, J., Jiang, Y., Shang, Z., Zhao, B. Q., Jiao, M., Liu, W., Cheng, M., Zhai, B., Guo, Y., Liu, B., et al. (2021) · Bioactive Materials · doi:10.1016/j.bioactmat.2021.03.035 · Cited by 132
Quality assessment tools for evidence from environmental science
Bilotta, G. S., Milner, A. M., & Boyd, I. L. (2014) · Environmental Evidence · doi:10.1186/2047-2382-3-14 · Cited by 107
“Background There are significant concerns about the low reproducibility of scientific studies [1,2], and the limited effectiveness of peer-review as a quality-control mechanism [3,4].”
BackgroundThermoneutrality, mice, and cancer: a heated opinion
Hylander, B. L., & Repasky, E. A. (2016) · Trends in cancer · doi:10.1016/j.trecan.2016.03.005 · Cited by 107
“Many authors have asserted that a major problem in research is irreproducibility [80,81,84,85].”
Targeting tumor suppressor genes for cancer therapy
Liu, Y., Hu, X., Han, C., Wang, L., Zhang, X., He, X., & Lu, X. (2015) · BioEssays · doi:10.1002/bies.201500093 · Cited by 107
Common misconceptions about data analysis and statistics
Motulsky, H. J. (2014) · Pharmacology Research & Perspectives · doi:10.1002/prp2.93 · Cited by 106
Bowen, A., & Casadevall, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504955112 · Cited by 104
“One such study found that only 11% of findings could be confirmed in 53 “landmark” hematology and oncology publications (6).”
BackgroundImage-based high-content screening in drug discovery
Lin, S., Schorpp, K., Rothenaigner, I., & Hadian, K. (2020) · Drug Discovery Today · doi:10.1016/j.drudis.2020.06.001 · Cited by 124
Keshari, K. R., Sriram, R., Van Criekinge, M., Wilson, D. M., Wang, Z. J., Vigneron, D. B., Peehl, D. M., & Kurhanewicz, J. (2013) · The Prostate · doi:10.1002/pros.22665 · Cited by 105
“In many recent reviews, the lack of clinically relevant models has been cited as one of the limiting factors in the development of robust cancer drugs [7,8].”
BackgroundReproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104
“This has been studied in retrospective observational [2,3] and prospective [4] studies.”
BackgroundModeling and predicting clinical efficacy for drugs targeting the tumor milieu
Singh, M., & Ferrara, N. (2012) · Nature Biotechnology · doi:10.1038/nbt.2286 · Cited by 115
Enhancing Reuse of Data and Biological Material in Medical Research: From FAIR to FAIR-Health
Holub, P., Kohlmayer, F., Prasser, F., Mayrhofer, M. T., Schlünder, I., Martin, G. M., Casati, S., Koumakis, L., Wutte, A., Kozera, Ł., et al. (2018) · Biopreservation and Biobanking · doi:10.1089/bio.2017.0110 · Cited by 102
Naldi, A., Hernandez, C., Levy, N., Stoll, G., Monteiro, P. T., Chaouiya, C., Helikar, T., Zinovyev, A. Y., Calzone, L., Cohen‐Boulakia, S., et al. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00680 · Cited by 113
Cancer reproducibility project releases first results
Baker, M., & Dolgin, E. (2017) · Nature · doi:10.1038/541269a · Cited by 103
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Quality of reporting of confounding remained suboptimal after the STROBE guideline
Pouwels, K. B., Widyakusuma, N. N., Groenwold, R. H. H., & Hak, E. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.08.009 · Cited by 112
Reproducibility of research and preclinical validation: problems and solutions
Pusztai, L., Hatzis, C., & André, F. (2013) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2013.171 · Cited by 114
Buonincontri, G., Biagi, L., Retico, A., Cecchi, P. C., Cosottini, M., Gallagher, F. A., Gómez, P. A., Graves, M. J., McLean, M. A., Riemer, F., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.047 · Cited by 104
Draeger, E., Sawant, A. R., Johnstone, C. D., Koger, B., Becker, S. J., Vujašković, Ž., Jackson, I., & Poirier, Y. P. (2019) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2019.06.2545 · Cited by 111
Advancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 140
Reproducibility2020: Progress and priorities
Freedman, L. P., Venugopalan, G., & Wisman, R. (2017) · F1000Research · doi:10.12688/f1000research.11334.1 · Cited by 102
Did a change in Nature journals’ editorial policy for life sciences research improve reporting?
The NPQIP Collaborative group, Study steering committee, Macleod, M., Sena, E., Howells, D., Macleod, M., Sena, E., Howells, D., Kiermer, V., Swaminathan, S., et al. (2019) · BMJ Open Science · doi:10.1136/bmjos-2017-000035 · Cited by 102
Overview of the Use of Murine Models in Leukemia and Lymphoma Research
Kohnken, R. A., Porcu, P., & Mishra, A. (2017) · Frontiers in Oncology · doi:10.3389/fonc.2017.00022 · Cited by 122
“This attrition rate can be improved by careful consideration, selection, and design of murine models (91, 92).”
BackgroundEstimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 101
Recent advances in microfluidics for drug screening
Sun, J., Warden, A. R., & Ding, X. (2019) · Biomicrofluidics · doi:10.1063/1.5121200 · Cited by 115
Cait, J., Cait, A., Scott, R. W., Winder, C. B., & Mason, G. J. (2022) · BMC Biology · doi:10.1186/s12915-021-01184-0 · Cited by 133
Gundersen, O. E., Gil, Y., & Aha, D. W. (2018) · AI Magazine · doi:10.1609/aimag.v39i3.2816 · Cited by 99
“Reproducibility is challenging for many sciences, for example when the variability of physical samples and reagents can significantly affect the outcome (Begley and Ellis 2012; Lithgow et al. 2017). In computer science, a large portion of the experiments are fully conducted on computers, making the experiments more st…”
BackgroundInfluentialBrommage, R., Liu, J., Hansen, G. M., Kirkpatrick, L. L., Potter, D. G., Sands, A. T., Zambrowicz, B., Powell, D. R., & Vogel, P. (2014) · Bone Research · doi:10.1038/boneres.2014.34 · Cited by 100
Fundamental Concepts for Semiquantitative Tissue Scoring in Translational Research
Meyerholz, D. K., & Beck, A. P. (2018) · ILAR Journal · doi:10.1093/ilar/ily025 · Cited by 112
“[23] [24] [25] There are several ways to constrain bias when scoring tissues, and by using these precepts investigators can acquire more objective data.”
BackgroundBoth Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 110
Reproducibility: The risks of the replication drive
Bissell, M. (2013) · Nature · doi:10.1038/503333a · Cited by 99
Bridging Translation by Improving Preclinical Study Design in AKI
de Caestecker, M., Humphreys, B. D., Liu, K. D., Fissell, W. H., Cerda, J., Nolin, T. D., Askenazi, D., Mour, G., Harrell, F. E., Pullen, N., et al. (2015) · Journal of the American Society of Nephrology · doi:10.1681/asn.2015070832 · Cited by 99
Metaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F., Chee, Y. E., Wintle, B. C., Burgman, M. A., McCarthy, M. A., & Gordon, A. (2017) · BioScience · doi:10.1093/biosci/biw159 · Cited by 94
“Similar metaresearch evaluations of biomedical research have produced a range of equally discouraging reproducibility estimates, from approximately 11% (Begley and Ellis 2012) to 49% (Freedman et al.”
BackgroundArtificially Expanded Genetic Information Systems for New Aptamer Technologies
Biondi, E., & Benner, S. A. (2018) · Biomedicines · doi:10.3390/biomedicines6020053 · Cited by 99
“involve animals, have also become quite preferred in the biological community, in part because of their reproducibility [27].”
BackgroundAddressing Unmet Clinical Needs with 3D Printing Technologies
Ghosh, U., Ning, S., Wang, Y., & Kong, Y. L. (2018) · Advanced Healthcare Materials · doi:10.1002/adhm.201800417 · Cited by 107
Bernier, A., Matte-Gagné, C., Bélanger, M.-È., & Whipple, N. (2014) · Child Development · doi:10.1111/cdev.12236 · Cited by 98
Sukhai, M. A., Craddock, K. J., Thomas, M., Hansen, A. R., Zhang, T., Siu, L., Bedard, P., Stockley, T. L., & Kamel-Reid, S. (2016) · Genetics in Medicine · doi:10.1038/gim.2015.47 · Cited by 97
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 108
“C. G. Begley and Ellis (2012) found that unreplicated studies were cited more frequently, on average, than those with reproduced results.”
BackgroundFirst-in-human Phase 1 CRISPR Gene Editing Cancer Trials: Are We Ready?
Baylis, F. E., & McLeod, M. (2017) · Current Gene Therapy · doi:10.2174/1566523217666171121165935 · Cited by 120
“In our view, there should have been several pre-clinical studies with independent replication by research teams that are financially disinterested in the outcome [44-46].”
BackgroundAppraising iniparib, the PARP inhibitor that never was—what must we learn?
Mateo, J., Ong, M., Tan, D. S. P., Gonzalez, M. A., & de Bono, J. S. (2013) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2013.177 · Cited by 96
Approaches to modernize the combination drug development paradigm
Day, D., & Siu, L. L. (2016) · Genome Medicine · doi:10.1186/s13073-016-0369-x · Cited by 104
“Additionally, low rates (11–25 %) of reproducibility of published laboratory data, including those from high-impact journals, have been identified, despite attempts to recreate the experimental environment, suggesting limitations in the validity of scientific findings and probable publication bias [76, 77].”
BackgroundIs it time to share qualitative research data?
DuBois, J. M., Strait, M., & Walsh, H. A. (2017) · Qualitative Psychology · doi:10.1037/qup0000076 · Cited by 120
“Which conditions 1This is not so different from the field of biomedical research: several years ago, in their attempts to replicate findings from 53 landmark cancer studies, scientists at Amgen succeeded replicating findings in only 11% of cases (Begley & Ellis, 2012).”
BackgroundSomatic Mutation Theory - Why it's Wrong for Most Cancers
Brücher, B. L. D. M., & Jamall, I. S. (2016) · Cellular Physiology and Biochemistry · doi:10.1159/000443106 · Cited by 105
Lessons Learned from Two Decades of Anticancer Drugs
Liu, Z., Delavan, B., Roberts, R., & Tong, W. (2017) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2017.06.005 · Cited by 94
How Genes and the Social Environment Moderate Each Other
Reiss, D., Leve, L. D., & Neiderhiser, J. M. (2013) · American Journal of Public Health · doi:10.2105/ajph.2013.301408 · Cited by 94
“correspondence in Nature on the irreproducibility of preclinical cancer research.(48,49) Many authors regard nonreplication as a crisis in the behavioral sciences as well (see, for example, the special section in the November 2012 issue of Perspectives on Psychological Science(50)).”
BackgroundUnderstanding the roles of mutations in the amyloid precursor protein in Alzheimer disease
Hunter, S., & Brayne, C. (2018) · Molecular Psychiatry · doi:10.1038/mp.2017.218 · Cited by 94
Improving Reproducibility in Synthetic Biology
Jessop-Fabre, M. M., & Sonnenschein, N. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00018 · Cited by 96
“One exception is within cancer biology, where it was reported that only 11% of landmark cancer studies could be reproduced (Begley and Ellis, 2012).”
BackgroundMichel, M. C., Murphy, T. J., & Motulsky, H. J. (2019) · Journal of Pharmacology and Experimental Therapeutics · doi:10.1124/jpet.119.264143 · Cited by 97
Data publication consensus and controversies
Kratz, J. E., & Strasser, C. A. (2014) · F1000Research · doi:10.12688/f1000research.3979.3 · Cited by 100
Hydrogel-Based 3D Model of Patient-Derived Prostate Xenograft Tumors Suitable for Drug Screening
Fong, E., Martinez, M., Yang, J., Mikos, A., Navone, N., Harrington, D., & Farach-Carson, M. (2014) · Molecular Pharmaceutics · doi:10.1021/mp500085p · Cited by 91
Jarrett, A. M., Faghihi, D., Hormuth, D. A., Lima, E. A. B. F., Virostko, J. M., Biros, G., Patt, D. A., & Yankeelov, T. E. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9051314 · Cited by 105
Proceedings of the 15th Python in Science Conference
(2016) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-629e541a-014 · Cited by 115
The Increasing Urgency for Standards in Basic Biologic Research
Freedman, L. P., & Inglese, J. (2014) · Cancer Research · doi:10.1158/0008-5472.can-14-0925 · Cited by 91
“As Begley and Ellis (1) note "Unquestionably, a significant contributor to failure in oncology trials is the quality of published preclinical data.”
InfluentialBBMRI-ERIC Directory: 515 Biobanks with Over 60 Million Biological Samples
Holub, P., Swertz, M. A., Reihs, R., van Enckevort, D., Müller, H., & Litton, J. (2016) · Biopreservation and Biobanking · doi:10.1089/bio.2016.0088 · Cited by 95
Reproducibility Crisis: Are We Ignoring Reaction Norms?
Voelkl, B., & Würbel, H. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2016.05.003 · Cited by 103
Machery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 110
Targeting class I histone deacetylase 2 in MYC amplified group 3 medulloblastoma
Ecker, J., Oehme, I., Mazitschek, R., Korshunov, A., Kool, M., Hielscher, T., Kiss, J., Selt, F., Konrad, C., Lodrini, M., et al. (2015) · Acta Neuropathologica Communications · doi:10.1186/s40478-015-0201-7 · Cited by 92
“The root of failure to translate preclinical findings in general has been extensively discussed [59-61].”
BackgroundMethodThe Economics of Ownership, Access and Trade in Digital DataPreprint
Duch‐Brown, N., Martens, B., & Mueller‐Langer, F. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2914144 · Cited by 157
Sources of error in the retracted scientific literature
Casadevall, A., Steen, R. G., & Fang, F. C. (2014) · The FASEB Journal · doi:10.1096/fj.14-256735 · Cited by 89
“Fields that rely heavily on the use of cell lines may be particularly affected by this problem, including cancer research, in which irreproducibility has been documented to be a major concern (2).”
BackgroundThe decline and fall of the IGF‐I receptor
Baserga, R. (2012) · Journal of Cellular Physiology · doi:10.1002/jcp.24217 · Cited by 98
“Begley and Ellis (2012) are even more pessimistic than I am.”
BackgroundLean Big Data integration in systems biology and systems pharmacology
Ma’ayan, A., Rouillard, A. D., Clark, N. R., Wang, Z., Duan, Q., & Kou, Y. (2014) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2014.07.001 · Cited by 91
Z-curve 2.0: Estimating Replication Rates and Discovery Rates
Bartos̆, F., & Schimmack, U. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2720 · Cited by 126
Parsing interindividual drug variability: an emerging role for systems pharmacology
Turner, R. M., Park, B. K., & Pirmohamed, M. (2015) · WIREs Systems Biology and Medicine · doi:10.1002/wsbm.1302 · Cited by 90
Andrews, N. A., Latrémolière, A., Basbaum, A. I., Mogil, J. S., Porreca, F., Rice, A. S. C., Woolf, C. J., Currie, G. L., Dworkin, R. H., Eisenach, J. C., et al. (2015) · Pain · doi:10.1097/j.pain.0000000000000458 · Cited by 99
Theoretical principles for biology: Variation
Montévil, M., Mossio, M., Pocheville, A., & Longo, G. O. (2016) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2016.08.005 · Cited by 112
A statistical definition for reproducibility and replicability
Patil, P., Peng, R. D., & Leek, J. T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/066803 · Cited by 87
Improved reproducibility by assuring confidence in measurements in biomedical research
Plant, A. L., Locascio, L. E., May, W. E., & Gallagher, P. D. (2014) · Nature Methods · doi:10.1038/nmeth.3076 · Cited by 89
Guidelines for the automated evaluation of Elispot assays
Janetzki, S., Price, L., Schroeder, H., Britten, C. M., Welters, M. J. P., & Hoos, A. (2015) · Nature Protocols · doi:10.1038/nprot.2015.068 · Cited by 88
“Such an impact is especially important in light of the recent alarming reports about nonreproducibility of preclinical cancer research studies 42 .”
BackgroundThe registration continuum in clinical science: A guide toward transparent practices
Benning, S. D., Bachrach, R. L., Smith, E. A., Freeman, A. J., & Wright, A. G. C. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000451 · Cited by 77
“This scientific ecology has led to an unfortunate plethora of studies whose findings could not be replicated in psychology (Open Science Collaboration, 2015) and other clinical disciplines (e.g., Begley & Ellis, 2012).”
BackgroundA validated antibody panel for the characterization of tau post-translational modifications
Ercan, E., Eid, S., Weber, C., Kowalski, A., Bichmann, M., Behrendt, A., Matthes, F., Krauss, S., Reinhardt, P., Fulle, S., et al. (2017) · Molecular Neurodegeneration · doi:10.1186/s13024-017-0229-1 · Cited by 87
“The so-called ‘reproducibility crisis’ has drawn public attention to problematic practices in pre-clinical research [15], and efforts to improve the robustness and reproducibility of primary research findings include guidelines for the thorough validation of antibody reagents [16].”
Failures in Phase III: Causes and Consequences
Seruga, B., Ocana, A., Amir, E., & Tannock, I. F. (2015) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-15-0124 · Cited by 88
“25 BSC or Placebo 8 (13) 12 (23) Small molecule 6 (10) 7 (13) Other 5 (8) 1 (2) Primary endpoint OS 39 (65) 12 (23) <0.”
BackgroundInfluentialThe promise and perils of industry‐funded science
Holman, B., & Elliott, K. C. (2018) · Philosophy Compass · doi:10.1111/phc3.12544 · Cited by 88
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
When Will Mendelian Randomization Become Relevant for Clinical Practice and Public Health?
Davey Smith, G., Paternoster, L., & Relton, C. (2017) · JAMA · doi:10.1001/jama.2016.21189 · Cited by 87
Methods for Heart Rate Variability Biofeedback (HRVB): A Systematic Review and Guidelines
Lalanza, J. F., Lorente, S., Bullich, R., García, C., Losilla, J.-M., & Capdevila, L. (2023) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-023-09582-6 · Cited by 117
“It could indeed be hypothesized that this lack of methodological information could exemplify the low levels of replication in psychology (Open Science, 2015) and biomedical disciplines (Begley & Ellis, 2012; Ioannidis et al., 2009).”
BackgroundKepes, S., Bennett, A. A., & McDaniel, M. A. (2014) · Academy of Management Learning and Education · doi:10.5465/amle.2013.0193 · Cited by 110
Is it time to reinvent basic cell culture medium?
McKee, T. J., & Komarova, S. V. (2017) · American Journal of Physiology-Cell Physiology · doi:10.1152/ajpcell.00336.2016 · Cited by 85
“In conclusion, we suggest that the nonphysiological electrolyte and carbohydrate microenvironment of the most commonly used cell culture media may result in unintended and uncontrolled changes in cell behavior, thereby contributing to the difficulties in reproducibility observed in modern publications (4).”
BackgroundLevelling the Translational Gap for Animal to Human Efficacy Data
Ferreira, G. S., Veening-Griffioen, D. H., Boon, W., Moors, E. H. M., & van Meer, P. J. K. (2020) · Animals · doi:10.3390/ani10071199 · Cited by 104
Pre-Clinical Testing of Therapies for Traumatic Brain Injury
DeWitt, D. S., Hawkins, B. E., Dixon, C. E., Kochanek, P. M., Armstead, W. M., Bass, C. R., Bramlett, H. M., Büki, A., Dietrich, W. D., Ferguson, A. R., et al. (2018) · Journal of Neurotrauma · doi:10.1089/neu.2018.5778 · Cited by 93
“particularly in the absence of adequate rigor, can mislead other investigators to pursue unfruitful paths.(150,151) Because many of the negative studies and their unnecessary duplication were supported”
Chiarella, A. M., Butler, K. V., Gryder, B. E., Lu, D., Wang, T. A., Yu, X., Pomella, S., Khan, J., Jin, J., & Hathaway, N. A. (2020) · Nature Biotechnology · doi:10.1038/s41587-019-0296-7 · Cited by 84
MISpheroID: a knowledgebase and transparency tool for minimum information in spheroid identity
Peirsman, A., Blondeel, E., Ahmed, T., Anckaert, J., Audenaert, D., Boterberg, T., Buzás, K., Carragher, N. O., Castellani, G., Castro, F., et al. (2021) · Nature Methods · doi:10.1038/s41592-021-01291-4 · Cited by 105
Opportunities and challenges in combination gene cancer therapy
Nastiuk, K. L., & Krolewski, J. J. (2016) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2015.12.005 · Cited by 85
“Problems with pre-clinical trial reproducibility may be a cause of the low rate of drug development success [75,76], but it could also be that murine models, even those genetically engineered to resemble human cancers, are not adequate models, and therefore restoration of a single tumor suppressor gene is simply not s…”
Background
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.