Raise standards for preclinical cancer research
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This Nature Comment argues that many landmark preclinical oncology findings are not reproducible and calls for higher standards in methods, reporting, and incentives. It reports that Amgen confirmed only 6 of 53 landmark papers (11%) and cites Bayer’s finding that about 25% of published preclinical studies could be validated.
Key findings
- In Amgen’s ten-year retrospective analysis, scientific findings were confirmed in only 6 of 53 “landmark” preclinical cancer papers (11%); non-reproduced studies often lacked blinding, relied on single experiments such as one Western blot, or presented selected data.
- Bayer HealthCare reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue; published cancer research represented 70% of the studies analysed in that report, with possible overlap with Amgen’s 53 papers.
- Citation data showed non-reproduced articles in journals with impact factor >20 had a mean of 248 citations (range 3–800) versus 231 (range 82–519) for reproduced articles; for impact factor 5–19, non-reproduced articles had a mean of 169 citations (range 6–1,909) versus 13 (range 3–24) for reproduced articles.
- The authors recommend reporting negative data, blinding preclinical experiments, validating reagents, repeating critical experiments, avoiding single cell lines or models, requiring patient-selection biomarkers, and changing academic incentives.
Methods
- This is a Nature Comment/perspective, not a primary experimental study; it synthesizes Amgen’s ten-year retrospective analysis of experiments performed prospectively and cites a Bayer HealthCare report.
- Amgen selected 53 “landmark” papers describing completely new approaches or alternative clinical uses, attempted confirmation, used additional models, contacted original authors, exchanged reagents, and repeated experiments under the authors’ direction, occasionally in the original investigator’s laboratory.
- The analysis reports counts of confirmed findings and citation metrics from Google Scholar, May 2011; the recommendations are proposed changes rather than tested interventions.
Limitations
- Stated: validation attempts may have failed because of technical differences or difficulties despite efforts to ensure otherwise; additional models were used; when findings could not be reproduced, authors were contacted and experiments were repeated under their direction.
- Stated: papers were deliberately selected as “landmark” and described something completely new, so some data might not hold up; Bayer’s analysed studies may overlap with Amgen’s 53 papers.
- Evident: as a Comment, the text provides no detailed systematic protocol, inclusion criteria, statistical methods, or full dataset; the citation analysis is descriptive and limited to two impact-factor strata.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 6 Not checked · 12 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas.”
What the sources say
- Partly supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). sourceExperimental — accuracy not yet measured
The opening passage of our copy (labelled Abstract; page unavailable) says cancer-drug attrition is much higher than in other therapeutic areas, citing 5% of anticancer agents versus 20% of cardiovascular agents reaching licensure. That comparison covers development from preclinical activity through phase III, not failure rates specifically among clinical trials; it also does not explicitly establish a highest-rate ranking. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although we in the cancer field hoped that this would lead to more effective drugs, historically, our ability to translate cancer research to clinical success has been remarkably low 1 .”
What the sources say
- Supportedcites [1] Hutchinson, L. & Kirk, R. Nature Rev. Clin. Oncol. 8, 189–190 (2011). source
The opening passage (labeled Abstract in our copy; no page given) reports that only 5% of agents with preclinical anticancer activity are licensed after demonstrating sufficient efficacy in phase III testing. It also says cancer drug attrition is much higher than in other therapeutic areas, supporting the claim of historically low translation to clinical success. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Investigators frequently presented the results of one experiment, such as a single Western-blot analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 22 claims and every source verdict.
- It may state two separate claims as one. 2
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there are other more common sources of unreliability in the literature (41)(42)(43)(44)”
From Misconduct accounts for the majority of retracted scientific publications (2012) · cites this paper as [42]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A landmark study conducted by Glenn Begley and his team in 2012 concluded that most cancer research studies were not reproducible, by alarmingly reporting that 47 of 53 published findings could not be replicated [13].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [10]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
High drug attrition rates—where are we going wrong?
Hutchinson, L., & Kirk, R. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.34 · Cited by 517
Drug development and clinical trials—the path to an approved cancer drug
Rubin, E. H., & Gilliland, D. G. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.22 · Cited by 135
Raising the bar for cancer therapy models
Francia, G., & Kerbel, R. S. (2010) · Nature Biotechnology · doi:10.1038/nbt0610-561 · Cited by 54
Cited by
Items 201–300 of 2,889
Anti-cancer effects of cinnamon: Insights into its apoptosis effects
Sadeghi, S., Davoodvandi, A., Pourhanifeh, M. H., Sharifi, N., ArefNezhad, R., Sahebnasagh, R., Moghadam, S. A., Sahebkar, A., & Mirzaei, H. (2019) · European Journal of Medicinal Chemistry · doi:10.1016/j.ejmech.2019.05.067 · Cited by 176
Ahmad, R., Riaz, M., Khan, A., Aljamea, A., Algheryafi, M., Sewaket, D., & Alqathama, A. (2021) · Phytotherapy Research · doi:10.1002/ptr.7215 · Cited by 181
Mouse models of Down syndrome: gene content and consequences
Gupta, M., Dhanasekaran, A. R., & Gardiner, K. J. (2016) · Mammalian Genome · doi:10.1007/s00335-016-9661-8 · Cited by 171
“This is true in a broad range of diseases, e.g., cancer, disorders of the central nervous system, inflammation, Fragile X, etc. (Begley and Ellis 2012; McGonigle and Ruggeri 2014, McGonigle 2014).”
BackgroundWaste, Leaks, and Failures in the Biomarker Pipeline
Ioannidis, J. P. A., & Bossuyt, P. M. M. (2017) · Clinical Chemistry · doi:10.1373/clinchem.2016.254649 · Cited by 172
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175
Use and Abuse of RNAi to Study Mammalian Gene Function
Kaelin, W. G. (2012) · Science · doi:10.1126/science.1225787 · Cited by 169
Complexity, uncertainty-reduction strategies, and project performance
Floricel, S., Michela, J. L., & Piperca, S. (2016) · International Journal of Project Management · doi:10.1016/j.ijproman.2015.11.007 · Cited by 169
“But the frequent experiment replication failures (Begley & Ellis 2012), the fragility of statistical inference (Taleb 2007), and snags in high-reliability technical systems (Perrow 1984; Leveson et al. 2009; Saunders 2015) all show that they hardly solve the correspondence problem.”
BackgroundAdvancing Biological Understanding and Therapeutics Discovery with Small-Molecule Probes
Schreiber, S. L., Kotz, J. D., Li, M., Aubé, J., Austin, C. P., Reed, J. C., Rosen, H., White, E. L., Sklar, L. A., Lindsley, C. W., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.023 · Cited by 169
“Hits straight from screens rarely have the potency and selectivity necessary to be effective tool compounds – and if used with insufficient caution can lead to non-robust conclusions (Begley and Ellis, 2012).”
Mobley, A., Linder, S. K., Braeuer, R., Ellis, L. M., & Zwelling, L. (2013) · PLoS ONE · doi:10.1371/journal.pone.0063221 · Cited by 166
“Several recent publications suggested that the seminal findings from academic laboratories could only be reproduced 11–50% of the time [1,2].”
BackgroundSelf-citationChanging Trends in Computational Drug Repositioning
Yella, J. K., Yaddanapudi, S., Wang, Y., & Jegga, A. G. (2018) · Pharmaceuticals · doi:10.3390/ph11020057 · Cited by 166
“Reproducibility particularly is a major issue especially when scientific papers publish unexpected, positive results and other researchers or an independent research group is unable to replicate the same results even after using same or similar methods as reported by the original study [151,152].”
ResultNext-Generation Treatments for Mental Disorders
Insel, T. R. (2012) · Science Translational Medicine · doi:10.1126/scitranslmed.3004873 · Cited by 164
“With 80% of compounds failing in phase 2 (2) and 70% of preclinical target validation claims appearing nonreplicable (75) the f eld needs more rigor, with progress def ned by informative negative data as well as f ndings that promise a new therapeutic breakthrough.”
BackgroundSerum protein-based nanoparticles for cancer diagnosis and treatment
Iqbal, H., Yang, T., Li, T., Zhang, M., Ke, H., Ding, D., Deng, Y., & Chen, H. (2021) · Journal of Controlled Release · doi:10.1016/j.jconrel.2020.10.030 · Cited by 163
Polymer therapeutics: Top 10 selling pharmaceuticals — What next?
Duncan, R. (2014) · Journal of Controlled Release · doi:10.1016/j.jconrel.2014.05.001 · Cited by 161
Effects of Rodent Thermoregulation on Animal Models in the Research Environment
Hankenson, F. C., Marx, J. O., Gordon, C. J., & David, J. M. (2018) · Comparative Medicine · doi:10.30802/aalas-cm-18-000049 · Cited by 159
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161
“The controversial matter of insufficient reproducibility was, in fact, communicated openly in oncology and cardiovascular biology (Begley and Ellis, 2012; Errington et al., 2014; Libby, 2015).”
BackgroundShah, D. K., Haddish-Berhane, N., & Betts, A. (2012) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-012-9276-y · Cited by 160
“One of the main reasons for the aforementioned scenario is a poor appreciation of PK/PD [29], and the use of empirical approaches and mathematical Fig.”
Method(2014) · Springer eBooks · doi:10.1007/978-3-319-00026-8 · Cited by 156
Robust research: Institutions must do their part for reproducibility
Begley, C. G., Buchan, A. M., & Dirnagl, U. (2015) · Nature · doi:10.1038/525025a · Cited by 162
Self-citationValidation of Molecular Simulation: An Overview of Issues
van Gunsteren, W. F., Daura, X., Hansen, N., Mark, A. E., Oostenbrink, C., Riniker, S., & Smith, L. J. (2017) · Angewandte Chemie International Edition · doi:10.1002/anie.201702945 · Cited by 158
Kaplan, H. M., Mishra, P., & Kohn, J. (2015) · Journal of Materials Science Materials in Medicine · doi:10.1007/s10856-015-5558-4 · Cited by 161
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Towards more effective advanced drug delivery systems1
Crommelin, D. J. A., & Florence, A. T. (2013) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2013.02.020 · Cited by 173
On the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 149
“Reproducibility of experimental research is essential to the progress of science, but there is growing concern over the failure of scientific studies to replicate (e.g., Ioannidis 2005; Prinz et al. 2011; Begley and Ellis 2012; Pashler and Wagenmakers 2012;McNutt 2014).”
BackgroundCooper, J. R., Abdullatif, M. B., Burnett, E. C., Kempsell, K. E., Conforti, F., Tolley, H., Collins, J. E., & Davies, D. E. (2016) · PLoS ONE · doi:10.1371/journal.pone.0164438 · Cited by 174
Oxygenation in Cell Culture: Critical Parameters for Reproducibility Are Routinely Not Reported
Al-Ani, A., Toms, D., Kondro, D., Thundathil, J., Yu, Y., & Ungrin, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204269 · Cited by 152
More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research
Würbel, H. (2017) · Lab Animal · doi:10.1038/laban.1220 · Cited by 152
Simultaneous cross-evaluation of heterogeneous E. coli datasets via mechanistic simulation
Macklin, D. N., Ahn-Horst, T. A., Choi, H., Ruggero, N. A., Carrera, J., Mason, J. C., Sun, G., Agmon, E., DeFelice, M. M., Maayan, I., et al. (2020) · Science · doi:10.1126/science.aav3751 · Cited by 151
“This is most readily apparent when studies of the same system perform the same measurements but obtain different results, an issue that has led high-profile journals to question the reproducibility of results in multiple scientific fields ( 3 , 4 ).”
ResultGuidelines for in vivo mouse models of myocardial infarction
Lindsey, M. L., Brunt, K. R., Kirk, J. A., Kleinbongard, P., Calvert, J. W., Brás, L. E. D. C., DeLeon‐Pennell, K. Y., Re, D. P. D., Frangogiannis, N. G., Frantz, S., et al. (2021) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00459.2021 · Cited by 166
Ebner-Priemer, U. W., Houben, M., Santangelo, P., Kleindienst, N., Tuerlinckx, F., Oravecz, Z., Verleysen, G., Van Deun, K., Bohus, M., & Kuppens, P. (2015) · Journal of Abnormal Psychology · doi:10.1037/abn0000021 · Cited by 144
Reproducibility in Scientific Computing
Ivie, P., & Thain, D. (2018) · ACM Computing Surveys · doi:10.1145/3186266 · Cited by 198
“Some argue that this situation has reached crisis proportions: In the biotech industry, Amgen [14] attempted to confirm the findings in 53 “landmark” articles in cancer research.”
BackgroundOver half of psychology studies fail reproducibility test
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.18248 · Cited by 162
The replicability crisis and public trust in psychological science
Anvari, F., & Lakens, D. (2018) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2019.1684822 · Cited by 176
“In cancer biology, of 53 landmark studies investigated, 89% (47) could not be confirmed (Begley & Ellis, 2012; see also Davis et al., 2017).”
BackgroundPhilosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 184
“• Amgen and Bayer Healthcare reports.”
BackgroundLearning from our mistakes: The ‘unknown knowns’ in fragment screening
Davis, B. J., & Erlanson, D. A. (2013) · Bioorganic & Medicinal Chemistry Letters · doi:10.1016/j.bmcl.2013.03.028 · Cited by 146
Methods to increase reproducibility in differential gene expression via meta-analysis
Sweeney, T. E., Haynes, W. A., Vallania, F., Ioannidis, J. P. A., & Khatri, P. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw797 · Cited by 146
Targets, pitfalls and reference materials for liquid biopsy tests in cancer diagnostics
Geeurickx, E., & Hendrix, A. (2019) · Molecular Aspects of Medicine · doi:10.1016/j.mam.2019.10.005 · Cited by 157
“This lack of reproducibility is particularly higher for cancer research compared to other research fields, supposedly due to the higher urgency for finding solutions in oncology (Begley and Ellis, 2012).”
ResultOverlooked Benefits of using Polyclonal Antibodies
Ascoli, C. A., & Aggeler, B. (2018) · BioTechniques · doi:10.2144/btn-2018-0065 · Cited by 142
Clayton, A., Boilard, E., Buzas, E. I., Cheng, L., Falcón‐Perez, J. M., Gardiner, C., Gustafson, D., Gualerzi, A., Hendrix, A., Hoffman, A., et al. (2019) · Journal of Extracellular Vesicles · doi:10.1080/20013078.2019.1647027 · Cited by 140
von Kortzfleisch, V. T., Karp, N. A., Palme, R., Kaiser, S., Sachser, N., & Richter, S. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-73503-4 · Cited by 154
Editorial: preclinical data reproducibility for R&D - the challenge for neuroscience
Steckler, T. (2015) · SpringerPlus · doi:10.1186/2193-1801-4-1 · Cited by 140
Assessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139
Bikard, M. A., & Marx, M. (2019) · Management Science · doi:10.1287/mnsc.2019.3385 · Cited by 165
“The rate of replicability of published academic studies is estimated to be between 11% and 49% (Begley and Ellis 2012, Hartshorne and Schachner 2012, Freedman et al. 2015).”
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138
“portending a waning confidence in animal model data to identify promising therapeutic targets.(5,7,11,12,22,23) Cardiovascular research and development, in particular, has fared poorly in recent years relative to other disease categories, exhibiting the greatest decline as a percentage of total projects(4) and among t…”
InfluentialNonhuman Primates and Translational Research: Progress, Opportunities, and Challenges
Harding, J. D. (2017) · ILAR Journal · doi:10.1093/ilar/ilx033 · Cited by 150
“As reported by company scientists, estimates of failure rate range from 89% (Begley and Ellis 2012) to 75–80% (Prinz et al. 2011).”
BackgroundRethinking Reproducibility as a Criterion for Research Quality
Leonelli, S. (2018) · Research in the history of economic thought and methodology · doi:10.1108/s0743-41542018000036b009 · Cited by 188
“…result of the failure of studies aiming to reproduce the results of well-established, published research in the fields of molecular biology (Ioannidis et al 2012), experimental psychology (Open Science Collaboration 2015; Baker 2015) and clinical medicine (Prinz et al 2011; Begley and Ellis 2012).”
ResultIs Target-Based Drug Discovery Efficient? Discovery and “Off-Target” Mechanisms of All Drugs
Sadri, A. (2023) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.2c01737 · Cited by 162
Han, S., Olonisakin, T. F., Pribis, J. P., Zupetic, J., Yoon, J. H., Holleran, K. M., Jeong, K., Shaikh, N., Rubio, D. M., & Lee, J. S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0183591 · Cited by 131
Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 171
Frické, M. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23212 · Cited by 134
Manipulating the Gut Microbiota: Methods and Challenges: Figure 1
Ericsson, A. C., & Franklin, C. L. (2015) · ILAR Journal · doi:10.1093/ilar/ilv021 · Cited by 153
Early phase clinical trials to identify optimal dosing and safety
Cook, N., Hansen, A. R., Siu, L. L., & Abdul Razak, A. R. (2015) · Molecular Oncology · doi:10.1016/j.molonc.2014.07.025 · Cited by 134
Validation of cluster analysis results on validation data: A systematic framework
Ullmann, T., Hennig, C. M., & Boulesteix, A. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1444 · Cited by 177
Camerer, C., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117
A Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines
Niepel, M., Hafner, M., Mills, C. E., Subramanian, K., Williams, E. H., Chung, M., Gaudio, B., Barrette, A. M., Stern, A. D., Hu, B., et al. (2019) · Cell Systems · doi:10.1016/j.cels.2019.06.005 · Cited by 133
“Adoption of FAIR approaches also responds to concerns from industrial and academic groups about the reproducibility and utility of biomedical research (Arrowsmith, 2011; Baker, 2016; Begley and Ellis, 2012; Prinz et al., 2011) and the adequacy of data-reporting standards (Errington et al.”
BackgroundAchieving human and machine accessibility of cited data in scholarly publications
Starr, J., Castro, E., Crosas, M., Dumontier, M., Downs, R. R., Duerr, R., Haak, L. L., Haendel, M., Herman, I., Hodson, S., et al. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.1 · Cited by 128
“This finding reinforces significant concerns recently expressed in the scientific literature about reproducibility and whether many false positives are being reported as fact (Colquhoun (2014); Rekdal (2014); Begley and Ellis (2012); Prinz et al. (2011); Greenberg (2009); Ioannidis (2005)).”
BackgroundWelner, R. S., Amabile, G., Bararia, D., Czibere, A., Yang, H., Zhang, H., Pontes, L. L. D. F., Ye, M., Levantini, E., Di Ruscio, A., et al. (2015) · Cancer Cell · doi:10.1016/j.ccell.2015.04.004 · Cited by 129
“Error bars represent mean ± SD.”
ResultInfluentialClark, T., Ciccarese, P. N., & Goble, C. A. (2014) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-5-28 · Cited by 127
“These ultimately all address the issue of scientific reproducibility [8,55,59,60].”
InfluentialIrreproducibility in hydrogen storage material research
Broom, D. P., & Hirscher, M. (2016) · Energy & Environmental Science · doi:10.1039/c6ee01435f · Cited by 130
Lin, Z., Luo, G., Du, W., Kong, T., Liu, C., & Liu, Z. (2019) · Small · doi:10.1002/smll.201903899 · Cited by 138
Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Jarvis, M. F., & Williams, M. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2015.12.001 · Cited by 129
Objective assessment of cancer genes for drug discovery
Patel, M. N., Halling‐Brown, M., Tym, J. E., Workman, P., & Al‐Lazikani, B. (2012) · Nature Reviews Drug Discovery · doi:10.1038/nrd3913 · Cited by 149
“Out of the 46 targets, 26 are annotated by the Census as oncogenes in the reported cancer type, 19 are annotated as tumour suppressors 28 , and one — CBL , encoding an E3 ubiquitin protein ligase — is annotated as having both functions.”
BackgroundInfluentialChen, H., & Liu, R. H. (2018) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.7b04975 · Cited by 136
State of the art review: the data revolution in critical care
Ghassemi, M., Celi, L. A., & Stone, D. J. (2015) · Critical Care · doi:10.1186/s13054-015-0801-4 · Cited by 128
“A 2012 Nature review of cancer research found reproducibility of findings in only 11% of 53 published papers [45].”
BackgroundReproZip: Computational Reproducibility With Ease
Chirigati, F., Rampin, R., Shasha, D., & Freire, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2899401 · Cited by 128
Shortened TDP43 isoforms upregulated by neuronal hyperactivity drive TDP43 pathology in ALS
Weskamp, K., Tank, E. M., Miguez, R., McBride, J. P., Gómez, N. B., White, M., Lin, Z., Gonzalez, C. M., Serio, A., Sreedharan, J., et al. (2020) · Journal of Clinical Investigation · doi:10.1172/jci130988 · Cited by 126
COVID-19 research: pandemic versus “paperdemic”, integrity, values and risks of the “speed science”
Dinis‐Oliveira, R. J. (2020) · Forensic Sciences Research · doi:10.1080/20961790.2020.1767754 · Cited by 165
Pamies, D. (2018) · ALTEX · doi:10.14573/altex.1710081 · Cited by 127
Kriel, J., & Loos, B. (2019) · Cell Death and Differentiation · doi:10.1038/s41418-018-0267-4 · Cited by 139
Making Open Science Work for Science and Society
Elliott, K. C., & Resnik, D. B. (2019) · Environmental Health Perspectives · doi:10.1289/ehp4808 · Cited by 141
Neoclassic Drug Discovery: The Case for Lead Generation Using Phenotypic and Functional Approaches
Lee, J. A., & Berg, E. L. (2013) · SLAS Discovery · doi:10.1177/1087057113506118 · Cited by 126
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
Kepes, S., & McDaniel, M. A. (2013) · Industrial and Organizational Psychology · doi:10.1111/iops.12045 · Cited by 124
“Finally, evidence from other scientific disciplines indicates that the reproducibility of findings from published articles is shockingly low (e.g., Begley & Ellis, 2012; Ioannidis et al., 2009), likely due to incomplete reporting of relevant statistical information in the published articles…”
BackgroundQuantitative structure–activity relationship: promising advances in drug discovery platforms
Wang, T., Wu, M.-B., Lin, J.-P., & Yang, L.-R. (2015) · Expert Opinion on Drug Discovery · doi:10.1517/17460441.2015.1083006 · Cited by 124
“However, traditional drug design and development is a complex, costly and laborious process [5-7].”
BackgroundMichel, M. C., Murphy, T. J., & Motulsky, H. J. (2020) · Molecular Pharmacology · doi:10.1124/mol.119.118927 · Cited by 127
Biopsies: next-generation biospecimens for tailoring therapy
Basik, M., Aguilar‐Mahecha, A., Rousseau, C., Diaz, Z., Tejpar, S., Spatz, A., Greenwood, C. M. T., & Batist, G. (2013) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2013.101 · Cited by 126
Is mindfulness research methodology improving over time? A systematic review
Goldberg, S. B., Tucker, R. P., Greene, P. A., Simpson, T. L., Kearney, D. J., & Davidson, R. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0187298 · Cited by 125
Understanding the Replication Crisis as a Base Rate Fallacy
Bird, A. (2018) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axy051 · Cited by 145
Bias in the reporting of sex and age in biomedical research on mouse models
Flórez-Vargas, O., Brass, A., Karystianis, G., Bramhall, M., Stevens, R., Cruickshank, S., & Nenadic, G. (2016) · eLife · doi:10.7554/elife.13615 · Cited by 124
“years there have been growing concerns over the lack of reproducibility of results in certain areas of 37 biomedical research (Begley and Ellis, 2012; Prinz et al., 2011; Collins and Tabak, 2014).”
BackgroundCorrelated Multimodal Imaging in Life Sciences: Expanding the Biomedical Horizon
Walter, A., Paul‐Gilloteaux, P., Plochberger, B., Šefc, L., Verkade, P., Mannheim, J. G., Slezak, P., Unterhuber, A., Marchetti‐Deschmann, M., Ogris, M., et al. (2020) · Frontiers in Physics · doi:10.3389/fphy.2020.00047 · Cited by 142
Yang, Y., Yang, H. H., Hu, Y., Watson, P. H., Liu, H., Geiger, T. R., Anver, M. R., Haines, D. C., Martin, P., Green, J. E., et al. (2017) · Oncotarget · doi:10.18632/oncotarget.15695 · Cited by 121
Biobanking 3.0: Evidence based and customer focused biobanking
Simeon‐Dubach, D., & Watson, P. H. (2014) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2013.12.018 · Cited by 133
“While this issue has been known for a long time [33] to biobankers, recent papers have highlighted this issue for the broader scientific community as it relates to the very low reproducibility of scientific data published even in high impact factor journals [34,35].”
BackgroundReplication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 145
Generating Biologically Stable TNA Aptamers that Function with High Affinity and Thermal Stability
Dunn, M. R., McCloskey, C. M., Buckley, P. E., Rhea, K. A., & Chaput, J. C. (2020) · Journal of the American Chemical Society · doi:10.1021/jacs.0c00641 · Cited by 130
Integration of phytochemicals and phytotherapy into cancer precision medicine
Efferth, T., Saeed, M. E. M., Mirghani, E., Alim, A., Yassin, Z., Saeed, E. M., Khalid, H. E., & Daak, S. (2017) · Oncotarget · doi:10.18632/oncotarget.17466 · Cited by 143
Rigor and reproducibility in rodent behavioral research
Gulinello, M. E., Mitchell, H. A., Chang, Q., O’Brien, W. T., Zhou, Z., Abel, T. G., Wang, L., Corbin, J. G., Veeraragavan, S., Samaco, R. C., et al. (2018) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2018.01.001 · Cited by 132
“…tool to evaluate potential clinical treatments, well-replicated and highly robust phenotypes are necessary to detect drug responses, over and above innate biological variability (Begley & Ellis, 2012; Cole et al., 2013; Drucker, 2016; Schulz, Cookson, & Hausmann, 2016; Silverman et al., 2012).”
Open Access to Data: An Ideal Professed but Not Practised
Andreoli-Versbach, P., & Mueller‐Langer, F. (2013) · Research Policy · doi:10.1016/j.respol.2014.04.008 · Cited by 153
Gupta, A. D., Chu, W. H., Howell, S., Chakraborty, S., Koblar, S., Visvanathan, R., Cameron, I., & Wilson, D. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0670-9 · Cited by 120
“It is not clear why this approach was taken as AUC is generally used for other purposes, for example, in pharmacology, to plot concentrations of drugs in blood plasma over time [25].”
MethodTumor-on-a-chip for integrating a 3D tumor microenvironment: chemical and mechanical factors
Wan, L., Neumann, C. A., & LeDuc, P. R. (2020) · Lab on a Chip · doi:10.1039/c9lc00550a · Cited by 120
A survey of early-career researchers in Australia
Christian, K., Johnstone, C., Larkins, J., Wright, W., & Doran, M. R. (2021) · eLife · doi:10.7554/elife.60613 · Cited by 141
Quality Issues of Research Antibodies
Weller, M. G. (2016) · Analytical Chemistry Insights · doi:10.4137/aci.s31614 · Cited by 118
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G. N., & Clarke, R. M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 116
“However, we would suggest that there are reasonably straightforward solutions to this problem and a number of specific solutions were suggested by Begley and Ellis (2012) in their discussion of replication failure in preclinical cancer trials.”
BackgroundHuman-Derived Organ-on-a-Chip for Personalized Drug Development
Jodat, Y. A., Kang, M., Kiaee, K., Kim, G. J., Martinez, A. F. H., Rosenkranz, A., Bae, H., & Shin, S. R. (2019) · Current Pharmaceutical Design · doi:10.2174/1381612825666190308150055 · Cited by 136
Bauer, M., Strom, M., Hammond, D., & Shigdar, S. L. (2019) · Molecules · doi:10.3390/molecules24234377 · Cited by 131
Jarantow, S. W., Pisors, E. D., & Chiu, M. L. (2023) · Current Protocols · doi:10.1002/cpz1.801 · Cited by 162
“For some time, there has been an apparent crisis in the scientific community regarding the low reproducibility of experimental data (Begley & Ellis, 2012; Flier, 2017; Ioannidis, 2005; MacCoun & Perlmutter, 2015; Nature, 2016; Nuzzo, 2015).”
BackgroundThe Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118
Replication, Communication, and the Population Dynamics of Scientific Discovery
McElreath, R., & Smaldino, P. E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0136088 · Cited by 117
Never waste a good crisis: confronting reproducibility in translational research
Drucker, D. J. (2016) · Cell Metabolism · doi:10.1016/j.cmet.2016.08.006 · Cited by 117
Grant, R. P., & Hoofnagle, A. N. (2014) · Clinical Chemistry · doi:10.1373/clinchem.2014.224840 · Cited by 117
“Given the lessons of the past (1), we cannot overemphasize the importance of this level of transparency and rigor in the publication of novel scientific discoveries.”
Predictive validity in drug discovery: what it is, why it matters and how to improve it
Scannell, J. W., Bosley, J., Hickman, J. A., Dawson, G. R., Truebel, H., Ferreira, G. S., Richards, D. B., & Treherne, J. M. (2022) · Nature Reviews Drug Discovery · doi:10.1038/s41573-022-00552-x · Cited by 142
How to make more published research true
Ioannidis, J. P. A. (2015) · Acimed · Cited by 134
Lalu, M. M., Sullivan, K., Mei, S. H. J., Moher, D., Straus, A., Fergusson, D. A., Stewart, D. J., Jazi, M., Macleod, M. R., Winston, B. W., et al. (2016) · eLife · doi:10.7554/elife.17850 · Cited by 133
Metadata
- Authors
- Begley, C. G., & Ellis, L. M.
- Affiliations
- Amgen (United States) (US)
- The University of Texas MD Anderson Cancer Center (US)
- Corresponding author
- Ellis, Lee M.
- Venue
- Nature
- Published
- 2012-03-28
- Volume
- 483
- Issue
- 7391
- Pages
- 531-533
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1987777080
- Semantic Scholar
- 1a81d3d1e8a983e0551edf485f4ac1a8cdd7fea4
- Cited by
- 2,979
- References
- 5
- Citations by year
- 2026: 106 · 2025: 116 · 2024: 127 · 2023: 169 · 2022: 179 · 2021: 206 · 2020: 236 · 2019: 255 · 2018: 262 · 2017: 271 · 2016: 283 · 2015: 262 · 2014: 226 · 2013: 133 · 2012: 62
- FWCI
- 210.63
- Type
- journal article
- Language
- en
- Update notices
- correction (2012-05-02)
- Open access
- bronze
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Nursing › Issues, ethics and legal aspects › Biomedical Ethics and Regulation
- Keywords
- preclinical cancer research, research standards, publication practices, research incentives, translational research, clinical trials
- MeSH terms
- Animals, Clinical Trials as Topic, Drug Evaluation, Preclinical, Humans, Neoplasms, Research Design, Reproducibility of Results, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.