An open investigation of the reproducibility of cancer biology research
Open accessAbstract
It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Reproducibility Project: Cancer Biology was unable to find sufficient protocol information for reproducing experiments in any of the original manuscripts, and reconstructed findings were often of greater variability or lower efficacy than the original ones.
“The”
From The history, landscape, and outlook of human cell line authentication and security (2024) · cites this paper as [43]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredInsufficient reproducibility has been openly communicated in oncology and cardiovascular biology.
“The controversial matter of insufficient reproducibility was, in fact, communicated openly in oncology and cardiovascular biology (Begley and Ellis, 2012; Errington et al., 2014; Libby, 2015).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Errington et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 4× · see citing works
“observed the median power of studies to be 21% (Button et al., 2013), which means that assuming the finding being investigated is true and accurately estimated, then only 21 of every 100 studies investigating that effect would detect statistically significant evidence for the effect.”
InfluentialSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1× · see citing works
“Instead, it suggests systematic exclusion of negative results ( Greenwald, 1975 ) and possibly the exaggeration of positive results by employing flexibility in analytic and reporting practices that inflate the likelihood of false positives ( Simmons et al., 2011 ).”
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1× · see citing works
“Across disciplines, most published studies demonstrate positive results—results that indicate an expected association between variables or a difference between experimental conditions ( Sterling, 1959 ; Fanelli, 2010 , 2012 ).”
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · see citing works
Rescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
Self-citationThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · see citing works
Inconsistency in large pharmacogenomic studies
Haibe-Kains, B., El-Hachem, N., Birkbak, N. J., Jin, A. C., Beck, A. H., Aerts, H. J. W. L., & Quackenbush, J. (2013) · Nature · doi:10.1038/nature12831 · Cited by 499 · see citing works
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384 · see citing works
Vasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Replication and reproducibility in spinal cord injury research
Steward, O., Popovich, P. G., Dietrich, W. D., & Kleitman, N. (2012) · Experimental Neurology · doi:10.1016/j.expneurol.2011.06.017 · Cited by 186 · see citing works
Flagged first, then by citations
Research works printed but not linked: 29
- Asendorpf JB Conner M De Fruyt F De Houwer J Denissen JJ Fiedler K Fiedler S Funder DC Kliegl R Nosek BA Perugini M Roberts BW Schmitt M van Aken MA Weber H Wicherts JM 2013 Recommendations for increasing replicability in psychology European Journal of Personality 27 108 119 10.1002/per.1919 · DOI
- Bissell M 2013 The risks of the replication drive Nature 503 333 334 10.1038/503333a 24273798 · DOI · Paper page
- Collins FS Tabak LA 2014 NIH plans to enhance reproducibility Nature 503 612 613 10.1038/505612a 24482835 PMC4058759 · DOI · Paper page
- Fanelli D 2010 “Positive” results increase down the Hierarchy of the Sciences PLOS ONE 5 e10068 10.1371/journal.pone.0010068 20383332 PMC2850928 · DOI · Paper page
- Fanelli D 2012 Negative results are disappearing from most disciplines and countries Scientometrics 90 891 904 10.1007/s11192-011-0494-7 · DOI · Paper page
- Forscher BK 1963 Chaos in the Brickyard Science 142 339 10.1126/science.142.3590.339 17799464 · DOI
- Hackam DG Redelmeier DA 2006 Translation of research evidence from animals to humans Jama 296 1727 1732 10.1001/jama.296.14.1731 17032985 · DOI · Paper page
- Hess KR 2011 Statistical design considerations in animal studies published recently in cancer research Cancer Research 71 625 10.1158/0008-5472.can-10-3296 21239476 · DOI · Paper page
- Hines WC Su Y Kuhn I Polyak K Bissell MJ 2014 Sorting out the FACS: a devil in the details Cell Reports 6 779 781 10.1016/j.celrep.2014.02.021 24630040 · DOI
- Ioannidis JP Khoury MJ 2011 Improving validation practices in “Omics” research Science 334 1230 1232 10.1126/science.1211811 22144616 PMC5624327 · DOI
- Ioannidis JP Munafo MR Fusar-Poli P Nosek BA David SP 2014 Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention Trends in Cognitive Sciences 18 235 241 10.1016/j.tics.2014.02.010 24656991 PMC4078993 · DOI · Paper page
- Jasny BR Chin G Chong L Vignieri S 2011 Data replication & reproducibility. Again, and again, and again… Science 334 1225 10.1126/science.334.6060.1225 22144612 · DOI
- Kuhn TS 1962 The Structure of Scientific Revolutions Chicago, Illinois University of Chicago Press
- Loscalzo J 2012 Irreproducible experimental results: causes, (mis)interpretations, and consequences Circulation 125 1211 1214 10.1161/circulationaha.112.098244 22412087 PMC3319669 · DOI
- Makel MC Plucker JA Hagerty B 2012 Replications in psychology research: how often do they really occur? Perspectives on Psychological Science 7 537 542 10.1177/1745691612460688 26168110 · DOI · Paper page
- Merton RK 1942 Science and technology in a democratic order Journal of Legal and Political Sociology 1 115 126
- Miguel E Camerer C Casey K Cohen J Esterling KM Gerber A Glennerster R Green DP Humphreys M Imbens G Laitin D Madon T Nelson L Nosek BA Petersen M Sedlmayr R Simmons JP Simonsohn U Van der Laan M 2014 Promoting transparency in social science research Science 343 30 31 10.1126/science.1245317 24385620 PMC4103621 · DOI
- Mobley A Linder SK Braeuer R Ellis LM Zwelling L 2013 A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic PLOS ONE 8 e63221 10.1371/journal.pone.0063221 23691000 PMC3655010 · DOI
- Nature 2012 Collateral damage Nature 485 41 10.1038/485041e · DOI
- Nosek BA Spies JR Motyl M 2012 Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability Perspectives on Psychological Science 7 615 631 10.1177/1745691612459058 PMC10540222 26168121 · DOI
- Open Science Collaboration 2012 An open, large-scale, collaborative effort to estimate the reproducibility of psychological science Perspectives on Psychological Science 7 657 660 10.1177/1745691612462588 26168127 · DOI
- Open Science Collaboration 2014 The Reproducibility Project: a model of large-scale collaboration for empirical research on reproducibility Stodden V Leish VF Peng R Implementing Reproducible Computational Research (A Volume in the R Series) New York Taylor & Francis
- Perrin S 2014 Preclinical research: Make mouse studies work Nature 507 423 425 10.1038/507423a 24678540 · DOI · Paper page
- Popper K 1934/1992 The Logic of Scientific Discovery New York Routledge
- Pusztai L Hatzis C Andre F 2013 Reproducibility of research and preclinical validation: problems and solutions Nature Reviews Clinical Oncology 10 720 724 10.1038/nrclinonc.2013.171 24080600 · DOI
- Schmidt S 2009 Shall we really do it again? The powerful concept of replication is neglected in the social sciences Review of General Psychology 13 90 100 10.1037/a0015108 · DOI · Paper page
- Valentine JC Biglan A Boruch RF Castro FG Collins LM Flay BR Kellam S Moscicki EK Schinke SP 2011 Replication in prevention science Prevention Science 12 103 117 10.1007/s11121-011-0217-6 21541692 · DOI · Paper page
- Vines TH Albert AY Andrew RL Debarre F Bock DG Franklin MT Gilbert KJ Moore JS Renaut S Rennison DJ 2014 The availability of research data declines rapidly with article age Current Biology 24 94 97 10.1016/j.cub.2013.11.014 24361065 · DOI · Paper page
- Wadmann M 2013 NIH mulls rules for validating key results Nature 500 14 16 10.1038/500014a 23903729 · DOI
Cited by
Sorted by citations
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Fusobacterium nucleatum — symbiont, opportunist and oncobacterium
Brennan, C. A., & Garrett, W. S. (2019) · Nature Reviews Microbiology · doi:10.1038/s41579-018-0129-6 · Cited by 1,205 · see citing works
Growth rate inhibition metrics correct for confounders in measuring sensitivity to cancer drugs
Hafner, M., Niepel, M., Chung, M., & Sorger, P. K. (2016) · Nature Methods · doi:10.1038/nmeth.3853 · Cited by 681 · see citing works
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
“In the Reproducibility Project: Cancer Biology, we sought to acquire evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from 53 highimpact papers published in 2010, 2011, and 2012 (Errington et al., 2014).”
MethodSelf-citationMisinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391 · see citing works
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · see citing works
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266 · see citing works
Nosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242 · see citing works
Self-citationReproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215 · see citing works
Common pitfalls in preclinical cancer target validation
Kaelin, W. G. (2017) · Nature Reviews Cancer · doi:10.1038/nrc.2017.32 · Cited by 175 · see citing works
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161 · see citing works
Oxygenation in Cell Culture: Critical Parameters for Reproducibility Are Routinely Not Reported
Al-Ani, A., Toms, D., Kondro, D., Thundathil, J., Yu, Y., & Ungrin, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204269 · Cited by 152 · see citing works
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131 · see citing works
Reproducible research practices are underused in systematic reviews of biomedical interventions
Page, M. J., Altman, D. G., Shamseer, L., McKenzie, J. E., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.017 · Cited by 115 · see citing works
Bustin, S., & Nolan, T. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12801 · Cited by 113 · see citing works
Pollet, T. V., & Saxton, T. K. (2019) · Evolutionary Psychological Science · doi:10.1007/s40806-019-00192-2 · Cited by 89 · see citing works
Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.0549 · Cited by 78 · see citing works
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72 · see citing works
Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66 · see citing works
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49 · see citing works
The best time to argue about what a replication means? Before you do it
Nosek, B. A., & Errington, T. M. (2020) · Nature · doi:10.1038/d41586-020-02142-6 · Cited by 47 · see citing works
Self-citationOverflow in science and its implications for trust
Siebert, S., Machesky, L. M., & Insall, R. H. (2015) · eLife · doi:10.7554/elife.10825 · Cited by 34 · see citing works
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32 · see citing works
Identification and characterization of cancer vulnerabilities via targeted protein degradation
Mayor-Ruiz, C., & Winter, G. E. (2019) · Drug Discovery Today: Technologies · doi:10.1016/j.ddtec.2018.12.003 · Cited by 33 · see citing works
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28 · see citing works
The history, landscape, and outlook of human cell line authentication and security
Harbut, E., Makris, Y., Pertsemlidis, A., & Bleris, L. (2024) · SLAS Discovery · doi:10.1016/j.slasd.2024.100194 · Cited by 8 · see citing works
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.07.239855 · Cited by 131 · see citing works
- Authors
- Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A.
- Affiliations
- Center for Open Science (US)
- Science Exchange (United States) (US)
- Mendeley (United Kingdom) (GB)
- Corresponding author
- Errington, Timothy M
- Nosek, Brian A
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2014-12-10
- Volume
- 3
- ISSN
- 2050-084X (ISSN-L)
- PMC
- PMC4270077
- OpenAlex
- W2142148275
- Cited by
- 312
- References
- 19
- Citations by year
- 2026: 14 · 2025: 9 · 2024: 13 · 2023: 19 · 2022: 12 · 2021: 33 · 2020: 22 · 2019: 38 · 2018: 35 · 2017: 53 · 2016: 34 · 2015: 28 · 2014: 1
- FWCI
- 72.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- eLife (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- reproducibility, replication study, preclinical cancer research, research reproducibility
- MeSH terms
- Humans, Neoplasms, Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
