Can cancer researchers accurately judge whether preclinical reports will reproduce?
Open accessAbstract
There is vigorous debate about the reproducibility of research findings in cancer biology. Whether scientists can accurately assess which experiments will reproduce original findings is important to determining the pace at which science self-corrects. We collected forecasts from basic and preclinical cancer researchers on the first 6 replication studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) to assess the accuracy of expert judgments on specific replication outcomes. On average, researchers forecasted a 75% probability of replicating the statistical significance and a 50% probability of replicating the effect size, yet none of these studies successfully replicated on either criterion (for the 5 studies with results reported). Accuracy was related to expertise: experts with higher h-indices were more accurate, whereas experts with more topic-specific expertise were less accurate. Our findings suggest that experts, especially those with specialized knowledge, were overconfident about the RP:CB replicating individual experiments within published reports; researcher optimism likely reflects a combination of overestimating the validity of original studies and underestimating the difficulties of repeating their methodologies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Predicting the reproducibility of cancer treatments even by experts has not always been accurate.
“Predicting the reproducibility of cancer treatments even by experts has not always been accurate106.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [106]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
BET Bromodomain Inhibition as a Therapeutic Strategy to Target c-Myc
Delmore, J. E., Issa, G. C., Lemieux, M. E., Rahl, P. B., Shi, J., Jacobs, H. M., Kastritis, E., Gilpatrick, T., Paranal, R. M., Qi, J., et al. (2011) · Cell · doi:10.1016/j.cell.2011.08.017 · Cited by 2,820 · see citing works
Clinical development success rates for investigational drugs
Hay, M., Thomas, D. W., Craighead, J. L., Economides, C., & Rosenthal, J. (2014) · Nature Biotechnology · doi:10.1038/nbt.2786 · Cited by 2,536 · see citing works
Timid choices and bold forecasts: a cognitive perspective on risk taking.
Kahneman, D., & Lovallo, D. (1993) · Management Science · doi:10.1287/mnsc.39.1.17 · Cited by 2,043 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
(1991) · Cited by 1,767 · see citing works
Willingham, S. B., Volkmer, J.-P., Gentles, A. J., Sahoo, D., Dalerba, P. D., Mitra, S. S., Wang, J., Contreras-Trujillo, H., Martin, R. K., Cohen, J. D., et al. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1121623109 · Cited by 1,667 · see citing works
Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukaemia
Dawson, M. A., Prinjha, R. K., Dittmann, A., Giotopoulos, G., Bantscheff, M., Chan, W.-I., Robson, S. C., Chung, C., Hopf, C., Savitski, M. M., et al. (2011) · Nature · doi:10.1038/nature10509 · Cited by 1,512 · see citing works
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · see citing works
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking
Kahneman, D. J., & Lovallo, D. P. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511803475.023 · Cited by 1,304 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Expert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260 · see citing works
Coadministration of a Tumor-Penetrating Peptide Enhances the Efficacy of Cancer Drugs
Sugahara, K. N., Teesalu, T., Karmali, P. P., Kotamraju, V. R., Agemy, L., Greenwald, D. R., & Ruoslahti, E. I. (2010) · Science · doi:10.1126/science.1183057 · Cited by 1,080 · see citing works
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
Sirota, M., Dudley, J. T., Kim, J., Chiang, A. P.-F., Morgan, A. A., Sweet-Cordero, A., Sage, J., & Butte, A. J. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3001318 · Cited by 801 · see citing works
Melanoma genome sequencing reveals frequent PREX2 mutations
Berger, M. F., Hodis, E., Heffernan, T. P., Deribe, Y. L., Lawrence, M. S., Protopopov, A., Ivanova, E., Watson, I. R., Nickerson, E., Ghosh, P., et al. (2012) · Nature · doi:10.1038/nature11071 · Cited by 756 · see citing works
Evidential impact of base rates
Tversky, A., & Kahneman, D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.011 · Cited by 591 · see citing works
(1991) · CaltechAUTHORS (California Institute of Technology) · Cited by 340 · see citing works
The role of actively open-minded thinking in information acquisition, accuracy, and calibration
Haran, U., Ritov, I., & Mellers, B. A. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500005921 · Cited by 331 · see citing works
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331 · see citing works
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312 · see citing works
Scoring rules and the evaluation of probabilities
Winkler, R., Muñoz, J., Cervera, J. L., Bernardo, J., Blattenberger, G., Kadane, J., Lindley, D., Murphy, A. H., Oliver, R. M., & Insua, D. R. (1996) · Test · doi:10.1007/bf02562681 · Cited by 238 · see citing works
Fischhoff, B., & de Bruin, W. B. (1999) · Journal of Behavioral Decision Making · doi:10.1002/(sici)1099-0771(199906)12:2 · Cited by 212 · see citing works
The Moderating Effects of Problem Characteristics on Experts' and Novices' Judgments
Spence, M. T., & Brucks, M. L. (1997) · Journal of Marketing Research · doi:10.2307/3151861 · Cited by 203 · see citing works
Overconfidence in interval estimates: What does expertise buy you?
McKenzie, C. R. M., Liersch, M. J., & Yaniv, I. (2008) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2008.02.007 · Cited by 180 · see citing works
Cancer reproducibility project releases first results
Baker, M., & Dolgin, E. (2017) · Nature · doi:10.1038/541269a · Cited by 104 · see citing works
“Accuracy of Forecasts in Strategic Intelligence.”
Mandel, D. R., & Barnes, A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1406138111 · Cited by 101 · see citing works
Self-citationReproducibility: The risks of the replication drive
Bissell, M. (2013) · Nature · doi:10.1038/503333a · Cited by 99 · see citing works
Expertise and Decision Support
Wright, G., & Bolger, F. (1992) · Springer eBooks · doi:10.1007/b102410 · Cited by 86 · see citing works
On the Competence and Incompetence of Experts
Ayton, P. (2007) · Springer eBooks · doi:10.1007/978-0-585-34290-0_5 · Cited by 85 · see citing works
Evidential Impact of Base Rates
Tversky, A., & Kahneman, D. (1981) · doi:10.21236/ada099501 · Cited by 80 · see citing works
McBride, M., Fidler, F., & Burgman, M. (2012) · Diversity and Distributions · doi:10.1111/j.1472-4642.2012.00884.x · Cited by 73 · see citing works
Horrigan, S. K., & Reproducibility Project: Cancer Biology (2017) · eLife · doi:10.7554/elife.18173 · Cited by 72 · see citing works
Professional vs Amateur Judgment Accuracy: The Case of Foreign Exchange Rates
Önkal, D., Yates, J., Simga‐Mugan, C., & Öztin, Ş. (2003) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(03)00058-x · Cited by 70 · see citing works
Bringing probability judgments into policy debates via forecasting tournaments
Tetlock, P. E., Mellers, B. A., & Scoblic, J. P. (2017) · Science · doi:10.1126/science.aal3147 · Cited by 67 · see citing works
Mantis, C., Kandela, I. K., Aird, F., & Reproducibility Project: Cancer Biology (2017) · eLife · doi:10.7554/elife.17584 · Cited by 54 · see citing works
Kaiser, J. (2015) · Science · doi:10.1126/science.348.6242.1411 · Cited by 45 · see citing works
Replication Study: BET bromodomain inhibition as a therapeutic strategy to target c-Myc
Aird, F., Kandela, I. K., Mantis, C., & Reproducibility Project: Cancer Biology (2017) · eLife · doi:10.7554/elife.21253 · Cited by 45 · see citing works
Time to do something about reproducibility
Morrison, S. J. (2014) · eLife · doi:10.7554/elife.03981 · Cited by 44 · see citing works
Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations
Horrigan, S. K., Courville, P., Sampey, D. B., zhou, F., Cai, S., & Reproducibility Project: Cancer Biology (2017) · eLife · doi:10.7554/elife.21634 · Cited by 32 · see citing works
(2017) · eLife · doi:10.7554/elife.23693 · Cited by 27 · see citing works
Kandela, I. K., Aird, F., & Reproducibility Project: Cancer Biology (2017) · eLife · doi:10.7554/elife.17044 · Cited by 26 · see citing works
Rigorous replication effort succeeds for just two of five cancer papers
(2017) · AAAS Articles DO Group · doi:10.1126/science.aal0628 · Cited by 14 · see citing works
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Flagged first, then by citations
Research works printed but not linked: 1
- [23]Weissman I. Response to: “Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors.” eLife. 2017; 6: e18173. 28100392 PMC5245970
Cited by
Sorted by citations
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175 · see citing works
Predict science to improve science
DellaVigna, S., Pope, D. G., & Vivalt, E. (2019) · Science · doi:10.1126/science.aaz1704 · Cited by 140 · see citing works
Bringing probability judgments into policy debates via forecasting tournaments
Tetlock, P. E., Mellers, B. A., & Scoblic, J. P. (2017) · Science · doi:10.1126/science.aal3147 · Cited by 67 · see citing works
The anti-money laundering risk assessment: A probabilistic approach
Ogbeide, H., Thomson, M. E., Gonul, M. S., Pollock, A. C., Bhowmick, S., & Bello, A. U. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.113820 · Cited by 39 · see citing works
Gupta, A., Gautam, P., Wennerberg, K., & Aittokallio, T. A. (2020) · Communications Biology · doi:10.1038/s42003-020-0765-z · Cited by 35 · see citing works
Predicting and reasoning about replicability using structured groups
Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H. S., Hemming, V., Thorn, F. S., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221553 · Cited by 31 · see citing works
Fraser, H. S., Bush, M., Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Gould, E., Hemming, V., Hamilton, D. G., Rumpff, L., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0274429 · Cited by 30 · see citing works
Self-citationIrreproducible text-book “knowledge”: The effects of color bands on zebra finch fitness
Wang, D., Forstmeier, W., Ihle, M., Khadraoui, M., Jerónimo, S., Martin, K., & Kempenaers, B. (2018) · Evolution · doi:10.1111/evo.13459 · Cited by 30 · see citing works
Schetinin, V., Jakaite, L., & Krzanowski, W. J. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.009 · Cited by 27 · see citing works
Is preclinical research in cancer biology reproducible enough?
Kane, P. B., & Kimmelman, J. (2021) · eLife · doi:10.7554/elife.67527 · Cited by 21 · see citing works
Self-citationThe Mouse Hospital and Its Integration in Ultra-Precision Approaches to Cancer Care
Clohessy, J. G., & Pandolfi, P. P. (2018) · Frontiers in Oncology · doi:10.3389/fonc.2018.00340 · Cited by 21 · see citing works
Correcting Judgment Correctives in National Security Intelligence
Mandel, D. R., & Tetlock, P. E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02640 · Cited by 21 · see citing works
Self-citationGrossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 20 · see citing works
Self-citationImproving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments
Fan, Y., Budescu, D. V., Mandel, D. R., & Himmelstein, M. (2019) · Decision Analysis · doi:10.1287/deca.2018.0388 · Cited by 19 · see citing works
Self-citationXie, J. Y., Qu, C., Munro, G., Petersen, K. A., & Porreca, F. (2019) · Pain · doi:10.1097/j.pain.0000000000001569 · Cited by 18 · see citing works
Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
Collins, R. N., Mandel, D. R., & Budescu, D. V. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_7 · Cited by 16 · see citing works
Self-citationFrancioli, S. P., Shakeri, A., & North, M. S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311009121 · Cited by 15 · see citing works
Understanding of researcher behavior is required to improve data reliability
Wass, M. N., Ray, L., & Michaelis, M. (2019) · GigaScience · doi:10.1093/gigascience/giz017 · Cited by 15 · see citing works
Talent Spotting in Crowd Prediction
Atanasov, P. D., & Himmelstein, M. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_6 · Cited by 14 · see citing works
Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Marcoci, A., Wilkinson, D. P., Vercammen, A., Wintle, B. C., Abatayo, A. L., Baskin, E., Berkman, H., Buchanan, E. M., Capitán, S., Capitán, T., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01961-1 · Cited by 11 · see citing works
Issues and Challenges to Reproducibility of Cancer Research: A Commentary
da Silva, J. A. T. (2022) · Future Oncology · doi:10.2217/fon-2021-1378 · Cited by 8 · see citing works
Wisdom of the expert crowd prediction of response for 3 neurology randomized trials
Atanasov, P. D., Diamantaras, A. A., MacPherson, A., Vinarov, E., Benjamin, D. M., Shrier, I., Paul, F., Dirnagl, U., & Kimmelman, J. (2020) · Neurology · doi:10.1212/wnl.0000000000009819 · Cited by 7 · see citing works
Self-citationHaslberger, M., Schorr, S. G., Strech, D., & Haven, T. L. (2022) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15503 · Cited by 6 · see citing works
Science with no fiction: measuring the veracity of scientific reports by citation analysisPreprint
Grabitz, P., Lazebnik, Y. A., Nicholson, J. M., & Rife, S. C. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/172940 · Cited by 4 · see citing works
The impact of feedback training on prediction of cancer clinical trial results
Doussau, A., Kane, P. B., Peppercorn, J. M., Feustel, A. C., Ganeshamoorthy, S., Kekre, N. K., Benjamin, D. M., & Kimmelman, J. (2023) · Clinical Trials · doi:10.1177/17407745231203375 · Cited by 2 · see citing works
Self-citationDressman, M., & Rao, D. (2019) · English in Education · doi:10.1080/04250494.2019.1626712 · Cited by 2 · see citing works
Fibrin and Extracellular Matrix: Scaffolds and Network for Malignant Cells
Schardt, F., Schmaußer, B., Bedel, S., Bachmann, E., & Henke, E. (2021) · Medical Research Archives · doi:10.18103/mra.v9i9.2551 · Cited by 2 · see citing works
A comment on computational biology and connecting the dots.
Lortie, C. J. (2019) · Ideas in Ecology and Evolution · doi:10.24908/iee.2018.11.12.e · Cited by 1 · see citing works
Protzko, J., Holcombe, A. O., & Simons, D. J. (2019) · doi:10.31234/osf.io/pq8m4
Towards a “Wisdom of Scientific Crowds”
Jorm, A. F. (2024) · Springer eBooks · doi:10.1007/978-981-97-9222-1_10
Knowledge and Awareness of Nursing Students Regarding Breast Cancer: A Cross-Sectional Study
Alaqidia, J. M., Helala, S., Hamdana, F., Alkhalidia, K., Alenzia, A. H., Alenzia, R. F., Ayeda, A. H., Alanzia, S. H., & Mohammed, H. M. (2025) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.6.0725.25131
- Authors
- Benjamin, D., Mandel, D. R., & Kimmelman, J.
- Affiliations
- McGill University (CA)
- York University (CA)
- Corresponding author
- Kimmelman, Jonathan
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-06-29
- Volume
- 15
- Issue
- 6
- Pages
- e2002212
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5490935
- OpenAlex
- W2732894280
- Cited by
- 32
- References
- 46
- Citations by year
- 2024: 3 · 2023: 8 · 2022: 2 · 2021: 3 · 2020: 2 · 2019: 7 · 2018: 5 · 2017: 2
- FWCI
- 7.04
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship@McGill (McGill) (submittedVersion, cc-by)
- Figshare (submittedVersion, cc-by)
- Funders
- Canadian Institutes of Health Research (EOG 201303)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- reproducibility, replication study, expert opinion, overconfidence, preclinical cancer research, statistical significance, effect size, h-index, researcher expertise
- MeSH terms
- Animals, Data Collection, Expert Testimony, Humans, Judgment, Neoplasms, Professional Competence, Research Personnel, Science, Reproducibility of Results, Xenograft Model Antitumor Assays, Biomedical Research, Mice, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
