Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) has shown promise for diagnosing prostate cancer in biopsies. However, results have been limited to individual studies, lacking validation in multinational settings. Competitions have been shown to be accelerators for medical imaging innovations, but their impact is hindered by lack of reproducibility and independent validation. With this in mind, we organized the PANDA challenge—the largest histopathology competition to date, joined by 1,290 developers—to catalyze development of reproducible AI algorithms for Gleason grading using 10,616 digitized prostate biopsies. We validated that a diverse set of submitted algorithms reached pathologist-level performance on independent cross-continental cohorts, fully blinded to the algorithm developers. On United States and European external validation sets, the algorithms achieved agreements of 0.862 (quadratically weighted κ, 95% confidence interval (CI), 0.840–0.884) and 0.868 (95% CI, 0.835–0.900) with expert uropathologists. Successful generalization across different patient populations, laboratories and reference standards, achieved by a variety of algorithmic approaches, warrants evaluating AI-based Gleason grading in prospective clinical trials.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Self-citationEhteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 1×
“GG 1 852 (17) 1,814 (33) 24 (12) 48 (24) 48 (14) 53 (25) 247 (33) 65 (20) 3,151 (25)”
Self-citationEpstein, J. I., Egevad, L., Amin, M. B., Delahunt, B., Srigley, J. R., & Humphrey, P. A. (2016) · American Journal of Surgical Pathology · doi:10.1097/pas.0000000000000530 · Cited by 3,236
Self-citationKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Self-citationSingularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740
Prostate Cancer, Version 2.2019, NCCN Clinical Practice Guidelines in Oncology
Mohler, J. L., Antonarakis, E. S., Armstrong, A. J., D’Amico, A. V., Davis, B. J., Dorff, T., Eastham, J. A., Enke, C. A., Farrington, T. A., Higano, C. S., et al. (2019) · Journal of the National Comprehensive Cancer Network · doi:10.6004/jnccn.2019.0023 · Cited by 1,532
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
A Contemporary Prostate Cancer Grading System: A Validated Alternative to the Gleason Score
Epstein, J. I., Zelefsky, M. J., Sjoberg, D. D., Nelson, J. B., Egevad, L., Magi-Galluzzi, C., Vickers, A. J., Parwani, A. V., Reuter, V. E., Fine, S. W., et al. (2016) · European Urology · doi:10.1016/j.eururo.2015.06.046 · Cited by 1,374
Self-citationEnsuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341 · cited 1×
“Nontumor 967 (19) 1,925 (35) 95 (49) 58 (29) 155 (47) 66 (31) 254 (34) 108 (33) 3,628 (29)”
Self-citationNagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Artificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956
Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl
Caicedo, J. C., Goodman, A., Karhohs, K. W., Cimini, B. A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0612-7 · Cited by 809
Prognostic G leason grade grouping: data based on the modified G leason scoring system
Pierorazio, P. M., Walsh, P. C., Partin, A. W., & Epstein, J. I. (2013) · British Journal of Urology · doi:10.1111/j.1464-410x.2012.11611.x · Cited by 751
van Leenders, G. J. L. H., van der Kwast, T. H., Grignon, D. J., Evans, A. J., Kristiansen, G., Kweldam, C. F., Litjens, G., McKenney, J. K., Melamed, J., Mottet, N., et al. (2020) · American Journal of Surgical Pathology · doi:10.1097/pas.0000000000001497 · Cited by 713
Self-citationBulten, W., Pinckaers, H., van Boven, H., Vink, R., de Bel, T., van Ginneken, B., van der Laak, J., Hulsbergen-van de Kaa, C., & Litjens, G. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30739-9 · Cited by 705
Self-citationStröm, P., Kartasalo, K., Olsson, H., Solorzano, L., Delahunt, B., Berney, D. M., Bostwick, D. G., Evans, A. J., Grignon, D. J., Humphrey, P. A., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30738-7 · Cited by 620
Self-citationBandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Self-citationAn Update of the Gleason Grading System
Epstein, J. I. (2010) · Journal of Urology · doi:10.1016/j.juro.2009.10.046 · Cited by 533
A new era: artificial intelligence and machine learning in prostate cancer
Goldenberg, S. L., Nir, G., & Salcudean, S. E. (2019) · Nature Reviews Urology · doi:10.1038/s41585-019-0193-3 · Cited by 511
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495
Self-citationWhy rankings of biomedical image analysis competitions should be interpreted with care
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07619-7 · Cited by 380
Interobserver reproducibility of Gleason grading of prostatic carcinoma: Urologic pathologists
Allsbrook, W. C., Mangold, K. A., Johnson, M. H., Lane, R. B., Lane, C. G., Amin, M. B., Bostwick, D. G., Humphrey, P. A., Jones, E. C., Reuter, V. E., et al. (2001) · Human Pathology · doi:10.1053/hupa.2001.21134 · Cited by 332
Self-citationDeep learning and artificial intelligence in radiology: current applications and future directions
Yasaka, K., & Abe, O. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002707 · Cited by 238
Nagpal, K., Foote, D., Tan, F., Liu, Y., Chen, P.-H. C., Steiner, D. F., Manoj, N., Olson, N., Smith, J. L., Mohtashamian, A., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.2485 · Cited by 221
Self-citationInterobserver variability in Gleason histological grading of prostate cancer
Ozkan, T. A., Eruyar, A. T., Cebeci, O. O., Memik, O., Ozcan, L., & Kuskonmaz, I. (2016) · Scandinavian Journal of Urology · doi:10.1080/21681805.2016.1206619 · Cited by 195
Standardization of Gleason grading among 337 European pathologists
Egevad, L., Ahmad, A. S., Algaba, F., Berney, D. M., Boccon‐Gibod, L., Compérat, E., Evans, A. J., Griffiths, D., Grobholz, R., Kristiansen, G., et al. (2013) · Histopathology · doi:10.1111/his.12008 · Cited by 186
Self-citationBulten, W., Balkenhol, M., Belinga, J.-J. A., Brilhante, A., Çakır, A., Egevad, L., Eklund, M., Farré, X., Geronatsiou, K., Molinié, V., et al. (2021) · Modern Pathology · doi:10.1038/s41379-020-0640-y · Cited by 174
Self-citationSteiner, D. F., Nagpal, K., Sayres, R., Foote, D., Wedin, B., Pearce, A., Cai, C. J., Winter, S. R., Symonds, M., Yatziv, L., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.23267 · Cited by 145
Self-citationMelia, J., Moseley, R., Ball, R. Y., Griffiths, D. F. R., Grigor, K. M., Harnden, P., Jarmulowicz, M., McWilliam, L. J., Montironi, R., Waller, M., et al. (2006) · Histopathology · doi:10.1111/j.1365-2559.2006.02393.x · Cited by 133
Nir, G., Karimi, D., Goldenberg, S. L., Fazli, L., Skinnider, B. F., Tavassoli, P., Turbin, D., Villamil, C. F., Wang, G., Thompson, D. J. S., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0442 · Cited by 106
Kott, O., Linsley, D. A., Amin, A., Karagounis, A., Jeffers, C., Golijanin, D., Serre, T., & Gershman, B. (2019) · European Urology Focus · doi:10.1016/j.euf.2019.11.003 · Cited by 98
Value of Public Challenges for the Development of Pathology Deep Learning Algorithms
Hartman, D. J., van der Laak, J. A. W. M., Gürcan, M. N., & Pantanowitz, L. (2020) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_64_19 · Cited by 46
(2018) · Nature · doi:10.1038/d41586-018-03067-x · Cited by 44
Han, W., Johnson, C., Gaed, M., Gómez, J. A., Moussa, M., Chin, J. L., Pautler, S. E., Bauman, G. S., & Ward, A. D. (2020) · Scientific Reports · doi:10.1038/s41598-020-66849-2 · Cited by 42
No pixel-level annotations needed
van der Laak, J. A. W. M., Ciompi, F., & Litjens, G. J. S. (2019) · Nature Biomedical Engineering · doi:10.1038/s41551-019-0472-6 · Cited by 21
Self-citation
Cited by
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
A pathology foundation model for cancer diagnosis and prognosis prediction
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07894-z · Cited by 580
A multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461
Metadata
- Authors
- Bulten, W., Kartasalo, K., Chen, P.-H. C., Ström, P., Pinckaers, H., Nagpal, K., Cai, Y., Steiner, D. F., van Boven, H., Vink, R., Hulsbergen-van de Kaa, C., van der Laak, J., Amin, M. B., Evans, A. J., van der Kwast, T., Allan, R., Humphrey, P. A., Grönberg, H., Samaratunga, H., Delahunt, B., Tsuzuki, T., Häkkinen, T., Egevad, L., Demkin, M., Dane, S., Tan, F., Valkonen, M., Corrado, G. S., Peng, L., Mermel, C. H., Ruusuvuori, P., Litjens, G., Eklund, M., the PANDA challenge consortium, Brilhante, A., Çakır, A., Farré, X., Geronatsiou, K., Molinié, V., Pereira, G., Roy, P., Saile, G., Salles, P. G. O., Schaafsma, E., Tschui, J., Billoch-Lima, J., Pereira, E. M., Zhou, M., He, S., Song, S., Sun, Q., Yoshihara, H., Yamaguchi, T., Ono, K., Shen, T., Ji, J., Roussel, A., Zhou, K., Chai, T., Weng, N., Grechka, D., Shugaev, M. V., Kiminya, R., Kovalev, V., Voynov, D., Malyshev, V., Lapo, E., Campos, M., Ota, N., Yamaoka, S., Fujimoto, Y., Yoshioka, K., Juvonen, J., Tukiainen, M., Karlsson, A., Guo, R., Hsieh, C.-L., Zubarev, I., Bukhar, H. S. T., Li, W., Li, J., Speier, W., Arnold, C., Kim, K., Bae, B., Kim, Y. W., Lee, H.-S., & Park, J.
- Venue
- Nature Medicine
- Published
- 2022-01
- Volume
- 28
- Issue
- 1
- Pages
- 154-163
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4205900565
- Cited by
- 543
- References
- 36
- FWCI
- 33.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.