A Novel Deep Learning Based Computer-Aided Diagnosis System Improves the Accuracy and Efficiency of Radiologists in Reading Biparametric Magnetic Resonance Images of the Prostate: Results of a Multireader, Multicase Study
Abstract
OBJECTIVE: The aim of this study was to evaluate the effect of a deep learning based computer-aided diagnosis (DL-CAD) system on radiologists' interpretation accuracy and efficiency in reading biparametric prostate magnetic resonance imaging scans. MATERIALS AND METHODS: We selected 100 consecutive prostate magnetic resonance imaging cases from a publicly available data set (PROSTATEx Challenge) with and without histopathologically confirmed prostate cancer. Seven board-certified radiologists were tasked to read each case twice in 2 reading blocks (with and without the assistance of a DL-CAD), with a separation between the 2 reading sessions of at least 2 weeks. Reading tasks were to localize and classify lesions according to Prostate Imaging Reporting and Data System (PI-RADS) v2.0 and to assign a radiologist's level of suspicion score (scale from 1-5 in 0.5 increments; 1, benign; 5, malignant). Ground truth was established by consensus readings of 3 experienced radiologists. The detection performance (receiver operating characteristic curves), variability (Fleiss κ), and average reading time without DL-CAD assistance were evaluated. RESULTS: The average accuracy of radiologists in terms of area under the curve in detecting clinically significant cases (PI-RADS ≥4) was 0.84 (95% confidence interval [CI], 0.79-0.89), whereas the same using DL-CAD was 0.88 (95% CI, 0.83-0.94) with an improvement of 4.4% (95% CI, 1.1%-7.7%; P = 0.010). Interreader concordance (in terms of Fleiss κ) increased from 0.22 to 0.36 (P = 0.003). Accuracy of radiologists in detecting cases with PI-RADS ≥3 was improved by 2.9% (P = 0.10). The median reading time in the unaided/aided scenario was reduced by 21% from 103 to 81 seconds (P < 0.001). CONCLUSIONS: Using a DL-CAD system increased the diagnostic accuracy in detecting highly suspicious prostate lesions and reduced both the interreader variability and the reading time.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A Siemens commercial prototype increased diagnostic accuracy for clinically significant prostate cancer in a retrospective reader study with seven radiologists.
“a commercial prototype from Siemens was shown to increase the diagnostic accuracy of clinically significant prostate cancer in a retrospective reader study involving seven radiologists (37).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [37]
Cites
Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R. L., Torre, L. A., & Jemal, A. (2018) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21492 · Cited by 84,109 · see citing works
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · see citing works
Ahmed, H. U., Bosaily, A. E., Brown, L. C., Gabe, R., Kaplan, R. S., Parmar, M., Collaço‐Moraes, Y., Ward, K., Hindley, R. G., Freeman, A., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(16)32401-1 · Cited by 3,292 · see citing works
MRI-Targeted or Standard Biopsy for Prostate-Cancer Diagnosis
Kasivisvanathan, V., Rannikko, A. S., Borghi, M., Panebianco, V., Mynderse, L. A., Vaarala, M. H., Briganti, A., Budäus, L., Hellawell, G. O., Hindley, R. G., et al. (2018) · New England Journal of Medicine · doi:10.1056/nejmoa1801993 · Cited by 3,102 · see citing works
PI-RADS Prostate Imaging – Reporting and Data System: 2015, Version 2
Weinreb, J. C., Barentsz, J. O., Choyke, P. L., Cornud, F., Haider, M. A., Macura, K. J., Margolis, D., Schnall, M. D., Shtern, F., Tempany, C. M., et al. (2015) · European Urology · doi:10.1016/j.eururo.2015.08.052 · Cited by 2,955 · see citing works
Heidenreich, A., Joaquim Bellmunt, B., Bolla, M., Joniau, S., Mason, M. D., Matveev, V. B., Mottet, N., Schmid, H., van der Kwast, T., Wiegel, T., et al. (2010) · European Urology · doi:10.1016/j.eururo.2010.10.039 · Cited by 1,494 · see citing works
前立腺癌のEAUガイドライン。パート1:臨床的限局性疾患のスクリーニング,診断および治療
Heidenreich, A., Bellmunt, J., Bolla, M., et al. (2011) · European Urology · Cited by 866 · see citing works
Rosenkrantz, A. B., Ginocchio, L. A., Cornfeld, D., Froemming, A. T., Gupta, R. T., Türkbey, B., Westphalen, A. C., Babb, J. S., & Margolis, D. (2016) · Radiology · doi:10.1148/radiol.2016152542 · Cited by 501 · see citing works
Computer-Aided Detection of Prostate Cancer in MRI
Litjens, G. J. S., Debats, O. A., Barentsz, J. O., Karssemeijer, N., & Huisman, H. J. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2303821 · Cited by 453 · see citing works
Self-citationWestphalen, A. C., McCulloch, C. E., Anaokar, J. M., Arora, S., Barashi, N. S., Barentsz, J. O., Bathala, T. K., Bittencourt, L. K., Booker, M. T., Braxton, V. G., et al. (2020) · Radiology · doi:10.1148/radiol.2020190646 · Cited by 426 · see citing works
Muller, B. G., Shih, J. H., Sankineni, S., Marko, J., Rais‐Bahrami, S., George, A. K., de la Rosette, J. J. M. C. H., Merino, M. J., Wood, B. J., Pinto, P. A., et al. (2015) · Radiology · doi:10.1148/radiol.2015142818 · Cited by 331 · see citing works
Classification of Cancer at Prostate MRI: Deep Learning versus Clinical PI-RADS Assessment
Schelb, P., Köhl, S., Radtke, J. P., Wiesenfarth, M., Kickingereder, P., Bickelhaupt, S., Kuder, T. A., Stenzinger, A., Hohenfellner, M., Schlemmer, H.-P., et al. (2019) · Radiology · doi:10.1148/radiol.2019190938 · Cited by 326 · see citing works
de Rooij, M., Israël, B., Tummers, M., Ahmed, H. U., Barrett, T., Giganti, F., Hamm, B., Løgager, V., Padhani, A. R., Panebianco, V., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06929-z · Cited by 326 · see citing works
Deep learning and artificial intelligence in radiology: current applications and future directions
Yasaka, K., & Abe, O. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002707 · Cited by 238 · see citing works
Automatic Liver Segmentation Using an Adversarial Image-to-Image Network
Yang, D., Xu, D., Zhou, S. K., Georgescu, B., Chen, M., Grbić, S., Metaxas, D., & Comaniciu, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_58 · Cited by 233 · see citing works
Self-citationJoint Prostate Cancer Detection and Gleason Score Prediction in mp-MRI via FocalNet
Cao, R., Bajgiran, A. M., Mirak, S. A., Shakeri, S., Zhong, X., Enzmann, D. R., Raman, S. S., & Sung, K. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901928 · Cited by 219 · see citing works
Song, Y. S., Zhang, Y., Yan, X., Liu, H., Zhou, M., Hu, B., & Yang, G. (2018) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26047 · Cited by 187 · see citing works
Deep Learning Regression for Prostate Cancer Detection and Grading in Bi-Parametric MRI
de Vente, C., Vos, P., Hosseinzadeh, M., Pluim, J. P. W., & Veta, M. (2020) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.2993528 · Cited by 177 · see citing works
Woo, S., Suh, C. H., Kim, S. Y., Cho, J. Y., Kim, S. H., & Moon, M. H. (2018) · American Journal of Roentgenology · doi:10.2214/ajr.18.19880 · Cited by 175 · see citing works
Assessment of PI-RADS v2 for the Detection of Prostate Cancer
Kasel-Seibert, M., Lehmann, T., Aschenbach, R., et al. (2016) · European Journal of Radiology · doi:10.1016/j.ejrad.2016.01.011 · Cited by 169 · see citing works
van der Leest, M., Israël, B., Cornel, E. B., et al. (2019) · European Urology · doi:10.1016/j.eururo.2019.05.029 · Cited by 162 · see citing works
Mehralivand, S., Bednarova, S., Shih, J. H., et al. (2017) · The Journal of Urology · doi:10.1016/j.juro.2017.03.131 · Cited by 151 · see citing works
Hambrock, T., Vos, P., de Kaa, C. A. H., Barentsz, J. O., & Huisman, H. J. (2012) · Radiology · doi:10.1148/radiol.12111634 · Cited by 123 · see citing works
Self-citationSanford, T., Harmon, S. A., Turkbey, E. B., Kesani, D., Azamat, S., Madariaga, M., Yang, C. X., Sackett, J., Mehralivand, S., Yan, P., et al. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27204 · Cited by 90 · see citing works
Winkel, D. J., Heye, T. J., Weikert, T., Boll, D. T., & Stieltjes, B. (2018) · Investigative Radiology · doi:10.1097/rli.0000000000000509 · Cited by 83 · see citing works
Self-citationEvaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA
Gallas, B. D., Chan, H.-P., D’Orsi, C. J., Dodd, L. E., Giger, M. L., Gur, D., Krupinski, E. A., Metz, C. E., Myers, K. J., Obuchowski, N. A., et al. (2012) · Academic Radiology · doi:10.1016/j.acra.2011.12.016 · Cited by 81 · see citing works
Litjens, G. J. S., Barentsz, J. O., Karssemeijer, N., & Huisman, H. J. (2015) · European Radiology · doi:10.1007/s00330-015-3743-y · Cited by 74 · see citing works
Self-citationNiaf, É., Lartizien, C., Bratan, F., et al. (2014) · Radiology · doi:10.1148/radiol.14130448 · Cited by 72 · see citing works
Vos, E. K., Kobus, T., Litjens, G. J. S., et al. (2015) · Investigative Radiology · doi:10.1097/rli.0000000000000157 · Cited by 69 · see citing works
Giannini, V., Mazzetti, S., Armando, E., Carabalona, S., Russo, F., Giacobbe, A., Muto, G., & Regge, D. (2017) · European Radiology · doi:10.1007/s00330-017-4805-0 · Cited by 69 · see citing works
Weiß, J., Martirosian, P., Notohamiprodjo, M., et al. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000427 · Cited by 53 · see citing works
Winkel, D. J., Wetterauer, C., Matthias, M., Lou, B., Shi, B., Kamen, A., Comaniciu, D., Seifert, H., Rentsch, C. A., & Boll, D. T. (2020) · Diagnostics · doi:10.3390/diagnostics10110951 · Cited by 49 · see citing works
Self-citationA Novel Projection Based Approach for Medical Image Registration
Khamene, A., Chisu, R., Wein, W., Navab, N., & Sauer, F. (2006) · Workshop on Biomedical Image Registration · doi:10.1007/11784012_30 · Cited by 37 · see citing works
Yu, X. F., Lou, B., Shi, B., et al. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098338 · Cited by 32 · see citing works
Self-citationDetecting Prostate Cancer with Deep Learning for MRI: A Small Step Forward
Padhani, A. R., & Türkbey, B. (2019) · Radiology · doi:10.1148/radiol.2019192012 · Cited by 28 · see citing works
Zhu, L., Gao, G., Liu, Y., Han, C., Liu, J., Zhang, X., & Wang, X. (2019) · Clinical Imaging · doi:10.1016/j.clinimag.2019.12.010 · Cited by 26 · see citing works
Winkel, D. J., Breit, H., Block, K. T., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06849-y · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196 · see citing works
Interactive explainable deep learning model informs prostate cancer diagnosis at MRI
Hamm, C. A., Baumgärtner, G. L., Biessmann, F., Beetz, N. L., Hartenstein, A., Savic, L. J., Froböse, K., Dräger, F., Schallenberg, S., Rudolph, M., et al. (2023) · Radiology · doi:10.1148/radiol.222276 · Cited by 133 · see citing works
Sunoqrot, M. R. S., Saha, A., Hosseinzadeh, M., Elschot, M., & Huisman, H. (2022) · European Radiology Experimental · doi:10.1186/s41747-022-00288-8 · Cited by 76 · see citing works
Self-citation
- Authors
- Winkel, D. J., Tong, A., Lou, B., Kamen, A., Comaniciu, D., Disselhorst, J. A., Rodríguez-Ruiz, A., Huisman, H., Szolar, D., Shabunin, I., Choi, M. H., Xing, P., Penzkofer, T., Grimm, R., von Busch, H., & Boll, D. T.
- Affiliations
- University Hospital of Basel (CH)
- NYU Langone Health (US)
- Siemens (United States) (US)
- Siemens Healthcare AG Advanced Clinical Imaging Technology, Lausanne, Vaud, Switzerland
- ScreenPoint Medical
- Radboud University Medical Center (NL)
- Diagnostikum Graz, Graz, Austria
- Patero Clinic, Moscow, Russia
- The Catholic University of Korea Eunpyeong St. Mary's Hospital (KR)
- Changhai Hospital (CN)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Siemens Healthineers (Germany) (DE)
- Siemens Healthineers Diagnostic Imaging, Erlangen, Germany.
- Corresponding author
- Winkel, David J.
- Shabunin, Ivan
- Boll, Daniel T.
- ORCID
- Venue
- Investigative Radiology
- Published
- 2021-04-02
- Volume
- 56
- Issue
- 10
- Pages
- 605-613
- ISSN
- 0020-9996 (ISSN-L)
- 1536-0210
- OpenAlex
- W3148634016
- Cited by
- 118
- References
- 37
- Citations by year
- 2026: 14 · 2025: 34 · 2024: 22 · 2023: 24 · 2022: 15 · 2021: 9
- FWCI
- 6.15
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Investigative Radiology (publishedVersion)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Funders
- Universitätsspital Basel
- Schweizerische Gesellschaft für Radiologie
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Prostate Cancer Diagnosis and Treatment
- Keywords
- computer-aided diagnosis, prostate cancer detection, inter-reader variability, reading time reduction, deep learning, clinically significant prostate cancer
- MeSH terms
- Radiologists, Deep Learning, Computers, Humans, Magnetic Resonance Imaging, Male, Prostatic Neoplasms, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
