A Deep Learning Model to Triage Screening Mammograms: A Simulation Study
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Background Recent deep learning (DL) approaches have shown promise in improving sensitivity but have not addressed limitations in radiologist specificity or efficiency. Purpose To develop a DL model to triage a portion of mammograms as cancer free, improving performance and workflow efficiency. Materials and Methods In this retrospective study, 223 109 consecutive screening mammograms performed in 66 661 women from January 2009 to December 2016 were collected with cancer outcomes obtained through linkage to a regional tumor registry. This cohort was split by patient into 212 272, 25 999, and 26 540 mammograms from 56 831, 7021, and 7176 patients for training, validation, and testing, respectively. A DL model was developed to triage mammograms as cancer free and evaluated on the test set. A DL-triage workflow was simulated in which radiologists skipped mammograms triaged as cancer free (interpreting them as negative for cancer) and read mammograms not triaged as cancer free by using the original interpreting radiologists’ assessments. Sensitivities, specificities, and percentage of mammograms read were calculated, with and without the DL-triage–simulated workflow. Statistics were computed across 5000 bootstrap samples to assess confidence intervals (CIs). Specificities were compared by using a two-tailed t test (P < .05) and sensitivities were compared by using a one-sided t test with a noninferiority margin of 5% (P < .05). Results The test set included 7176 women (mean age, 57.8 years ± 10.9 [standard deviation]). When reading all mammograms, radiologists obtained a sensitivity and specificity of 90.6% (173 of 191; 95% CI: 86.6%, 94.7%) and 93.5% (24 625 of 26 349; 95% CI: 93.3%, 93.9%). In the DL-simulated workflow, the radiologists obtained a sensitivity and specificity of 90.1% (172 of 191; 95% CI: 86.0%, 94.3%) and 94.2% (24 814 of 26 349; 95% CI: 94.0%, 94.6%) while reading 80.7% (21 420 of 26 540) of the mammograms. The simulated workflow improved specificity (P = .002) and obtained a noninferior sensitivity with a margin of 5% (P < .001). Conclusion This deep learning model has the potential to reduce radiologist workload and significantly improve specificity without harming sensitivity. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Kontos and Conant in this issue.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [106]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In brief, we implemented our model as a deep convolutional neural network (ResNet18 [22]) with PyTorch (version 0.31; https://pytorch.org ).”
MethodInfluentialREDUCTION IN MORTALITY FROM BREAST CANCER AFTER MASS SCREENING WITH MAMMOGRAPHY
Tabár, L., Gad, A., Holmberg, L. H., Ljungquist, U., Fagerberg, C. J. G., Baldetorp, L., Gröntoft, O., Lundström, B., Månson, J. C., Eklund, G., et al. (1985) · The Lancet · doi:10.1016/s0140-6736(85)92204-4 · Cited by 1,781
The benefits and harms of breast cancer screening: an independent review
(2012) · The Lancet · doi:10.1016/s0140-6736(12)61611-0 · Cited by 1,492
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Self-citationLehman, C. D., Arao, R. F., Sprague, B. L., Lee, J. M., Buist, D. S. M., Kerlikowske, K., Henderson, L. M., Onega, T., Tosteson, A. N. A., Rauscher, G. H., et al. (2017) · Radiology · doi:10.1148/radiol.2016161174 · Cited by 665
Self-citationInfluence of Computer-Aided Detection on Performance of Screening Mammography
Fenton, J. J., Taplin, S. H., Carney, P. A., Abraham, L., Sickles, E. A., D'Orsi, C., Berns, E. A., Cutter, G., Hendrick, R. E., Barlow, W. E., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa066099 · Cited by 619
Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System
Rodríguez-Ruiz, A., Krupinski, E., Mordang, J.-J., Schilling, K., Heywang-Köbrunner, S. H., Sechopoulos, I., & Mann, R. M. (2019) · Radiology · doi:10.1148/radiol.2018181371 · Cited by 602 · cited 3×
“Although traditional approaches have not demonstrated improved radiologist performance in sensitivity or specificity in clinical practice (14–16), a more recent deep learning approach to CAD has shown promise in improving sensitivity in a reader study (17).”
BackgroundMethodField, C. A., & Welsh, A. H. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00593.x · Cited by 426 · cited 1×
“To account for patients appearing in our test set multiple times, we used the cluster bootstrap procedure (23).”
MethodMammographic Breast Density Assessment Using Deep Learning: Clinical Implementation
Lehman, C. D., Yala, A., Schuster, T., Dontchos, B., Bahl, M., Swanson, K., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2018180694 · Cited by 294 · cited 1×
“Mammograms from 60 886 of the 80 818 women in our patient population were previously reported (20,21).”
Self-citationRodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Teuwen, J., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Mertelmeier, T., et al. (2019) · European Radiology · doi:10.1007/s00330-019-06186-9 · Cited by 210 · cited 2×
“Recall rates from screening mammography have increased steadily concurrently explored in a recent reader study by Rodriguez-Ruiz et al (24), which used a commercial system and showed a simulated workload reduction of 17% with a drop of 1% in sensitivity and a noninferior radiologist AUC of 0.05 on an enriched data set.”
MethodInfluential
Research works printed but not linked: 3
- Shapiro S, 1982, J Natl Cancer Inst, 69, 349
- Miller AB, 1981, Clin Invest Med, 4, 227
- Yala A, 2019, Radiology, 182716
Cited by
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“Workflow improvements include prioritizing worklists for radiologists (2,3), triaging screening mammograms (4), reducing or eliminating gadolinium-based contrast media for MRI (5,6), and reducing the radiation dose of CT imaging by advancing image noise reduction (7–9).”
BackgroundDiagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Swanson, K., Wu, E., Zhang, A., Alizadeh, A. A., & Zou, J. (2023) · Cell · doi:10.1016/j.cell.2023.01.035 · Cited by 597
“, a ResNet-18 model was built to triage mammograms by setting a high-sensitivity prediction threshold so that nearly all predicted negative caseswere truly negative.(36) In a simulation study, these predicted negative cases were skipped by radiologists, allowing radiologists to only read 80.”
MethodLotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423
Sechopoulos, I., Teuwen, J., & Mann, R. (2021) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.06.002 · Cited by 341
“The same pre-selection strategy was subsequently investigated by Yala et al., using another deep learning-based system specifically developed to classify screening DM cases as cancer free , and therefore to not be human-interpreted, or not [82].”
MethodDembrower, K., Wåhlin, E., Liu, Y., Salim, M., Smith, K., Lindholm, P., Eklund, M., & Strand, F. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30185-0 · Cited by 290
Toward robust mammography-based models for breast cancer risk
Yala, A., Mikhael, P. G., Strand, F., Lin, G., Smith, K., Wan, Y.-L., Lamb, L., Hughes, K., Lehman, C., & Barzilay, R. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.aba4373 · Cited by 287
Self-citationContinuous Learning AI in Radiology: Implementation Principles and Early Applications
Pianykh, O. S., Langs, G., Dewey, M., Enzmann, D. R., Herold, C. J., Schoenberg, S. O., & Brink, J. A. (2020) · Radiology · doi:10.1148/radiol.2020200038 · Cited by 237
Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis
Leibig, C., Brehmer, M., Bunk, S., Byng, D., Pinker, K., & Umutlu, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00070-x · Cited by 202
“Exact values are provided in the appendix (pp 9, 12, 13).”
Machine learning-based models for the prediction of breast cancer recurrence risk
Zuo, D., Yang, L., Jin, Y. Q., Qi, H., Liu, Y., & Li, R. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02377-z · Cited by 195
“Yala et al. [27]proposed a DL model was built to tri-age mammograms by setting a high-sensitivity prediction threshold so that nearly all predicted negative cases were truly negative.”
MethodRaya-Povedano, J. L., Romero-Martín, S., Elías-Cabot, E., Gubern-Mérida, A., Rodríguez-Ruiz, A., & Álvarez-Benito, M. (2021) · Radiology · doi:10.1148/radiol.2021203555 · Cited by 177
CAD and AI for breast cancer—recent development and challenges
Chan, H.-P., Samala, R. K., & Hadjiiski, L. M. (2019) · The British Journal of Radiology · doi:10.1259/bjr.20190580 · Cited by 173
Novel Approaches to Screening for Breast Cancer
Mann, R. M., Hooley, R., Barr, R. G., & Moy, L. (2020) · Radiology · doi:10.1148/radiol.2020200172 · Cited by 167
Lauritzen, A. D., Rodríguez-Ruiz, A., von Euler-Chelpin, M. C., Lynge, E., Vejborg, I., Nielsen, M., Karssemeijer, N., & Lillholm, M. (2022) · Radiology · doi:10.1148/radiol.210948 · Cited by 166
“The inferiority margin, d , was set to 0.05 in accordance with clinicians and recent literature (24,25,32).”
BackgroundMethodInfluentialApplication of Deep Learning in Breast Cancer Imaging
Balkenende, L., Teuwen, J., & Mann, R. M. (2022) · Seminars in Nuclear Medicine · doi:10.1053/j.semnuclmed.2022.02.003 · Cited by 160
Hickman, S. E., Woitek, R., Le, E., Im, Y. R., Luxhøj, C. M., Avilés-Rivero, A. I., Baxter, G. C., Mackay, J., & Gilbert, F. J. (2021) · Radiology · doi:10.1148/radiol.2021210391 · Cited by 139
Kim, H., Goo, J. M., Lee, K. H., Kim, Y. T., & Park, C. M. (2020) · Radiology · doi:10.1148/radiol.2020192764 · Cited by 130
Identifying normal mammograms in a large screening population using artificial intelligence
Lång, K., Dustler, M., Dahlblom, V., Åkesson, A., Andersson, I., & Zackrisson, S. (2020) · European Radiology · doi:10.1007/s00330-020-07165-1 · Cited by 137
“The magnitude of normal exams identified in this study was similar to the results presented by Rodriguez-Ruiz et al using the same AI system, but on a study population with both clinical and screening mammography exams [25], and by Yala et al using a different AI system than the one used in this study, on a large scre…”
ResultChen, M., Wang, Y., Wang, Q., Shi, J., Wang, H., Ye, Z., Xue, P., & Qiao, Y. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01328-w · Cited by 144
A review of explainable and interpretable AI with applications in COVID‐19 imaging
Fuhrman, J. D., Gorre, N., Hu, Q., Li, H., El Naqa, I., & Giger, M. L. (2022) · Medical Physics · doi:10.1002/mp.15359 · Cited by 120
Can artificial intelligence reduce the interval cancer rate in mammography screening?
Lång, K., Hofvind, S., Rodríguez-Ruiz, A., & Andersson, I. (2021) · European Radiology · doi:10.1007/s00330-021-07686-3 · Cited by 109
van Winkel, S. L., Rodríguez‐Ruiz, A., Appelman, L., Gubern‐Mérida, A., Karssemeijer, N., Teuwen, J., Wanders, A. J. T., Sechopoulos, I., & Mann, R. M. (2021) · European Radiology · doi:10.1007/s00330-021-07992-w · Cited by 118
“This may result in tools that sustain current mammography-based breast cancer screening programs with less human interaction or even improve the overall quality of screening [14, 15].”
BackgroundMethodRomero-Martín, S., Elías-Cabot, E., Raya-Povedano, J. L., Gubern‐Mérida, A., Rodríguez‐Ruiz, A., & Benito, M. Á. (2021) · Radiology · doi:10.1148/radiol.211590 · Cited by 107
Deep Learning-Based Artificial Intelligence for Mammography
Yoon, J. H., & Kim, E. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2020.1210 · Cited by 110
“In the other two studies, DL algorithms were constructed to triage mammography examinations as cancerfree using either imaging features [44] or imaging features combined with non-imaging features and pathologic outcomes [42].”
MethodSystematic reviews of machine learning in healthcare: a literature review
Kolasa, K., Admassu, B. Y., Hołownia-Voloskova, M. E., Kędzior, K. J., Poirrier, J., & Perni, S. (2023) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2023.2279107 · Cited by 117
Liu, H., Chen, Y., Zhang, Y., Wang, L., Luo, R., Wu, H., Wu, C., Zhang, H., Tan, W., Yin, H., et al. (2021) · European Radiology · doi:10.1007/s00330-020-07659-y · Cited by 88
Lamb, L. R., Lehman, C. D., Gastounioti, A., Conant, E. F., & Bahl, M. (2022) · American Journal of Roentgenology · doi:10.2214/ajr.21.27071 · Cited by 96
Self-citationGoncharov, M., Pisov, M., Shevtsov, A. E., Shirokikh, B., Kurmukov, A. G., Blokhin, I. A., Chernina, V., Solovev, A. V., Gombolevskiy, V. А., Morozov, S. P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102054 · Cited by 98
Maghsoudi, O. H., Gastounioti, A., Scott, C. G., Pantalone, L., Wu, F.-F., Cohen, E. A., Winham, S. J., Conant, E. F., Vachon, C. M., & Kontos, D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102138 · Cited by 96
“AI, including deep learning, has shown great potential in substantially improving medical image segmentation, classification, risk assessment and breast cancer detection [4, 17, 21, 23, 24, 26, 31, 32, 33, 39, 41, 48, 49].”
Computational Radiology in Breast Cancer Screening and Diagnosis Using Artificial Intelligence
Tran, W. T., Naini, A. S., Lu, F., Gandhi, S., Meti, N., Brackstone, M., Rakovitch, E., & Curpen, B. N. (2020) · Canadian Association of Radiologists Journal · doi:10.1177/0846537120949974 · Cited by 89
Shoshan, Y., Bakalo, R., Gilboa-Solomon, F., Ratner, V., Barkan, E., Ozery-Flato, M., Amit, M., Khapun, D., Ambinder, E. B., Oluyemi, E. T., et al. (2022) · Radiology · doi:10.1148/radiol.211105 · Cited by 76
Advancements in Oncology with Artificial Intelligence—A Review Article
Vobugari, N., Raja, V., Sethi, U., Gandhi, K. D., Raja, K., & Surani, S. (2022) · Cancers · doi:10.3390/cancers14051349 · Cited by 85
“0 screenpoint medical BV, Nijmegen, the Netherlands, expedites interpretation and reduces workload by 20–50% by excluding mammograms with a low likelihood of cancer, allowing radiologists to concentrate on challenging cases [2,51].”
BackgroundStand-alone artificial intelligence - The future of breast cancer screening?
Sechopoulos, I., & Mann, R. M. (2020) · The Breast · doi:10.1016/j.breast.2019.12.014 · Cited by 79
Workflow Applications of Artificial Intelligence in Radiology and an Overview of Available Tools
Kapoor, N., Lacson, R. C., & Khorasani, R. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.08.016 · Cited by 78
AI-enhanced breast imaging: Where are we and where are we heading?
Bitencourt, A. G. V., Naranjo, I. D., Gullo, R. L., Saccarelli, C. R., & Pinker, K. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109882 · Cited by 76
Artificial Intelligence: A Primer for Breast Imaging Radiologists
Bahl, M. (2020) · Journal of Breast Imaging · doi:10.1093/jbi/wbaa033 · Cited by 74
“In one recent study, investigators developed a DL model to identify mammograms as cancer-free with high confidence, in order to improve workflow efficiency and improve performance (26).”
Frazer, H. M. L., Peña‐Solórzano, C. A., Kwok, C. F., Elliott, M. S., Chen, Y., Wang, C., The BRAIx Team, Al-Qershi, O. M., Fox, S., Hill, B., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51725-8 · Cited by 68
“Taken together, our work provides actionable information relevant to current directions for AI implementation 21 , improves on previous simulation efforts 15 – 17,”
Tong, W.-J., Wu, S.-H., Cheng, M.-Q., Huang, H., Liang, J.-Y., Li, C.-Q., Guo, H.-L., He, D.-N., Liu, Y.-H., Xiao, H., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.13674 · Cited by 52
Yirgin, İ. K., Köylüoğlu, Y. O., Şeker, M. E., Gürdal, S. Ö., Özaydın, A., Özçınar, B., Cabıoğlu, N., Özmen, V., & Arıbal, E. (2022) · Technology in Cancer Research & Treatment · doi:10.1177/15330338221075172 · Cited by 75
“Tumor size (mm) (median, range) 15 (3-37)”
BackgroundDeep learning in breast radiology: current progress and future directions
Ou, W. C., Polat, D. S., & Doğan, B. E. (2021) · European Radiology · doi:10.1007/s00330-020-07640-9 · Cited by 62
“In a retrospective simulation of this scenario, allowing a DL model to identify cancer-free mammograms would have excluded 19% of mammograms from review, with non-inferior sensitivity and superior specificity compared to the original radiologists’ assessments [86].”
ResultLiu, J., Lei, J., Ou, Y., Zhao, Y., Tuo, X., Zhang, B., & Shen, M. (2022) · Clinical and Experimental Medicine · doi:10.1007/s10238-022-00895-0 · Cited by 60
“Therefore, a large number of researches have gradually focused on the automatic reading of mammograms [17–19].”
BackgroundDeep learning performance for detection and classification of microcalcifications on mammography
Pesapane, F., Trentin, C., Ferrari, F., Signorelli, G., Tantrige, P. M., Montesano, M., Cicala, C., Virgoli, R., D'Acquisto, S., Nicosia, L., et al. (2023) · European Radiology Experimental · doi:10.1186/s41747-023-00384-3 · Cited by 57
“Particularly, AI systems may have clinical value for early detection and treatment of breast cancer—by filtering out cancer-free mammograms, resulting in lower recall rates and reducing the number of unnecessary biopsies [15–18].”
BackgroundApplications of Artificial Intelligence in Breast Imaging
Morgan, M. B., & Mates, J. (2020) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2020.08.007 · Cited by 59
Uwimana, A., Gnecco, G., & Riccaboni, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109391 · Cited by 44
Patchless Multi-Stage Transfer Learning for Improved Mammographic Breast Mass Classification
Ayana, G., Park, J., & Choe, S. (2022) · Cancers · doi:10.3390/cancers14051280 · Cited by 51
Clinical Artificial Intelligence Applications
Hu, Q., & Giger, M. L. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.010 · Cited by 49
Artificial Intelligence Algorithm for Subclinical Breast Cancer Detection
Gjesvik, J., Moshina, N., Lee, C. I., Miglioretti, D. L., & Hofvind, S. S. H. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.37402 · Cited by 46
Jing, X., Wielema, M., Cornelissen, L. J., van Gent, M., Iwema, W. M., Zheng, S., Sijens, P. E., Oudkerk, M., Dorrius, M. D., & van Ooijen, P. M. A. (2022) · European Radiology · doi:10.1007/s00330-022-08863-8 · Cited by 47
“Rodriguez-Ruiz et al [28] and Yala et al [9] showed that AI could help reduce mammogram screening workload by 17% or 19.3% with a sensitivity of 90.6% or 90.1%, respectively.”
BackgroundInfluentialDong, F., She, R., Cui, C., Shi, S., Hu, X., Zeng, J., Wu, H., Xu, J., & Zhang, Y. (2021) · European Radiology · doi:10.1007/s00330-020-07561-7 · Cited by 43
Elías-Cabot, E., Romero-Martín, S., Raya-Povedano, J. L., Brehl, A.-K., & Benito, M. Á. (2023) · European Radiology · doi:10.1007/s00330-023-10426-4 · Cited by 46
Inteligencia artificial y aprendizaje automático en medicina
Vega, M. Á., Mora, L. M. Q., & Badilla, M. V. C. (2020) · Revista Medica Sinergia · doi:10.31434/rms.v5i8.557 · Cited by 68
A Review of Applications of Machine Learning in Mammography and Future Challenges
Batchu, S., Liu, F., Amireh, A., Waller, J., & Umair, M. (2021) · Oncology · doi:10.1159/000515698 · Cited by 41
“The radiologists in the study showed a significant improvement in specificity (93.5–94.2%; p = 0.002) and a non-inferior sensitivity (90.6 vs. 90.1%; p < 0.001) at a margin of 0.05 [20].”
BackgroundInfluentialDiagnostic test accuracy of AI-assisted mammography for breast imaging: a narrative review
Dave, D., Akhunzada, A., Ivković, N., Gyawali, S., Cengiz, K., Ahmed, A. R., & Al‐Shamayleh, A. S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2476 · Cited by 42
“AI tools can expedite image analysis, identify subtle patterns, and assist in diagnosis, allowing radiologists to focus on complex cases and patient care ( Hickman et al., 2022 ).”
Mammography Breast Cancer Screening Triage Using Deep Learning: A UK Retrospective Study
Hickman, S. E., Payne, N. R., Black, R. T., Huang, Y., Priest, A. N., Hudson, S. M., Kasmai, B., Juette, A., Nanaa, M., Aniq, M. I., et al. (2023) · Radiology · doi:10.1148/radiol.231173 · Cited by 42
Koch, H. W., Larsen, M., Bartsch, H., Kurz, K. D., & Hofvind, S. S. H. (2023) · European Radiology · doi:10.1007/s00330-023-09461-y · Cited by 40
Şeker, M. E., Köylüoğlu, Y. O., Özaydın, A., Gürdal, S. Ö., Özçınar, B., Cabıoğlu, N., Özmen, V., & Arıbal, E. (2024) · European Radiology · doi:10.1007/s00330-024-10661-3 · Cited by 39
Deep learning in breast imaging
Bhowmik, A., & Eskreis‐Winkler, S. (2022) · BJR|Open · doi:10.1259/bjro.20210060 · Cited by 46
Feng, S., Liu, Q., Patel, A., Bazai, S. U., Jin, C., Kim, J. S., Sarrafzadeh, M., Azzollini, D., Yeoh, J., Kim, E., et al. (2022) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13393 · Cited by 40
“3,13 Considering the promising potential of current medical AI systems, the implementation of AI algorithms in the current radiology work fl ow offers great potential to reduce the time taken for urgent fi ndings to be reported in the New Zealand healthcare system.”
BackgroundArtificial intelligence-based diagnosis of breast cancer by mammography microcalcification
Lin, Q., Tan, W., Ge, J., Huang, Y., Xiao, Q., Xu, Y., Jin, Y., Shao, Z., Gu, Y., Yan, B., et al. (2023) · Fundamental Research · doi:10.1016/j.fmre.2023.04.018 · Cited by 39
“Yala et al. [24] demonstrated that deep-learning models could classify even a fraction of mammography scans as cancer-free, thereby improving the performance and workflow efficiency.”
van Winkel, S. L., Peters, J., Janssen, N. N. Y., Kroes, J., Loehrer, E. A., Gommers, J. J. J., Sechopoulos, I., de Munck, L., Teuwen, J., Broeders, M. J. M., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100882 · Cited by 42
New Frontiers in Breast Cancer Imaging: The Rise of AI
Shamir, S. B., Sasson, A. L., Margolies, L. R., & Mendelson, D. S. (2024) · Bioengineering · doi:10.3390/bioengineering11050451 · Cited by 35
Beyond fine-tuning: Classifying high resolution mammograms using function-preserving transformations
Wei, T., Avilés-Rivero, A. I., Wang, S., Huang, Y., Gilbert, F. J., Schönlieb, C., & Chen, C. W. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102618 · Cited by 37
Wu, M., Wu, H., Wu, L. J., Cui, C., Shi, S., Xu, J., Liu, Y., & Dong, F. (2020) · Journal of Clinical Ultrasound · doi:10.1002/jcu.23143 · Cited by 32
“15,16 Furthermore, results provided by DL model can be fast, reproducible, and self-improving as data increases.”
Artificial intelligence in mammography: a systematic review of the external validation
Branco, P. E. S. C., Franco, A. H. S., de Oliveira, A. P., Carneiro, I. M. C., de Carvalho, L. S., de Souza, J. I. N., Leandro, D. R., & Cândido, E. B. (2024) · Revista Brasileira Ginecologia e Obstetrícia · doi:10.61622/rbgo/2024rbgo71 · Cited by 37
“(11) In general, AI algorithms used in breast cancer screening have come to be considered an instrument to help improve the efficiency and precision of radiologists.”
BackgroundOmoleye, O. J., Woodard, A. E., Howard, F. M., Zhao, F., Yoshimatsu, T. F., Zheng, Y., Pearson, A. T., Levental, M., Aribisala, B. S., Kulkarni, K., et al. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.220299 · Cited by 37
Thrall, J. H., Fessell, D. P., & Pandharipande, P. V. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.07.010 · Cited by 30
Classification of Multiple Diseases on Body CT Scans Using Weakly Supervised Deep Learning
Tushar, F. I., D’Anniballe, V. M., Hou, R., Mazurowski, M. A., Fu, W., Samei, E., Rubin, G. D., & Lo, J. Y.-C. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.210026 · Cited by 37
“Many of the existing studies in medical image artificial intelligence have focused on a single abnormality or organ system (1, 4-6, 9, 23-25).”
BackgroundFisches, Z. V., Ball, M., Mukama, T., Štih, V., Payne, N. R., Hickman, S. E., Gilbert, F. J., Bunk, S., & Leibig, C. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00173-0 · Cited by 29
van Nijnatten, T. J. A., Payne, N. R., Hickman, S. E., Ashrafian, H., & Gilbert, F. J. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.111087 · Cited by 29
A Semiautonomous Deep Learning System to Reduce False Positives in Screening Mammography
Pedemonte, S., Tsue, T. T., Mombourquette, B., Vu, Y. N. T., Matthews, T. P., Hoil, R. M., Shah, M., Ghare, N., Zingman-Daniels, N., Holley, S., et al. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.230033 · Cited by 28
Anomaly Detection of Calcifications in Mammography Based on 11,000 Negative Cases
Hou, R., Peng, Y., Grimm, L. J., Ren, Y., Mazurowski, M. A., Marks, J. R., King, L. M., Maley, C. C., Hwang, E. S., & Lo, J. Y. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3126281 · Cited by 32
A review of artificial intelligence in mammography
Jairam, M. P., & Ha, R. S. (2022) · Clinical Imaging · doi:10.1016/j.clinimag.2022.05.005 · Cited by 29
Multi-task Learning for Detection and Classification of Cancer in Screening Mammography
de Cea, M. V. S., Diedrich, K., Bakalo, R., Ness, L., & Richmond, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59725-2_24 · Cited by 27
Xavier, D. P., Miyawaki, I. A., Jorge, C. A. C., Silva, G. B. F., Lloyd, M. R., de Moraes, F. Y., Patel, B., & Batalini, F. (2023) · Journal of Medical Screening · doi:10.1177/09691413231219952 · Cited by 26
Siira, E., Johansson, H., & Nygren, J. (2025) · Journal of Medical Internet Research · doi:10.2196/53741 · Cited by 27
Ozcan, B. B., Doğan, B. E., Xi, Y., & Knippa, E. E. (2025) · Radiology Imaging Cancer · doi:10.1148/rycan.240290 · Cited by 27
Yoo, H., Kim, E. Y., Kim, H., Choi, Y. R., Kim, M. Y., Hwang, S. H., Kim, Y. J., Cho, Y. J., & Jin, K. N. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0189 · Cited by 27
“For this system to be successfully implemented, it is important to minimize the number of abnormal images that are incorrectly removed from the radiology worklist because of normal triaging [26,28].”
Computer‐Aided Detection AI Reduces Interreader Variability in Grading Hip Abnormalities With MRI
Tibrewala, R., Ozhinsky, E., Shah, R., Flament, I., Crossley, K. M., Srinivasan, R., Souza, R. B., Link, T. M., Pedoia, V., & Majumdar, S. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27164 · Cited by 25
“25,26 Additionally, the results of the all the networks could be combined in order to fi nd which patients have cartilage lesions, bone marrow edema-like lesions, and subchondral cyst-like lesions, classifying those as patients of higher priority.”
BackgroundQuality control system for mammographic breast positioning using deep learning
Watanabe, H., Hayashi, S., Kondo, Y., Matsuyama, E., Hayashi, N., Ogura, T., & Shimosegawa, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-34380-9 · Cited by 23
“In recent years, artificial intelligence related to computer vision has been used in the medical field 15,16 .”
MethodUse of artificial intelligence in breast surgery: a narrative review
Seth, I., Lim, B., Joseph, K., Gracias, D., Xie, Y., Ross, R. J., & Rozen, W. M. (2024) · Gland Surgery · doi:10.21037/gs-23-414 · Cited by 22
“AI models can be used to support radiologists in their work by preselecting suspicious lesions for subsequent assessment by radiologists (51,52).”
A 2021 update on cancer image analytics with deep learning
Kurian, N. C., Sethi, A., Konduru, A. R., Mahajan, A., & Rane, S. U. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1410 · Cited by 24
“…to screen images for abnormalities with high speed and accuracy (Ardila et al., 2019; Pasa, Golkov, Pfeiffer, Cremers, & Pfeiffer, 2019; Shen, 2017; Yala, Schuster, Miles, Barzilay, & Lehman, 2019) provides the ability to increase radiologist's productivity and allows to focus expert attention…”
BackgroundRetson, T., & Eghtedari, M. (2023) · Diagnostics · doi:10.3390/diagnostics13132133 · Cited by 24
“AI is also gaining increasing traction for a role as a digital second reader [10,11], with some studies proposing the use of AI to sort confidently negative studies for worklist reduction [12,13].”
BackgroundBolocan, V.-O., Mitoi, A., Nicu-Canareica, O., Băean, M.-L., Medar, C., & Popa, G.-A. (2025) · Journal of Imaging · doi:10.3390/jimaging11100335 · Cited by 22
“Optimal integration might involve AI performing initial triage and quantification, with radiologists focusing on interpretation and clinical correlation [112].”
Yi, C., Tang, Y., Ouyang, R., Zhang, Y., Cao, Z., Yang, Z., Wu, S., Han, M., Xiao, J. Z., Chang, P., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08275-0 · Cited by 20
Advancements in AI-driven diagnostic radiology: Enhancing accuracy and efficiency
Alanazi, M. M. F., Almutairi, S., Alarjani, N., Alghaylan, M. Y. A., Aljawhari, M. S. M., & Alkhulaifi, A. A. S. (2024) · International Journal of Health Sciences · doi:10.53730/ijhs.v8ns1.14928 · Cited by 30
Chen, Y., Wang, L., Luo, R., Wang, S., Wang, H., Gao, F., & Wang, D. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.943415 · Cited by 22
“Yala et al. (9) reported that the Deep Learning (DL) model had the potential to improve speci fi city to 94.2% for triaging the mammogram as cancer-free and reduce the workload by 19.3%.”
BackgroundInfluentialKerschke, L., Weigel, S., Rodríguez‐Ruiz, A., Karssemeijer, N., & Heindel, W. L. (2021) · European Radiology · doi:10.1007/s00330-021-08217-w · Cited by 20
Tan, H., Wu, Q., Wu, Y., Zheng, B., Wang, B., Chen, Y., Du, L., Zhou, J., Fu, F., Guo, H., et al. (2025) · Insights into Imaging · doi:10.1186/s13244-025-01983-x · Cited by 23
“Recent advances in deep learning-based arti fi cial intelligence (AI) methods [13 – 17] have exhibited great potential in breast cancer detection and diagnosis.”
Machine Learning and the Pursuit of High-Value Health Care
Ganguli, I., Gordon, W. J., Lupo, C., Sands-Lincoln, M., George, J. K., Jackson, G. P., Rhee, K. B. L., & Bates, D. W. (2020) · NEJM Catalyst · doi:10.1056/cat.20.0094 · Cited by 22
Baughan, N. M., Douglas, L., & Giger, M. L. (2022) · Journal of Breast Imaging · doi:10.1093/jbi/wbac052 · Cited by 22
Role of calibration in uncertainty-based referral for deep learning
Zhang, R., Gatsonis, C. A., & Steingrimsson, J. A. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231158811 · Cited by 21
“…in cardiovascular imaging; Yao et al. 5 summarized recent literature in neuroradiology and highlighted directions for future work; Yala et al. 6,7 studied mammography-based models for improving breast cancer risk prediction and triage efficiency, and Geras et al. 8 outlined current status and…”
BackgroundDeepCAT: Deep Computer-Aided Triage of Screening Mammography
Yi, P. H., Singh, D., Harvey, S. C., Hager, G. D., & Mullen, L. A. (2021) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-020-00407-0 · Cited by 19
Deep learning models and the limits of explainable artificial intelligence
Bjerring, J. C., Mainz, J. T., & Munch, L. A. (2025) · Asian Journal of Philosophy · doi:10.1007/s44204-024-00238-8 · Cited by 18
“…inputs, the outputs, and the mathematical functions between 1 For examples, see Esteva et al. (2019), Esteva et al. (2017), Gulshan et al. (2016), Yala et al. (2019), Hannun et al. (2019), Nemati et al. (2018), Jiang et al. (2017), Liu et al. (2019), Obermeyer and Emanuel (2016), Topol (2019),…”
The Economic Impact of AI on Breast Imaging
Smetherman, D. H., Golding, L. P., Moy, L. P., & Rubin, E. J. (2022) · Journal of Breast Imaging · doi:10.1093/jbi/wbac012 · Cited by 17
Choi, W. J., An, J. K., Woo, J. J., & Kwak, H. Y. (2022) · Diagnostics · doi:10.3390/diagnostics13010117 · Cited by 20
“A reduction in radiologist’s workload is also expected [13,18,19,34].”
BackgroundThe Promise of Artificial Intelligence in Cardiothoracic Surgery
Salna, M. P. (2022) · The Korean Journal of Thoracic and Cardiovascular Surgery · doi:10.5090/jcs.22.083 · Cited by 17
“[9], whose deep learning algorithm was significantly more accurate at predicting the presence or absence of cancer in mammograms than general radiologists.”
BackgroundKim, H., Choi, J. S., Kim, K., Ko, E. S., Ko, E. Y., & Han, B. (2023) · European Radiology · doi:10.1007/s00330-023-09692-z · Cited by 17
Lee, K. J., Ryoo, I., Choi, D., Sunwoo, L., You, S., & Jung, H. N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241796 · Cited by 16
Factors Contributing to Disproportionate Burnout in Women Breast Imaging Radiologists: A Review
Dodelzon, K., Milch, H. S., Mullen, L. A., Dialani, V. M., Jacobs, S. A., Parikh, J. R., & Grimm, L. J. (2024) · Journal of Breast Imaging · doi:10.1093/jbi/wbad104 · Cited by 13
Adaptive Dual-Task Deep Learning for Automated Thyroid Cancer Triaging at Screening US
Wu, S., Li, M.-D., Tong, W., Liu, Y., Cui, R., Hu, J.-B., Cheng, M.-Q., Ke, W.-P., Lin, X.-X., Lv, J.-Y., et al. (2025) · Radiology Artificial Intelligence · doi:10.1148/ryai.240271 · Cited by 14
Metadata
- Authors
- Yala, A., Schuster, T., Miles, R., Barzilay, R., & Lehman, C.
- Affiliations
- Harvard University (US)
- Venue
- Radiology
- Published
- 2019-10
- Volume
- 293
- Issue
- 1
- Pages
- 38-46
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2964642168
- Cited by
- 210
- References
- 22
- Citations by year
- 2026: 13 · 2025: 33 · 2024: 31 · 2023: 37 · 2022: 31 · 2021: 36 · 2020: 26 · 2019: 3
- FWCI
- 10.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Breast Cancer Alliance
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Digital Radiography and Breast Imaging
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- deep learning, mammography screening, triage, breast cancer screening, sensitivity and specificity, breast cancer detection, simulation study, non-inferiority margin, bootstrap confidence intervals, breast cancer mortality reduction, cost-effectiveness, false positive reduction
- MeSH terms
- Deep Learning, Adult, Aged, Aged, 80 and over, Breast, Breast Neoplasms, Computer Simulation, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Registries, Retrospective Studies, Triage, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.