Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms
Open accessAbstract
Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Digital Mammography DREAM Challenge was an early catalyst in AI for breast cancer detection, allowing groups to benchmark and compare approaches.
“An early catalyst in this space was the Digital Mammography DREAM Challenge, an open AI competition where groups were able to benchmark and compare different AI approaches (8).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652 · see citing works
Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning
Poplin, R., Varadarajan, A. V., Blumer, K., Liu, Y., McConnell, M. V., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0195-0 · Cited by 1,865 · see citing works
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,349 · see citing works
Screening for Breast Cancer: An Update for the U.S. Preventive Services Task Force
Nelson, H. D., Tyne, K., Naik, A. M., Bougatsos, C., Chan, B. K. S., & Humphrey, L. L. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-10-200911170-00009 · Cited by 1,328 · see citing works
Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
(2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9 · Cited by 1,149 · see citing works
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.246280 · Cited by 984 · see citing works
Detecting and classifying lesions in mammograms with Deep Learning
Ribli, D., Horváth, A., Unger, Z., Pollner, P., & Csabai, I. (2018) · Scientific Reports · doi:10.1038/s41598-018-22437-z · Cited by 696 · see citing works
Self-citationDeep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening
Wu, N., Phang, J., Park, J., Shen, Y., Huang, Z., Zorin, M., Jastrzȩbski, S., Févry, T., Katsnelson, J., Kim, E. S., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945514 · Cited by 695 · see citing works
Self-citationRodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Tan, T., Mertelmeier, T., et al. (2019) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djy222 · Cited by 692 · see citing works
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678 · see citing works
Self-citationLehman, C. D., Arao, R. F., Sprague, B. L., Lee, J. M., Buist, D. S. M., Kerlikowske, K. M., Henderson, L. M., Onega, T., Tosteson, A. N. A., Rauscher, G. H., et al. (2016) · Radiology · doi:10.1148/radiol.2016161174 · Cited by 667 · see citing works
Self-citationInfluence of Computer-Aided Detection on Performance of Screening Mammography
Fenton, J. J., Taplin, S. H., Carney, P. A., Abraham, L., Sickles, E. A., D’Orsi, C. J., Berns, E. A., Cutter, G., Hendrick, R. E., Barlow, W. E., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa066099 · Cited by 619 · see citing works
Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System
Rodríguez-Ruiz, A., Krupinski, E., Mordang, J.-J., Schilling, K., Heywang-Köbrunner, S. H., Sechopoulos, I., & Mann, R. M. (2019) · Radiology · doi:10.1148/radiol.2018181371 · Cited by 602 · see citing works
Nelson, H. D., Pappas, M., Cantor, A. G., Griffin, J. C., Daeges, M., & Humphrey, L. L. (2016) · Annals of Internal Medicine · doi:10.7326/m15-0970 · Cited by 409 · see citing works
Conant, E. F., Toledano, A. Y., Periaswamy, S., Fotin, S. V., Go, J., Boatsman, J. E., & Hoffmeister, J. W. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180096 · Cited by 243 · see citing works
Predicting Breast Cancer by Applying Deep Learning to Linked Health Records and Mammograms
Akselrod-Ballin, A., Chorev, M., Shoshan, Y., Spiro, A., Hazan, A., Melamed, R., Barkan, E., Herzel, E., Naor, S., Karavani, E., et al. (2019) · Radiology · doi:10.1148/radiol.2019182622 · Cited by 189 · see citing works
Crowdsourcing biomedical research: leveraging communities as innovation engines
Sáez-Rodríguez, J., Costello, J. C., Friend, S. H., Kellen, M., Mangravite, L. M., Meyer, P., Norman, T., & Stolovitzky, G. A. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.69 · Cited by 165 · see citing works
Self-citationA Multi-scale CNN and Curriculum Learning Strategy for Mammogram Classification
Lotter, W. E., Sorensen, G., & Cox, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_20 · Cited by 155 · see citing works
Self-citationArtificial intelligence for breast cancer screening: Opportunity or hype?
Houssami, N., Lee, C. I., Buist, D. S. M., & Tao, D. (2017) · The Breast · doi:10.1016/j.breast.2017.09.003 · Cited by 106 · see citing works
Self-citationO’Donoghue, C., Eklund, M., Ozanne, E. M., & Esserman, L. J. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1217 · Cited by 95 · see citing works
Will Machine Learning Tip the Balance in Breast Cancer Screening?
Trister, A. D., Buist, D. S. M., & Lee, C. I. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2017.0473 · Cited by 92 · see citing works
Self-citationMayo, R. C., Kent, D., Sen, L. Q. C., Kapoor, M. M., Leung, J. W. T., & Watanabe, A. T. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0168-6 · Cited by 87 · see citing works
Alternative models for sharing confidential biomedical data
Guinney, J., & Sáez-Rodríguez, J. (2018) · Nature Biotechnology · doi:10.1038/nbt.4128 · Cited by 72 · see citing works
Self-citationPredicting protein function and other biomedical characteristics with heterogeneous ensembles
Whalen, S., Pandey, O. P., & Pandey, G. S. (2015) · Methods · doi:10.1016/j.ymeth.2015.08.016 · Cited by 27 · see citing works
Self-citationChallenges: Crowdsourced solutions
Bender, E. (2016) · Nature · doi:10.1038/533s62a · Cited by 26 · see citing works
Artificial Intelligence for Breast Cancer Imaging: The New Frontier?
Lee, C. I., & Elmore, J. G. (2018) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djy223 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [20]SAGE Bionetworks. Digital mammography DREAM challenge. https://sagebionetworks.org/research-projects/digital-mammography-dream-challenge/. Accessed January 22, 2020.
- [24]SAGE Bionetworks. Synapse. https://www.synapse.org/. Accessed January 22, 2020.
Cited by
Sorted by citations
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 455 · see citing works
Lotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423 · see citing works
Self-citationThe Role of Artificial Intelligence in Early Cancer Diagnosis
Hunter, B., Hindocha, S., & Lee, R. W. (2022) · Cancers · doi:10.3390/cancers14061524 · Cited by 367 · see citing works
Sechopoulos, I., Teuwen, J., & Mann, R. M. (2020) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.06.002 · Cited by 364 · see citing works
Freeman, K., Geppert, J., Stinton, C., Todkill, D., Johnson, S. A., Clarke, A., & Taylor‐Phillips, S. (2021) · BMJ · doi:10.1136/bmj.n1872 · Cited by 352 · see citing works
Dembrower, K., Wåhlin, E., Liu, Y., Salim, M., Smith, K., Lindholm, P., Eklund, M., & Strand, F. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30185-0 · Cited by 305 · see citing works
Self-citationSalim, M., Wåhlin, E., Dembrower, K., Azavedo, E., Foukakis, T., Liu, Y., Smith, K., Eklund, M., & Strand, F. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.3321 · Cited by 288 · see citing works
Self-citationToward robust mammography-based models for breast cancer risk
Yala, A., Mikhael, P. G., Strand, F., Lin, G., Smith, K., Wan, Y.-L., Lamb, L., Hughes, K., Lehman, C., & Barzilay, R. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.aba4373 · Cited by 287 · see citing works
Self-citationSezgın, E. (2023) · Digital Health · doi:10.1177/20552076231186520 · Cited by 279 · see citing works
Malgaroli, M., Hull, T. D., Zech, J. M., & Althoff, T. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02592-2 · Cited by 217 · see citing works
Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis
Leibig, C., Brehmer, M., Bunk, S., Byng, D., Pinker, K., & Umutlu, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00070-x · Cited by 213 · see citing works
Nguyen, H. T., Nguyen, H. Q., Pham, H., Lam, K., Le, L. T., Dao, M., & Vu, V. (2023) · Scientific Data · doi:10.1038/s41597-023-02100-7 · Cited by 200 · see citing works
Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196 · see citing works
Self-citationConvolutional neural networks for breast cancer detection in mammography: A survey
Abdelrahman, L., Alghamdi, M., Collado‐Mesa, F., & Abdel-Mottaleb, M. S. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104248 · Cited by 194 · see citing works
Deep Learning in Breast Cancer Imaging: State of the Art and Recent Advancements in Early 2024
Carriero, A., Groenhoff, L., Vologina, E., Basile, P., & Albera, M. (2024) · Diagnostics · doi:10.3390/diagnostics14080848 · Cited by 183 · see citing works
Barnett, A. J., Schwartz, F. R., Tao, C., Chen, C., Ren, Y., Lo, J. Y.-C., & Rudin, C. D. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00423-x · Cited by 162 · see citing works
Self-citationAn Overview of Artificial Intelligence in Oncology
Farina, E. M. J. D. M., Nabhen, J. J., Dacoregio, M. I., Batalini, F., & de Moraes, F. Y. (2022) · Future Science OA · doi:10.2144/fsoa-2021-0074 · Cited by 137 · see citing works
Xue, P., Wang, J., Qin, D., Yan, H., Qu, Y., Seery, S., Jiang, Y., & Qiao, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00559-z · Cited by 134 · see citing works
Buda, M., Saha, A., Walsh, R., Ghate, S. V., Li, N., Swiecicki, A., Lo, J. Y.-C., & Mazurowski, M. A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.19100 · Cited by 131 · see citing works
Self-citation“Shortcuts” Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation
Banerjee, I., Bhattacharjee, K., Burns, J. L., Trivedi, H., Purkayastha, S., Seyyed-Kalantari, L., Patel, B. N., Shiradkar, R., & Gichoya, J. W. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.025 · Cited by 131 · see citing works
Self-citationLukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · see citing works
“AI already accompanies the area of breast cancer, especially in radiology in so-called computer-assisted diagnosis in mammography screening [5] or in radiation oncology [6].”
BackgroundUses and Limitations of Artificial Intelligence for Oncology
Kolla, L., & Parikh, R. B. (2024) · Cancer · doi:10.1002/cncr.35307 · Cited by 126 · see citing works
Hickman, S. E., Woitek, R., Le, E., Im, Y. R., Luxhøj, C. M., Avilés-Rivero, A. I., Baxter, G. C., Mackay, J., & Gilbert, F. J. (2021) · Radiology · doi:10.1148/radiol.2021210391 · Cited by 126 · see citing works
Artificial intelligence and machine learning in emergency medicine: a narrative review
Mueller, B., Kinoshita, T., Peebles, A., Graber, M. A., & Lee, S. (2022) · Acute Medicine & Surgery · doi:10.1002/ams2.740 · Cited by 118 · see citing works
Artificial Intelligence-Empowered Radiology—Current Status and Critical Review
Obuchowicz, R., Lasek, J., Wodziński, M., Piórkowski, A., Strzelecki, M., & Nurzyǹska, K. (2025) · Diagnostics · doi:10.3390/diagnostics15030282 · Cited by 113 · see citing works
Hickman, S. E., Baxter, G. C., & Gilbert, F. J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01333-w · Cited by 110 · see citing works
Can artificial intelligence reduce the interval cancer rate in mammography screening?
Lång, K., Hofvind, S. S. H., Rodríguez‐Ruiz, A., & Andersson, I. (2021) · European Radiology · doi:10.1007/s00330-021-07686-3 · Cited by 108 · see citing works
Marinovich, M. L., Wylie, E. J., Lotter, W. E., Lund, H., Waddell, A., Madeley, C., Pereira, G. F., & Houssami, N. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104498 · Cited by 108 · see citing works
Self-citationArtificial Intelligence for the Future Radiology Diagnostic Service
Mun, S. K., Wong, K. H., Lo, S. B., Li, Y., & Bayarsaikhan, S. (2021) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2020.614258 · Cited by 106 · see citing works
Machine Learning and Big Data Provide Crucial Insight for Future Biomaterials Discovery and Research
Kerner, J., Dogan, A. B., & von Recum, H. A. (2020) · Acta Biomaterialia · doi:10.1016/j.actbio.2021.05.053 · Cited by 106 · see citing works
Tarca, A. L., Pataki, B., Romero, R. J., Sirota, M., Guan, Y., Kutum, R., Gomez‐Lopez, N., Done, B., Bhatti, G., Yu, T. V., et al. (2021) · Cell Reports Medicine · doi:10.1016/j.xcrm.2021.100323 · Cited by 103 · see citing works
Self-citationArasu, V. A., Habel, L. A., Achacoso, N. S., Buist, D. S. M., Cord, J. B., Esserman, L. J., Hylton, N. M., Glymour, M. M., Kornak, J., Kushi, L. H., et al. (2023) · Radiology · doi:10.1148/radiol.222733 · Cited by 98 · see citing works
Self-citationArtificial Intelligence Applications in Breast Imaging: Current Status and Future Directions
Taylor, C. R., Monga, N., Johnson, C., Hawley, J. R., & Patel, M. J. (2023) · Diagnostics · doi:10.3390/diagnostics13122041 · Cited by 94 · see citing works
Jeong, J. J., Vey, B. L., Bhimireddy, A., Kim, T., Santos, T., Correa, R., Dutt, R., Mosunjac, M., Oprea-Ilies, G., Smith, G., et al. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.220047 · Cited by 92 · see citing works
Self-citationArtificial Intelligence in Breast Cancer Screening
Potnis, K. C., Ross, J. S., Aneja, S., Gross, C. P., & Richman, I. B. (2022) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2022.4969 · Cited by 90 · see citing works
Browning, L., Colling, R. T., Rakha, E. A., Rajpoot, N., Rittscher, J., James, J. A., Salto‐Tellez, M., Snead, D. R. J., & Verrill, C. L. (2020) · Journal of Clinical Pathology · doi:10.1136/jclinpath-2020-206854 · Cited by 88 · see citing works
Deep Learning-Based Artificial Intelligence for Mammography
Yoon, J. H., & Kim, E. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2020.1210 · Cited by 88 · see citing works
Systematic reviews of machine learning in healthcare: a literature review
Kolasa, K., Admassu, B. Y., Hołownia-Voloskova, M. E., Kędzior, K. J., Poirrier, J., & Perni, S. (2023) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2023.2279107 · Cited by 87 · see citing works
Petrini, D. G. P., Shimizu, C., Roela, R. A., Valente, G. V., Folgueira, M. A. A. K., & Kim, H. Y. (2022) · IEEE Access · doi:10.1109/access.2022.3193250 · Cited by 85 · see citing works
Self-citationAdvancements in Oncology with Artificial Intelligence—A Review Article
Vobugari, N., Raja, V., Sethi, U., Gandhi, K. D., Raja, K., & Surani, S. (2022) · Cancers · doi:10.3390/cancers14051349 · Cited by 80 · see citing works
Basic of machine learning and deep learning in imaging for medical physicists
Manco, L., Maffei, N., Strolin, S., Vichi, S., Bottazzi, L., & Strigari, L. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.026 · Cited by 75 · see citing works
Baccouche, A., García-Zapirain, B., Zheng, Y., & Elmaghraby, A. S. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106884 · Cited by 71 · see citing works
On the role of artificial intelligence in medical imaging of COVID-19
Born, J., Beymer, D., Rajan, D., Coy, A., Mukherjee, V. V., Manica, M., Prasanna, P. M., Ballah, D., Guindy, M., Shaham, D., et al. (2021) · Patterns · doi:10.1016/j.patter.2021.100269 · Cited by 69 · see citing works
Improved Productivity Using Deep Learning–assisted Reporting for Lumbar Spine MRI
Lim, D. S. W., Makmur, A., Zhu, L., Zhang, W., Cheng, A. J. L., Sia, S. Y., Eide, S. E., Ong, H. Y., Jagmohan, P., Tan, W. C., et al. (2022) · Radiology · doi:10.1148/radiol.220076 · Cited by 68 · see citing works
A Comparison of Techniques for Class Imbalance in Deep Learning Classification of Breast Cancer
Walsh, R., & Tardy, M. (2022) · Diagnostics · doi:10.3390/diagnostics13010067 · Cited by 67 · see citing works
AI-enhanced breast imaging: Where are we and where are we heading?
Bitencourt, A. G. V., Naranjo, I. D., Gullo, R. L., Saccarelli, C. R., & Pinker, K. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109882 · Cited by 65 · see citing works
Holste, G., Partridge, S. C., Rahbar, H., Biswas, D., Lee, C. I., & Alessio, A. M. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00368 · Cited by 63 · see citing works
Self-citationAnderson, A. W., Marinovich, M. L., Houssami, N., Lowry, K. P., Elmore, J. G., Buist, D. S. M., Hofvind, S. S. H., & Lee, C. I. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.11.008 · Cited by 63 · see citing works
Self-citationHsu, W. W. Y., Hippe, D. S., Nakhaei, N., Wang, P., Zhu, B., Siu, N., Ahsen, M. E., Lotter, W. E., Sorensen, A. G., Naeim, A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.42343 · Cited by 61 · see citing works
Self-citationArtificial intelligence with deep learning in nuclear medicine and radiology
Decuyper, M., Maebe, J., Holen, R. V., & Vandenberghe, S. (2021) · EJNMMI Physics · doi:10.1186/s40658-021-00426-y · Cited by 60 · see citing works
Taylor‐Phillips, S., Seedat, F., Kijauskaite, G., Marshall, J., Halligan, S., Hyde, C., Given-Wilson, R., Wilkinson, L., Denniston, A. K., Glocker, B., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00088-7 · Cited by 60 · see citing works
de Vries, C. F., Colosimo, S. J., Staff, R. T., Dymiter, J., Yearsley, J. E., Dinneen, D., Boyle, M., Harrison, D. J., Anderson, L. A., Lip, G., et al. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.220146 · Cited by 60 · see citing works
Regulatory Issues and Challenges to Artificial Intelligence Adoption
Harvey, H. B., & Gowda, V. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.007 · Cited by 59 · see citing works
Jones, M. A., Islam, W., Faiz, R., Chen, X., & Zheng, B. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.980793 · Cited by 59 · see citing works
Manigrasso, F., Milazzo, R., Russo, A. S., Lamberti, F., Strand, F., Pagnani, A., & Morra, L. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103320 · Cited by 59 · see citing works
Self-citationSarno, L. D., Caroselli, A., Tonin, G., Graglia, B., Pansini, V., Causio, F. A., Gatto, A., & Chiaretti, A. (2024) · Biomedicines · doi:10.3390/biomedicines12061220 · Cited by 58 · see citing works
Advances in systems biology modeling: 10 years of crowdsourcing DREAM challenges
Meyer, P., & Sáez-Rodríguez, J. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.015 · Cited by 57 · see citing works
Cè, M., Caloro, E., Pellegrino, M. E., Basile, M., Sorce, A., Fazzini, D., Oliva, G., & Cellina, M. I. (2022) · Exploration of Targeted Anti-tumor Therapy · doi:10.37349/etat.2022.00113 · Cited by 57 · see citing works
Lokaj, B., Pugliese, M., Kinkel, K., Lovis, C., & Schmid, J. (2023) · European Radiology · doi:10.1007/s00330-023-10181-6 · Cited by 57 · see citing works
Deep learning in breast radiology: current progress and future directions
Ou, W. C., Polat, D. S., & Doğan, B. E. (2021) · European Radiology · doi:10.1007/s00330-020-07640-9 · Cited by 57 · see citing works
Lee, J. H., Kim, K. H., Lee, E. H., Ahn, J. S., Ryu, J. K., Park, Y. M., Shin, G. W., Kim, Y. J., & Choi, H. Y. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0476 · Cited by 56 · see citing works
Garrucho, L., Kushibar, K., Jouide, S., Díaz, O., Igual, L., & Lekadir, K. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102386 · Cited by 56 · see citing works
Koçak, B., Kuş, E. A., & Kılıçkesmez, Ö. (2020) · European Radiology · doi:10.1007/s00330-020-07324-4 · Cited by 55 · see citing works
Frazer, H. M. L., Peña‐Solórzano, C. A., Kwok, C. F., Elliott, M. S., Chen, Y., Wang, C., The BRAIx Team, Al-Qershi, O. M., Fox, S., Hill, B., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51725-8 · Cited by 52 · see citing works
Artificial intelligence in breast cancer screening: primary care provider preferences
Hendrix, N., Hauber, B., Lee, C. I., Bansal, A., & Veenstra, D. L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa292 · Cited by 52 · see citing works
Self-citationHanna, M. G., Olson, N. H., Zarella, M. D., Dash, R. C., Herrmann, M. D., Furtado, L. V., Stram, M. N., Raciti, P. M., Hassell, L. A., Mays, A., et al. (2023) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2023-0042-cp · Cited by 51 · see citing works
Applications of Artificial Intelligence in Breast Imaging
Morgan, M. B., & Mates, J. (2020) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2020.08.007 · Cited by 51 · see citing works
Artificial Intelligence in Emergency Medicine: Benefits, Risks, and Recommendations
Vearrier, L., Derse, A. R., Basford, J. B., Larkin, G. L., & Moskop, J. C. (2022) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2022.01.001 · Cited by 49 · see citing works
Impact of artificial intelligence in breast cancer screening with mammography
Dang, L.-A., Chazard, E., Poncelet, É., Serb, T., Rusu, A., Pauwels, X., Parsy, C., Poclet, T., Cauliez, H., Engelaere, C., et al. (2022) · Breast Cancer · doi:10.1007/s12282-022-01375-9 · Cited by 49 · see citing works
Liu, J., Lei, J., Ou, Y., Zhao, Y., Tuo, X., Zhang, B., & Shen, M. (2022) · Clinical and Experimental Medicine · doi:10.1007/s10238-022-00895-0 · Cited by 47 · see citing works
Korfiatis, P., Kline, T. L., Meyer, H. M., Khalid, S., Leiner, T., Loufek, B. T., Blezek, D., Vidal, D. E., Hartman, R. P., Joppa, L. J., et al. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.100188 · Cited by 46 · see citing works
Lu, G., Tian, R., Yang, W. P., Liu, R., Liu, D., Xiang, Z., & Zhang, G. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1402967 · Cited by 46 · see citing works
Basurto-Hurtado, J. A., Cruz-Albarrán, I. A., Toledano‐Ayala, M., Ibarra-Manzano, M.-A., Morales-Hernandez, L. A., & Perez-Ramirez, C. A. (2022) · Cancers · doi:10.3390/cancers14143442 · Cited by 46 · see citing works
Johansson, J. V., Dembrower, K., Strand, F., & Grauman, Å. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-084014 · Cited by 45 · see citing works
Self-citationArtificial intelligence for breast cancer: Implications for diagnosis and management
AlSamhori, J. F., AlSamhori, A. R. F., Duncan, L. A., Qalajo, A., Alshahwan, H. F., Al-abbadi, M., Soudi, M. A., Zakraoui, R., AlSamhori, A. F., AlRyalat, S. A., et al. (2024) · Journal of Medicine Surgery and Public Health · doi:10.1016/j.glmedi.2024.100120 · Cited by 44 · see citing works
Dahlblom, V., Dustler, M., Tingberg, A., & Zackrisson, S. (2022) · European Radiology · doi:10.1007/s00330-022-09316-y · Cited by 43 · see citing works
Buddenkotte, T., Escudero Sanchez, L., Crispin-Ortuzar, M., Woitek, R., McCague, C., Brenton, J. D., Öktem, O., Sala, E., & Rundo, L. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107096 · Cited by 42 · see citing works
Frazer, H. M. L., Qin, A. K., Pan, H., & Brotchie, P. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13278 · Cited by 39 · see citing works
Diagnostic test accuracy of AI-assisted mammography for breast imaging: a narrative review
Dave, D., Akhunzada, A., Ivković, N., Gyawali, S., Cengiz, K., Ahmed, A. R., & Al‐Shamayleh, A. S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2476 · Cited by 38 · see citing works
Petrick, N., Akbar, S., Kenny, H., Nofech‐Mozes, S., Sahiner, B., Gavrielides, M. A., Kalpathy–Cramer, J., Drukker, K., & Martel, A. L. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.3.034501 · Cited by 38 · see citing works
Self-citationRabinovici‐Cohen, S., Fernández, X. M., Grandal, B., Hexter, E., Cubelos, O. H., Pajula, J., Pölönen, H., Reyal, F., & Rosen‐Zvi, M. (2022) · Cancers · doi:10.3390/cancers14163848 · Cited by 38 · see citing works
Self-citationLee, S. E., Han, K., Yoon, J. H., Youk, J. H., & Kim, E. (2022) · European Radiology · doi:10.1007/s00330-022-08718-2 · Cited by 37 · see citing works
Şeker, M. E., Köylüoğlu, Y. O., Özaydın, A., Gürdal, S. Ö., Özçınar, B., Cabıoğlu, N., Özmen, V., & Arıbal, E. (2024) · European Radiology · doi:10.1007/s00330-024-10661-3 · Cited by 36 · see citing works
Deep learning in breast imaging
Bhowmik, A., & Eskreis‐Winkler, S. (2022) · BJR|Open · doi:10.1259/bjro.20210060 · Cited by 36 · see citing works
Role of Machine Learning and Artificial Intelligence in Interventional Oncology
D’Amore, B., Smolinski‐Zhao, S., Daye, D., & Uppot, R. N. (2021) · Current Oncology Reports · doi:10.1007/s11912-021-01054-6 · Cited by 35 · see citing works
Feng, S., Liu, Q., Patel, A., Bazai, S. U., Jin, C., Kim, J. S., Sarrafzadeh, M., Azzollini, D., Yeoh, J., Kim, E., et al. (2022) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13393 · Cited by 35 · see citing works
Lee, S. E., Yoon, J. H., Son, N., Han, K., & Moon, H. J. (2023) · American Journal of Roentgenology · doi:10.2214/ajr.23.29655 · Cited by 34 · see citing works
Applications of Artificial Intelligence in Medicine
Srivastava, R. (2023) · Exploratory Research and Hypothesis in Medicine · doi:10.14218/erhm.2023.00048 · Cited by 33 · see citing works
Chang, S., Kim, H., Suh, Y. J., Choi, D. M., Kim, H., Kim, D. K., Kim, J. Y., Yoo, J. Y., & Choi, B. W. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109582 · Cited by 33 · see citing works
Self-citationArtificial Intelligence-Driven Morphology-Based Enrichment of Malignant Cells from Body Fluid
Mavropoulos, A., Johnson, C., Lu, V., Nieto, J., Schneider, E. C., Saini, K., Phelan, M. L., Hsie, L., Wang, M. J., Cruz, J., et al. (2023) · Modern Pathology · doi:10.1016/j.modpat.2023.100195 · Cited by 33 · see citing works
Santus, E., Marino, N., Cirillo, D., Chersoni, E., Montagud, A., Chadha, A. S., Valencia, A., Hughes, K. S., & Lindvall, C. J. (2021) · Journal of Medical Internet Research · doi:10.2196/22453 · Cited by 32 · see citing works
Bhandari, A., Purchuri, S. N., Sharma, C., Ibrahim, M., & Prior, M. (2021) · Clinical Imaging · doi:10.1016/j.clinimag.2021.08.004 · Cited by 32 · see citing works
Armato, S. G., Drukker, K., & Hadjiiski, L. M. (2023) · British Journal of Radiology · doi:10.1259/bjr.20221152 · Cited by 31 · see citing works
Ramwala, O. A., Lowry, K. P., Cross, N. M., Hsu, W. W. Y., Austin, C. C., Mooney, S. D., & Lee, C. I. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.04.027 · Cited by 31 · see citing works
Self-citationEmpowering cancer research in Europe: the EUCAIM cancer imaging infrastructure
Martí‐Bonmatí, L., Blanquer, I., Tsiknakis, M., Tsakou, G., Martínez, R. M., Capella-Gutiérrez, S., Zullino, S., Meszaros, J. G., Bron, E. E., Gelpí, J. L., et al. (2025) · Insights into Imaging · doi:10.1186/s13244-025-01913-x · Cited by 31 · see citing works
Self-citationDifferences between human and machine perception in medical diagnosis
Makino, T., Jastrzębski, S., Oleszkiewicz, W., Chacko, C., Ehrenpreis, R., Samreen, N., Chhor, C., Kim, E., Lee, J., Pysarenko, K., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10526-z · Cited by 30 · see citing works
Self-citationKwon, M.-R., Chang, Y., Ham, S., Cho, Y., Kim, E. Y., Kang, J., Park, E. K., Kim, K. H., Kim, M., Kim, T. S., et al. (2024) · Breast Cancer Research · doi:10.1186/s13058-024-01821-w · Cited by 30 · see citing works
Rajendran, A., Rajan, R. A., Balasubramaniyam, S., & Elumalai, K. (2025) · iRadiology · doi:10.1002/ird3.70027 · Cited by 30 · see citing works
Brahimetaj, R., Willekens, I., Massart, A., Forsyth, R. G., Cornelis, J. P., Mey, J. D., & Jansen, B. (2022) · BMC Cancer · doi:10.1186/s12885-021-09133-4 · Cited by 30 · see citing works
Ng, A., Glocker, B., Oberije, C., Fox, G., Sharma, N., James, J. J., Ambrózay, É., Nash, J., Karpati, E., Kerruish, S., et al. (2023) · Journal of Breast Imaging · doi:10.1093/jbi/wbad010 · Cited by 30 · see citing works
- Authors
- Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., Feng, J., Feng, M., Kim, H.-E., Albiol, F., Albiol, A., Morrell, S., Wojna, Z., Ahsen, M. E., Asif, U., Jimeno Yepes, A., Yohanandan, S., Rabinovici-Cohen, S., Yi, D., Hoff, B., Yu, T., Chaibub Neto, E., Rubin, D. L., Lindholm, P., Margolies, L. R., McBride, R. B., Rothstein, J. H., Sieh, W., Ben-Ari, R., Harrer, S., Trister, A., Friend, S., Norman, T., Sahiner, B., Strand, F., Guinney, J., Stolovitzky, G., and the DM DREAM Consortium, Mackey, L., Cahoon, J., Shen, L., Sohn, J. H., Trivedi, H., Shen, Y., Buturovic, L., Pereira, J. C., Cardoso, J. S., Castro, E., Kalleberg, K. T., Pelka, O., Nedjar, I., Geras, K. J., Nensa, F., Goan, E., Koitka, S., Caballero, L., Cox, D. D., Krishnaswamy, P., Pandey, G., Friedrich, C. M., Perrin, D., Fookes, C., Shi, B., Cardoso Negrie, G., Kawczynski, M., Cho, K., Khoo, C. S., Lo, J. Y., Sorensen, A. G., & Jung, H.
- Affiliations
- Sage Bionetworks (US)
- Kaiser Permanente Washington Health Research Institute (US)
- University of Washington (US)
- Therapixel, Paris, France
- Department of Physics of Complex Systems, ELTE Eötvös Loránd University, Budapest, Hungary
- University of Michigan (US)
- IBM (United States) (US)
- Tencent (China) (CN)
- National University of Singapore (SG)
- Synapxe (SG)
- National University Health System (SG)
- Lunit Inc, Seoul, Korea
- Consejo Superior de Investigaciones Científicas (ES)
- Universitat Politècnica de València (ES)
- University College London (GB)
- NortonLifeLock (United States) (US)
- University of Illinois Urbana-Champaign (US)
- IBM Research - Australia (AU)
- IBM Research Australia, Melbourne, Australia
- IBM Research Haifa, Haifa University Campus, Mount Carmel, Haifa, Israel
- Stanford University (US)
- Computational Oncology, Sage Bionetworks, Seattle, Washington
- Karolinska Institutet (SE)
- Icahn School of Medicine at Mount Sinai (US)
- Fred Hutch Cancer Center (US)
- Gates Foundation (US)
- United States Food and Drug Administration (US)
- Karolinska University Hospital (SE)
- IBM Research - Thomas J. Watson Research Center (US)
- Microsoft (United States) (US)
- North Carolina State University (US)
- University of California, San Francisco (US)
- Emory University (US)
- New York University (US)
- Clinical Persona, East Palo Alto, California
- Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal
- INESC TEC (PT)
- KolibriFX, Oslo, Norway
- Dortmund University of Applied Sciences and Arts (DE)
- University of Abou Bekr Belkaïd (DZ)
- NYU Langone Health (US)
- Essen University Hospital (DE)
- Queensland University of Technology (AU)
- MIT-IBM Computing Research Lab (US)
- Agency for Science, Technology and Research (SG)
- Duke University (US)
- Satalia, London, United Kingdom
- Korea University (KR)
- Corresponding author
- Stolovitzky, Gustavo
- ORCID
- Schaffter, Thomas
- Buist, Diana S. M.
- Lee, Christoph I.
- Nikulin, Yaroslav
- Guan, Yuanfang
- Lotter, William
- Jie, Zequn
- Du, Hao
- Wang, Sijia
- Feng, Jiashi
- Feng, Mengling
- Albiol, Francisco
- Albiol, Alberto
- Morrell, Stephen
- Wojna, Zbigniew
- Ahsen, Mehmet Eren
- Asif, Umar
- Yi, Darvin
- Hoff, Bruce
- Yu, Thomas
- Rubin, Daniel L.
- Lindholm, Peter
- Margolies, Laurie R.
- McBride, Russell Bailey
- Rothstein, Joseph H.
- Sieh, Weiva
- Harrer, Stefan
- Trister, Andrew
- Friend, Stephen
- Norman, Thea
- Sahiner, Berkman
- Strand, Fredrik
- Guinney, Justin
- Stolovitzky, Gustavo
- Mackey, Lester
- Cahoon, Joyce
- Shen, Li
- Sohn, Jae Ho
- Trivedi, Hari
- Shen, Yiqiu
- Cardoso, Jaime S.
- Castro, Eduardo
- Kalleberg, Karl Trygve
- Pelka, Obioma
- Nedjar, Imane
- Geras, Krzysztof J.
- Nensa, Felix
- Goan, Ethan
- Koitka, Sven
- Caballero, Luis
- Cox, David D.
- Krishnaswamy, Pavitra
- Pandey, Gaurav
- Friedrich, Christoph M.
- Perrin, Dimitri
- Fookes, Clinton
- Shi, Bibo
- Cho, Kyunghyun
- Lo, Joseph Y.
- Sorensen, A. Gregory
- Jung, Hwejin
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2020-03-02
- Volume
- 3
- Issue
- 3
- Pages
- e200265
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC7052735
- OpenAlex
- W3009692632
- Cited by
- 409
- References
- 32
- Citations by year
- 2026: 29 · 2025: 55 · 2024: 92 · 2023: 74 · 2022: 71 · 2021: 66 · 2020: 21
- FWCI
- 22.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DIGITAL.CSIC (Spanish National Research Council (CSIC)) (acceptedVersion, other-oa)
- QUT ePrints (Queensland University of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (1482827)
- National Cancer Institute (P01CA154292, R37CA240403)
- National Institute of General Medical Sciences (R35GM133346)
- National Institute of Biomedical Imaging and Bioengineering (P41EB017183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Digital Radiography and Breast Imaging
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- mammography screening, breast cancer detection, artificial intelligence, radiologist interpretation, ensemble learning, false positives, breast cancer risk factors
- MeSH terms
- Radiologists, Deep Learning, Adult, Aged, Algorithms, Artificial Intelligence, Breast Neoplasms, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Radiology, Sensitivity and Specificity, Sweden, United States, Early Detection of Cancer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
