Stand-Alone Artificial Intelligence for Breast Cancer Detection in Mammography: Comparison With 101 Radiologists
Open accessAbstract
Background: Artificial intelligence (AI) systems performing at radiologist-like levels in the evaluation of digital mammography (DM) would improve breast cancer screening accuracy and efficiency. We aimed to compare the stand-alone performance of an AI system to that of radiologists in detecting breast cancer in DM. Methods: Nine multi-reader, multi-case study datasets previously used for different research purposes in seven countries were collected. Each dataset consisted of DM exams acquired with systems from four different vendors, multiple radiologists’ assessments per exam, and ground truth verified by histopathological analysis or follow-up, yielding a total of 2652 exams (653 malignant) and interpretations by 101 radiologists (28 296 independent interpretations). An AI system analyzed these exams yielding a level of suspicion of cancer present between 1 and 10. The detection performance between the radiologists and the AI system was compared using a noninferiority null hypothesis at a margin of 0.05. Results: The performance of the AI system was statistically noninferior to that of the average of the 101 radiologists. The AI system had a 0.840 (95% confidence interval [CI] = 0.820 to 0.860) area under the ROC curve and the average of the radiologists was 0.814 (95% CI = 0.787 to 0.841) (difference 95% CI = −0.003 to 0.055). The AI system had an AUC higher than 61.4% of the radiologists. Conclusions: The evaluated AI system achieved a cancer detection accuracy comparable to an average breast radiologist in this retrospective setting. Although promising, the performance and impact of such a system in a screening setting needs further investigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sixteen breast studies compared algorithm performance against healthcare professionals.
“Sixteen studies62,91,92,94,97–107 compared algorithm performance against healthcare professionals.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [98]
Eight breast studies validated algorithms on external data.
“eight91–98 studies validated algorithms on external data”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cancer incidence and mortality worldwide: Sources, methods and major patterns in GLOBOCAN 2012
Ferlay, J., Soerjomataram, I., Dikshit, R., Eser, S., Mathers, C., Rebelo, M., Parkin, D. M., Forman, D., & Bray, F. (2015) · International Journal of Cancer · doi:10.1002/ijc.29210 · Cited by 28,766 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 1× · see citing works
“In medical imaging, deep learning-based AI is also rapidly closing the gap between humans and computers (15,16).”
BackgroundThe benefits and harms of breast cancer screening: an independent review
Screening, T. I. U. P. O. B. C., Marmot, M. G., Altman, D. G., Cameron, D., Dewar, J. A., Thompson, S. G., & Wilcox, M. (2013) · British Journal of Cancer · doi:10.1038/bjc.2013.177 · Cited by 1,076 · see citing works
Perry, N., Broeders, M. J. M., de Wolf, C., Törnberg, S., Holland, R., & von Karsa, L. (2007) · Annals of Oncology · doi:10.1093/annonc/mdm481 · Cited by 1,018 · see citing works
Self-citationBreast-Cancer Screening — Viewpoint of the IARC Working Group
Secretan, B., Scoccianti, C., LOOMIS, D. P., Benbrahim‐Tallaa, L., Bouvard, V., Bianchini, F., & Straíf, K. (2015) · New England Journal of Medicine · doi:10.1056/nejmsr1504363 · Cited by 966 · see citing works
Large scale deep learning for computer aided detection of mammographic lesions
Kooi, T., Litjens, G., van Ginneken, B., Gubern-Mérida, A., Sánchez, C. I., Mann, R., den Heeten, A., & Karssemeijer, N. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.07.007 · Cited by 965 · see citing works
Self-citation“Proving the null hypothesis” in clinical trials
Blackwelder, W. C. (1982) · Controlled Clinical Trials · doi:10.1016/0197-2456(82)90024-1 · Cited by 890 · see citing works
Analysis of cancers missed at screening mammography.
Bird, R. E., Wallace, T., & Yankaskas, B. C. (1992) · Radiology · doi:10.1148/radiology.184.3.1509041 · Cited by 822 · see citing works
Smith, R. A., Andrews, K. S., Brooks, D. D., Fedewa, S. A., Manassaram‐Baptiste, D., Saslow, D., Brawley, O. W., & Wender, R. C. (2017) · CA A Cancer Journal for Clinicians · doi:10.3322/caac.21392 · Cited by 721 · see citing works
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678 · see citing works
Influence of Computer-Aided Detection on Performance of Screening Mammography
Fenton, J. J., Taplin, S. H., Carney, P. A., Abraham, L., Sickles, E. A., D’Orsi, C. J., Berns, E. A., Cutter, G., Hendrick, R. E., Barlow, W. E., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa066099 · Cited by 619 · see citing works
Broeders, M. J. M., Moss, S., Nyström, L., Njor, S. H., Jönsson, H., Paap, E., Massat, N. J., Duffy, S. W., Lynge, E., & Paci, E. (2012) · Journal of Medical Screening · doi:10.1258/jms.2012.012078 · Cited by 505 · see citing works
Self-citationBecker, A. S., Marcon, M., Ghafoor, S., Wurnig, M. C., Frauenfelder, T., & Boss, A. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000358 · Cited by 352 · see citing works
Huynh, P. T., Jarolimek, A. M., & Daye, S. (1998) · Radiographics · doi:10.1148/radiographics.18.5.9747612 · Cited by 320 · see citing works
Single Reading with Computer-Aided Detection for Screening Mammography
Gilbert, F. J., Astley, S., Gillan, M. G., Agbaje, O. F., Wallis, M., James, J. J., Boggis, C. R. M., & Duffy, S. W. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0803545 · Cited by 296 · see citing works
Self-citationAutomated classification of parenchymal patterns in mammograms
Karssemeijer, N. (1998) · Physics in Medicine and Biology · doi:10.1088/0031-9155/43/2/011 · Cited by 278 · see citing works
Missed Breast Carcinoma: Pitfalls and Pearls
Majid, A. S., de Paredes, E. S., Doherty, R. D., Sharma, N. R., & Salvador, X. (2003) · Radiographics · doi:10.1148/rg.234025083 · Cited by 274 · see citing works
Detection of stellate distortions in mammograms
Karssemeijer, N., & Brake, G. M. T. (1996) · IEEE Transactions on Medical Imaging · doi:10.1109/42.538938 · Cited by 256 · see citing works
Evans, K. K., Birdwell, R. L., & Wolfe, J. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0064366 · Cited by 238 · see citing works
Accuracy of Screening Mammography Interpretation by Characteristics of Radiologists
Barlow, W. E., Chi, C., Carney, P. A., et al. (2004) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djh333 · Cited by 218 · see citing works
Radiologist shortage leaves patient care at risk, warns royal college
Rimmer, A. (2017) · BMJ · doi:10.1136/bmj.j4683 · Cited by 193 · see citing works
Wallis, M., Moa, E., Zanca, F., et al. (2012) · Radiology · doi:10.1148/radiol.11103514 · Cited by 186 · see citing works
Self-citationGur, D., Bandos, A. I., Cohen, C. S., Hakim, C. M., Hardesty, L. A., Ganott, M. A., Perrin, R. L., Poller, W. R., Shah, R., Sumkin, J. H., et al. (2008) · Radiology · doi:10.1148/radiol.2491072025 · Cited by 184 · see citing works
Will Machine Learning Tip the Balance in Breast Cancer Screening?
Trister, A. D., Buist, D. S. M., & Lee, C. I. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2017.0473 · Cited by 92 · see citing works
Learning from unbalanced data: A cascade-based approach for detecting clustered microcalcifications
Bria, A., Karssemeijer, N., & Tortorella, F. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.10.014 · Cited by 81 · see citing works
Workforce Shortages in Breast Imaging: Impact on Mammography Utilization
Wing, P., & Langelier, M. H. (2009) · American Journal of Roentgenology · doi:10.2214/ajr.08.1665 · Cited by 80 · see citing works
A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators
Gallas, B. D., Bandos, A. I., Samuelson, F. W., & Wagner, R. F. (2009) · Communication in Statistics- Theory and Methods · doi:10.1080/03610920802610084 · Cited by 73 · see citing works
Breast cancer screening with digital breast tomosynthesis
Skaane, P. (2016) · Breast Cancer · doi:10.1007/s12282-016-0699-y · Cited by 70 · see citing works
Broeders, M. J. M., Onland‐Moret, N. C., Rijken, H., Hendriks, J. H. C. L., Verbeek, A. L. M., & Holland, R. (2003) · European Journal of Cancer · doi:10.1016/s0959-8049(03)00311-3 · Cited by 67 · see citing works
Self-citationUse of Normal Tissue Context in Computer-Aided Detection of Masses in Mammograms
Hupse, R., & Karssemeijer, N. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2028611 · Cited by 62 · see citing works
Weber, R. J. P., van Bommel, R. M. G., Louwman, M. W. J., Nederend, J., Voogd, A. C., Jansen, F. H., Tjan‐Heijnen, V. C. G., & Duijm, L. E. M. (2016) · Breast Cancer Research and Treatment · doi:10.1007/s10549-016-3882-0 · Cited by 61 · see citing works
Computer-aided Detection of Masses at Mammography: Interactive Decision Support versus Prompts
Hupse, R., Samulski, M. R. M., Lobbes, M. B. I., Mann, R. M., Mus, R. D. M., den Heeten, G. J., Beijerinck, D., Pijnappel, R. M., Boetes, C., & Karssemeijer, N. (2012) · Radiology · doi:10.1148/radiol.12120218 · Cited by 59 · see citing works
Self-citationGennaro, G., Hendrick, R. E., Ruppel, P., et al. (2012) · European Radiology · doi:10.1007/s00330-012-2649-1 · Cited by 51 · see citing works
Self-citationHypothesis Testing in Noninferiority and Equivalence MRMC ROC Studies
Chen, W., Petrick, N., & Sahiner, B. (2012) · Academic Radiology · doi:10.1016/j.acra.2012.04.011 · Cited by 42 · see citing works
The average receiver operating characteristic curve in multireader multicase imaging studies
Chen, W., & Samuelson, F. W. (2014) · British Journal of Radiology · doi:10.1259/bjr.20140016 · Cited by 42 · see citing works
BI-RADS Data Should Not Be Used to Estimate ROC Curves
Jiang, Y., & Metz, C. E. (2010) · Radiology · doi:10.1148/radiol.10091394 · Cited by 39 · see citing works
Rodríguez‐Ruiz, A., Gubern‐Mérida, A., Imhof-Tas, M. W., et al. (2017) · European Radiology · doi:10.1007/s00330-017-5167-3 · Cited by 31 · see citing works
Reader studies for validation of CAD systems
Gallas, B. D., & Brown, D. G. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.12.013 · Cited by 27 · see citing works
Diagnostic value of the stand-alone synthetic image in digital breast tomosynthesis examinations
Garayoa, J., Chevalier, M. L., Castillo, M. D. P., et al. (2017) · European Radiology · doi:10.1007/s00330-017-4991-9 · Cited by 24 · see citing works
Self-citationGennaro, G. (2018) · European Journal of Radiology · doi:10.1016/j.ejrad.2018.03.014 · Cited by 22 · see citing works
Self-citationVisser, R., Veldkamp, W. J. H., Beijerinck, D., et al. (2011) · European Radiology · doi:10.1007/s00330-011-2320-2 · Cited by 20 · see citing works
Gallas, B. D., & Hillis, S. L. (2014) · Journal of medical imaging · doi:10.1117/1.jmi.1.3.031006 · Cited by 20 · see citing works
Clauser, P., Baltzer, P., Kapetas, P., et al. (2018) · Investigative Radiology · doi:10.1097/rli.0000000000000513 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- [3]3. Lauby-Secretan B, Scoccianti C, Loomis D, et al. Breast cancer screeningviewpoint of the IARC Working Group. N Engl J Med . 2015;372(24): 2353-2358.
- [12]12. Wing P, Langelier MH. Workforce shortages in breast imaging: impact on mammographyutilization. AJR Am J Roentgenol . 2009;192(2):370-378.
- [20]20. Mordang J-J, Janssen T, Bria A, Kooi T, Gubern-M erida A, Karssemeijer N. Automatic microcalcification detection in multi-vendor mammography using convolutional neural networks. Paper presented at International Workshop on Digital Mammography (IWDM); 2016; Malmo, Sweden.
Cited by
Sorted by citations
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946 · see citing works
Artificial intelligence for clinical trial design
Harrer, S., Shah, P., Antony, B., & Hu, J. (2019) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2019.05.005 · Cited by 672 · see citing works
Lång, K., Josefsson, V., Larsson, A.-M., Larsson, S., Högberg, C., Sartor, H., Hofvind, S., Andersson, I., & Rosso, A. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(23)00298-x · Cited by 598 · see citing works
Self-citationLotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423 · see citing works
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.0265 · Cited by 409 · see citing works
Application of Artificial Intelligence in Medicine: An Overview
Liu, P.-R., Lu, L., Zhang, J.-Y., Huo, T.-T., Liu, S.-X., & Ye, Z.-W. (2021) · Current Medical Science · doi:10.1007/s11596-021-2474-3 · Cited by 258 · see citing works
How does artificial intelligence in radiology improve efficiency and health outcomes?
van Leeuwen, K. G., de Rooij, M., Schalekamp, S., van Ginneken, B., & Rutten, M. J. C. M. (2022) · Pediatric Radiology · doi:10.1007/s00247-021-05114-8 · Cited by 236 · see citing works
Noseworthy, P. A., Attia, Z. I., Brewer, L. C., Hayes, S. N., Yao, X., Kapa, S., Friedman, P. A., & Lopez-Jimenez, F. (2020) · Circulation: Arrhythmia and Electrophysiology · doi:10.1161/circep.119.007988 · Cited by 220 · see citing works
Xu, H., Tang, R. S. Y., Lam, T. Y. T., Zhao, G., Lau, J. Y. W., Liu, Y., Wu, Q., Rong, L., Xu, W., Li, X., et al. (2023) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2022.07.006 · Cited by 209 · see citing works
Artificial intelligence in oncology: path to implementation
Chua, I. S., Gaziel‐Yablowitz, M., Korach, Z. T., Kehl, K. L., Levitan, N. A., Arriaga, Y. E., Jackson, G. P., Bates, D. W., & Hassett, M. (2021) · Cancer Medicine · doi:10.1002/cam4.3935 · Cited by 179 · see citing works
Bickman, L. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01065-8 · Cited by 184 · see citing works
Lennartz, S., Dratsch, T., Zopfs, D., Persigehl, T., Maintz, D., Große Hokamp, N., & Pinto dos Santos, D. (2021) · Journal of Medical Internet Research · doi:10.2196/24221 · Cited by 153 · see citing works
Artificial Intelligence in Imaging: The Radiologist’s Role
Rubin, D. L. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.036 · Cited by 110 · see citing works
Artificial Intelligence in Screening Mammography: A Population Survey of Women’s Preferences
Ongena, Y. P., Yakar, D., Haan, M., & Kwee, T. C. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.09.042 · Cited by 105 · see citing works
Advancements in AI based healthcare techniques with FOCUS ON diagnostic techniques
Kalra, N., Verma, P., & Verma, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108917 · Cited by 95 · see citing works
Is it time to get rid of black boxes and cultivate trust in AI?
Gastounioti, A., & Kontos, D. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200088 · Cited by 47 · see citing works
A Review of Applications of Machine Learning in Mammography and Future Challenges
Batchu, S., Liu, F., Amireh, A., Waller, J., & Umair, M. (2021) · Oncology · doi:10.1159/000515698 · Cited by 41 · see citing works
Galsgaard, A., Doorschodt, T., Holten, A.-L., Müller, F. C., Ploug Boesen, M., & Maas, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110231 · Cited by 38 · see citing works
Goel, I., Bhaskar, Y., Kumar, N., Singh, S., Amanullah, M., Dhar, R., & Karmakar, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1550407 · Cited by 31 · see citing works
Differences between human and machine perception in medical diagnosis
Makino, T., Jastrzębski, S., Oleszkiewicz, W., Chacko, C., Ehrenpreis, R., Samreen, N., Chhor, C., Kim, E., Lee, J., Pysarenko, K., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10526-z · Cited by 30 · see citing works
Anticipating artificial intelligence in mammography screening: views of Swedish breast radiologists
Högberg, C., Larsson, S., & Lång, K. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100712 · Cited by 28 · see citing works
Self-citation
- Authors
- Rodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Tan, T., Mertelmeier, T., Wallis, M. G., Andersson, I., Zackrisson, S., Mann, R. M., & Sechopoulos, I.
- Affiliations
- Radboud University Medical Center (NL)
- Corresponding author
- Sechopoulos, Ioannis
- ORCID
- Venue
- JNCI: Journal of the National Cancer Institute
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-03-05
- Volume
- 111
- Issue
- 9
- Pages
- 916-922
- ISSN
- 0027-8874 (ISSN-L)
- 1460-2105
- PMC
- PMC6748773
- OpenAlex
- W2918598741
- Cited by
- 692
- References
- 44
- Citations by year
- 2026: 81 · 2025: 104 · 2024: 112 · 2023: 120 · 2022: 108 · 2021: 82 · 2020: 60 · 2019: 23 · 2018: 1
- FWCI
- 23.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- JNCI Journal of the National Cancer Institute (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Lund University Publications (Lund University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, other-oa)
- Apollo
- ETH Zürich Research Collection
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Digital Radiography and Breast Imaging
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- breast cancer detection, digital mammography, artificial intelligence, radiologist comparison, non-inferiority, breast cancer screening, computer-aided detection, multi-reader multi-case study, mammography interpretation
- MeSH terms
- Radiologists, Algorithms, Artificial Intelligence, Breast Neoplasms, Female, Humans, Image Processing, Computer-Assisted, Mammography, ROC Curve, Reproducibility of Results, Area Under Curve, Early Detection of Cancer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
