The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms
Open accessAbstract
Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An early catalyst in this space was the Digital Mammography DREAM Challenge, an open AI competition where groups were able to benchmark and compare different AI approaches (8).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning
Poplin, R., Varadarajan, A. V., Blumer, K., Liu, Y., McConnell, M. V., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0195-0 · Cited by 1,865
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,360
Screening for Breast Cancer: An Update for the U.S. Preventive Services Task Force
Nelson, H. D., Tyne, K., Naik, A. M., Bougatsos, C., Chan, B. K. S., & Humphrey, L. L. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-10-200911170-00009 · Cited by 1,328
Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
(2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9 · Cited by 1,149
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.246280 · Cited by 984
Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening
Wu, N., Phang, J., Park, J., Shen, Y., Huang, Z., Zorin, M., Jastrzȩbski, S., Févry, T., Katsnelson, J., Kim, E. S., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945514 · Cited by 736
Self-citationDetecting and classifying lesions in mammograms with Deep Learning
Ribli, D., Horváth, A., Unger, Z., Pollner, P., & Csabai, I. (2018) · Scientific Reports · doi:10.1038/s41598-018-22437-z · Cited by 696
Self-citationRodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Tan, T., Mertelmeier, T., et al. (2019) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djy222 · Cited by 692
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Self-citationLehman, C. D., Arao, R. F., Sprague, B. L., Lee, J. M., Buist, D. S. M., Kerlikowske, K. M., Henderson, L. M., Onega, T., Tosteson, A. N. A., Rauscher, G. H., et al. (2016) · Radiology · doi:10.1148/radiol.2016161174 · Cited by 667
Self-citationInfluence of Computer-Aided Detection on Performance of Screening Mammography
Fenton, J. J., Taplin, S. H., Carney, P. A., Abraham, L., Sickles, E. A., D’Orsi, C. J., Berns, E. A., Cutter, G., Hendrick, R. E., Barlow, W. E., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa066099 · Cited by 619
Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System
Rodríguez-Ruiz, A., Krupinski, E., Mordang, J.-J., Schilling, K., Heywang-Köbrunner, S. H., Sechopoulos, I., & Mann, R. M. (2019) · Radiology · doi:10.1148/radiol.2018181371 · Cited by 602
Nelson, H. D., Pappas, M., Cantor, A. G., Griffin, J. C., Daeges, M., & Humphrey, L. L. (2016) · Annals of Internal Medicine · doi:10.7326/m15-0970 · Cited by 409
Conant, E. F., Toledano, A. Y., Periaswamy, S., Fotin, S. V., Go, J., Boatsman, J. E., & Hoffmeister, J. W. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180096 · Cited by 243
Predicting Breast Cancer by Applying Deep Learning to Linked Health Records and Mammograms
Akselrod-Ballin, A., Chorev, M., Shoshan, Y., Spiro, A., Hazan, A., Melamed, R., Barkan, E., Herzel, E., Naor, S., Karavani, E., et al. (2019) · Radiology · doi:10.1148/radiol.2019182622 · Cited by 200
Crowdsourcing biomedical research: leveraging communities as innovation engines
Sáez-Rodríguez, J., Costello, J. C., Friend, S. H., Kellen, M., Mangravite, L. M., Meyer, P., Norman, T., & Stolovitzky, G. A. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.69 · Cited by 173
Self-citationA Multi-scale CNN and Curriculum Learning Strategy for Mammogram Classification
Lotter, W. E., Sorensen, G., & Cox, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_20 · Cited by 155
Self-citationArtificial intelligence for breast cancer screening: Opportunity or hype?
Houssami, N., Lee, C. I., Buist, D. S. M., & Tao, D. (2017) · The Breast · doi:10.1016/j.breast.2017.09.003 · Cited by 106
Self-citationO’Donoghue, C., Eklund, M., Ozanne, E. M., & Esserman, L. J. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1217 · Cited by 95
Will Machine Learning Tip the Balance in Breast Cancer Screening?
Trister, A. D., Buist, D. S. M., & Lee, C. I. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2017.0473 · Cited by 92
Self-citationMayo, R. C., Kent, D., Sen, L. Q. C., Kapoor, M. M., Leung, J. W. T., & Watanabe, A. T. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0168-6 · Cited by 87
Alternative models for sharing confidential biomedical data
Guinney, J., & Sáez-Rodríguez, J. (2018) · Nature Biotechnology · doi:10.1038/nbt.4128 · Cited by 72
Self-citationPredicting protein function and other biomedical characteristics with heterogeneous ensembles
Whalen, S., Pandey, O. P., & Pandey, G. S. (2015) · Methods · doi:10.1016/j.ymeth.2015.08.016 · Cited by 29
Self-citationChallenges: Crowdsourced solutions
Bender, E. (2016) · Nature · doi:10.1038/533s62a · Cited by 26
Artificial Intelligence for Breast Cancer Imaging: The New Frontier?
Lee, C. I., & Elmore, J. G. (2018) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djy223 · Cited by 14
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Kuhn, A Short Introduction to the Caret Package
- SAGE Bionetworks. Digital mammography DREAM challenge. https://sagebionetworks.org/research-projects/digital-mammography-dream-challenge/. Accessed January 22, 2020.
- SAGE Bionetworks. Synapse. https://www.synapse.org/. Accessed January 22, 2020.
Cited by
Sorted by citations
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 477
Lotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423
Self-citationThe Role of Artificial Intelligence in Early Cancer Diagnosis
Hunter, B., Hindocha, S., & Lee, R. W. (2022) · Cancers · doi:10.3390/cancers14061524 · Cited by 384
Sechopoulos, I., Teuwen, J., & Mann, R. M. (2020) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.06.002 · Cited by 376
Freeman, K., Geppert, J., Stinton, C., Todkill, D., Johnson, S. A., Clarke, A., & Taylor‐Phillips, S. (2021) · BMJ · doi:10.1136/bmj.n1872 · Cited by 365
Metadata
- Authors
- Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., Feng, J., Feng, M., Kim, H.-E., Albiol, F., Albiol, A., Morrell, S., Wojna, Z., Ahsen, M. E., Asif, U., Jimeno Yepes, A., Yohanandan, S., Rabinovici-Cohen, S., Yi, D., Hoff, B., Yu, T., Chaibub Neto, E., Rubin, D. L., Lindholm, P., Margolies, L. R., McBride, R. B., Rothstein, J. H., Sieh, W., Ben-Ari, R., Harrer, S., Trister, A., Friend, S., Norman, T., Sahiner, B., Strand, F., Guinney, J., Stolovitzky, G., and the DM DREAM Consortium, Mackey, L., Cahoon, J., Shen, L., Sohn, J. H., Trivedi, H., Shen, Y., Buturovic, L., Pereira, J. C., Cardoso, J. S., Castro, E., Kalleberg, K. T., Pelka, O., Nedjar, I., Geras, K. J., Nensa, F., Goan, E., Koitka, S., Caballero, L., Cox, D. D., Krishnaswamy, P., Pandey, G., Friedrich, C. M., Perrin, D., Fookes, C., Shi, B., Cardoso Negrie, G., Kawczynski, M., Cho, K., Khoo, C. S., Lo, J. Y., Sorensen, A. G., & Jung, H.
- Affiliations
- Sage Bionetworks (US)
- Kaiser Permanente Washington Health Research Institute (US)
- University of Washington (US)
- Therapixel, Paris, France
- Department of Physics of Complex Systems, ELTE Eötvös Loránd University, Budapest, Hungary
- University of Michigan (US)
- IBM (United States) (US)
- Tencent (China) (CN)
- National University of Singapore (SG)
- Synapxe (SG)
- National University Health System (SG)
- Lunit Inc, Seoul, Korea
- Consejo Superior de Investigaciones Científicas (ES)
- Universitat Politècnica de València (ES)
- University College London (GB)
- NortonLifeLock (United States) (US)
- University of Illinois Urbana-Champaign (US)
- IBM Research - Australia (AU)
- IBM Research Australia, Melbourne, Australia
- IBM Research Haifa, Haifa University Campus, Mount Carmel, Haifa, Israel
- Stanford University (US)
- Computational Oncology, Sage Bionetworks, Seattle, Washington
- Karolinska Institutet (SE)
- Icahn School of Medicine at Mount Sinai (US)
- Fred Hutch Cancer Center (US)
- Gates Foundation (US)
- United States Food and Drug Administration (US)
- Karolinska University Hospital (SE)
- IBM Research - Thomas J. Watson Research Center (US)
- Microsoft (United States) (US)
- North Carolina State University (US)
- University of California, San Francisco (US)
- Emory University (US)
- New York University (US)
- Clinical Persona, East Palo Alto, California
- Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal
- INESC TEC (PT)
- KolibriFX, Oslo, Norway
- Dortmund University of Applied Sciences and Arts (DE)
- University of Abou Bekr Belkaïd (DZ)
- NYU Langone Health (US)
- Essen University Hospital (DE)
- Queensland University of Technology (AU)
- MIT-IBM Computing Research Lab (US)
- Agency for Science, Technology and Research (SG)
- Duke University (US)
- Satalia, London, United Kingdom
- Korea University (KR)
- Corresponding author
- Stolovitzky, Gustavo
- ORCID
- Schaffter, Thomas
- Buist, Diana S. M.
- Lee, Christoph I.
- Nikulin, Yaroslav
- Guan, Yuanfang
- Lotter, William
- Jie, Zequn
- Du, Hao
- Wang, Sijia
- Feng, Jiashi
- Feng, Mengling
- Albiol, Francisco
- Albiol, Alberto
- Morrell, Stephen
- Wojna, Zbigniew
- Ahsen, Mehmet Eren
- Asif, Umar
- Yi, Darvin
- Hoff, Bruce
- Yu, Thomas
- Rubin, Daniel L.
- Lindholm, Peter
- Margolies, Laurie R.
- McBride, Russell Bailey
- Rothstein, Joseph H.
- Sieh, Weiva
- Harrer, Stefan
- Trister, Andrew
- Friend, Stephen
- Norman, Thea
- Sahiner, Berkman
- Strand, Fredrik
- Guinney, Justin
- Stolovitzky, Gustavo
- Mackey, Lester
- Cahoon, Joyce
- Shen, Li
- Sohn, Jae Ho
- Trivedi, Hari
- Shen, Yiqiu
- Cardoso, Jaime S.
- Castro, Eduardo
- Kalleberg, Karl Trygve
- Pelka, Obioma
- Nedjar, Imane
- Geras, Krzysztof J.
- Nensa, Felix
- Goan, Ethan
- Koitka, Sven
- Caballero, Luis
- Cox, David D.
- Krishnaswamy, Pavitra
- Pandey, Gaurav
- Friedrich, Christoph M.
- Perrin, Dimitri
- Fookes, Clinton
- Shi, Bibo
- Cho, Kyunghyun
- Lo, Joseph Y.
- Sorensen, A. Gregory
- Jung, Hwejin
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2020-03-02
- Volume
- 3
- Issue
- 3
- Pages
- e200265
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC7052735
- OpenAlex
- W3009692632
- Cited by
- 409
- References
- 32
- Citations by year
- 2026: 29 · 2025: 55 · 2024: 92 · 2023: 74 · 2022: 71 · 2021: 66 · 2020: 21
- FWCI
- 22.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DIGITAL.CSIC (Spanish National Research Council (CSIC)) (acceptedVersion, other-oa)
- QUT ePrints (Queensland University of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (1482827)
- National Cancer Institute (P01CA154292, R37CA240403)
- National Institute of General Medical Sciences (R35GM133346)
- National Institute of Biomedical Imaging and Bioengineering (P41EB017183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Digital Radiography and Breast Imaging
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- mammography screening, breast cancer detection, artificial intelligence, radiologist interpretation, ensemble learning, false positives, breast cancer risk factors
- MeSH terms
- Radiologists, Deep Learning, Adult, Aged, Algorithms, Artificial Intelligence, Breast Neoplasms, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Radiology, Sensitivity and Specificity, Sweden, United States, Early Detection of Cancer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
