CoChat
Sign Up
Cited by 420AI in cancer detectionDigital Radiography and Breast ImagingArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.

Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms

Open access
Schaffter, Thomas, Buist, Diana S. M., Lee, Christoph I., Nikulin, Yaroslav, Ribli, Dezso, Guan, Yuanfang, Lotter, William, Jie, Zequn, Du, Hao, Wang, Sijia, Feng, Jiashi, Feng, Mengling, Kim, Hyo-Eun, Albiol, Francisco, Albiol, Alberto, Morrell, Stephen, Wojna, Zbigniew, Ahsen, Mehmet Eren, Asif, Umar, Jimeno Yepes, Antonio, Yohanandan, Shivanthan, Rabinovici-Cohen, Simona, Yi, Darvin, Hoff, Bruce, Yu, Thomas, Chaibub Neto, Elias, Rubin, Daniel L., Lindholm, Peter, Margolies, Laurie R., McBride, Russell Bailey, Rothstein, Joseph H., Sieh, Weiva, Ben-Ari, Rami, Harrer, Stefan, Trister, Andrew, Friend, Stephen, Norman, Thea, Sahiner, Berkman, Strand, Fredrik, Guinney, Justin, Stolovitzky, Gustavo, and the DM DREAM Consortium, Mackey, Lester, Cahoon, Joyce, Shen, Li, Sohn, Jae Ho, Trivedi, Hari, Shen, Yiqiu, Buturovic, Ljubomir, Pereira, Jose Costa, Cardoso, Jaime S., Castro, Eduardo, Kalleberg, Karl Trygve, Pelka, Obioma, Nedjar, Imane, Geras, Krzysztof J., Nensa, Felix, Goan, Ethan, Koitka, Sven, Caballero, Luis, Cox, David D., Krishnaswamy, Pavitra, Pandey, Gaurav, Friedrich, Christoph M., Perrin, Dimitri, Fookes, Clinton, Shi, Bibo, Cardoso Negrie, Gerard, Kawczynski, Michael, Cho, Kyunghyun, Khoo, Can Son, Lo, Joseph Y., Sorensen, A. Gregory, Jung, Hwejin·2020·JAMA Network Open
DOI 10.1001/jamanetworkopen.2020.0265PMC PMC7052735
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2020.0265

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Training and Evaluation of Algorithms During the Digital Mammography DREAM Challenge Training and evaluation Kaiser Permamente Washington (KPW) and Karolinskia Institute data were not directly available to challenge participants; they were stored behind a firewall in a secure cloud (gray box). To access the data, participants submitted models to be run behind the firewall, in the graphics processing unit (GPU)–accelerated cloud (IBM). A, Training and evaluation of models submitted by teams during the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Challenge. B, A subset of the 8 best models in the evaluation KPW dataset were combined into the Challenge Ensemble Method (CEM), trained using the KPW training set and evaluated in the KPW and Karolinska Institute evaluation datasets. C, We developed a final ensemble method incorporating radiologists' interpretation in a method called CEM+radiologist (CEM+R). AUC indicates area under the curve; pAUC, partial area under the curve.
Figure 1. Training and Evaluation of Algorithms During the Digital Mammography DREAM Challenge Training and evaluation Kaiser Permamente Washington (KPW) and Karolinskia Institute data were not directly available to challenge participants; they were stored behind a firewall in a secure cloud (gray box). To access the data, participants submitted models to be run behind the firewall, in the graphics processing unit (GPU)–accelerated cloud (IBM). A, Training and evaluation of models submitted by teams during the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Challenge. B, A subset of the 8 best models in the evaluation KPW dataset were combined into the Challenge Ensemble Method (CEM), trained using the KPW training set and evaluated in the KPW and Karolinska Institute evaluation datasets. C, We developed a final ensemble method incorporating radiologists' interpretation in a method called CEM+radiologist (CEM+R). AUC indicates area under the curve; pAUC, partial area under the curve.CC BY · PubMed Central
Figure 2. Performance of the Algorithms Submitted at the End of the Competitive Phase Individual algorithm performance submitted at the end of the competitive phase on Kaiser Permamente Washington (KPW) and Karolinska Institute data. A, Area under the curve (AUC) and specificity computed at KPW radiologists' sensitivity of 85.9% of 31 methods submitted to the Digital Mammography Digital Mammography Dialogue on Reverse Engineering Assessment and Methods Challenge and evaluated on KPW evaluation set. B, The performance of methods is not significantly higher when clinical, demographic, and longitudinal data are provided. C-D, The AUC and specificity computed at Breast Cancer Surveillance Consortium’s sensitivity of 86.9% of methods evaluated on the KPW evaluation set generalize to the Karolinska Institute data.
Figure 2. Performance of the Algorithms Submitted at the End of the Competitive Phase Individual algorithm performance submitted at the end of the competitive phase on Kaiser Permamente Washington (KPW) and Karolinska Institute data. A, Area under the curve (AUC) and specificity computed at KPW radiologists' sensitivity of 85.9% of 31 methods submitted to the Digital Mammography Digital Mammography Dialogue on Reverse Engineering Assessment and Methods Challenge and evaluated on KPW evaluation set. B, The performance of methods is not significantly higher when clinical, demographic, and longitudinal data are provided. C-D, The AUC and specificity computed at Breast Cancer Surveillance Consortium’s sensitivity of 86.9% of methods evaluated on the KPW evaluation set generalize to the Karolinska Institute data.CC BY · PubMed Central
Figure 3. Receiver Operating Characteristic Curves of the Best Individual CEM and CEM+R Methods Receiver operating characteristic curves of the best individual method (orange), challenge ensemble method (CEM) (light blue), and challenge ensemble method + radiologist (CEM+R) method (dark blue) in Kaiser Permamente Washington (KPW) (A) and Karolinska Institute (KI) (B-C) data sets for single radiologist and consensus. The black cross reports the sensitivity and specificity achieved by the radiologist(s) in the corresponding cohort. AUC indicates area under the curve; FPR, false-positive rate; TPR, true-positive rate.
Figure 3. Receiver Operating Characteristic Curves of the Best Individual CEM and CEM+R Methods Receiver operating characteristic curves of the best individual method (orange), challenge ensemble method (CEM) (light blue), and challenge ensemble method + radiologist (CEM+R) method (dark blue) in Kaiser Permamente Washington (KPW) (A) and Karolinska Institute (KI) (B-C) data sets for single radiologist and consensus. The black cross reports the sensitivity and specificity achieved by the radiologist(s) in the corresponding cohort. AUC indicates area under the curve; FPR, false-positive rate; TPR, true-positive rate.CC BY · PubMed Central
Figure 4. Comparison of the Specificity of Radiologist(s) and CEM+R on Kaiser Permanente Washington (KPW) and Karolinska (KI) Data Comparison of the specificity of radiologist(s) and challenge ensemble method + radiologist (CEM+R) for different clinical/demographic conditions on KPW and KI data. For each condition, we report the CI of the specificity of radiologist (blue) and CEM+R (orange) computed at the sensitivity of radiologists. A best performing approach can be identified when the 2 CIs do not overlap. DCIS indicates ductal carcinoma in situ.
Figure 4. Comparison of the Specificity of Radiologist(s) and CEM+R on Kaiser Permanente Washington (KPW) and Karolinska (KI) Data Comparison of the specificity of radiologist(s) and challenge ensemble method + radiologist (CEM+R) for different clinical/demographic conditions on KPW and KI data. For each condition, we report the CI of the specificity of radiologist (blue) and CEM+R (orange) computed at the sensitivity of radiologists. A best performing approach can be identified when the 2 CIs do not overlap. DCIS indicates ductal carcinoma in situ.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 32 references

Research works printed but not linked: 3

  1. Kuhn, A Short Introduction to the Caret Package
  2. SAGE Bionetworks. Digital mammography DREAM challenge. https://sagebionetworks.org/research-projects/digital-mammography-dream-challenge/. Accessed January 22, 2020.
  3. SAGE Bionetworks. Synapse. https://www.synapse.org/. Accessed January 22, 2020.

Cited by

Sorted by citations

Items 301–400 of 420

Related

See all

Metadata

Authors
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., Feng, J., Feng, M., Kim, H.-E., Albiol, F., Albiol, A., Morrell, S., Wojna, Z., Ahsen, M. E., Asif, U., Jimeno Yepes, A., Yohanandan, S., Rabinovici-Cohen, S., Yi, D., Hoff, B., Yu, T., Chaibub Neto, E., Rubin, D. L., Lindholm, P., Margolies, L. R., McBride, R. B., Rothstein, J. H., Sieh, W., Ben-Ari, R., Harrer, S., Trister, A., Friend, S., Norman, T., Sahiner, B., Strand, F., Guinney, J., Stolovitzky, G., and the DM DREAM Consortium, Mackey, L., Cahoon, J., Shen, L., Sohn, J. H., Trivedi, H., Shen, Y., Buturovic, L., Pereira, J. C., Cardoso, J. S., Castro, E., Kalleberg, K. T., Pelka, O., Nedjar, I., Geras, K. J., Nensa, F., Goan, E., Koitka, S., Caballero, L., Cox, D. D., Krishnaswamy, P., Pandey, G., Friedrich, C. M., Perrin, D., Fookes, C., Shi, B., Cardoso Negrie, G., Kawczynski, M., Cho, K., Khoo, C. S., Lo, J. Y., Sorensen, A. G., & Jung, H.
Affiliations
  • Sage Bionetworks (US)
  • Kaiser Permanente Washington Health Research Institute (US)
  • University of Washington (US)
  • Therapixel, Paris, France
  • Department of Physics of Complex Systems, ELTE Eötvös Loránd University, Budapest, Hungary
  • University of Michigan (US)
  • IBM (United States) (US)
  • Tencent (China) (CN)
  • National University of Singapore (SG)
  • Synapxe (SG)
  • National University Health System (SG)
  • Lunit Inc, Seoul, Korea
  • Consejo Superior de Investigaciones Científicas (ES)
  • Universitat Politècnica de València (ES)
  • University College London (GB)
  • NortonLifeLock (United States) (US)
  • University of Illinois Urbana-Champaign (US)
  • IBM Research - Australia (AU)
  • IBM Research Australia, Melbourne, Australia
  • IBM Research Haifa, Haifa University Campus, Mount Carmel, Haifa, Israel
  • Stanford University (US)
  • Computational Oncology, Sage Bionetworks, Seattle, Washington
  • Karolinska Institutet (SE)
  • Icahn School of Medicine at Mount Sinai (US)
  • Fred Hutch Cancer Center (US)
  • Gates Foundation (US)
  • United States Food and Drug Administration (US)
  • Karolinska University Hospital (SE)
  • IBM Research - Thomas J. Watson Research Center (US)
  • Microsoft (United States) (US)
  • North Carolina State University (US)
  • University of California, San Francisco (US)
  • Emory University (US)
  • New York University (US)
  • Clinical Persona, East Palo Alto, California
  • Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal
  • INESC TEC (PT)
  • KolibriFX, Oslo, Norway
  • Dortmund University of Applied Sciences and Arts (DE)
  • University of Abou Bekr Belkaïd (DZ)
  • NYU Langone Health (US)
  • Essen University Hospital (DE)
  • Queensland University of Technology (AU)
  • MIT-IBM Computing Research Lab (US)
  • Agency for Science, Technology and Research (SG)
  • Duke University (US)
  • Satalia, London, United Kingdom
  • Korea University (KR)
Corresponding author
Stolovitzky, Gustavo
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2020-03-02
Volume
3
Issue
3
Pages
e200265
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2020.0265
PMC
PMC7052735
OpenAlex
W3009692632
Cited by
409
References
32
Citations by year
2026: 29 · 2025: 55 · 2024: 92 · 2023: 74 · 2022: 71 · 2021: 66 · 2020: 21
FWCI
22.80
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DIGITAL.CSIC (Spanish National Research Council (CSIC)) (acceptedVersion, other-oa)
  • QUT ePrints (Queensland University of Technology) (submittedVersion)
  • National University of Singapore (submittedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
Funders
  • Engineering and Physical Sciences Research Council (1482827)
  • National Cancer Institute (P01CA154292, R37CA240403)
  • National Institute of General Medical Sciences (R35GM133346)
  • National Institute of Biomedical Imaging and Bioengineering (P41EB017183)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Digital Radiography and Breast Imaging
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
mammography screening, breast cancer detection, artificial intelligence, radiologist interpretation, ensemble learning, false positives, breast cancer risk factors
MeSH terms
Radiologists, Deep Learning, Adult, Aged, Algorithms, Artificial Intelligence, Breast Neoplasms, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Radiology, Sensitivity and Specificity, Sweden, United States, Early Detection of Cancer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review