CoChat
Sign Up
Cited by 420AI in cancer detectionDigital Radiography and Breast ImagingArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.

Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms

Open access
Schaffter, Thomas, Buist, Diana S. M., Lee, Christoph I., Nikulin, Yaroslav, Ribli, Dezso, Guan, Yuanfang, Lotter, William, Jie, Zequn, Du, Hao, Wang, Sijia, Feng, Jiashi, Feng, Mengling, Kim, Hyo-Eun, Albiol, Francisco, Albiol, Alberto, Morrell, Stephen, Wojna, Zbigniew, Ahsen, Mehmet Eren, Asif, Umar, Jimeno Yepes, Antonio, Yohanandan, Shivanthan, Rabinovici-Cohen, Simona, Yi, Darvin, Hoff, Bruce, Yu, Thomas, Chaibub Neto, Elias, Rubin, Daniel L., Lindholm, Peter, Margolies, Laurie R., McBride, Russell Bailey, Rothstein, Joseph H., Sieh, Weiva, Ben-Ari, Rami, Harrer, Stefan, Trister, Andrew, Friend, Stephen, Norman, Thea, Sahiner, Berkman, Strand, Fredrik, Guinney, Justin, Stolovitzky, Gustavo, and the DM DREAM Consortium, Mackey, Lester, Cahoon, Joyce, Shen, Li, Sohn, Jae Ho, Trivedi, Hari, Shen, Yiqiu, Buturovic, Ljubomir, Pereira, Jose Costa, Cardoso, Jaime S., Castro, Eduardo, Kalleberg, Karl Trygve, Pelka, Obioma, Nedjar, Imane, Geras, Krzysztof J., Nensa, Felix, Goan, Ethan, Koitka, Sven, Caballero, Luis, Cox, David D., Krishnaswamy, Pavitra, Pandey, Gaurav, Friedrich, Christoph M., Perrin, Dimitri, Fookes, Clinton, Shi, Bibo, Cardoso Negrie, Gerard, Kawczynski, Michael, Cho, Kyunghyun, Khoo, Can Son, Lo, Joseph Y., Sorensen, A. Gregory, Jung, Hwejin·2020·JAMA Network Open
DOI 10.1001/jamanetworkopen.2020.0265PMC PMC7052735
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2020.0265

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 32 references

Research works printed but not linked: 3

  1. Kuhn, A Short Introduction to the Caret Package
  2. SAGE Bionetworks. Digital mammography DREAM challenge. https://sagebionetworks.org/research-projects/digital-mammography-dream-challenge/. Accessed January 22, 2020.
  3. SAGE Bionetworks. Synapse. https://www.synapse.org/. Accessed January 22, 2020.

Cited by

Sorted by citations

Items 401–420 of 420

Related

See all

Metadata

Authors
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., Feng, J., Feng, M., Kim, H.-E., Albiol, F., Albiol, A., Morrell, S., Wojna, Z., Ahsen, M. E., Asif, U., Jimeno Yepes, A., Yohanandan, S., Rabinovici-Cohen, S., Yi, D., Hoff, B., Yu, T., Chaibub Neto, E., Rubin, D. L., Lindholm, P., Margolies, L. R., McBride, R. B., Rothstein, J. H., Sieh, W., Ben-Ari, R., Harrer, S., Trister, A., Friend, S., Norman, T., Sahiner, B., Strand, F., Guinney, J., Stolovitzky, G., and the DM DREAM Consortium, Mackey, L., Cahoon, J., Shen, L., Sohn, J. H., Trivedi, H., Shen, Y., Buturovic, L., Pereira, J. C., Cardoso, J. S., Castro, E., Kalleberg, K. T., Pelka, O., Nedjar, I., Geras, K. J., Nensa, F., Goan, E., Koitka, S., Caballero, L., Cox, D. D., Krishnaswamy, P., Pandey, G., Friedrich, C. M., Perrin, D., Fookes, C., Shi, B., Cardoso Negrie, G., Kawczynski, M., Cho, K., Khoo, C. S., Lo, J. Y., Sorensen, A. G., & Jung, H.
Affiliations
  • Sage Bionetworks (US)
  • Kaiser Permanente Washington Health Research Institute (US)
  • University of Washington (US)
  • Therapixel, Paris, France
  • Department of Physics of Complex Systems, ELTE Eötvös Loránd University, Budapest, Hungary
  • University of Michigan (US)
  • IBM (United States) (US)
  • Tencent (China) (CN)
  • National University of Singapore (SG)
  • Synapxe (SG)
  • National University Health System (SG)
  • Lunit Inc, Seoul, Korea
  • Consejo Superior de Investigaciones Científicas (ES)
  • Universitat Politècnica de València (ES)
  • University College London (GB)
  • NortonLifeLock (United States) (US)
  • University of Illinois Urbana-Champaign (US)
  • IBM Research - Australia (AU)
  • IBM Research Australia, Melbourne, Australia
  • IBM Research Haifa, Haifa University Campus, Mount Carmel, Haifa, Israel
  • Stanford University (US)
  • Computational Oncology, Sage Bionetworks, Seattle, Washington
  • Karolinska Institutet (SE)
  • Icahn School of Medicine at Mount Sinai (US)
  • Fred Hutch Cancer Center (US)
  • Gates Foundation (US)
  • United States Food and Drug Administration (US)
  • Karolinska University Hospital (SE)
  • IBM Research - Thomas J. Watson Research Center (US)
  • Microsoft (United States) (US)
  • North Carolina State University (US)
  • University of California, San Francisco (US)
  • Emory University (US)
  • New York University (US)
  • Clinical Persona, East Palo Alto, California
  • Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal
  • INESC TEC (PT)
  • KolibriFX, Oslo, Norway
  • Dortmund University of Applied Sciences and Arts (DE)
  • University of Abou Bekr Belkaïd (DZ)
  • NYU Langone Health (US)
  • Essen University Hospital (DE)
  • Queensland University of Technology (AU)
  • MIT-IBM Computing Research Lab (US)
  • Agency for Science, Technology and Research (SG)
  • Duke University (US)
  • Satalia, London, United Kingdom
  • Korea University (KR)
Corresponding author
Stolovitzky, Gustavo
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2020-03-02
Volume
3
Issue
3
Pages
e200265
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2020.0265
PMC
PMC7052735
OpenAlex
W3009692632
Cited by
409
References
32
Citations by year
2026: 29 · 2025: 55 · 2024: 92 · 2023: 74 · 2022: 71 · 2021: 66 · 2020: 21
FWCI
22.80
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DIGITAL.CSIC (Spanish National Research Council (CSIC)) (acceptedVersion, other-oa)
  • QUT ePrints (Queensland University of Technology) (submittedVersion)
  • National University of Singapore (submittedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
Funders
  • Engineering and Physical Sciences Research Council (1482827)
  • National Cancer Institute (P01CA154292, R37CA240403)
  • National Institute of General Medical Sciences (R35GM133346)
  • National Institute of Biomedical Imaging and Bioengineering (P41EB017183)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Digital Radiography and Breast Imaging
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
mammography screening, breast cancer detection, artificial intelligence, radiologist interpretation, ensemble learning, false positives, breast cancer risk factors
MeSH terms
Radiologists, Deep Learning, Adult, Aged, Algorithms, Artificial Intelligence, Breast Neoplasms, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Radiology, Sensitivity and Specificity, Sweden, United States, Early Detection of Cancer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review