A Deep Learning Model to Triage Screening Mammograms: A Simulation Study
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Background Recent deep learning (DL) approaches have shown promise in improving sensitivity but have not addressed limitations in radiologist specificity or efficiency. Purpose To develop a DL model to triage a portion of mammograms as cancer free, improving performance and workflow efficiency. Materials and Methods In this retrospective study, 223 109 consecutive screening mammograms performed in 66 661 women from January 2009 to December 2016 were collected with cancer outcomes obtained through linkage to a regional tumor registry. This cohort was split by patient into 212 272, 25 999, and 26 540 mammograms from 56 831, 7021, and 7176 patients for training, validation, and testing, respectively. A DL model was developed to triage mammograms as cancer free and evaluated on the test set. A DL-triage workflow was simulated in which radiologists skipped mammograms triaged as cancer free (interpreting them as negative for cancer) and read mammograms not triaged as cancer free by using the original interpreting radiologists’ assessments. Sensitivities, specificities, and percentage of mammograms read were calculated, with and without the DL-triage–simulated workflow. Statistics were computed across 5000 bootstrap samples to assess confidence intervals (CIs). Specificities were compared by using a two-tailed t test (P < .05) and sensitivities were compared by using a one-sided t test with a noninferiority margin of 5% (P < .05). Results The test set included 7176 women (mean age, 57.8 years ± 10.9 [standard deviation]). When reading all mammograms, radiologists obtained a sensitivity and specificity of 90.6% (173 of 191; 95% CI: 86.6%, 94.7%) and 93.5% (24 625 of 26 349; 95% CI: 93.3%, 93.9%). In the DL-simulated workflow, the radiologists obtained a sensitivity and specificity of 90.1% (172 of 191; 95% CI: 86.0%, 94.3%) and 94.2% (24 814 of 26 349; 95% CI: 94.0%, 94.6%) while reading 80.7% (21 420 of 26 540) of the mammograms. The simulated workflow improved specificity (P = .002) and obtained a noninferior sensitivity with a margin of 5% (P < .001). Conclusion This deep learning model has the potential to reduce radiologist workload and significantly improve specificity without harming sensitivity. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Kontos and Conant in this issue.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [106]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In brief, we implemented our model as a deep convolutional neural network (ResNet18 [22]) with PyTorch (version 0.31; https://pytorch.org ).”
MethodInfluentialREDUCTION IN MORTALITY FROM BREAST CANCER AFTER MASS SCREENING WITH MAMMOGRAPHY
Tabár, L., Gad, A., Holmberg, L. H., Ljungquist, U., Fagerberg, C. J. G., Baldetorp, L., Gröntoft, O., Lundström, B., Månson, J. C., Eklund, G., et al. (1985) · The Lancet · doi:10.1016/s0140-6736(85)92204-4 · Cited by 1,781
The benefits and harms of breast cancer screening: an independent review
(2012) · The Lancet · doi:10.1016/s0140-6736(12)61611-0 · Cited by 1,492
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Self-citationLehman, C. D., Arao, R. F., Sprague, B. L., Lee, J. M., Buist, D. S. M., Kerlikowske, K., Henderson, L. M., Onega, T., Tosteson, A. N. A., Rauscher, G. H., et al. (2017) · Radiology · doi:10.1148/radiol.2016161174 · Cited by 665
Self-citationInfluence of Computer-Aided Detection on Performance of Screening Mammography
Fenton, J. J., Taplin, S. H., Carney, P. A., Abraham, L., Sickles, E. A., D'Orsi, C., Berns, E. A., Cutter, G., Hendrick, R. E., Barlow, W. E., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa066099 · Cited by 619
Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System
Rodríguez-Ruiz, A., Krupinski, E., Mordang, J.-J., Schilling, K., Heywang-Köbrunner, S. H., Sechopoulos, I., & Mann, R. M. (2019) · Radiology · doi:10.1148/radiol.2018181371 · Cited by 602 · cited 3×
“Although traditional approaches have not demonstrated improved radiologist performance in sensitivity or specificity in clinical practice (14–16), a more recent deep learning approach to CAD has shown promise in improving sensitivity in a reader study (17).”
BackgroundMethodField, C. A., & Welsh, A. H. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00593.x · Cited by 426 · cited 1×
“To account for patients appearing in our test set multiple times, we used the cluster bootstrap procedure (23).”
MethodMammographic Breast Density Assessment Using Deep Learning: Clinical Implementation
Lehman, C. D., Yala, A., Schuster, T., Dontchos, B., Bahl, M., Swanson, K., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2018180694 · Cited by 294 · cited 1×
“Mammograms from 60 886 of the 80 818 women in our patient population were previously reported (20,21).”
Self-citationRodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Teuwen, J., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Mertelmeier, T., et al. (2019) · European Radiology · doi:10.1007/s00330-019-06186-9 · Cited by 210 · cited 2×
“Recall rates from screening mammography have increased steadily concurrently explored in a recent reader study by Rodriguez-Ruiz et al (24), which used a commercial system and showed a simulated workload reduction of 17% with a drop of 1% in sensitivity and a noninferior radiologist AUC of 0.05 on an enriched data set.”
MethodInfluential
Research works printed but not linked: 3
- Shapiro S, 1982, J Natl Cancer Inst, 69, 349
- Miller AB, 1981, Clin Invest Med, 4, 227
- Yala A, 2019, Radiology, 182716
Cited by
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“Workflow improvements include prioritizing worklists for radiologists (2,3), triaging screening mammograms (4), reducing or eliminating gadolinium-based contrast media for MRI (5,6), and reducing the radiation dose of CT imaging by advancing image noise reduction (7–9).”
BackgroundDiagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Swanson, K., Wu, E., Zhang, A., Alizadeh, A. A., & Zou, J. (2023) · Cell · doi:10.1016/j.cell.2023.01.035 · Cited by 597
“, a ResNet-18 model was built to triage mammograms by setting a high-sensitivity prediction threshold so that nearly all predicted negative caseswere truly negative.(36) In a simulation study, these predicted negative cases were skipped by radiologists, allowing radiologists to only read 80.”
MethodLotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423
Sechopoulos, I., Teuwen, J., & Mann, R. (2021) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.06.002 · Cited by 341
“The same pre-selection strategy was subsequently investigated by Yala et al., using another deep learning-based system specifically developed to classify screening DM cases as cancer free , and therefore to not be human-interpreted, or not [82].”
Method
Metadata
- Authors
- Yala, A., Schuster, T., Miles, R., Barzilay, R., & Lehman, C.
- Affiliations
- Harvard University (US)
- Venue
- Radiology
- Published
- 2019-10
- Volume
- 293
- Issue
- 1
- Pages
- 38-46
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2964642168
- Cited by
- 210
- References
- 22
- Citations by year
- 2026: 13 · 2025: 33 · 2024: 31 · 2023: 37 · 2022: 31 · 2021: 36 · 2020: 26 · 2019: 3
- FWCI
- 10.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Breast Cancer Alliance
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Digital Radiography and Breast Imaging
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- deep learning, mammography screening, triage, breast cancer screening, sensitivity and specificity, breast cancer detection, simulation study, non-inferiority margin, bootstrap confidence intervals, breast cancer mortality reduction, cost-effectiveness, false positive reduction
- MeSH terms
- Deep Learning, Adult, Aged, Aged, 80 and over, Breast, Breast Neoplasms, Computer Simulation, Female, Humans, Image Interpretation, Computer-Assisted, Mammography, Middle Aged, Registries, Retrospective Studies, Triage, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Deep Learning Model to Triage Screening Mammograms: A Simulation Study. https://app.cochat.ai/papers/10.1148/radiol.2019182908
- The paper
- Yala, A., Schuster, T., Miles, R., Barzilay, R., & Lehman, C. (2019). A Deep Learning Model to Triage Screening Mammograms: A Simulation Study. Radiology. https://doi.org/10.1148/radiol.2019182908
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.