The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy
Open accessAbstract
Objective: To examine the accuracy of artificial intelligence (AI) for the detection of breast cancer in mammography screening practice. Design: Systematic review of test accuracy studies. Data sources: Medline, Embase, Web of Science, and Cochrane Database of Systematic Reviews from 1 January 2010 to 17 May 2021. Eligibility criteria: Studies reporting test accuracy of AI algorithms, alone or in combination with radiologists, to detect cancer in women’s digital mammograms in screening practice, or in test sets. Reference standard was biopsy with histology or follow-up (for screen negative women). Outcomes included test accuracy and cancer type detected. Study selection and synthesis: Two reviewers independently assessed articles for inclusion and assessed the methodological quality of included studies using the QUality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. A single reviewer extracted data, which were checked by a second reviewer. Narrative data synthesis was performed. Results: Twelve studies totalling 131 822 screened women were included. No prospective studies measuring test accuracy of AI in screening practice were found. Studies were of poor methodological quality. Three retrospective studies compared AI systems with the clinical decisions of the original radiologist, including 79 910 women, of whom 1878 had screen detected cancer or interval cancer within 12 months of screening. Thirty four (94%) of 36 AI systems evaluated in these studies were less accurate than a single radiologist, and all were less accurate than consensus of two or more radiologists. Five smaller studies (1086 women, 520 cancers) at high risk of bias and low generalisability to the clinical context reported that all five evaluated AI systems (as standalone to replace radiologist or as a reader aid) were more accurate than a single radiologist reading a test set in the laboratory. In three studies, AI used for triage screened out 53%, 45%, and 50% of women at low risk but also 10%, 4%, and 0% of cancers detected by radiologists. Conclusions: Current evidence for AI does not yet allow judgement of its accuracy in breast cancer screening programmes, and it is unclear where on the clinical pathway AI might be of most benefit. AI systems are not sufficiently specific to replace radiologist double reading in screening programmes. Promising results in smaller studies are not replicated in larger studies. Prospective studies are required to measure the effect of AI in clinical practice. Such studies will require clear stopping rules to ensure that AI does not reduce programme specificity. Study registration: Protocol registered as PROSPERO CRD42020213590.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“there is little high quality evidence for improved clinician performance or patient outcomes in clinical studies.3 4”
From Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI (2022) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4_9 · Cited by 12,516
Global Burden of Disease Cancer Collaboration, Fitzmaurice, C., Allen, C., Barber, R. M., Barregard, L., Bhutta, Z. A., Brenner, H., Dicker, D. J., Chimed-Orchir, O., Dandona, R., et al. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2016.5688 · Cited by 6,215
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
McInnes, M. D. F., Moher, D., Thombs, B. D., McGrath, T. A., Bossuyt, P. M., and the PRISMA-DTA Group, Clifford, T., Cohen, J. F., Deeks, J. J., Gatsonis, C., et al. (2018) · JAMA · doi:10.1001/jama.2017.19163 · Cited by 3,210
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773
Morrison, A., Polisena, J., Husereau, D., Moulton, K., Clark, M., Fiander, M., Mierzwinski-Urban, M., Clifford, T., Hutton, B., & Rabb, D. (2012) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462312000086 · Cited by 1,257
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224
Ethical and legal challenges of artificial intelligence-driven healthcare
Gerke, S., Minssen, T., & Cohen, G. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00012-5 · Cited by 1,170
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Youlden, D. R., Cramb, S. M., Dunn, N. A. M., Muller, J. M., Pyke, C. M., & Baade, P. D. (2012) · Cancer Epidemiology · doi:10.1016/j.canep.2012.02.007 · Cited by 709
Rodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Tan, T., Mertelmeier, T., et al. (2019) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djy222 · Cited by 692
Deep Learning Is Effective for Classifying Normal versus Age-Related Macular Degeneration OCT Images
Lee, C. S., Baughman, D. M., & Lee, A. Y. (2017) · Ophthalmology Retina · doi:10.1016/j.oret.2016.12.009 · Cited by 657
Deep learning ensembles for melanoma recognition in dermoscopy images
Codella, N. C. F., Nguyen, Q.-B., Pankanti, S., Gutman, D. A., Helba, B., Halpern, A. C., & Smith, J. R. (2017) · IBM Journal of Research and Development · doi:10.1147/jrd.2017.2708299 · Cited by 644
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System
Rodríguez-Ruiz, A., Krupinski, E., Mordang, J.-J., Schilling, K., Heywang-Köbrunner, S. H., Sechopoulos, I., & Mann, R. M. (2019) · Radiology · doi:10.1148/radiol.2018181371 · Cited by 602
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W., Westwood, M. E., Mallett, S. V., Deeks, J. J., Reitsma, J. B., Leeflang, M., Sterne, J. A. C., & Bossuyt, P. M. (2011) · Data Archiving and Networked Services (DANS) · doi:10.1059/0003-4819-155-8-201110180-00009 · Cited by 601
Case–Control and Two-Gate Designs in Diagnostic Accuracy Studies
Rutjes, A. W., Reitsma, J. B., Vandenbroucke, J. P., Glas, A. S., & Bossuyt, P. M. (2005) · Clinical Chemistry · doi:10.1373/clinchem.2005.048595 · Cited by 501
McBee, M. P., Awan, O. A., Colucci, A. T., Ghobadi, C. W., Kadom, N., Kansagra, A. P., Tridandapani, S., & Auffermann, W. F. (2018) · Academic Radiology · doi:10.1016/j.acra.2018.02.018 · Cited by 496
McDonald, R. J., Schwartz, K. M., Eckel, L. J., Diehn, F. E., Hunt, C. H., Bartholmai, B. J., Erickson, B. J., & Kallmes, D. F. (2015) · Academic Radiology · doi:10.1016/j.acra.2015.05.007 · Cited by 460
Nussbaumer-Streit, B., Klerings, I., Dobrescu, A. I., Persad, E., Stevens, A., Garritty, C., Kamel, C., Affengruber, L., King, V. J., & Gartlehner, G. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.10.011 · Cited by 440
Lotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., Boyer, Y., Boxerman, J. L., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-01174-9 · Cited by 423
Glaucoma detection based on deep convolutional neural network
Chen, X., Xu, Y., Kee Wong, D. W., Wong, T. Y., & Liu, J. (2015) · 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7318462 · Cited by 412
Pourbabaee, B., Roshtkhari, M. J., & Khorasani, K. (2018) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2017.2705582 · Cited by 411
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.0265 · Cited by 409
Artificial intelligence for diabetic retinopathy screening: a review
Grzybowski, A., Brona, P., Lim, G., Ruamviboonsuk, P., Tan, G. S. W., Abramoff, M., & Ting, D. S. W. (2020) · Eye · doi:10.1038/s41433-019-0566-0 · Cited by 376 · cited 1×
“47 48 Beyond conventional screening programmes, the use of deep learning in medicine is increasing, and has been considered in the diagnosis of melanoma, 49 ophthalmic diseases (age-related macular degeneration 50 and glaucoma 51 ), and in interpretation of histological, 52 radiological, 53 and…”
BackgroundCarter, S. M., Rogers, W., Win, K. T., Frazer, H., Richards, B., & Houssami, N. (2020) · The Breast · doi:10.1016/j.breast.2019.10.001 · Cited by 312
Dembrower, K., Wåhlin, E., Liu, Y., Salim, M., Smith, K., Lindholm, P., Eklund, M., & Strand, F. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30185-0 · Cited by 290
Salim, M., Wåhlin, E., Dembrower, K., Azavedo, E., Foukakis, T., Liu, Y., Smith, K., Eklund, M., & Strand, F. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.3321 · Cited by 288
Automated Diabetic Retinopathy Image Assessment Software
Tufail, A., Rudisill, C., Egan, C., Kapetanakis, V. V., Salas-Vega, S., Owen, C. G., Lee, A., Louw, V., Anderson, J., Liew, G., et al. (2017) · Ophthalmology · doi:10.1016/j.ophtha.2016.11.014 · Cited by 283
Houssami, N., & Hunter, K. (2017) · npj Breast Cancer · doi:10.1038/s41523-017-0014-x · Cited by 223
Gur, D., Bandos, A. I., Cohen, C. S., Hakim, C. M., Hardesty, L. A., Ganott, M. A., Perrin, R. L., Poller, W. R., Shah, R., Sumkin, J. H., et al. (2008) · Radiology · doi:10.1148/radiol.2491072025 · Cited by 186
Houssami, N., Kirkpatrick-Jones, G., Noguchi, N., & Lee, C. I. (2019) · Expert Review of Medical Devices · doi:10.1080/17434440.2019.1610387 · Cited by 185
Heydon, P., Egan, C., Bolter, L., Chambers, R., Anderson, J., Aldington, S., Stratton, I. M., Scanlon, P. H., Webster, L., Mann, S., et al. (2021) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2020-316594 · Cited by 183
Raya-Povedano, J. L., Romero-Martín, S., Elías-Cabot, E., Gubern-Mérida, A., Rodríguez-Ruiz, A., & Álvarez-Benito, M. (2021) · Radiology · doi:10.1148/radiol.2021203555 · Cited by 177
Pacilè, S., Lopez, J., Chone, P., Bertinotti, T., Grouin, J. M., & Fillard, P. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190208 · Cited by 161
Yen, M.-F., Tabár, L. K., Viták, B., Smith, R. A., Chen, H.-H., & Duffy, S. W. (2003) · European Journal of Cancer · doi:10.1016/s0959-8049(03)00260-0 · Cited by 152
The End of Radiology? Three Threats to the Future Practice of Radiology
Chockley, K., & Emanuel, E. (2016) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2016.07.010 · Cited by 139
Nielsen, K. B., Lautrup, M. L., Andersen, J., Savarimuthu, T. R., & Grauslund, J. (2018) · Ophthalmology Retina · doi:10.1016/j.oret.2018.10.014 · Cited by 133
Identifying normal mammograms in a large screening population using artificial intelligence
Lång, K., Dustler, M., Dahlblom, V., Åkesson, A., Andersson, I., & Zackrisson, S. (2020) · European Radiology · doi:10.1007/s00330-020-07165-1 · Cited by 131
Breast Cancer Surveillance Consortium
National Cancer Institute (2020) · Definitions · doi:10.32388/7oifu8 · Cited by 100
Watanabe, A. T., Lim, V., Vu, H., Chim, C. Y., Weise, E., Liu, J., Bradley, W. G., & Comstock, C. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00192-5 · Cited by 96
van Luijt, P. A., Heijnsdijk, E. A. M., Fracheboud, J., Overbeek, L. I. H., Broeders, M. J. M., Wesseling, J., den Heeten, G. J., & de Koning, H. J. (2016) · Breast Cancer Research · doi:10.1186/s13058-016-0705-5 · Cited by 91
Interval cancers in the NHS breast cancer screening programme in England, Wales and Northern Ireland
Bennett, R. L., Sellars, S., & Moss, S. M. (2011) · British Journal of Cancer · doi:10.1038/bjc.2011.3 · Cited by 90
Thomas, E. T., Mar, C. D., Glasziou, P., Wright, R. G., Barratt, A. L., & Bell, K. (2017) · BMC Cancer · doi:10.1186/s12885-017-3808-1 · Cited by 66
The AI Index 2021 Annual Report
Zhang, D., Mishra, S., Brynjolfsson, E., Etchemendy, J., Ganguli, D., Grosz, B., Lyons, T., Manyika, J., Niebles, J. C., Sellitto, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.06312 · Cited by 48
Effect of Mammography Screening on Mortality by Histological Grade
Tabár, L. K., Chen, T. H.-H., Yen, A. M., Chen, S. L., Fann, J. C., Chiu, S. Y., Ku, M. M., Wu, W. Y.-Y., Hsu, C.-Y., Chen, Y.-Y., et al. (2017) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-17-0487 · Cited by 43
Hupse, R., Samulski, M. R. M., Lobbes, M. B. I., den Heeten, A., Tas, M. W., Beijerinck, D., Pijnappel, R. M., Boetes, C., & Karssemeijer, N. (2012) · European Radiology · doi:10.1007/s00330-012-2562-7 · Cited by 23
Sharma, N., Ng, A., James, J. J., Khara, G. S., Ambrózay, É., Austin, C. C., Forrai, G., Fox, G., Glocker, B., Heindl, A., et al. (2021) · medRxiv · doi:10.1101/2021.02.26.21252537 · Cited by 18
Rodríguez‐Ruiz, A., Mordang, J., Karssemeijer, N., Sechopoulos, I., & Mann, R. M. (2018) · International Workshop on Breast Imaging (IWBI) · doi:10.1117/12.2317937 · Cited by 12
Evaluating AI in breast cancer screening: a complex task
Dustler, M. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30019-4 · Cited by 10
Cited by
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665
Lång, K., Josefsson, V., Larsson, A.-M., Larsson, S., Högberg, C., Sartor, H., Hofvind, S., Andersson, I., & Rosso, A. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(23)00298-x · Cited by 598
Dembrower, K., Crippa, A., Colón, E., Eklund, M., & Strand, F. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00153-x · Cited by 358
The Role of Artificial Intelligence in Early Cancer Diagnosis
Hunter, B., Hindocha, S., & Lee, R. W. (2022) · Cancers · doi:10.3390/cancers14061524 · Cited by 332
Metadata
- Authors
- Freeman, K., Geppert, J., Stinton, C., Todkill, D., Johnson, S., Clarke, A., & Taylor-Phillips, S.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2021-09-01
- Volume
- 374
- Pages
- n1872
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W3197840494
- Cited by
- 338
- References
- 56
- FWCI
- 5.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.