The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): An international, multicenter analysis
Open accessAbstract
OBJECTIVES: AI-based algorithms for medical image analysis showed comparable performance to human image readers. However, in practice, diagnoses are made using multiple imaging modalities alongside other data sources. We determined the importance of this multi-modal information and compared the diagnostic performance of routine breast cancer diagnosis to breast ultrasound interpretations by humans or AI-based algorithms. METHODS: Patients were recruited as part of a multicenter trial (NCT02638935). The trial enrolled 1288 women undergoing routine breast cancer diagnosis (multi-modal imaging, demographic, and clinical information). Three physicians specialized in ultrasound diagnosis performed a second read of all ultrasound images. We used data from 11 of 12 study sites to develop two machine learning (ML) algorithms using unimodal information (ultrasound features generated by the ultrasound experts) to classify breast masses which were validated on the remaining study site. The same ML algorithms were subsequently developed and validated on multi-modal information (clinical and demographic information plus ultrasound features). We assessed performance using area under the curve (AUC). RESULTS: Of 1288 breast masses, 368 (28.6%) were histopathologically malignant. In the external validation set (n = 373), the performance of the two unimodal ultrasound ML algorithms (AUC 0.83 and 0.82) was commensurate with performance of the human ultrasound experts (AUC 0.82 to 0.84; p for all comparisons > 0.05). The multi-modal ultrasound ML algorithms performed significantly better (AUC 0.90 and 0.89) but were statistically inferior to routine breast cancer diagnosis (AUC 0.95, p for all comparisons ≤ 0.05). CONCLUSIONS: The performance of humans and AI-based algorithms improves with multi-modal information. KEY POINTS: • The performance of humans and AI-based algorithms improves with multi-modal information. • Multimodal AI-based algorithms do not necessarily outperform expert humans. • Unimodal AI-based algorithms do not represent optimal performance to classify breast masses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Missing clinical context in unimodal approaches only focusing on image analysis was a limitation to AI usage in breast US.
“Limitations to AI usage in breast US were the missing clinical context in unimodal approaches only focusing on image analysis [202]”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570 · see citing works
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883 · see citing works
Machine learning in medicine: a practical introduction
Sidey-Gibbons, J. A. M., & Sidey-Gibbons, C. J. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0681-4 · Cited by 1,308 · see citing works
Self-citationHow to Read Articles That Use Machine Learning
Liu, Y., Chen, P.-H. C., Krause, J., & Peng, L. H. (2019) · JAMA · doi:10.1001/jama.2019.16489 · Cited by 530 · see citing works
McDonald, R. J., Schwartz, K. M., Eckel, L. J., Diehn, F. E., Hunt, C. H., Bartholmai, B. J., Erickson, B. J., & Kallmes, D. F. (2015) · Academic Radiology · doi:10.1016/j.acra.2015.05.007 · Cited by 460 · see citing works
Probabilistic prediction in patient management and clinical trials
Spiegelhalter, D. J. (1986) · Statistics in Medicine · doi:10.1002/sim.4780050506 · Cited by 362 · see citing works
Becker, A. S., Marcon, M., Ghafoor, S., Wurnig, M. C., Frauenfelder, T., & Boss, A. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000358 · Cited by 352 · see citing works
Byra, M., Galperin, M., Ojeda‐Fournier, H., Olson, L., O'Boyle, M., Comstock, C., & Andre, M. (2019) · Medical Physics · doi:10.1002/mp.13361 · Cited by 253 · see citing works
Fujioka, T., Kubota, K., Mori, M., Kikuchi, Y., Katsuta, L., Kasahara, M., Oda, G., Ishiba, T., Nakagawa, T., & Tateishi, U. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-019-00831-5 · Cited by 177 · see citing works
Becker, A. S., Mueller, M., Stoffel, E., Marcon, M., Ghafoor, S., & Boss, A. (2018) · The British Journal of Radiology · doi:10.1259/bjr.20170576 · Cited by 157 · see citing works
Lin, W., Hasenstab, K. A., Cunha, G. M., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-77264-y · Cited by 121 · see citing works
Choi, J. S., Han, B.-K., Ko, E. S., Bae, J. M., Ko, E. Y., Song, S. H., Kwon, M.-R., Shin, J. H., & Hahn, S. Y. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2018.0530 · Cited by 104 · see citing works
Performance of ultrasonography screening for breast cancer: a systematic review and meta-analysis
Yang, L., Wang, S., Zhang, L., Sheng, C., Song, F., Wang, P., & Huang, Y. (2020) · BMC Cancer · doi:10.1186/s12885-020-06992-1 · Cited by 102 · see citing works
Is the false-positive rate in mammography in North America too high?
Le, M., Mothersill, C. E., Seymour, C. B., et al. (2016) · British Journal of Radiology · doi:10.1259/bjr.20160045 · Cited by 92 · see citing works
Wöckel, A., Festl, J., Stüber, T. N., Brust, K., Stangl, S., Heuschmann, P. U., Albert, U., Budach, W., Follmann, M., Janni, W. J., et al. (2018) · Geburtshilfe und Frauenheilkunde · doi:10.1055/a-0646-4522 · Cited by 91 · see citing works
Self-citationGolatta, M., Franz, D., Harcos, A., et al. (2013) · European Journal of Radiology · doi:10.1016/j.ejrad.2013.03.005 · Cited by 74 · see citing works
Self-citationTowards Patient-centered Decision-making in Breast Cancer Surgery
Pfob, A., Mehrara, B. J., Nelson, J. A., et al. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000004862 · Cited by 73 · see citing works
Self-citationPfob, A., Sidey-Gibbons, C. J., Lee, H., Tasoulis, M. K., Koelbel, V., Golatta, M., Rauch, G. M., Smith, B. D., Valero, V., Han, W., et al. (2020) · European Journal of Cancer · doi:10.1016/j.ejca.2020.11.006 · Cited by 72 · see citing works
Self-citationSidey-Gibbons, C. J., Pfob, A., Asaad, M., Boukovalas, S., Lin, Y.-L., Selber, J. C., Butler, C. E., & Offodile, A. C. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00088 · Cited by 60 · see citing works
Self-citationBreast Nodules Computer-Aided Diagnostic System Design Using Fuzzy Cerebellar Model Neural Networks
Lin, C., Hou, Y.-L., Chen, T.-Y., et al. (2013) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2013.2269149 · Cited by 56 · see citing works
Stoffel, E., Becker, A. S., Wurnig, M. C., Marcon, M., Ghafoor, S., Berger, N., & Boss, A. (2018) · European Journal of Radiology Open · doi:10.1016/j.ejro.2018.09.002 · Cited by 47 · see citing works
Youk, J. H., Jung, I., Yoon, J. H., et al. (2016) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2016.05.006 · Cited by 29 · see citing works
Golatta, M., Pfob, A., Büsch, C. A., et al. (2021) · Ultraschall in der Medizin - European Journal of Ultrasound · doi:10.1055/a-1543-6156 · Cited by 24 · see citing works
Self-citationEvaluation of the FUSION-X-US-II prototype to combine automated breast ultrasound and tomosynthesis
Schäfgen, B., Juskic, M., Radicke, M., et al. (2020) · European Radiology · doi:10.1007/s00330-020-07573-3 · Cited by 18 · see citing works
Kim, S. M., Han, H. S., Park, J. M., Choi, Y. J., Yoon, H. S., Sohn, J. H., Baek, M. H., Kim, Y. N., Chae, Y. M., June, J. J., et al. (2012) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-012-9457-7 · Cited by 16 · see citing works
Breast Cancer Screening and Diagnosis
Jørgensen, K. J., & Paludan‐Müller, A. S. (2020) · Annals of Internal Medicine · doi:10.7326/l20-0253 · Cited by 14 · see citing works
Best Practice Guideline – DEGUM Recommendations on Breast Ultrasound
Bader, W., Vogel-Minea, C. M., Blohmer, J., et al. (2021) · Ultraschall in der Medizin - European Journal of Ultrasound · doi:10.1055/a-1634-5021 · Cited by 13 · see citing works
New developments in the diagnostic procedures to reduce prospective biopsies breast
Ayed, N. G. B., Masmoudi, A. D., Sellami, D., et al. (2015) · International Conference on Advances in Biomedical Engineering (ICABME) · doi:10.1109/icabme.2015.7323288 · Cited by 11 · see citing works
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Tibshirani (1997) · doi:10.1002/(sici)1097-0258(19970228)16:4
Flagged first, then by citations
Research works printed but not linked: 1
- [4]American College of Radiology. Subject: (Docket No. FDA-2019-N-5592) “Public Workshop - Evolving Role of Artificial Intelligence in Radiological Imaging;” Comments of the American College of Radiology. https://www.acr.org/-/media/ACR/NOINDEX/Advocacy/acr_rsna_comments_fda-ai-evolvingrole-ws_6-30-2020.pdf. Published 2020. Accessed 3 Apr 2021
Cited by
Sorted by citations
Jost, E., Kosian, P., Jimenez Cruz, J., Albarqouni, S., Gembruch, U., Strizek, B., & Recker, F. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12216833 · Cited by 44 · see citing works
“Limitations to AI usage in breast US were the missing clinical context in unimodal approaches only focusing on image analysis [202], small data sets for algorithm training, and a lower accuracy in borderline findings [201].”
BackgroundPfob, A., Sidey-Gibbons, C., Barr, R. G., Duda, V., Alwafai, Z., Balleyguier, C., Clevert, D.-A., Fastner, S., Gomez, C., Goncalo, M., et al. (2022) · European Journal of Cancer · doi:10.1016/j.ejca.2022.09.018 · Cited by 27 · see citing works
Self-citation
- Authors
- Pfob, A., Sidey-Gibbons, C., Barr, R. G., Duda, V., Alwafai, Z., Balleyguier, C., Clevert, D.-A., Fastner, S., Gomez, C., Goncalo, M., Gruber, I., Hahn, M., Hennigs, A., Kapetas, P., Lu, S.-C., Nees, J., Ohlinger, R., Riedel, F., Rutten, M., Schaefgen, B., Schuessler, M., Stieber, A., Togawa, R., Tozaki, M., Wojcinski, S., Xu, C., Rauch, G., Heil, J., & Golatta, M.
- Affiliations
- The University of Texas MD Anderson Cancer Center (US)
- Northeast Ohio Medical University (US)
- Philipps University of Marburg (DE)
- Universitätsmedizin Greifswald (DE)
- Institut Gustave Roussy (FR)
- Ludwig-Maximilians-Universität München (DE)
- Heidelberg University (DE)
- University of Tübingen (DE)
- Medical University of Vienna (AT)
- Radboud University Medical Center (NL)
- Sagara Hospital (JP)
- Klinikum Bielefeld (DE)
- Humboldt-Universität zu Berlin (DE)
- Corresponding author
- Golatta, Michael
- ORCID
- Pfob, André
- Sidey-Gibbons, Chris
- Alwafai, Zaher
- Balleyguier, Corinne
- Clevert, Dirk-André
- Fastner, Sarah
- Gomez, Christina
- Gruber, Ines
- Hahn, Markus
- Hennigs, André
- Kapetas, Panagiotis
- Lu, Sheng-Chieh
- Nees, Juliane
- Riedel, Fabian
- Rutten, Matthieu
- Schaefgen, Benedikt
- Schuessler, Maximilian
- Tozaki, Mitsuhiro
- Wojcinski, Sebastian
- Xu, Cai
- Rauch, Geraldine
- Heil, Joerg
- Golatta, Michael
- Venue
- European Radiology
- Published
- 2022-02-17
- Volume
- 32
- Issue
- 6
- Pages
- 4101-4115
- ISSN
- 0938-7994 (ISSN-L)
- 1432-1084
- PMC
- PMC9123064
- OpenAlex
- W4213279130
- Cited by
- 29
- References
- 34
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 9 · 2023: 5 · 2022: 4
- FWCI
- 2.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- European Radiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Funders
- Medizinische Fakultät Heidelberg der Universität Heidelberg
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Pathology and Forensic Medicine › Breast Lesions and Carcinomas
- Keywords
- breast cancer diagnosis, breast mass, breast ultrasound, artificial intelligence, machine learning, multimodal data, clinical data, breast cancer screening
- MeSH terms
- Algorithms, Artificial Intelligence, Breast, Breast Neoplasms, Female, Humans, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
