Cited by 147AI in cancer detectionRadiomics and Machine Learning in Medical ImagingArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index, and 1 are not checked yet.

Ultrasound image analysis using deep neural networks for discriminating between benign and malignant ovarian tumors: comparison with expert subjective assessment

Open access

Christiansen, F., Epstein, E. L., Smedberg, E., Åkerlund, M., Smith, K., Epstein, E.·2021·Ultrasound in Obstetrics & Gynecology

DOI 10.1002/uog.23530PMC PMC7839489

Open this paper in CoChat — free account

Abstract

OBJECTIVES: To develop and test the performance of computerized ultrasound image analysis using deep neural networks (DNNs) in discriminating between benign and malignant ovarian tumors and to compare its diagnostic accuracy with that of subjective assessment (SA) by an ultrasound expert. METHODS: We included 3077 (grayscale, n = 1927; power Doppler, n = 1150) ultrasound images from 758 women with ovarian tumors, who were classified prospectively by expert ultrasound examiners according to IOTA (International Ovarian Tumor Analysis) terms and definitions. Histological outcome from surgery (n = 634) or long-term (≥ 3 years) follow-up (n = 124) served as the gold standard. The dataset was split into a training set (n = 508; 314 benign and 194 malignant), a validation set (n = 100; 60 benign and 40 malignant) and a test set (n = 150; 75 benign and 75 malignant). We used transfer learning on three pre-trained DNNs: VGG16, ResNet50 and MobileNet. Each model was trained, and the outputs calibrated, using temperature scaling. An ensemble of the three models was then used to estimate the probability of malignancy based on all images from a given case. The DNN ensemble classified the tumors as benign or malignant (Ovry-Dx1 model); or as benign, inconclusive or malignant (Ovry-Dx2 model). The diagnostic performance of the DNN models, in terms of sensitivity and specificity, was compared to that of SA for classifying ovarian tumors in the test set. RESULTS: At a sensitivity of 96.0%, Ovry-Dx1 had a specificity similar to that of SA (86.7% vs 88.0%; P = 1.0). Ovry-Dx2 had a sensitivity of 97.1% and a specificity of 93.7%, when designating 12.7% of the lesions as inconclusive. By complimenting Ovry-Dx2 with SA in inconclusive cases, the overall sensitivity (96.0%) and specificity (89.3%) were not significantly different from using SA in all cases (P = 1.0). CONCLUSION: Ultrasound image analysis using DNNs can predict ovarian malignancy with a diagnostic accuracy comparable to that of human expert examiners, indicating that these models may have a role in the triage of women with an ovarian tumor. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1002/uog.23530

Cites

34 of 35 references verified0 citation errors · 0 not checkable

Statistics

Figure 1 Diagnostic performance for distinguishing between benign and malignant ovarian tumors of deep‐learning models VGG16 (), ResNet50 (), MobileNet () and ensemble of the three models (Ovry‐Dx1 in (a) and Ovry‐Dx2 in (b); ()), in all patients in test set (a) and when excluding cases with predicted probability of malignancy between 0.4 and 0.6, corresponding to high uncertainty (b). Percentage of cases excluded was 10.7% for VGG16, 22.7% for ResNet50, 14.0% for MobileNet and 12.7% for Ovry‐Dx2. AUC, area under receiver‐operating‐characteristics curve.CC BY · PubMed Central

Figure 2 Receiver‐operating‐characteristics (ROC) curves for distinguishing between benign and malignant ovarian tumors of deep‐learning models VGG16 (), ResNet50 () and MobileNet () and ensemble of the three models (Ovry‐Dx1 in (a) and Ovry‐Dx2 in (b); ()), in all patients in test set (a) and when excluding inconclusive cases (predicted probability of malignancy between 0.4 and 0.6) (b). In (a), ROC curve of IOTA simple‐rules risk () and operating point for expert subjective assessment () are also shown.CC BY · PubMed Central

Figure 3 (a,b) Ten ovarian lesions that were difficult to classify by both expert subjective assessment and deep neural network model Ovry‐Dx2; final histological diagnosis was malignant or borderline in four (a) and benign in six (b). (c,d) Further nine ovarian lesions classified as inconclusive by Ovry‐Dx2; final histological diagnosis was malignant or borderline in three (c) and benign in six (d). Images marked with () were also classified as inconclusive by IOTA simple rules.CC BY · PubMed Central

Figure 4 Six cases misdiagnosed by deep neural network model Ovry‐Dx2: (a,b) malignant cases classified as benign; (c–f) benign cases classified as malignant. (a) Clear‐cell carcinoma with 6‐mm papillary projection; case was also misdiagnosed by expert subjective assessment. (b) Mucinous intestinal borderline tumor. (c,d) Dermoid. (e) Fibroma; case was also misdiagnosed by expert subjective assessment. (f) Leydig cell tumor.CC BY · PubMed Central

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Deep learning
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T., & Dasgupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,746
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
A method for solving the convex programming problem with convergence rate O(1/k^2)
Nesterov, Y. E. (1983) · Medical Entomology and Zoology · Cited by 1,770
Effect of Screening on Ovarian Cancer Mortality
Buys, S. S. (2011) · JAMA · doi:10.1001/jama.2011.766 · Cited by 1,326
Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) group
Timmerman, D., Valentin, L., Bourne, T., Collins, W. P., Verrelst, H., & Vergote, I. B. (2000) · Ultrasound in Obstetrics and Gynecology · doi:10.1046/j.1469-0705.2000.00287.x · Cited by 999
Epidemiology of epithelial ovarian cancer
Webb, P. M., & Jordan, S. J. (2016) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2016.08.006 · Cited by 998
Transfusion: Understanding Transfer Learning for Medical ImagingPreprint
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07208 · Cited by 901
Simple ultrasound-based rules for the diagnosis of ovarian cancer
Timmerman, D., Testa, A. C., Bourne, T., Ameye, L., Jurkovic, D., Van Holsbeke, C., Paladini, D., Van Calster, B., Vergote, I., Van Huffel, S., et al. (2008) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.5365 · Cited by 613
Simple ultrasound rules to distinguish between benign and malignant adnexal masses before surgery: prospective validation by IOTA group
Timmerman, D., Ameye, L., Fischerová, D., Epstein, E., Melis, G. B., Guerriero, S., Holsbeke, C. V., Savelli, L., Fruscio, R., Lissoni, A. A., et al. (2010) · BMJ · doi:10.1136/bmj.c6839 · Cited by 486
Self-citation
Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study
Li, X., Zhang, S., Zhang, Q., Wei, X., Pan, Y., Zhao, J., Xin, X., Qin, C., Wang, X., Li, J., et al. (2018) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30762-9 · Cited by 439
From BoW to CNN: Two Decades of Texture Representation for Texture Classification
Liu, L., Chen, J., Fieguth, P., Zhao, G., Chellappa, R., & Pietikäinen, M. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1125-z · Cited by 400
A deep learning framework for supporting the classification of breast lesions in ultrasound images
Han, S., Kang, H.-K., Jeong, J.-Y., Park, M.-H., Kim, W., Bang, W.-C., & Seong, Y.-K. (2017) · Physics in Medicine & Biology · doi:10.1088/1361-6560/aa82ec · Cited by 358
Predicting the risk of malignancy in adnexal masses based on the Simple Rules from the International Ovarian Tumor Analysis group
Timmerman, D., CALSTER, B. V., Testa, A. C., Savelli, L., Fischerová, D., Froyman, W., Wynants, L., Holsbeke, C. V., Epstein, E., Franchi, D., et al. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2016.01.007 · Cited by 337
Self-citation
Surgery by consultant gynecologic oncologists improves survival in patients with ovarian carcinoma
Engelen, M. J. A., Kos, H. E., Willemse, P. H. B., Aalders, J. G., de Vries, E. G. E., Schaapveld, M., Otter, R., & van der Zee, A. G. J. (2005) · Cancer · doi:10.1002/cncr.21616 · Cited by 305
Subjective assessment versus ultrasound models to diagnose ovarian cancer: A systematic review and meta-analysis
Meys, E. M. J., Kaijser, J., Kruitwagen, R. F. P. M., Slangen, B. F. M., CALSTER, B. V., Aertgeerts, B., Verbakel, J. Y. J., Timmerman, D., & Gorp, T. V. (2016) · European Journal of Cancer · doi:10.1016/j.ejca.2016.01.007 · Cited by 281
Machine learning for medical ultrasound: status, methods, and future opportunities
Brattain, L. J., Telfer, B. A., Dhyani, M., Grajo, J. R., & Samir, A. E. (2018) · Abdominal Radiology · doi:10.1007/s00261-018-1517-0 · Cited by 262
Risk of complications in patients with conservatively managed ovarian tumours (IOTA5): a 2-year interim analysis of a multicentre, prospective, cohort study
Froyman, W., Landolfo, C., Campo, B. D. C., Wynants, L., Sladkevicius, P., Testa, A. C., Holsbeke, C. V., Domali, E., Fruscio, R., Epstein, E., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30837-4 · Cited by 192
Self-citation
Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound
Ko, S. Y., Lee, J. H., Yoon, J. H., Na, H., Hong, E., Han, K., Jung, I., Kim, E., Moon, H. J., Park, V. Y., et al. (2019) · Head & Neck · doi:10.1002/hed.25415 · Cited by 104
Adnexal masses difficult to classify as benign or malignant using subjective assessment of gray‐scale and Doppler ultrasound findings: logistic regression models do not help
Valentin, L., Ameye, L., Savelli, L., Fruscio, R., Leone, F. P., Czekierdowski, A., Lissoni, A., Fischerová, D., Guerriero, S., Holsbeke, C. V., et al. (2011) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.9030 · Cited by 90
Risk of epithelial ovarian cancer in asymptomatic women with ultrasound‐detected ovarian masses: a prospective cohort study within the UK collaborative trial of ovarian cancer screening (UKCTOCS)
Sharma, A., Apostolidou, S., Burnell, M., Campbell, S. C., Habib, M., Gentry‐Maharaj, A., Amso, N. N., Seif, M. W., Fletcher, G., Singh, N., et al. (2012) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.12270 · Cited by 74
Evaluation of machine learning methods with Fourier Transform features for classifying ovarian tumors based on ultrasound images
Martínez-Más, J., Bueno-Crespo, A., Khazendar, S., Remezal-Solano, M., Martínez-Cendán, J.-P., Jassim, S., Du, H., Al Assam, H., Bourne, T., & Timmerman, D. (2019) · PLOS ONE · doi:10.1371/journal.pone.0219388 · Cited by 70
Ultrasound Experience Substantially Impacts on Diagnostic Performance and Confidence when Adnexal Masses Are Classified Using Pattern Recognition
Holsbeke, C. V., Daemen, A., Yazbek, J., Holland, T., Bourne, T., Mesens, T., Lannoo, L., Boes, A., Joos, A., van de Vijver, A., et al. (2009) · Gynecologic and Obstetric Investigation · doi:10.1159/000265012 · Cited by 69
When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
Zhang, L., Wang, X., Yang, D., Sanford, T., Harmon, S., Turkbey, B., Roth, H., Myronenko, A., Xu, D., & Xu, Z. (2019) · arXiv · doi:10.48550/arxiv.1906.03347 · Cited by 36
Automated characterisation of ultrasound images of ovarian tumours: the diagnostic accuracy of a support vector machine and image processing with a local binary pattern operator.
Khazendar, S., Sayasneh, A., Al-Assam, H., Du, H., Kaijser, J., Ferrara, L., Timmerman, D., Jassim, S., & Bourne, T. (2015) · PubMed Central · Cited by 30
Automated classification of static ultrasound images of ovarian tumours based on decision level fusion
Khazendar, S., Al-Assam, H., Du, H., Jassim, S., Sayasneh, A., Bourne, T., Kaijser, J., & Timmerman, D. (2014) · Computer Science and Electronic Engineering Conference · doi:10.1109/ceec.2014.6958571 · Cited by 16
Logistic models and artificial intelligence in the sonographic assessment of adnexal masses – a systematic review of the literature
Grigore, M., Popovici, R. M., Gafițanu, D., Himiniuc, L. M., Murarasu, M., & Micu, R. (2020) · Medical Ultrasonography · doi:10.11152/mu-2538 · Cited by 10

Research works printed but not linked: 1

Nesterov Y., 1983, A method of solving a convex programming problem with convergence rate O(1/k2), Sov Math Dokl, 27, 372

Sorted by citations

Artificial Intelligence Performance in Image-Based Ovarian Cancer Identification: A Systematic Review and Meta-Analysis
Xu, H.-L., Gong, T., Liu, F.-H., Chen, H.-Y., Xiao, Q., Hou, Y., Huang, Y., Sun, H., Shi, Y., Gao, S., et al. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101662 · Cited by 135
Towards Clinical Application of Artificial Intelligence in Ultrasound Imaging
Komatsu, M., Sakai, A., Dozen, A., Shozu, K., Yasutomi, S., Machino, H., Asada, K., Kaneko, S., & Hamamoto, R. (2021) · Biomedicines · doi:10.3390/biomedicines9070720 · Cited by 122
A Deep Learning Fusion Approach to Diagnosis the Polycystic Ovary Syndrome (PCOS)
Alamoudi, A., Khan, I. U., Aslam, N., Alqahtani, N., Alsaif, H. S., Dandan, O. S. A., Gadeeb, M. A., & Bahrani, R. A. (2023) · Applied Computational Intelligence and Soft Computing · doi:10.1155/2023/9686697 · Cited by 111
Deep Learning Prediction of Ovarian Malignancy at US Compared with O-RADS and Expert Assessment
Chen, H., Yang, B.-W., Qian, L., Meng, Y.-S., Bai, X.-H., Hong, X.-W., He, X., Jiang, M.-J., Yuan, F., Du, Q.-W., et al. (2022) · Radiology · doi:10.1148/radiol.211367 · Cited by 110
A systematic review on the use of artificial intelligence in gynecologic imaging – Background, state of the art, and future directions
Shrestha, P., Poudyal, B., Yadollahi, S., Wright, D., Gregory, A., Warner, J., Korfiatis, P. D., Green, I. C., Rassier, S. L. C., Mariani, A., et al. (2022) · Gynecologic Oncology · doi:10.1016/j.ygyno.2022.07.024 · Cited by 100

See all 147

Authors: Christiansen, F., Epstein, E. L., Smedberg, E., Åkerlund, M., Smith, K., & Epstein, E.
Affiliations: KTH Royal Institute of Technology (SE)
Karolinska Institutet (SE)
Harvard University (US)
Science for Life Laboratory (SE)
Corresponding author: Epstein, E.
ORCID: Christiansen, F.
Smedberg, E.
Smith, K.
Epstein, E.
Venue: Ultrasound in Obstetrics & Gynecology
Publisher: Wiley
Published: 2021-01
Volume: 57
Issue: 1
Pages: 155-163
ISSN: 0960-7692 (ISSN-L)
1469-0705
DOI: 10.1002/uog.23530
PMC: PMC7839489
OpenAlex: W3096898300
Cited by: 142
References: 34
Citations by year: 2026: 17 · 2025: 27 · 2024: 46 · 2023: 31 · 2022: 13 · 2021: 8
FWCI: 5.11
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/ (vor, 1 d)
doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
Locations: Ultrasound in Obstetrics and Gynecology (publishedVersion, cc-by)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
Funders: Radiumhemmets Forskningsfonder (181115)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords: ovarian neoplasms, benign versus malignant differentiation, computer-aided diagnosis, deep neural network, transfer learning, ensemble learning, temperature scaling, subjective evaluation, power Doppler, B-mode ultrasound, ovarian cancer, diagnostic accuracy, histological outcomes, expert comparison, VGG16, ResNet50, MobileNet
MeSH terms: Deep Learning, Adnexal Diseases, Diagnosis, Differential, Female, Humans, Image Processing, Computer-Assisted, Ovarian Neoplasms, Retrospective Studies, Sensitivity and Specificity, Reproducibility of Results, Ultrasonography, Doppler

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Ultrasound image analysis using deep neural networks for discriminating between benign and malignant ovarian tumors: comparison with expert subjective assessment

Abstract

Overview

Integrity signals

Figures

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks