A pretrained neural network shows similar diagnostic accuracy to medical students in categorizing dermatoscopic images after comparable training conditions
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Dear Editor, Automated image classification by machine vision is evolving, largely because of the increasing efficiency of neural networks with special architectures such as convolutional neural networks (CNNs).1 A recent study comparing board‐certified dermatologists with CNNs suggested that CNNs may achieve diagnostic accuracies similar to those of human experts.2 We demonstrated that medical students without prior knowledge of dermatoscopy learn equally well from an analytical or heuristic teaching approach.3 During heuristic teaching, representative images of each class are presented together with the correct diagnosis without any further explanations, mimicking training of a neural network. In this experiment we trained a CNN (pretrained for general‐purpose image classification) and a group of previously untrained medical students to diagnose pigmented skin lesions on dermatoscopic images, and compared their diagnostic performance after training. We invited final‐year medical students without prior knowledge of dermatoscopy to participate in a 1‐h training session.3 During this training, the students saw dermatoscopic images of 298 pigmented lesions: 40 basal cell carcinomas (BCCs), seven dermatofibromas, 62 melanomas, 129 melanocytic naevi, 38 seborrhoeic keratoses and 22 vascular lesions. The instructor mentioned the diagnosis but did not explain any diagnostic features. After the training session the students had to rate a test set of 50 randomly selected pigmented lesions including 10 melanomas, 10 BCCs, 14 naevi, nine seborrhoeic keratoses and five vascular lesions or dermatofibromas. The students were asked to make a specific diagnosis and to indicate the probability of malignancy (scale from 1 to 5). No additional information (age, site or medical history) was provided. All images from the students' training session were also used to retrain the last layer of the ‘GoogLeNet Inception v3' neural network,4 without any kind of test‐set augmentation (4000 epochs, learning rate 0·001, batch size 50). We used the retrained network on the same test set as the students. The diagnosis was considered to indicate a malignant lesion if the top class of the softmax outputs was BCC or melanoma. The softmax outputs indicate probabilities for each class and were used to calculate receiver operating characteristics (ROC) curves. We calculated the areas under the curves (AUCs) and their confidence intervals (CIs) with the package pROC,5 and used epiR (in R, https://www.r-project.org) for the calculation of diagnostic values. We assessed the inter‐rater agreement by calculating Cohen's kappa for each class and each reader pair using the irr package in R. The study was approved by the institutional review board of the Medical University of Vienna (no. 1285/2014). Twenty‐seven students (mean age 24·9 ± 2·3 years, 74% female) completed the training session and evaluated the test set. The retrained neural network (R‐NN) correctly identified malignant lesions (BCC and melanoma) with a sensitivity of 90% (95% CI 68–99) and a specificity of 71% (95% CI 51–87). The sensitivity was slightly higher than that achieved by the students (mean 86%, 95% CI 83–88) but the R‐NN specificity was slightly lower than that of the students (mean 79%, 95% CI 74–83). For the R‐NN, the AUC for correctly predicting malignancy was 0·91 (95% CI 0·81–0·97), which was similar to the AUC of the pooled student ratings of 0·85 (95% CI 0·87–0·91, P = 0·64; Fig. 1a). (a) Diagnostic accuracy. The retrained neural network (R‐NN) and students (pooled ratings) showed similar diagnostic accuracy in detecting skin cancer (basal cell carcinoma and melanoma) as measured by the areas under the receiver operating characteristic curves. (b) Misclassified lesions. Correct (black) and incorrect (red) specific diagnoses of every case (rows 1–48) by every student (columns 1–37) and the R‐NN, and the mean correct ratings of the students. A striking similarity is seen between misdiagnosed lesions of the students and the R‐NN. The R‐NN made a correct specific diagnosis in 69% of lesions. Two of 10 melanomas were misclassified as naevi and two as BCC. Cases misdiagnosed by the R‐NN were also misdiagnosed by 46% of students. Of the 13 cases that were misdiagnosed by more than half of the students, nine were also misclassified by the R‐NN (Fig. 1b). Agreement on specific diagnoses between the R‐NN and students was moderate (mean Cohen's kappa 0·57, 95% CI 0·52–0·61) and similar to the average agreement between students. We show that an R‐NN achieves a diagnostic accuracy similar to that of medical students under comparable training conditions. The neural network (GoogLeNet Inception v3)4 was pretrained on more than 1 million images for the purpose of general image classification but was not pretrained to diagnose skin cancer. During training, students and the R‐NN had to extract features that discriminate benign from malignant pigmented skin lesions. Given the small trai
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213
Cited by
Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V., & Zou, J. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3129 · Cited by 403
Expert-Level Diagnosis of Nonpigmented Skin Cancer by Combined Convolutional Neural Networks
Tschandl, P., Rosendahl, C., Akay, B. N., Argenziano, G., Blum, A., Braun, R. P., Cabo, H., Gourhant, J.-Y., Kreusch, J., Lallas, A., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.4378 · Cited by 328
Self-citationMaron, R. C., Weichenthal, M., Utikal, J. S., Hekler, A., Berking, C., Hauschild, A., Enk, A. H., Haferkamp, S., Klode, J., Schadendorf, D., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.06.013 · Cited by 219
Artificial intelligence applications in dermatology: Where do we stand?
Gomolin, A., Netchiporouk, E., Gniadecki, R., & Litvinov, I. V. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2020.00100 · Cited by 142
Muñoz‐López, C., Ramírez‐Cornejo, C., Marchetti, M. A., Han, S. S., Del Barrio‐Díaz, P., Jaque, A., Uribe, P., Majerson, D., Curi, M., Del Puerto, C., et al. (2021) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.16979 · Cited by 70
Metadata
- Authors
- Tschandl, P., Kittler, H., & Argenziano, G.
- Venue
- British Journal of Dermatology
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-07-19
- Volume
- 177
- Issue
- 3
- Pages
- 867-869
- ISSN
- 0007-0963 (ISSN-L)
- 1365-2133
- OpenAlex
- W2621161783
- Cited by
- 25
- References
- 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.