Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists

Open access
Haenssle, H.A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. Ben Hadj, Thomas, L., Enk, A., Uhlmann, L., Alt, Christina, Arenbergerova, Monika, Bakos, Renato, Baltzer, Anne, Bertlich, Ines, Blum, Andreas, Bokor-Billmann, Therezia, Bowling, Jonathan, Braghiroli, Naira, Braun, Ralph, Buder-Bakhaya, Kristina, Buhl, Timo, Cabo, Horacio, Cabrijan, Leo, Cevic, Naciye, Classen, Anna, Deltgen, David, Fink, Christine, Georgieva, Ivelina, Hakim-Meibodi, Lara-Elena, Hanner, Susanne, Hartmann, Franziska, Hartmann, Julia, Haus, Georg, Hoxha, Elti, Karls, Raimonds, Koga, Hiroshi, Kreusch, Jürgen, Lallas, Aimilios, Majenka, Pawel, Marghoob, Ash, Massone, Cesare, Mekokishvili, Lali, Mestel, Dominik, Meyer, Volker, Neuberger, Anna, Nielsen, Kari, Oliviero, Margaret, Pampena, Riccardo, Paoli, John, Pawlik, Erika, Rao, Barbar, Rendon, Adriana, Russo, Teresa, Sadek, Ahmed, Samhaber, Kinga, Schneiderbauer, Roland, Schweizer, Anissa, Toberer, Ferdinand, Trennheuser, Lukas, Vlahova, Lyobomira, Wald, Alexander, Winkler, Julia, Wölbing, Priscila, Zalaudek, Iris · 2018 · Annals of Oncology

Abstract

Background Deep learning convolutional neural networks (CNN) may facilitate melanoma detection, but data comparing a CNN's diagnostic performance to larger groups of dermatologists are lacking. Methods Google's Inception v4 CNN architecture was trained and validated using dermoscopic images and corresponding diagnoses. In a comparative cross-sectional reader study a 100-image test-set was used (level-I: dermoscopy only; level-II: dermoscopy plus clinical information and images). Main outcome measures were sensitivity, specificity and area under the curve (AUC) of receiver operating characteristics (ROC) for diagnostic classification (dichotomous) of lesions by the CNN versus an international group of 58 dermatologists during level-I or -II of the reader study. Secondary end points included the dermatologists' diagnostic performance in their management decisions and differences in the diagnostic performance of dermatologists during level-I and -II of the reader study. Additionally, the CNN's performance was compared with the top-five algorithms of the 2016 International Symposium on Biomedical Imaging (ISBI) challenge. Results In level-I dermatologists achieved a mean (±standard deviation) sensitivity and specificity for lesion classification of 86.6% (±9.3%) and 71.3% (±11.2%), respectively. More clinical information (level-II) improved the sensitivity to 88.9% (±9.6%, P = 0.19) and specificity to 75.7% (±11.7%, P < 0.05). The CNN ROC curve revealed a higher specificity of 82.5% when compared with dermatologists in level-I (71.3%, P < 0.01) and level-II (75.7%, P < 0.01) at their sensitivities of 86.6% and 88.9%, respectively. The CNN ROC AUC was greater than the mean ROC area of dermatologists (0.86 versus 0.79, P < 0.01). The CNN scored results close to the top three algorithms of the ISBI 2016 challenge. Conclusions For the first time we compared a CNN's diagnostic performance with a large international group of 58 dermatologists, including 30 experts. Most dermatologists were outperformed by the CNN. Irrespective of any physicians' experience, they may benefit from assistance by a CNN's image classification. Clinical trial number This study was registered at the German Clinical Trial Register (DRKS-Study-ID: DRKS00013570; https://www.drks.de/drks_web/).

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/annonc/mdy166

Cites

0 citation errors · 0 not checkable

Statistics

Figures

Image 1
From the PDF
Figure 1 The CNN's melanoma probability scores (range 0-1) for benign nevi (green online) in comparison to in situ (orange online) or invasive melanomas (red online) are depicted as boxplots for testset-300 and test-set-100. Scores closer to 1 indicated a higher probability of melanoma. The upper and lower bounds of boxes indicate the 25th and 75th percentiles while the median is indicated by the line intersection the upper and lower box. Whiskers indicate the full range of probability scores. Statistical analyses revealed significantly different melanoma probability scores when comparing benign lesions to in situ or invasive melanomas ( P < 0.001). However, melanoma probability scores for in situ and invasive melanomas showed no significant differences (set-300 P ¼ 0.84, set-100 P ¼ 0.24).
Figure 1 The CNN's melanoma probability scores (range 0-1) for benign nevi (green online) in comparison to in situ (orange online) or invasive melanomas (red online) are depicted as boxplots for testset-300 and test-set-100. Scores closer to 1 indicated a higher probability of melanoma. The upper and lower bounds of boxes indicate the 25th and 75th percentiles while the median is indicated by the line intersection the upper and lower box. Whiskers indicate the full range of probability scores. Statistical analyses revealed significantly different melanoma probability scores when comparing benign lesions to in situ or invasive melanomas ( P < 0.001). However, melanoma probability scores for in situ and invasive melanomas showed no significant differences (set-300 P ¼ 0.84, set-100 P ¼ 0.24).From the PDF
Image 3
From the PDF
Image 4
From the PDF
Figure 2 (A) ROC curve of the CNN in relation to the average ( 6 SD) sensitivity and specificity of all dermatologists [mean: green (online) circle; 6 SD: green (online) error bars] in set-100 (dichotomous classification, study level-I) and the dermatologists' mean sensitivity and specificity in relation to their level of experience. (B) ROC curve of the CNN in set-300.
Figure 2 (A) ROC curve of the CNN in relation to the average ( 6 SD) sensitivity and specificity of all dermatologists [mean: green (online) circle; 6 SD: green (online) error bars] in set-100 (dichotomous classification, study level-I) and the dermatologists' mean sensitivity and specificity in relation to their level of experience. (B) ROC curve of the CNN in set-300.From the PDF
Figure 3 Comparison of ROC curves of the CNN described in this study (dark green (online) line) to the top-five ranked individual algorithms of the 2016 International Symposium on Biomedical Imaging (ISBI) challenge [18]. ROC AUCs in descending order were as follows: ISBI team-2: 0.7956; ISBI team-1: 0.7928; ISBI team-3: 0.7892; CNN of this study: 0.7868; ISBI team-4: 0.5460; ISBI team-5: 0.5324.
Figure 3 Comparison of ROC curves of the CNN described in this study (dark green (online) line) to the top-five ranked individual algorithms of the 2016 International Symposium on Biomedical Imaging (ISBI) challenge [18]. ROC AUCs in descending order were as follows: ISBI team-2: 0.7956; ISBI team-1: 0.7928; ISBI team-3: 0.7892; CNN of this study: 0.7868; ISBI team-4: 0.5460; ISBI team-5: 0.5324.From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Cited by

See all 1,571

Metadata

Authors
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., Uhlmann, L., Alt, C., Arenbergerova, M., Bakos, R., Baltzer, A., Bertlich, I., Blum, A., Bokor-Billmann, T., Bowling, J., Braghiroli, N., Braun, R., Buder-Bakhaya, K., Buhl, T., Cabo, H., Cabrijan, L., Cevic, N., Classen, A., Deltgen, D., Fink, C., Georgieva, I., Hakim-Meibodi, L.-E., Hanner, S., Hartmann, F., Hartmann, J., Haus, G., Hoxha, E., Karls, R., Koga, H., Kreusch, J., Lallas, A., Majenka, P., Marghoob, A., Massone, C., Mekokishvili, L., Mestel, D., Meyer, V., Neuberger, A., Nielsen, K., Oliviero, M., Pampena, R., Paoli, J., Pawlik, E., Rao, B., Rendon, A., Russo, T., Sadek, A., Samhaber, K., Schneiderbauer, R., Schweizer, A., Toberer, F., Trennheuser, L., Vlahova, L., Wald, A., Winkler, J., Wölbing, P., & Zalaudek, I.
Affiliations
  • Heidelberg University (DE)
  • Memorial Sloan Kettering Cancer Center (US)
  • University of Passau (DE)
  • Lyon 1 Université (FR)
Corresponding author
Haenssle, H.A.
Venue
Annals of Oncology
Publisher
Elsevier BV
Published
2018-08
Volume
29
Issue
8
Pages
1836-1842
ISSN
0923-7534 (ISSN-L)
OpenAlex
W2806853752
Cited by
1,571
References
0
Citations by year
2026: 117 · 2025: 202 · 2024: 224 · 2023: 217 · 2022: 185 · 2021: 215 · 2020: 187 · 2019: 185 · 2018: 36
FWCI
58.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Annals of Oncology (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • ArTS Archivio della ricerca di Trieste (University of Trieste https://www.units.it/) (submittedVersion, other-oa)
  • Lund University Publications (Lund University) (submittedVersion)
  • GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, other-oa)
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
melanoma, dermoscopy, convolutional neural networks, Inception-v4, computer-aided diagnosis, skin lesion classification, sensitivity and specificity, clinical decision support, artificial intelligence in dermatology
MeSH terms
Dermatologists, Deep Learning, Clinical Competence, Cross-Sectional Studies, Humans, Image Processing, Computer-Assisted, International Cooperation, Melanoma, Retrospective Studies, ROC Curve, Skin, Skin Neoplasms, Dermoscopy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. https://app.cochat.ai/papers/10.1093/annonc/mdy166
The paper
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., Uhlmann, L., Alt, C., Arenbergerova, M., Bakos, R., Baltzer, A., Bertlich, I., Blum, A., Bokor-Billmann, T., Bowling, J., . . . Zalaudek, I. (2018). Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology. https://doi.org/10.1093/annonc/mdy166

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.