Cited by 267Cutaneous Melanoma Detection and ManagementAI in cancer detectionArtificial Intelligence in Healthcare and Education

Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

Open access
Haggenmüller, Sarah, Maron, Roman C., Hekler, Achim, Utikal, Jochen S., Barata, Catarina, Barnhill, Raymond L., Beltraminelli, Helmut, Berking, Carola +57 more·2021·European Journal of Cancer

Abstract

BACKGROUND: Multiple studies have compared the performance of artificial intelligence (AI)-based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. OBJECTIVE: The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. METHODS: PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. RESULTS: A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. CONCLUSIONS: All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ejca.2021.06.049

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI models have been proven to provide high sensitivity in identifying malignant tumors and differentiating various skin diseases, making them useful for dermatologists.

    “These models have been proven to provide high sensitivity in the identification of malignant tumors and differentiation of various skin diseases thus being useful for dermatologists (6, 39, 40).”

    From Revolutionizing dermatopathology using AI in skin diagnostics: scoping review (2026) · cites this paper as [6]

    Section 3.1 reports 95.0% sensitivity for a CNN distinguishing (pre)malignant from benign lesions and describes models differentiating melanoma from several other skin diseases. But the Conclusions (page not provided) call clinical usefulness a potential, not a proven outcome: most studies used experimental single-image settings, and prospective studies of effects on clinicians’ decisions are still needed. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 45 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Haggenmüller, S., Maron, R. C., Hekler, A., Utikal, J. S., Barata, C., Barnhill, R. L., Beltraminelli, H., Berking, C., Betz-Stablein, B., Blum, A., Braun, S. A., Carr, R., Combalia, M., Fernandez-Figueras, M.-T., Ferrara, G., Fraitag, S., French, L. E., Gellrich, F. F., Ghoreschi, K., Goebeler, M., Guitera, P., Haenssle, H. A., Haferkamp, S., Heinzerling, L., Heppt, M. V., Hilke, F. J., Hobelsberger, S., Krahl, D., Kutzner, H., Lallas, A., Liopyris, K., Llamas-Velasco, M., Malvehy, J., Meier, F., Müller, C. S. L., Navarini, A. A., Navarrete-Dechent, C., Perasole, A., Poch, G., Podlipnik, S., Requena, L., Rotemberg, V. M., Saggini, A., Sangueza, O. P., Santonja, C., Schadendorf, D., Schilling, B., Schlaak, M., Schlager, J. G., Sergon, M., Sondermann, W., Soyer, H. P., Starz, H., Stolz, W., Vale, E., Weyers, W., Zink, A., Krieghoff-Henning, E., Kather, J. N., von Kalle, C., Lipka, D. B., Fröhling, S., Hauschild, A., Kittler, H., & Brinker, T. J.
Affiliations
  • German Cancer Research Center (DE)
  • University of Lisbon (PT)
  • Institut Curie (FR)
  • University of Bern (CH)
  • Universitätsklinikum Erlangen (DE)
  • Klinikum Konstanz (DE)
  • University Hospital Münster (DE)
  • Warwick Hospital (GB)
  • Hospital Clínic de Barcelona (ES)
  • Ospedale di Macerata (IT)
  • Hôpital Necker-Enfants Malades (FR)
  • University of Miami (US)
  • University Hospital Carl Gustav Carus (DE)
  • Charité – Universitätsmedizin Berlin (DE)
  • Universitätsklinikum Würzburg (DE)
  • The University of Sydney (AU)
  • University Hospital Heidelberg (DE)
  • University Hospital Regensburg (DE)
  • LMU Klinikum (DE)
  • Dermatopathologie Friedrichshafen (DE)
  • Aristotle University of Thessaloniki (GR)
  • Memorial Sloan Kettering Cancer Center (US)
  • University Hospital of Basel (CH)
  • Ospedale San Bortolo (IT)
  • Hospital Universitario Fundación Jiménez Díaz (ES)
  • Essen University Hospital (DE)
  • The University of Queensland (AU)
  • München Klinik (DE)
  • Hospital da Luz (PT)
  • Dermatologie Freiburg (DE)
  • Technical University of Munich (DE)
  • Universitätsklinikum Aachen (DE)
  • University Hospital Schleswig-Holstein (DE)
  • Medical University of Vienna (AT)
Corresponding author
Brinker, Titus J.
Venue
European Journal of Cancer
Publisher
Elsevier BV
Published
2021-10
Volume
156
Pages
202-216
ISSN
0959-8049 (ISSN-L)
OpenAlex
W3196396697
Cited by
267
References
45
Citations by year
2026: 43 · 2025: 37 · 2024: 79 · 2023: 57 · 2022: 43 · 2021: 8
FWCI
5.66
Type
journal article
Language
en
Open access
Locations
  • European Journal of Cancer (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • University of Regensburg Publication Server (University of Regensburg) (submittedVersion, cc-by-nc-nd)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
  • RWTH Publications (RWTH Aachen)
Funders
  • National Institutes of Health (P30 CA008748)
  • National Cancer Institute (P30 CA008748)
  • Bundesministerium für Gesundheit (P30 CA008748)
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
skin cancer classification, melanoma, convolutional neural networks, reader study, human expert comparison, dermoscopic images, whole slide imaging, histopathology, out-of-distribution data, clinical translation of AI, digital biomarkers
MeSH terms
Dermatologists, Pathologists, Deep Learning, Automation, Biopsy, Clinical Competence, Diagnosis, Computer-Assisted, Humans, Image Interpretation, Computer-Assisted, Melanoma, Microscopy, Predictive Value of Tests, Skin Neoplasms, Reproducibility of Results, Neural Networks, Computer, Dermoscopy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review