Cited by 58Cutaneous Melanoma Detection and ManagementAI in cancer detectionAdvanced Neural Network Applications

Effects of Label Noise on Deep Learning-Based Skin Cancer Classification

Open access
Hekler, Achim, Kather, Jakob N., Krieghoff-Henning, Eva, Utikal, Jochen S., Meier, Friedegund, Gellrich, Frank F., Upmeier zu Belzen, Julius, French, Lars +16 more·2020·Frontiers in Medicine

Abstract

Recent studies have shown that deep learning is capable of classifying dermatoscopic images at least as well as dermatologists. However, many studies in skin cancer classification utilize non-biopsy-verified training images. This imperfect ground truth introduces a systematic error, but the effects on classifier performance are currently unknown. Here, we systematically examine the effects of label noise by training and evaluating convolutional neural networks (CNN) with 804 images of melanoma and nevi labeled either by dermatologists or by biopsy. The CNNs are evaluated on a test set of 384 images by means of 4-fold cross validation comparing the outputs with either the corresponding dermatological or the biopsy-verified diagnosis. With identical ground truths of training and test labels, high accuracies with 75.03% (95% CI: 74.39-75.66%) for dermatological and 73.80% (95% CI: 73.10-74.51%) for biopsy-verified labels can be achieved. However, if the CNN is trained and tested with different ground truths, accuracy drops significantly to 64.53% (95% CI: 63.12-65.94%, p<0.01) on a non-biopsy-verified and to 64.24% (95% CI: 62.66-65.83%, p<0.01) on a biopsy-verified test set. In conclusion, deep learning methods for skin cancer classification are highly sensitive to label noise and future work should use biopsy-verified training images to mitigate this problem.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fmed.2020.00177

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Label noise can also derive from human uncertainty when using the consensus of multiple annotators, often needed due to the absence of a pathologically confirmed diagnosis label.

    “Label noise can also derive from human uncertainty when using the consensus of multiple annotators, often needed due to the absence of a pathologically confirmed diagnosis label, thereby introducing inherent uncertainty35–37.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 16 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Hekler, A., Kather, J. N., Krieghoff-Henning, E., Utikal, J. S., Meier, F., Gellrich, F. F., Upmeier zu Belzen, J., French, L., Schlager, J. G., Ghoreschi, K., Wilhelm, T., Kutzner, H., Berking, C., Heppt, M. V., Haferkamp, S., Sondermann, W., Schadendorf, D., Schilling, B., Izar, B., Maron, R., Schmitt, M., Fröhling, S., Lipka, D. B., & Brinker, T. J.
Affiliations
  • German Cancer Research Center (DE)
  • National Center for Tumor Diseases (DE)
  • Ludwig-Maximilians-Universität München (DE)
  • Charité – Universitätsmedizin Berlin (DE)
  • Dermatopathologie Friedrichshafen (DE)
  • Universitätsklinikum Erlangen (DE)
  • University Hospital Regensburg (DE)
  • Essen University Hospital (DE)
  • Universitätsklinikum Würzburg (DE)
  • Dana-Farber Cancer Institute (US)
Corresponding author
Brinker, Titus J.
Venue
Frontiers in Medicine
Published
2020-05-06
Volume
7
Pages
177-177
ISSN
2296-858X (ISSN-L)
OpenAlex
W3021577186
Cited by
58
References
16
Citations by year
2026: 3 · 2025: 5 · 2024: 12 · 2023: 10 · 2022: 19 · 2021: 9
FWCI
3.43
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, cc-by)
  • University of Regensburg Publication Server (University of Regensburg) (submittedVersion)
  • Online Publication Service of Würzburg University (Würzburg University) (submittedVersion, cc-by)
  • OPUS FAU (Kooperativer Bibliotheksverbund Berlin-Brandenburg (KOBV), on behalf of the Universitätsbibliothek Erlangen-Nürnberg) (submittedVersion, cc-by)
Funders
  • Deutsches Krebsforschungszentrum
  • Bundesministerium für Gesundheit
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
label noise, skin cancer classification, melanoma, nevi, dermoscopic images, convolutional neural networks, classification accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review