Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
Open accessAbstract
This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DermaMNIST is a 2D dataset based on HAM10000, comprising 10,015 dermatoscopic images categorized as 7 different diseases for multi-class classification.
“The DermaMNIST is based on the HAM1000019–21, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. The dataset consists of 10,015 dermatoscopic images categorized as 7 different diseases, formulized as a multi-class classification task.”
From MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification (2023) · cites this paper as [21]
Section 2.3 (p. 4) reports 10,015 dermoscopic training images with seven classification labels and cites the HAM10000 paper, identified in the References (p. 8). It does not identify DermaMNIST, establish that DermaMNIST is based on HAM10000, or describe it as a 2D dataset; the 10,015 figure is stated for the challenge’s training set. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 11
- [1]1. Guy GP, Machlin S, Ekwueme DU, Yabroff KR. Prevalence and costs of skin cancer treatment in the US, 2002-2006 and 2007-2011. Am J Prev Med. 2015;48:183-7.
- [3]3. Gutman D, Codella NCF, Celebi E, Helba, B, Marchetti M, Mishra N, Halpern A. 'Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)'. eprint arXiv:1605.01397. 2016. · arXiv
- [4]4. Marchetti M, et al. 'Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images'. J Am Acad Dermatol. 2018 Feb;78(2):270-277
- [5]5. Codella N, et al. 'Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2017, hosted by the International Skin Imaging Collaboration (ISIC)'. IEEE International Symposium of Biomedical Imaging (ISBI) 2018.
- [6]6. Codella NCF, Nguyen B, Pankanti S, Gutman D, Helba B, Halpern A, Smith JR. 'Deep learning ensembles for melanoma recognition in dermoscopy images' In: IBM Journal of Research and Development, vol. 61, no. 4/5, 2017.
- [7]7. Esteva A,Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. 'Dermatologist-level classification of skin cancer with deep neural networks'. Nature, vol 542, pp 115-118. 2017.
- [8]8. Menegola A, Tavares J, Fornaciali M, Li LT, Avila S, Valle E. 'RECOD Titans at ISIC Challenge 2017'. 2017 International Symposium on Biomedical Imaging (ISBI) Challenge on Skin Lesion Analysis Towards Melanoma Detection. Available: https://arxiv.org/pdf/1703.04819.pdf · arXiv
- [9]9. Diaz, I.G. 'Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions. 2017 International Symposium on Biomedical Imaging (ISBI) Challenge on Skin Lesion Analysis Towards Melanoma Detection.' Available: https://arxiv.org/abs/1703.01976 · arXiv
- [10]10. Tschandl P, Sinz C, Kittler H. 'Domain-specific classification-pretrained fully convolutional network encoders for skin lesion segmentation.' Computers in Biology and Medicine, vol 104, pp 111-116, 2019
- [11]11. Tschandl P, Rosendahl C, Kittler H. 'The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.' Sci Data. 2018 Aug 14;5:180161.
- [12]12. Carrera C, et al. 'Validity and Reliability of Dermoscopic Criteria Used to Differentiate Nevi From Melanoma: A Web-Based International Dermoscopy Society Study.' JAMA Dermatol. 2016 July 01; 152(7): 798-806.
Cited by
Sorted by citations
MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification
Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., & Ni, B. (2023) · Scientific Data · doi:10.1038/s41597-022-01721-8 · Cited by 1,043 · see citing works
“The DermaMNIST is based on the HAM10000 19–21 , a large collection of multi-source dermatoscopic images of common pigmented skin lesions.”
MethodHuman-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024 · see citing works
Self-citationA Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities
Song, Y., Wang, T., Cai, P., Mondal, S. K., & Sahoo, J. P. (2023) · ACM Computing Surveys · doi:10.1145/3582688 · Cited by 721 · see citing works
Segment anything model for medical images?
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103061 · Cited by 536 · see citing works
“AbdomenCT-1K [21] Liver, kidney, spleen and pancreas CT ACDC [22] Left and right ventricle and left ventricular myocardium cine MRI AMOS 2022 [23] Abdominal multi-organ segmentation CT, MRI AutoLaparo [24] Integrated dataset with multiple image-based perception tasks Colonoscopy BrainPTM 2021 [25] white matter tracts…”
Disparities in dermatology AI performance on a diverse, curated clinical image set
Daneshjou, R., Vodrahalli, K., Novoa, R. A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S. M., Bailey, E. E., Gevaert, O., et al. (2022) · Science Advances · doi:10.1126/sciadv.abq6147 · Cited by 544 · see citing works
Self-citationFederated learning for medical image analysis: A survey
Guan, H., Yap, P.-T., Bozoki, A., & Liu, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110424 · Cited by 518 · see citing works
Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V., & Zou, J. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3129 · Cited by 403 · see citing works
Self-citationAnalysis of the ISIC image datasets: Usage, benchmarks and recommendations
Cassidy, B., Kendrick, C., Brodzicki, A., Jaworek-Korjakowska, J., & Yap, M. H. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102305 · Cited by 307 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
“However, this practice has not been ubiquitous due to the difficulty of obtaining data spanning multiple hospitals (Esteva et al., 2017; Bejnordi et al., 2017; Codella et al., 2019; Veta et al., 2019).”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267 · see citing works
Naeem, A., Farooq, M. S., Khelifi, A., & Abid, A. (2020) · IEEE Access · doi:10.1109/access.2020.3001507 · Cited by 189 · see citing works
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167 · see citing works
Transparent medical image AI via an image–text foundation model grounded in medical literature
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I. (2024) · Nature Medicine · doi:10.1038/s41591-024-02887-x · Cited by 125 · see citing works
Pham, T.-C., Luong, C.-M., Hoang, V.-D., & Doucet, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-96707-8 · Cited by 101 · see citing works
Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets
He, S., Bao, R., Li, J., Stout, J., Bjornerud, A., Grant, P. E., & Ou, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.09324 · Cited by 59 · see citing works
Hashmani, M. A., Jameel, S. M., Rizvi, S. S. H., & Shukla, S. (2021) · Applied Sciences · doi:10.3390/app11052145 · Cited by 53 · see citing works
Naeem, A., & Anees, T. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297667 · Cited by 48 · see citing works
De, A., Mishra, N., & Chang, H.-T. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1884 · Cited by 42 · see citing works
<i>auto</i>SMIM: Automatic Superpixel-Based Masked Image Modeling for Skin Lesion Segmentation
Wang, Z., Lyu, J., & Tang, X. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3290700 · Cited by 35 · see citing works
Yue, G., Wei, P., Zhou, T., Song, Y., Zhao, C., Wang, T., & Lei, B. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3319516 · Cited by 24 · see citing works
Aksoy, S., Demircioglu, P., & Bogrekci, I. (2024) · Dermato · doi:10.3390/dermato4040015 · Cited by 20 · see citing works
Multimodal Large Language Models are Generalist Medical Image Interpreters
Han, T., Adams, L. C., Nebelung, S., Kather, J. N., Bressem, K. K., & Truhn, D. (2023) · doi:10.1101/2023.12.21.23300146 · Cited by 18 · see citing works
Ahammed, S., Cui, X., Lu, W., & Yap, M. H. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00027 · Cited by 12 · see citing works
O.K., S., Riera-Marín, M., Galdran, A., López, J. G., Rodríguez-Comas, J., Piella, G., & Ballester, M. A. G. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102547 · Cited by 11 · see citing works
ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality
Cosarinsky, M., Billot, R., Mansilla, L., Jimenez, G., Gaggión, N., Fu, G., Tirer, T., & Ferrante, E. (2025) · arXiv
“2 including SCD [44], HC18 [45], PSFHS [46], PH2 [47], ISIC 2018 [48], NuCLS [49], JSRT [50], 3D-IRCAdB [51], WBC-CV and WBC-JTSC [52].”
- Authors
- Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., Kittler, H., & Halpern, A.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1902.03368
- OpenAlex
- W2932083555
- Cited by
- 984
- References
- 0
- Citations by year
- 2026: 41 · 2025: 164 · 2024: 229 · 2023: 186 · 2022: 118 · 2021: 148 · 2020: 76 · 2019: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- melanoma detection, dermoscopic images, skin lesion segmentation, skin lesion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
