The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images
Open accessAbstract
Background Computer vision may aid in melanoma detection. Objective We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images. Methods We conducted a cross‐sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into “fusion” algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant. Results The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P = .68) but lower than the best‐performing fusion algorithm (59% vs. 76%, P = .02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P = .001). Limitations The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice. Conclusion Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620
Self-citationA methodological approach to the classification of dermoscopy images
Celebi, M. E., Kingravi, H. A., Uddin, B., Iyatomi, H., Aslandogan, Y. A., Stoecker, W. V., & Moss, R. H. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.01.003 · Cited by 693
Self-citationDeep learning ensembles for melanoma recognition in dermoscopy images
Codella, N. C. F., Nguyen, Q.-B., Pankanti, S., Gutman, D. A., Helba, B., Halpern, A. C., & Smith, J. R. (2017) · IBM Journal of Research and Development · doi:10.1147/jrd.2017.2708299 · Cited by 644
Self-citationGutman, D. A., Codella, N., Celebi, E., Helba, B., Marchetti, M. A., Mishra, N. K., & Halpern, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01397 · Cited by 365
Self-citationScreening for Skin Cancer: US Preventive Services Task Force Recommendation Statement
US Preventive Services Task Force, Bibbins-Domingo, K., Grossman, D. C., Curry, S. J., Davidson, K. W., Ebell, M., Epling, J. W., García, F. A. R., Gillman, M. W., Kemper, A. R., et al. (2016) · JAMA · doi:10.1001/jama.2016.8465 · Cited by 322
The Performance of MelaFind: A Prospective Multicenter Study
Monheit, G., Cognetta, A. B., Ferris, L., Rabinovitz, H., Gross, K., Martini, M., Grichnik, J. M., Mihm, M., Prieto, V. G., Googe, P., et al. (2011) · Archives of Dermatology · doi:10.1001/archdermatol.2010.302 · Cited by 231
Iyatomi, H., Oka, H., Celebi, M. E., Hashimoto, M., Hagiwara, M., Tanaka, M., & Ogawa, K. (2008) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2008.06.005 · Cited by 230
Self-citationMalvehy, J., Hauschild, A., Curiel‐Lewandrowski, C., Mohr, P., Hofmann‐Wellenhof, R., Motley, R., Berking, C., Grossman, D., Paoli, J., Loquai, C., et al. (2014) · British Journal of Dermatology · doi:10.1111/bjd.13121 · Cited by 210
Cited by
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620
Self-citationHaenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571
Self-citationDeep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
Self-citationCodella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.03368 · Cited by 984
Self-citationBrinker, T. J., Hekler, A., Enk, A. H., Klode, J., Hauschild, A., Berking, C., Schilling, B., Haferkamp, S., Schadendorf, D., Holland-Letz, T., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.04.001 · Cited by 605
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571
“Studies have shown that the interpretation of medical images for the diagnosis of tumors performs equally well or better with AI compared with experts [53-56].”
Tschandl, P., Codella, N., Akay, B. N., Argenziano, G., Braun, R. P., Cabo, H., Gutman, D., Halpern, A., Helba, B., Hofmann-Wellenhof, R., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30333-x · Cited by 555
Self-citationMultimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528
Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review
Brinker, T. J., Hekler, A., Utikal, J. S., Grabe, N., Schadendorf, D., Klode, J., Berking, C., Steeb, T., Enk, A. H., & von Kalle, C. (2018) · Journal of Medical Internet Research · doi:10.2196/11936 · Cited by 500
Machine learning: applications of artificial intelligence to imaging and diagnosis
Nichols, J. A., Herbert Chan, H. W., & Baker, M. A. B. (2019) · Biophysical Reviews · doi:10.1007/s12551-018-0449-9 · Cited by 455
Deep neural networks are superior to dermatologists in melanoma image classification
Brinker, T. J., Hekler, A., Enk, A. H., Berking, C., Haferkamp, S., Hauschild, A., Weichenthal, M., Klode, J., Schadendorf, D., Holland-Letz, T., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.05.023 · Cited by 415
Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V., & Zou, J. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3129 · Cited by 403
Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365
What is AI? Applications of artificial intelligence to dermatology
Du‐Harpur, X., Watt, F. M., Luscombe, N. M., & Lynch, M. D. (2020) · British Journal of Dermatology · doi:10.1111/bjd.18880 · Cited by 355
Pacheco, A. G. C., Lima, G. R., Salomão, A. S., Krohling, B., Biral, I. P., de Angelo, G. G., Alves Jr, F. C. R., Esgario, J. G. M., Simora, A. C., Castro, P. B. C., et al. (2020) · Data in Brief · doi:10.1016/j.dib.2020.106221 · Cited by 354
Expert-Level Diagnosis of Nonpigmented Skin Cancer by Combined Convolutional Neural Networks
Tschandl, P., Rosendahl, C., Akay, B. N., Argenziano, G., Blum, A., Braun, R. P., Cabo, H., Gourhant, J.-Y., Kreusch, J., Lallas, A., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.4378 · Cited by 328
Self-citationFujisawa, Y., Otomo, Y., Ogata, Y., Nakamura, Y., Fujita, R., Ishitsuka, Y., Watanabe, R., Okiyama, N., Ohara, K., & Fujimoto, M. (2019) · British Journal of Dermatology · doi:10.1111/bjd.16924 · Cited by 317
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
Brinker, T. J., Hekler, A., Enk, A. H., Klode, J., Hauschild, A., Berking, C., Schilling, B., Haferkamp, S., Schadendorf, D., Fröhling, S., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.02.005 · Cited by 302
Hekler, A., Utikal, J. S., Enk, A. H., Solass, W., Schmitt, M., Klode, J., Schadendorf, D., Sondermann, W., Franklin, C., Bestvater, F., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.06.012 · Cited by 285
BCN20000: Dermoscopic Lesions in the Wild
Combalia, M., Codella, N., Rotemberg, V. M., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A. C., Puig, S., et al. (2019) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.1908.02288 · Cited by 284
Self-citationMachine Learning in Dermatology: Current Applications, Opportunities, and Limitations
Chan, S., Reddy, V., Myers, B., Thibodeaux, Q., Brownstone, N., & Liao, W. (2020) · Dermatology and Therapy · doi:10.1007/s13555-020-00372-0 · Cited by 267
Haggenmüller, S., Maron, R. C., Hekler, A., Utikal, J. S., Barata, C., Barnhill, R. L., Beltraminelli, H., Berking, C., Betz-Stablein, B., Blum, A., et al. (2021) · European Journal of Cancer · doi:10.1016/j.ejca.2021.06.049 · Cited by 267
Artificial Intelligence in Dermatology: A Primer
Young, A. T., Xiong, M., Pfau, J., Keiser, M. J., & Wei, M. L. (2020) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2020.02.026 · Cited by 249
Characteristics of publicly available skin cancer image datasets: a systematic review
Wen, D., Khan, S. M., Ji Xu, A., Ibrahim, H., Smith, L., Caballero, J., Zepeda, L., de Blas Perez, C., Denniston, A. K., Liu, X., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00252-1 · Cited by 243
Pathologist-level classification of histopathological melanoma images with deep neural networks
Hekler, A., Utikal, J. S., Enk, A. H., Berking, C., Klode, J., Schadendorf, D., Jansen, P., Franklin, C., Holland-Letz, T., Krahl, D., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.04.021 · Cited by 231
Naeem, A., Farooq, M. S., Khelifi, A., & Abid, A. (2020) · IEEE Access · doi:10.1109/access.2020.3001507 · Cited by 189
Skin disease diagnosis with deep learning: a review
Li, H., Pan, Y., Zhao, J., & Zhang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.096 · Cited by 178
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189
Brinker, T. J., Hekler, A., Hauschild, A., Berking, C., Schilling, B., Enk, A. H., Haferkamp, S., Karoglan, A., von Kalle, C., Weichenthal, M., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2018.12.016 · Cited by 157
Artificial intelligence applications in dermatology: Where do we stand?
Gomolin, A., Netchiporouk, E., Gniadecki, R., & Litvinov, I. V. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2020.00100 · Cited by 142
Automated Dermatological Diagnosis: Hype or Reality?
Navarrete-Dechent, C., Dusza, S. W., Liopyris, K., Marghoob, A. A., Halpern, A. C., & Marchetti, M. A. (2018) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2018.04.040 · Cited by 129
Self-citationAccuracy of Computer-Aided Diagnosis of Melanoma: A Meta-analysis
Dick, V., Sinz, C., Mittlböck, M., Kittler, H., & Tschandl, P. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1375 · Cited by 125
Kamulegeya, L., Bwanika, J., Okello, M., Rusoke, D., Nassiwa, F., Lubega, W., Musinguzi, D., & Börve, A. (2023) · African Health Sciences · doi:10.4314/ahs.v23i2.86 · Cited by 111
Fink, C., Blum, A., Buhl, T., Mitteldorf, C., Hofmann‐Wellenhof, R., Deinlein, T., Stolz, W., Trennheuser, L., Cussigh, C., Deltgen, D., et al. (2020) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.16165 · Cited by 71
“4,5 7 More recently, a first deep learning CNN for the classification of skin neoplasms has gained market access as a medical device in Europe (Moleanalyzer-Pro (cid:1) , FotoFinder GmbH, Bad Birnbach, Germany).”
BackgroundMuñoz‐López, C., Ramírez‐Cornejo, C., Marchetti, M. A., Han, S. S., Del Barrio‐Díaz, P., Jaque, A., Uribe, P., Majerson, D., Curi, M., Del Puerto, C., et al. (2021) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.16979 · Cited by 70
Self-citationJones, O. T., Calanzani, N., Saji, S., Duffy, S. W., Emery, J., Hamilton, W., Singh, H., de Wit, N. J., & Walter, F. M. (2021) · Journal of Medical Internet Research · doi:10.2196/23483 · Cited by 65
Cerminara, S. E., Cheng, P., Kostner, L., Huber, S., Kunz, M., Maul, J.-T., Böhm, J. S., Dettwiler, C. F., Geser, A., Jakopović, C., et al. (2023) · European Journal of Cancer · doi:10.1016/j.ejca.2023.112954 · Cited by 62
Zhao, Z., Wu, C.-M., Zhang, S., He, F., Liu, F., Wang, B., Huang, Y., Shi, W., Jian, D., Xie, H., et al. (2021) · JMIR Medical Informatics · doi:10.2196/23415 · Cited by 58
“Generally, the data sources for the interpretation of machine learning varies from digital medical records [35,36], histopathological pictures [37], and clinical photos [38-41] to dermoscopic images [42-50].”
BackgroundEffects of Label Noise on Deep Learning-Based Skin Cancer Classification
Hekler, A., Kather, J. N., Krieghoff-Henning, E., Utikal, J. S., Meier, F., Gellrich, F. F., Upmeier zu Belzen, J., French, L., Schlager, J. G., Ghoreschi, K., et al. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2020.00177 · Cited by 58
Felmingham, C. M., Adler, N. R., Ge, Z., Morton, R. L., Janda, M., & Mar, V. J. (2021) · American Journal of Clinical Dermatology · doi:10.1007/s40257-020-00574-4 · Cited by 57
Codella, N. C. F., Lin, C.-C., Halpern, A., Hind, M., Feris, R., & Smith, J. R. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-02628-8_11 · Cited by 52
Self-citationKamulegeya, L. H., Okello, M., Bwanika, J. M., Musinguzi, D., Lubega, W., Rusoke, D., Nassiwa, F., & Börve, A. (2019) · doi:10.1101/826057 · Cited by 111
Chen, J. Y., Fernandez, K., Fadadu, R. P., Reddy, R., Kim, M.-O., Tan, J., & Wei, M. L. (2025) · JAMA Dermatology · doi:10.1001/jamadermatol.2024.4382 · Cited by 43
Zhu, A.-Q., Wang, Q., Shi, Y.-L., Ren, W.-W., Cao, X., Ren, T.-T., Wang, J., Zhang, Y.-Q., Sun, Y.-K., Chen, X.-W., et al. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102391 · Cited by 25
Metadata
- Authors
- Marchetti, M. A., Codella, N. C. F., Dusza, S. W., Gutman, D. A., Helba, B., Kalloo, A., Mishra, N., Carrera, C., Celebi, M. E., DeFazio, J. L., Jaimes, N., Marghoob, A. A., Quigley, E., Scope, A., Yélamos, O., & Halpern, A. C.
- Venue
- Journal of the American Academy of Dermatology
- Publisher
- Elsevier BV
- Published
- 2018-02
- Volume
- 78
- Issue
- 2
- Pages
- 270-277.e1
- ISSN
- 0190-9622 (ISSN-L)
- OpenAlex
- W2757940437
- Cited by
- 330
- References
- 22
- FWCI
- 13.57
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.