Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study
Open accessAbstract
Background Whether machine-learning algorithms can diagnose all pigmented skin lesions as accurately as human experts is unclear. The aim of this study was to compare the diagnostic accuracy of state-of-the-art machine-learning algorithms with human readers for all clinically relevant types of benign and malignant pigmented skin lesions.Methods For this open, web-based, international, diagnostic study, human readers were asked to diagnose dermatoscopic images selected randomly in 30-image batches from a test set of 1511 images. The diagnoses from human readers were compared with those of 139 algorithms created by 77 machine-learning labs, who participated in the International Skin Imaging Collaboration 2018 challenge and received a training set of 10 015 images in advance. The ground truth of each lesion fell into one of seven predefined disease categories: intraepithelial carcinoma including actinic keratoses and Bowen's disease; basal cell carcinoma; benign keratinocytic lesions including solar lentigo, seborrheic keratosis and lichen planus-like keratosis; dermatofibroma; melanoma; melanocytic nevus; and vascular lesions. The two main outcomes were the differences in the number of correct specific diagnoses per batch between all human readers and the top three algorithms, and between human experts and the top three algorithms.Findings Between Aug 4, 2018, and Sept 30, 2018, 511 human readers from 63 countries had at least one attempt in the reader study. 283 (55·4%) of 511 human readers were board-certified dermatologists, 118 (23·1%) were dermatology residents, and 83 (16·2%) were general practitioners. When comparing all human readers with all machine-learning algorithms, the algorithms achieved a mean of 2·01 (95% CI 1·97 to 2·04; p<0·0001) more correct diagnoses (17·91 [SD 3·42] vs 19·92 [4·27]). 27 human experts with more than 10 years of experience achieved a mean of 18·78 (SD 3·15) correct answers, compared with 25·43 (1·95) correct answers for the top three machine algorithms (mean difference 6·65, 95% CI 6·06-7·25; p<0·0001). The difference between human experts and the top three algorithms was significantly lower for images in the test set that were collected from sources not included in the training set (human underperformance of 11·4%, 95% CI 9·9-12·9 vs 3·6%, 0·8-6·3; p<0·0001).Interpretation State-of-the-art machine-learning classifiers outperformed human experts in the diagnosis of pigmented skin lesions and should have a more important role in clinical practice. However, a possible limitation of these algorithms is their decreased performance for out-of-distribution images, which should be addressed in future research.Funding None.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A study on comparison of diagnostic performance for skin cancer recognition revealed that AI algorithms achieve better accuracy than human experts.
“A study on the comparison of diagnostic performance for skin cancer recognition revealed that AI algorithms achieve better accuracy than human experts [143].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [143]
The improvement of AI over human experts was particularly prominent for the diagnosis of pigmented actinic keratoses.
“This improvement was particularly prominent for the diagnosis of pigmented actinic keratoses.”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [143]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
ggplot2: Elegant Graphics for Data Analysis
Ginestet, C. E. (2011) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2010.00676_9.x · Cited by 14,484 · see citing works
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
Tschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628 · see citing works
Self-citationCodella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620 · see citing works
Self-citationIncidence Estimate of Nonmelanoma Skin Cancer (Keratinocyte Carcinomas) in the US Population, 2012
Rogers, H. W., Weinstock, M. A., Feldman, S. R., & Coldiron, B. M. (2015) · JAMA Dermatology · doi:10.1001/jamadermatol.2015.1187 · Cited by 1,815 · see citing works
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571 · see citing works
Self-citationDiagnostic accuracy of dermoscopy
Kittler, H., Pehamberger, H., Wolff, K., & Binder, M. (2002) · The Lancet Oncology · doi:10.1016/s1470-2045(02)00679-4 · Cited by 1,349 · see citing works
Self-citationHan, S. S., Kim, M. S., Lim, W., Park, G., Park, I., & Chang, S. E. (2018) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2018.01.028 · Cited by 715 · see citing works
Deep learning ensembles for melanoma recognition in dermoscopy images
Codella, N., Nguyen, Q.-B., Pankanti, S., Gutman, D. A., Helba, B., Halpern, A. C., & Smith, J. R. (2017) · IBM Journal of Research and Development · doi:10.1147/jrd.2017.2708299 · Cited by 665 · see citing works
Self-citationMarchetti, M. A., Codella, N., Dusza, S. W., Gutman, D. A., Helba, B., Kalloo, A., Mishra, N. K., Carrera, C., Celebi, M. E., DeFazio, J., et al. (2017) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2017.08.016 · Cited by 329 · see citing works
Self-citationMonheit, G. D., Cognetta, A. B., Ferris, L. K., Rabinovitz, H. S., Gross, K. G., Martini, M. C., Grichnik, J. M., Mihm, M. C., Prieto, V. G., Googe, P. B., et al. (2010) · Archives of Dermatology · doi:10.1001/archdermatol.2010.302 · Cited by 232 · see citing works
Diagnostic accuracy of dermatoscopy for melanocytic and nonmelanocytic pigmented lesions
Rosendahl, C. O., Tschandl, P., Cameron, A., & Kittler, H. (2011) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2010.03.039 · Cited by 215 · see citing works
Self-citationMalvehy, J., Hauschild, A., Curiel‐Lewandrowski, C. N., Mohr, P., Hofmann‐Wellenhof, R., Motley, R., Berking, C., Grossman, D., Paoli, J., Loquai, C., et al. (2014) · British Journal of Dermatology · doi:10.1111/bjd.13121 · Cited by 210 · see citing works
Self-citationManagement Reasoning: Beyond the Diagnosis
Cook, D. A., Sherbino, J., & Durning, S. J. (2018) · JAMA · doi:10.1001/jama.2018.4385 · Cited by 170 · see citing works
Menzies, S. W., Bischof, L., Talbot, H., Gutenev, A., Avramidis, M., Wong, L., Lo, S. K., Mackellar, G., Skladnev, V., McCarthy, W. H., et al. (2005) · Archives of Dermatology · doi:10.1001/archderm.141.11.1388 · Cited by 156 · see citing works
Self-citationBinder, M., Steiner, A., Schwarz, M., Knollmayer, S., Wolff, K., & Pehamberger, H. (1994) · British Journal of Dermatology · doi:10.1111/j.1365-2133.1994.tb03378.x · Cited by 110 · see citing works
Ugly Duckling Sign as a Major Factor of Efficiency in Melanoma Detection
Gaudy‐Marqueste, C., Wazaefi, Y., Bruneu, Y., et al. (2017) · JAMA Dermatology · doi:10.1001/jamadermatol.2016.5500 · Cited by 102 · see citing works
Proposed Technical Guidelines for the Acquisition of Clinical Images of Skin-Related Conditions
Finnane, A., Curiel-Lewandrowski, C., Wimberley, G., Caffery, L., Katragadda, C., Halpern, A., Marghoob, A. A., Malvehy, J., Kittler, H., Hofmann-Wellenhof, R., et al. (2017) · JAMA Dermatology · doi:10.1001/jamadermatol.2016.6214 · Cited by 89 · see citing works
Self-citationDreiseitl, S., Binder, M., Hable, K., & Kittler, H. (2009) · Melanoma Research · doi:10.1097/cmr.0b013e32832a1e41 · Cited by 72 · see citing works
Self-citationRosendahl, C. O., Williams, G. M., Eley, D. S., et al. (2012) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2011.12.030 · Cited by 59 · see citing works
Self-citationArtificial Swarm Intelligence employed to Amplify Diagnostic Accuracy in Radiology
Rosenberg, L. B., Lungren, M. P., Halabi, S. S., Willcox, G., Baltaxe, D., & Lyons, M. (2018) · IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) · doi:10.1109/iemcon.2018.8614883 · Cited by 58 · see citing works
Factors driving the use of dermoscopy in Europe: a pan-European survey
Forsea, A., Tschandl, P., del Mármol, V., et al. (2016) · British Journal of Dermatology · doi:10.1111/bjd.14895 · Cited by 40 · see citing works
Self-citationSaphier, J. (1921) · Archives of Dermatological Research · doi:10.1007/bf01991493 · Cited by 26 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [18]Stoyanov D, Taylor Z, Sarikaya D, OR 2.0 context-aware operating theaters, computer assisted robotic endoscopy, clinical image-based procedures, and skin image analysis. First International Workshop, OR 2.0 2018, 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, Third International Workshop, ISIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings. New York: Springer, 2018.
- [28]Codella NCF, Lin C-C, Halpern A, Hind M, Feris R, Smith JR. Collaborative human-AI (CHAI): Evidence-based interpretable melanoma classification in dermoscopic images. Understanding and interpreting machine learning in medical image computing applications. New York: Springer International Publishing, 2018.
Cited by
Sorted by citations
Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology
Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V., & Madabhushi, A. (2019) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-019-0252-y · Cited by 1,698 · see citing works
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024 · see citing works
Self-citationSalahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508 · see citing works
Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V., & Zou, J. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.3129 · Cited by 403 · see citing works
What is AI? Applications of artificial intelligence to dermatology
Du‐Harpur, X., Watt, F. M., Luscombe, N. M., & Lynch, M. D. (2020) · British Journal of Dermatology · doi:10.1111/bjd.18880 · Cited by 355 · see citing works
Analysis of the ISIC image datasets: Usage, benchmarks and recommendations
Cassidy, B., Kendrick, C., Brodzicki, A., Jaworek-Korjakowska, J., & Yap, M. H. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102305 · Cited by 307 · see citing works
Hekler, A., Utikal, J. S., Enk, A. H., Solass, W., Schmitt, M., Klode, J., Schadendorf, D., Sondermann, W., Franklin, C., Bestvater, F., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.06.012 · Cited by 285 · see citing works
Haggenmüller, S., Maron, R. C., Hekler, A., Utikal, J. S., Barata, C., Barnhill, R. L., Beltraminelli, H., Berking, C., Betz-Stablein, B., Blum, A., et al. (2021) · European Journal of Cancer · doi:10.1016/j.ejca.2021.06.049 · Cited by 267 · see citing works
Artificial Intelligence in Dermatology: A Primer
Young, A. T., Xiong, M., Pfau, J., Keiser, M. J., & Wei, M. L. (2020) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2020.02.026 · Cited by 249 · see citing works
Han, S. S., Park, I., Eun Chang, S., Lim, W., Kim, M. S., Park, G. H., Chae, J. B., Huh, C. H., & Na, J.-I. (2020) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2020.01.019 · Cited by 248 · see citing works
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212 · see citing works
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189 · see citing works
Jain, A., Way, D., Gupta, V., Gao, Y., de Oliveira Marinho, G., Hartford, J., Sayres, R., Kanada, K., Eng, C., Nagpal, K., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.7249 · Cited by 164 · see citing works
Brinker, T. J., Schmitt, M., Krieghoff-Henning, E. I., Barnhill, R., Beltraminelli, H., Braun, S. A., Carr, R., Fernandez-Figueras, M.-T., Ferrara, G., Fraitag, S., et al. (2022) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2021.02.009 · Cited by 70 · see citing works
The role of technology in melanoma screening and diagnosis
Young, A. T., Vora, N. B., Cortez, J., Tam, A., Yeniay, Y., Afifi, L., Yan, D., Nosrati, A., Wong, A., Johal, A., et al. (2021) · Pigment Cell & Melanoma Research · doi:10.1111/pcmr.12907 · Cited by 60 · see citing works
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58 · see citing works
Felmingham, C. M., Adler, N. R., Ge, Z., Morton, R. L., Janda, M., & Mar, V. J. (2021) · American Journal of Clinical Dermatology · doi:10.1007/s40257-020-00574-4 · Cited by 57 · see citing works
Sangers, T. E., Wakkee, M., Kramer‐Noels, E. C., Nijsten, T., & Lugtenberg, M. (2021) · British Journal of Dermatology · doi:10.1111/bjd.20441 · Cited by 52 · see citing works
The use and future perspective of Artificial Intelligence—A survey among German surgeons
Pecqueux, M., Riediger, C., Distler, M., Oehme, F., Bork, U., Kolbinger, F. R., Schöffski, O., van Wijngaarden, P., Weitz, J., Schweipert, J., et al. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.982335 · Cited by 41 · see citing works
Sangers, T. E., Wakkee, M., Moolenburgh, F. J., Nijsten, T., & Lugtenberg, M. (2022) · Archives of Dermatological Research · doi:10.1007/s00403-022-02492-3 · Cited by 38 · see citing works
Escalé-Besa, A., Vidal-Alaball, J., Miró Catalina, Q., Gracia, V. H. G., Marin-Gomez, F. X., & Fuster-Casanovas, A. (2024) · Healthcare · doi:10.3390/healthcare12121192 · Cited by 29 · see citing works
Zhu, A.-Q., Wang, Q., Shi, Y.-L., Ren, W.-W., Cao, X., Ren, T.-T., Wang, J., Zhang, Y.-Q., Sun, Y.-K., Chen, X.-W., et al. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102391 · Cited by 25 · see citing works
Enhancing skin lesion classification: a CNN approach with human baseline comparison
Ajabani, D., Shaikh, Z. A., Yousef, A., Ali, K., & Albahar, M. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2795 · Cited by 18 · see citing works
Mehta, D., Betz-Stablein, B., Nguyen, T. D., Gal, Y., Bowling, A., Haskett, M., Sashindranath, M., Bonnington, P., Mar, V., Soyer, H. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.01009 · Cited by 3 · see citing works
Self-citation
- Authors
- Tschandl, P., Codella, N., Akay, B. N., Argenziano, G., Braun, R. P., Cabo, H., Gutman, D., Halpern, A., Helba, B., Hofmann-Wellenhof, R., Lallas, A., Lapins, J., Longo, C., Malvehy, J., Marchetti, M. A., Marghoob, A., Menzies, S., Oakley, A., Paoli, J., Puig, S., Rinner, C., Rosendahl, C., Scope, A., Sinz, C., Soyer, H. P., Thomas, L., Zalaudek, I., & Kittler, H.
- Affiliations
- Medical University of Vienna (AT)
- IBM (United States) (US)
- Ankara University (TR)
- University of Campania "Luigi Vanvitelli" (IT)
- University Hospital Zurich (CH)
- Institute for Medical Research (AR)
- Emory University (US)
- Memorial Sloan Kettering Cancer Center (US)
- Kitware (United States) (US)
- Aristotle University of Thessaloniki (GR)
- Karolinska University Hospital (SE)
- University of Modena and Reggio Emilia (IT)
- Instituto de Salud Carlos III (ES)
- The University of Sydney (AU)
- University of Auckland (NZ)
- University of Gothenburg (SE)
- The University of Queensland (AU)
- Tel Aviv University (IL)
- Lyon 1 Université (FR)
- University of Trieste (IT)
- Corresponding author
- Kittler, Harald
- ORCID
- Tschandl, Philipp
- Codella, Noel
- Akay, Bengü Nisa
- Argenziano, Giuseppe
- Braun, Ralph P
- Cabo, Horacio
- Gutman, David
- Halpern, Allan
- Helba, Brian
- Lallas, Aimilios
- Lapins, Jan
- Longo, Caterina
- Malvehy, Josep
- Marchetti, Michael A
- Marghoob, Ashfaq
- Menzies, Scott
- Oakley, Amanda
- Paoli, John
- Puig, Susana
- Rosendahl, Cliff
- Scope, Alon
- Sinz, Christoph
- Soyer, H Peter
- Thomas, Luc
- Zalaudek, Iris
- Kittler, Harald
- Venue
- The Lancet Oncology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 20
- Issue
- 7
- Pages
- 938-947
- ISSN
- 1470-2045 (ISSN-L)
- PMC
- PMC8237239
- OpenAlex
- W2952971376
- Cited by
- 555
- References
- 27
- Citations by year
- 2026: 63 · 2025: 71 · 2024: 77 · 2023: 90 · 2022: 72 · 2021: 88 · 2020: 75 · 2019: 16 · 2014: 1
- FWCI
- 26.02
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- The Lancet Oncology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- HAL (submittedVersion)
- ArTS Archivio della ricerca di Trieste (University of Trieste https://www.units.it/) (submittedVersion, cc-by-nc-nd)
- Archivio Istituzionale della Ricerca - Università degli Studi della Campania "Luigi Vanvitelli" (submittedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- ResearchSpace (University of Auckland) (submittedVersion)
- Funders
- National Institutes of Health (P30 CA008748)
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- pigmented skin lesions, dermoscopic images, machine learning algorithms, diagnostic accuracy, melanoma, basal cell carcinoma, seborrheic keratosis, actinic keratosis, melanocytic nevi, dermatofibroma, vascular skin lesions, general practitioners, skin cancer diagnosis
- MeSH terms
- Machine Learning, Adult, Algorithms, Female, Humans, Male, Pigmentation Disorders, Retrospective Studies, Skin Neoplasms, Reproducibility of Results, Internet, Dermoscopy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
