CoChat
Sign Up
Cited by 87Cutaneous Melanoma Detection and ManagementAI in cancer detectionArtificial Intelligence in Healthcare and Education

Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis

Open access
Krakowski, Isabelle, Kim, Jiyeong, Cai, Zhuo Ran, Daneshjou, Roxana, Lapins, Jan, Eriksson, Hanna, Lykou, Anastasia, Linos, Eleni·2024·npj Digital Medicine
DOI 10.1038/s41746-024-01031-wPMC PMC11004168
Open this paper in CoChat — free accountOpen PDF

Abstract

The development of diagnostic tools for skin cancer based on artificial intelligence (AI) is increasing rapidly and will likely soon be widely implemented in clinical use. Even though the performance of these algorithms is promising in theory, there is limited evidence on the impact of AI assistance on human diagnostic decisions. Therefore, the aim of this systematic review and meta-analysis was to study the effect of AI assistance on the accuracy of skin cancer diagnosis. We searched PubMed, Embase, IEE Xplore, Scopus and conference proceedings for articles from 1/1/2017 to 11/8/2022. We included studies comparing the performance of clinicians diagnosing at least one skin cancer with and without deep learning-based AI assistance. Summary estimates of sensitivity and specificity of diagnostic accuracy with versus without AI assistance were computed using a bivariate random effects model. We identified 2983 studies, of which ten were eligible for meta-analysis. For clinicians without AI assistance, pooled sensitivity was 74.8% (95% CI 68.6-80.1) and specificity was 81.5% (95% CI 73.9-87.3). For AI-assisted clinicians, the overall sensitivity was 81.1% (95% CI 74.4-86.5) and specificity was 86.1% (95% CI 79.2-90.9). AI benefitted medical professionals of all experience levels in subgroup analyses, with the largest improvement among non-dermatologists. No publication bias was detected, and sensitivity analysis revealed that the findings were robust. AI in the hands of clinicians has the potential to improve diagnostic accuracy in skin cancer diagnosis. Given that most studies were conducted in experimental settings, we encourage future studies to further investigate these potential benefits in real-life settings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01031-w

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. [2]Wu E How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals Nat. Med. 2021 27 582 584 10.1038/s41591-021-01312-x 33820998 · DOI · Paper page
  2. [3]Yu K-H Beam AL Kohane IS Artificial intelligence in healthcare Nat. Biomed. Eng. 2018 2 719 731 10.1038/s41551-018-0305-z 31015651 · DOI · Paper page
  3. [4]Topol EJ High-performance medicine: the convergence of human and artificial intelligence Nat. Med. 2019 25 44 56 10.1038/s41591-018-0300-7 30617339 · DOI · Paper page
  4. [5]Kittler H Pehamberger H Wolff K Binder M Diagnostic accuracy of dermoscopy Lancet Oncol. 2002 3 159 165 10.1016/S1470-2045(02)00679-4 11902502 · DOI · Paper page
  5. [6]Marghoob AA Scope A The complexity of diagnosing melanoma J. Investig. Dermatol. 2009 129 11 13 10.1038/jid.2008.388 19078984 · DOI
  6. [7]Tschandl P Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study Lancet Oncol. 2019 20 938 947 10.1016/S1470-2045(19)30333-X 31201137 PMC8237239 · DOI · Paper page
  7. [8]Haenssle HA Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions Ann. Oncol. 2020 31 137 143 10.1016/j.annonc.2019.10.013 31912788 · DOI
  8. [9]Esteva A Dermatologist-level classification of skin cancer with deep neural networks Nature 2017 542 115 118 10.1038/nature21056 28117445 PMC8382232 · DOI · Paper page
  9. [10]Tschandl P Human–computer collaboration for skin cancer recognition Nat. Med. 2020 26 1229 1234 10.1038/s41591-020-0942-0 32572267 · DOI · Paper page
  10. [11]Ngiam KY Khor IW Big data and machine learning algorithms for health-care delivery Lancet Oncol. 2019 20 e262 e273 10.1016/S1470-2045(19)30149-4 31044724 · DOI · Paper page
  11. [12]Lee S Augmented decision-making for acral lentiginous melanoma detection using deep convolutional neural networks J. Eur. Acad. Dermatol. Venereol. 2020 34 1842 1850 10.1111/jdv.16185 31919901 · DOI
  12. [13]Cho SI Dermatologist-level classification of malignant lip diseases using a deep convolutional neural network Br. J. Dermatol. 2020 182 1388 1394 10.1111/bjd.18459 31449661 · DOI
  13. [15]Jain A Development and assessment of an artificial intelligence–based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices JAMA Netw. Open 2021 4 e217249 e217249 10.1001/jamanetworkopen.2021.7249 33909055 PMC8082316 · DOI · Paper page
  14. [16]Muñoz-López C Performance of a deep neural network in teledermatology: a single-centre prospective diagnostic study J. Eur. Acad. Dermatol. Venereol. 2021 35 546 553 10.1111/jdv.16979 33037709 PMC8274350 · DOI · Paper page
  15. [17]Jahn AS Over-detection of melanoma-suspect lesions by a CE-certified smartphone app: performance in comparison to dermatologists, 2D and 3D convolutional neural networks in a prospective data set of 1204 pigmented skin lesions involving patients’ perception Cancers 2022 14 3829 10.3390/cancers14153829 35954491 PMC9367531 · DOI
  16. [18]Lucius M Deep neural frameworks improve the accuracy of general practitioners in the classification of pigmented skin lesions Diagnostics 2020 10 969 10.3390/diagnostics10110969 33218060 PMC7698907 · DOI
  17. [19]Han SS Evaluation of artificial intelligence-assisted diagnosis of skin neoplasms: a single-center, paralleled, unmasked Randomized Controlled Trial. J. Investig. Dermatol. 2022 142 2353 2362.e2352 10.1016/j.jid.2022.02.003 35183551 · DOI
  18. [20]Kim YJ Augmenting the accuracy of trainee doctors in diagnosing skin lesions suspected of skin neoplasms in a real-world setting: a prospective controlled before-and-after study PLoS One 2022 17 e0260895 10.1371/journal.pone.0260895 35061692 PMC8782525 · DOI
  19. [21]Ba W Convolutional neural network assistance significantly improves dermatologists’ diagnosis of cutaneous tumours using clinical images Eur. J. Cancer 2022 169 156 165 10.1016/j.ejca.2022.04.015 35569282 · DOI
  20. [22]Maron RC Artificial intelligence and its effect on dermatologists’ accuracy in dermoscopic melanoma image classification: web-based survey study J. Med. Internet Res. 2020 22 e18091 10.2196/18091 32915161 PMC7519424 · DOI
  21. [23]Bowker AH A test for symmetry in contingency tables J. Am. Stat. Assoc. 1948 43 572 574 10.1080/01621459.1948.10483284 18123073 · DOI
  22. [24]Beltrami EJ Artificial intelligence in the detection of skin cancer J. Am. Acad. Dermatol. 2022 87 1336 1342 10.1016/j.jaad.2022.08.028 35998842 · DOI
  23. [25]Young AT Xiong M Pfau J Keiser MJ Wei ML Artificial intelligence in dermatology: a primer J. Investig. Dermatol. 2020 140 1504 1512 10.1016/j.jid.2020.02.026 32229141 · DOI · Paper page
  24. [26]Barata C A reinforcement learning model for AI-based decision support in skin cancer Nat. Med. 2023 29 1941 1946 10.1038/s41591-023-02475-5 37501017 PMC10427421 · DOI
  25. [28]Krakowski SM Haftor D Luger J Pashkevich N Raisch S Humans and algorithms in organizational decision making: evidence from a field experiment Acad. Manag. Proc. 2019 2019 16633 10.5465/AMBPP.2019.16633abstract · DOI
  26. [30]Vodrahalli, K., Daneshjou, R., Gerstenberg, T. & Zou, J. Do humans trust advice more if it comes from AI? An analysis of human-ai interactions. In Proc. 2022 AAAI/ACM Conference on AI, Ethics, and Society 763–777 (Association for Computing Machinery, Oxford, United Kingdom, 2022).
  27. [31]Ludolph R Schulz PJ Debiasing health-related judgments and decision making: a systematic review Med. Decis. Mak. 2018 38 3 13 10.1177/0272989X17716672 28649904 · DOI
  28. [33]Breck, E., Polyzotis, N., Roy, S., Whang, S. & Zinkevich, M. Data validation for machine learning. In Proceedings of the Conference on Systems and Machine Learning, (2019)
  29. [34]Daneshjou R Smith MP Sun MD Rotemberg V Zou J Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review JAMA Dermatol. 2021 157 1362 1369 10.1001/jamadermatol.2021.3129 34550305 PMC9379852 · DOI · Paper page
  30. [36]Groh, M. et al. Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1820-1828 (2021).
  31. [39]Daneshjou R Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR derm consensus guidelines from the international skin imaging collaboration artificial intelligence working group JAMA Dermatol. 2022 158 90 96 10.1001/jamadermatol.2021.4915 34851366 PMC9845064 · DOI
  32. [41]McInnes MDF Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement JAMA 2018 319 388 396 10.1001/jama.2017.19163 29362800 · DOI · Paper page
  33. [42]Whiting PF QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies Ann. Intern. Med. 2011 155 529 536 10.7326/0003-4819-155-8-201110180-00009 22007046 · DOI · Paper page
  34. [46]Kim KW Lee J Choi SH Huh J Park SH Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-Part I. General Guidance and Tips Korean J. Radio. 2015 16 1175 1187 10.3348/kjr.2015.16.6.1175 PMC4644738 26576106 · DOI
  35. [47]Takwoingi, Y. et al. Chapter 10: Undertaking meta-analysis. Draft version (4 October 2022) for inclusion in: Deeks, J. J., Bossuyt, P. M., Leeflang, M. M., Takwoingi, Y. In Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy 1–77 (Cochrane, London, 2022).
  36. [48]Zamora J Abraira V Muriel A Khan K Coomarasamy A Meta-DiSc: a software for meta-analysis of test accuracy data BMC Med. Res. Methodol. 2006 6 31 31 10.1186/1471-2288-6-31 16836745 PMC1552081 · DOI
  37. [50]Deeks JJ Macaskill P Irwig L The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed J. Clin. Epidemiol. 2005 58 882 893 10.1016/j.jclinepi.2005.01.016 16085191 · DOI · Paper page

Cited by

Sorted by citations

Authors
Krakowski, I., Kim, J., Cai, Z. R., Daneshjou, R., Lapins, J., Eriksson, H., Lykou, A., & Linos, E.
Affiliations
  • Karolinska Institutet (SE)
  • Stanford Medicine (US)
  • Karolinska University Hospital (SE)
  • University of Nicosia (CY)
Corresponding author
Linos, Eleni
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-04-09
Volume
7
Issue
1
Pages
78-78
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-024-01031-w
PMC
PMC11004168
OpenAlex
W4394620240
Cited by
87
References
12
Citations by year
2026: 45 · 2025: 35 · 2024: 7
FWCI
4.66
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Foundation for the National Institutes of Health (K24AR075060, R01AR082109)
  • Radiumhemmets Forskningsfonder (009614, 211063, 181083)
  • Cancerfonden (2111617Pj, 210406JCIA01)
  • Vetenskapsrådet (202201534)
  • Stockholms Läns Landsting (FoUI-962339, FoUI-972654)
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
skin cancer diagnosis, artificial intelligence, deep learning, AI-assisted diagnosis, diagnostic accuracy, sensitivity and specificity, human-AI interaction, dermatologists, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review