CoChat
Sign Up
Cited by 164Artificial Intelligence in Healthcare and EducationCutaneous Melanoma Detection and ManagementArtificial Intelligence in Healthcare

Development and assessment of an artificial intelligence–based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices

Open access
Jain, Ayush, Way, David, Gupta, Vishakha, Gao, Yi, de Oliveira Marinho, Guilherme, Hartford, Jay, Sayres, Rory, Kanada, Kimberly +12 more·2021·JAMA Network Open
DOI 10.1001/jamanetworkopen.2021.7249PMC PMC8082316
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Most dermatologic cases are initially evaluated by nondermatologists such as primary care physicians (PCPs) or nurse practitioners (NPs). Objective: To evaluate an artificial intelligence (AI)-based tool that assists with diagnoses of dermatologic conditions. Design, Setting, and Participants: This multiple-reader, multiple-case diagnostic study developed an AI-based tool and evaluated its utility. Primary care physicians and NPs retrospectively reviewed an enriched set of cases representing 120 different skin conditions. Randomization was used to ensure each clinician reviewed each case either with or without AI assistance; each clinician alternated between batches of 50 cases in each modality. The reviews occurred from February 21 to April 28, 2020. Data were analyzed from May 26, 2020, to January 27, 2021. Exposures: An AI-based assistive tool for interpreting clinical images and associated medical history. Main Outcomes and Measures: The primary analysis evaluated agreement with reference diagnoses provided by a panel of 3 dermatologists for PCPs and NPs. Secondary analyses included diagnostic accuracy for biopsy-confirmed cases, biopsy and referral rates, review time, and diagnostic confidence. Results: Forty board-certified clinicians, including 20 PCPs (14 women [70.0%]; mean experience, 11.3 [range, 2-32] years) and 20 NPs (18 women [90.0%]; mean experience, 13.1 [range, 2-34] years) reviewed 1048 retrospective cases (672 female [64.2%]; median age, 43 [interquartile range, 30-56] years; 41 920 total reviews) from a teledermatology practice serving 11 sites and provided 0 to 5 differential diagnoses per case (mean [SD], 1.6 [0.7]). The PCPs were located across 12 states, and the NPs practiced in primary care without physician supervision across 9 states. The NPs had a mean of 13.1 (range, 2-34) years of experience and practiced in primary care without physician supervision across 9 states. Artificial intelligence assistance was significantly associated with higher agreement with reference diagnoses. For PCPs, the increase in diagnostic agreement was 10% (95% CI, 8%-11%; P < .001), from 48% to 58%; for NPs, the increase was 12% (95% CI, 10%-14%; P < .001), from 46% to 58%. In secondary analyses, agreement with biopsy-obtained diagnosis categories of maglignant, precancerous, or benign increased by 3% (95% CI, -1% to 7%) for PCPs and by 8% (95% CI, 3%-13%) for NPs. Rates of desire for biopsies decreased by 1% (95% CI, 0-3%) for PCPs and 2% (95% CI, 1%-3%) for NPs; the rate of desire for referrals decreased by 3% (95% CI, 1%-4%) for PCPs and NPs. Diagnostic agreement on cases not indicated for a dermatologist referral increased by 10% (95% CI, 8%-12%) for PCPs and 12% (95% CI, 10%-14%) for NPs, and median review time increased slightly by 5 (95% CI, 0-8) seconds for PCPs and 7 (95% CI, 5-10) seconds for NPs per case. Conclusions and Relevance: Artificial intelligence assistance was associated with improved diagnoses by PCPs and NPs for 1 in every 8 to 10 cases, indicating potential for improving the quality of dermatologic care.

AI snapshot

Written from the paper’s full text.

TL;DR

In a multiple-reader, multiple-case diagnostic study, AI assistance improved primary care physicians’ and nurse practitioners’ top-1 diagnostic agreement with a dermatologist panel for teledermatology skin cases. The absolute improvement was 10% for PCPs and 12% for NPs, with modest increases in review time.

Key findings

  • Among 40 clinicians (20 PCPs and 20 NPs) reviewing 1048 cases, AI assistance increased top-1 agreement from 48% to 58% for PCPs (10%; 95% CI, 8%-11%; P < .001) and from 46% to 58% for NPs (12%; 95% CI, 10%-14%; P < .001), corresponding to benefit for 1 in every 8 to 10 cases.
  • For 141 biopsy-confirmed growth cases, accuracy at classifying lesions as malignant, precancerous, or benign trended upward by 3% for PCPs (95% CI, −1% to 7%) from 64% to 67% and by 8% for NPs (95% CI, 3%-13%) from 60% to 68%.
  • AI assistance was associated with lower rates of recommending biopsy (1% lower for PCPs; 2% lower for NPs) and lower referral desire (3% lower for both), higher self-reported diagnostic confidence, and higher agreement in nonreferred cases (PCPs 51% to 61%; NPs 47% to 59%).
  • Median review time increased from 89 to 94 seconds for PCPs and from 77 to 84 seconds for NPs.

Methods

  • Multiple-reader, multiple-case diagnostic study with 20 PCPs and 20 NPs; each reader reviewed each of 1048 retrospective teledermatology cases only once, randomized to AI-assisted or unassisted modality, with modality alternating every 50 cases.
  • Cases came from California and Hawaii data sets previously used to validate the AI algorithm, included 120 different skin conditions, 3935 images, and 1016 patients; reference diagnoses were provided by a panel of 3 US board-certified dermatologists using a collective intelligence approach.
  • Primary end points were top-1 agreement of PCPs and NPs with the reference diagnosis; analyses used permutation tests with 1000 iterations, with Bonferroni-corrected significance at P < .0125, and biopsy-confirmed subset analyses for 141 growth cases.

Limitations

  • The paper states that cases were teledermatology cases mixing primary care referrals and patient-requested submissions, potentially affecting difficulty; Fitzpatrick skin types I and V were underrepresented and type VI was absent; AI improvements for malignant neoplasms were lower than across all cases; the one-modality-per-case/reader design precludes case- and reader-specific inferences; and store-and-forward cases limited follow-up questions and tests.
  • The text makes evident that this was a retrospective diagnostic study, not a prospective clinical-impact study, and the authors state that prospective studies are warranted for telemedicine and in-person primary care.
  • The text also shows AI assistance could reduce agreement when the AI predictions were incorrect: when none of the AI tool’s predictions was correct (13% of cases), agreement was 8% lower for PCPs and 9% lower for NPs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2021.7249

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 32 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Similar improvements were observed beyond the primary diagnosis based on the top-3 agreement, average overlap, per-condition sensitivity, and κ value.

    “Similar improvements were observed beyond the primary diagnosis based on the top-3 agreement, average overlap, per-condition sensitivity, and κ value (eFigures 4 and 5 and eTable 3 in the Supplement).”
    Quote in the paperSubstantial claim
  • Findingown data

    For cases with available biopsy diagnoses (n = 141), NPs' accuracy at classifying lesions as malignant, precancerous, or benign trended upward by 8% (95% CI, 3%-13%) from 60% to 68%.

    “by 8% for NPs (95% CI, 3%-13%) from 60% to 68%”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The biopsy-confirmed diagnoses are accurate ground truth for the biopsy subset.
  • Findingown data

    For PCPs, AI assistance was associated with an increase in diagnostic agreement of 10% (95% CI, 8%-11%; P < .001), from 48% to 58%.

    “For PCPs, the increase in diagnostic agreement was 10% (95% CI, 8%-11%; P < .001), from 48% to 58%;”
    • Also stated in: § Findings · plain
      “with an increase from 48% to 58% for primary care physicians”
    Quote in the paperSubstantial claim
  • Findingown data

    For NPs, AI assistance was associated with an improvement of 12% (95% CI, 10%-14%; P < .001), from 46% to 58%.

    “for NPs, the improvement was 12% (95% CI, 10%-14%; P < .001), from 46% to 58%.”
    • Also stated in: § Findings · plain
      “an increase from 46% to 58% for nurse practitioners”
    Quote in the paperSubstantial claim
  • Findingown data

    For cases with available biopsy diagnoses (n = 141), PCPs' accuracy at classifying lesions as malignant, precancerous, or benign trended upward by 3% (95% CI, −1% to 7%) from 64% to 67%.

    “trended upward by 3% for PCPs (95% CI, −1% to 7%) from 64% to 67%”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The biopsy-confirmed diagnoses are accurate ground truth for the biopsy subset.
  • Classified as probably not a substantial claim. 12
  • May combine two claims. 15

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of 152 cases with available biopsy results, the diagnosis of 141 cases was growths.

    “of 152 cases with available biopsy results, the diagnosis of 141 cases was growths.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    AI assistance was associated with improvements for all 40 readers, although the magnitude varied by reader (range, 2%-22%; median, 10%).

    “Assistance was associated with improvements for all 40 readers, although the magnitude varied by reader (range, 2%-22%; median, 10%) (Figure 2B).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Assistance-associated benefits were consistent during the study and across several skin types.

    “Subanalyses also indicated that assistance-associated benefits were consistent during the study and across several skin types (eFigures 9 and 10 in the Supplement).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The top-1 agreement of cases rated with more than 90% confidence was substantially higher: 73% vs 64% for PCPs and 68% vs 58% for NPs.

    “The top-1 agreement of cases that were rated with more than 90% confidence was substantially higher (73% vs 64% for PCPs and 68% vs 58% for NPs)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Dermatologists' reference diagnoses are an accurate standard for skin condition diagnosis.

  • May combine two claims.

    Findingown data

    Subgroup analysis found that sensitivity for malignant lesions, precancerous lesions, infectious skin diseases, and categories of hair loss trended upward or remained similar with assistance for both NPs and PCPs, with improvements ranging from −1% to 36%.

    “Subgroup analysis further found that sensitivity for malignant lesions, precancerous lesions, infectious skin diseases, and categories of hair loss trended upward or remained similar with assistance for both NPs and PCPs, with improvements ranging from −1% to 36% (eTable 4 in the Supplement).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Examination of some skin conditions by dermatologists results in significantly higher diagnostic accuracy and is associated with better clinical outcomes than nondermatologist examination.

    “The examination of some skin conditions by dermatologists results in significantly higher diagnostic accuracy2,3,4 and is associated with better clinical outcomes5 than nondermatologist examination.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [2] Chen SC, Pennie ML, Kolm P,. Diagnosing and managing cutaneous pigmented lesions: primary care physicians versus dermatologists. J Gen Intern Med. 2006; 21(7): 678- 682. doi: 10.1111/j.1525-1497.2006.00462.x 16808765 PMC1924688 source

      Not checked in the free run.

    • cites [3] Goulding JMR, Levine S, Blizard RA, Deroide F, Swale VJ. Dermatological surgery: a comparison of activity and outcomes in primary and secondary care. Br J Dermatol. 2009; 161(1): 110- 114. doi: 10.1111/j.1365-2133.2009.09228.x 19438854 source

      Not checked in the free run.

    • cites [4] Federman DG, Concato J, Kirsner RS. Comparison of dermatologic diagnoses by primary care practitioners and dermatologists: a review of the literature. Arch Fam Med. 1999; 8(2): 170- 172. doi: 10.1001/archfami.8.2.170 10101989 source

      Not checked in the free run.

    • cites [5] Pennie ML, Soon SL, Risser JB, Veledar E, Culler SD, Chen SC. Melanoma outcomes for Medicare patients: association of stage and survival with detection by a dermatologist vs a nondermatologist. Arch Dermatol. 2007; 143(4): 488- 494. doi: 10.1001/archderm.143.4.488 17438181 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The 2 reader cohorts read the same cases but with the opposite assistance modalities for each case.

    “The 2 reader cohorts read the same cases but with the opposite assistance modalities (ie, unassisted vs AI assisted) for each case.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Three US board-certified dermatologists independently reviewed each case.

    “Briefly, 3 US board-certified dermatologists (from a pool of 12) independently reviewed each case.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each reader was randomly assigned to 1 of 2 reader cohorts.

    “Each reader was randomly assigned to 1 of 2 reader cohorts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Sensitivity analysis using a permutation test that preserved the reader cohorts' structure and another statistical analysis via a generalized linear mixed model produced similar results.

    “Sensitivity analysis using a permutation test that preserved the reader cohorts’ structure and another statistical analysis via a generalized linear mixed model produced similar results (see the Alternative Statistical Analyses section in the eMethods in the Supplement).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several algorithms incorporating artificial intelligence have been developed to help interpret clinical and dermoscopic images for a variety of skin conditions.

    “Several algorithms incorporating artificial intelligence (AI) have been developed to help interpret both clinical11,12,13,14,15 and dermoscopic16,17,18,19,20,21,22,23 images for a variety of skin conditions”
    Quote in the paperNot a substantial claim0 of 10 sources checked
    • cites [11] Esteva A, Kuprel B, Novoa RA,. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639): 115- 118. doi: 10.1038/nature21056 28117445 PMC8382232 source

      Not checked in the free run.

    • cites [12] Han SS, Park GH, Lim W,. Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network. PLoS One. 2018; 13(1): e0191493. doi: 10.1371/journal.pone.0191493 29352285 PMC5774804 source

      Not checked in the free run.

    • cites [13] Sun X, Yang J, Sun M, Wang K. A benchmark for automatic visual classification of clinical skin disease images. In: Liebe B, Matas J, Seba N, Welling M, eds. Computer Vision–ECCV 2016. Springer; 2016:206-222. Lecture Notes in Computer Science; vol 9910. https://link.springer.com/chapter/10.1007/978-3-319-46466-4_13 source

      Not checked in the free run.

    • cites [14] Derm101.com. Derm101. Accessed August 9, 2019. https://www.derm101.com/

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [15] Han SS, Park I, Eun Chang S,. Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders. J Invest Dermatol. 2020; 140(9): 1753- 1761. doi: 10.1016/j.jid.2020.01.019 32243882 source

      Not checked in the free run.

    • cites [16] Cruz-Roa AA, Arevalo Ovalle JE, Madabhushi A, González Osorio FA. A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. Med Image Comput Comput Assist Interv. 2013; 16(pt 2): 403- 410. doi: 10.1007/978-3-642-40763-5_50 24579166 source

      Not checked in the free run.

    • cites [17] Codella NCF, Gutman D, Emre Celebi M,. Skin lesion analysis toward melanoma detection: a challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE; 2018. https://faculty.uca.edu/ecelebi/documents/ISBI_2018.pdf

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [18] Yuan Y, Chao M, Lo YC. Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance. IEEE Trans Med Imaging. 2017; 36(9): 1876- 1886. doi: 10.1109/TMI.2017.2695227 28436853 source

      Not checked in the free run.

    • cites [19] Haenssle HA, Fink C, Schneiderbauer R,; Reader Study Level-I and Level-II Groups. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann Oncol. 2018; 29(8): 1836- 1842. doi: 10.1093/annonc/mdy166 29846502 source

      Not checked in the free run.

    • cites [20] Brinker TJ, Hekler A, Enk AH,. Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur J Cancer. 2019; 113: 47- 54. doi: 10.1016/j.ejca.2019.04.001 30981091 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The study was designed to evaluate 2 prespecified primary end points: the agreement rate of the primary differential diagnosis of PCPs with the reference diagnosis and the agreement rate of the primary differential diagnosis of NPs with the reference diagnosis.

    “Our study was designed to evaluate 2 prespecified primary end points: (1) the agreement rate of the primary differential diagnosis of the PCPs with the reference diagnosis and (2) the agreement rate of the primary differential diagnosis of the NPs with the reference diagnosis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Agreement was also assessed against biopsy-confirmed diagnoses when available.

    “Agreement was also assessed against biopsy-confirmed diagnoses when available.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Because the study had 2 prespecified primary end points (both 1-tailed superiority tests), the Bonferroni correction was applied, and P < .0125 was considered statistically significant.

    “Because this study had 2 prespecified primary end points (both 1-tailed superiority tests), we applied the Bonferroni correction, and P < .0125 was considered statistically significant (halved from α = 0.05 owing to 1-tailed tests and halved again owing to having 2 primary end points).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To reduce effects associated with switching modalities, the 1048 cases were divided into batches of 50 cases (except the last 48 cases, which were divided into 2 batches of 24 cases), and the assistance modality switched after each batch of cases.

    “To reduce effects associated with switching modalities, the 1048 cases were divided into batches of 50 cases (except the last 48 cases, which were divided into 2 batches of 24 cases), and the assistance modality switched after each batch of cases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The effect of AI-based support on dermoscopic images has been studied.

    “the effect of AI-based support on dermoscopic images has been studied.15,24”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [15] Han SS, Park I, Eun Chang S,. Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders. J Invest Dermatol. 2020; 140(9): 1753- 1761. doi: 10.1016/j.jid.2020.01.019 32243882 source

      Not checked in the free run.

    • cites [24] Tschandl P, Rinner C, Apalla Z,. Human-computer collaboration for skin cancer recognition. Nat Med. 2020; 26(8): 1229- 1234. doi: 10.1038/s41591-020-0942-0 32572267 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    Reference diagnoses were obtained using a previously described collective intelligence approach, which results in more reproducible diagnoses than diagnoses obtained by individual dermatologist review.

    “Reference diagnoses were obtained using a previously described collective intelligence approach, which results in more reproducible diagnoses than diagnoses obtained by individual dermatologist review (eTable 2 in the Supplement).26,30”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] Liu Y, Jain A, Eng C,. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26(6): 900- 908. doi: 10.1038/s41591-020-0842-3 32424212 source

      Not checked in the free run.

    • cites [30] Barnett ML, Boddupalli D, Nundy S, Bates DW. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA Netw Open. 2019; 2(3): e190096. doi: 10.1001/jamanetworkopen.2019.0096 30821822 PMC6484633 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Liu et al previously described an AI algorithm that provides a differential diagnosis given clinical photographs of skin conditions and the medical history.

    “Liu et al26 previously described an AI algorithm that provides a differential diagnosis given clinical photographs of skin conditions and the medical history (eTable 1 in the Supplement).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] Liu Y, Jain A, Eng C,. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26(6): 900- 908. doi: 10.1038/s41591-020-0842-3 32424212 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The AI model described by Liu et al was developed using 16 114 cases and used a convolutional neural network to output prediction scores across 419 skin conditions.

    “Their AI model was developed using 16 114 cases and used a convolutional neural network to output prediction scores across 419 skin conditions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Liu Y, Jain A, Eng C,. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26(6): 900- 908. doi: 10.1038/s41591-020-0842-3 32424212 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    None of the PCPs or NPs in this study previously reviewed the study cases, and the AI algorithm used was identical to the one used in the previous study.

    “None of the PCPs or NPs in this study previously reviewed these cases, and the AI algorithm used was identical to the one used in the previous study.26”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] Liu Y, Jain A, Eng C,. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26(6): 900- 908. doi: 10.1038/s41591-020-0842-3 32424212 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Nonspecialists have suboptimal diagnostic accuracy and have been shown to perform more biopsies while diagnosing fewer malignant neoplasms than dermatologists.

    “Nonspecialists have suboptimal diagnostic accuracy and have been shown to perform more biopsies while diagnosing fewer malignant neoplasms than dermatologists.37”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37] Anderson AM, Matsumoto M, Saul MI, Secrest AM, Ferris LK. Accuracy of skin cancer diagnosis by physician assistants compared with dermatologists in a large health care system. JAMA Dermatol. 2018; 154(5): 569- 573. doi: 10.1001/jamadermatol.2018.0212 29710082 PMC6128496

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Our randomized study design avoids any potential memory effects of reviewing the same case more than once.

    “Finally, our randomized study design avoids any potential memory effects of reviewing the same case more than once.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    These observations suggest that AI assistance improved skin condition diagnosis and diagnostic confidence of nonspecialists without incurring a reflexive increased use of referrals or biopsies.

    “These observations suggest that AI assistance improved skin condition diagnosis and diagnostic confidence of nonspecialists without incurring a reflexive increased use of referrals or biopsies.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Dermatologists' reference diagnoses are an accurate standard for skin condition diagnosis.
    • Rests on an assumption: The simulated clinical decisions (biopsy and referral) are valid proxies for actual clinical decisions.

  • May combine two claims.

    Study design and conductnone given

    In this study, 40 clinicians each reviewed 1048 teledermatology cases, with AI assistance for a random half of the cases and without AI assistance for the remaining half.

    “In this study, 40 clinicians each reviewed 1048 teledermatology cases, with AI assistance for a random half of the cases and without AI assistance for the remaining half.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The study's teledermatology cases were a mix of cases referred from primary care and cases submitted at the patient's request, which may have affected clinician diagnostic performance.

    “these were teledermatology cases that were a mix of cases that were referred from primary care and other cases that were submitted at the patient’s request. The potentially increased case difficulty and case enrichment may have affected clinician diagnostic performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    In the dataset, Fitzpatrick skin types I and V are underrepresented, and type VI is absent.

    “types I and V are underrepresented, and type VI is absent in this data set.”
    Quote in the paperSubstantial claim
26 of 26

  • The paper relies on this without support.

    The study cases are representative of skin conditions encountered in primary care teledermatology practices.

    “The study used cases from 2 retrospective data sets from California and Hawaii previously used to validate the AI algorithm.26”

  • The paper relies on this without support.

    The readers’ diagnostic performance in the study reflects their real-world clinical performance.

    “Primary care physicians and NPs reviewed these cases using a laptop and could consult additional resources as they would in clinical practice.”

  • The paper relies on this without support.

    The biopsy-confirmed diagnoses are accurate ground truth for the biopsy subset.

    “Agreement was also assessed against biopsy-confirmed diagnoses when available.”

  • The paper relies on this without support.

    Dermatologists' reference diagnoses are an accurate standard for skin condition diagnosis.

    “agreement rate with dermatologists’ reference diagnoses”
    “improved PCP and NP diagnostic agreement with dermatologists”

  • The paper relies on this without support.

    The simulated clinical decisions (biopsy and referral) are valid proxies for actual clinical decisions.

    “changes in several simulated clinical decisions”
    “Comparing Simulated Clinical Decisions by Clinicians When Assisted by Artificial Intelligence vs Unassisted”

  • The paper relies on this without support.

    Access to dermatologists is limited.

    “access to dermatologists is limited”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • Histologic diagnosis counts exceed the stated total

    Minor

    The counts in the histologic-diagnosis column sum to 180, although the column contains 152 cases. The categories are presented as primary diagnoses, so the counts should not exceed the number of cases.

    Reported 152 · recomputed 180

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. American Medical Association. Augmented intelligence in health care policy report. June 2018. Accessed June 3, 2020. https://www.ama-assn.org/system/files/2019-01/augmented-intelligence-policy-report.pdf

Cited by

Sorted by citations

Related

Authors
Jain, A., Way, D., Gupta, V., Gao, Y., de Oliveira Marinho, G., Hartford, J., Sayres, R., Kanada, K., Eng, C., Nagpal, K., DeSalvo, K. B., Corrado, G. S., Peng, L., Webster, D. R., Dunn, R. C., Coz, D., Huang, S. J., Liu, Y., Bui, P., & Liu, Y.
Affiliations
  • Google Health, Palo Alto, California
  • Google Health via Advanced Clinical, Deerfield, Illinois
  • University of California, San Francisco (US)
Corresponding author
Liu, Yun
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2021-04-28
Volume
4
Issue
4
Pages
e217249
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2021.7249
PMC
PMC8082316
OpenAlex
W3159814022
Semantic Scholar
f31ac31a7aff34bcc5759d0fa25d59e15db13fd2
Cited by
164
References
43
Citations by year
2026: 26 · 2025: 41 · 2024: 35 · 2023: 23 · 2022: 29 · 2021: 10
FWCI
6.98
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • JAMA Network Open (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
artificial intelligence, teledermatology, primary care physicians, nurse practitioners, diagnostic accuracy, diagnostic confidence, differential diagnosis, primary care dermatology, referral timing
MeSH terms
Adult, Artificial Intelligence, Dermatology, Diagnosis, Computer-Assisted, Female, Humans, Male, Middle Aged, Nurse Practitioners, Referral and Consultation, Skin Diseases, Telemedicine, Physicians, Primary Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review