Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies

May combine two claims.

Six included studies verified results by using histology or follow-up, with 725 lesions.

“Six verified results by using histology or follow-up (n=725 lesions)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two CE marked apps are available for download.

“Two CE (Conformit Europenne) marked apps are available for download.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Lesion selection and image acquisition were performed by clinicians rather than smartphone users.

“Lesion selection and image acquisition were performed by clinicians rather than smartphone users.”

May combine two claims.

Three included studies verified results by using expert recommendations, with 407 lesions.

“three verified results by using expert recommendations (n=407 lesions)”

May combine two claims.

A later revision of the SkinVision app to allow for non-pigmented lesions led to a 15 percentage point increase in sensitivity for detection of melanoma when applied to the original pigmented lesion dataset (88%, 95% CI 70% to 98%); specificity dropped by 3 percentage points (79%, 70% to 86%).

“A later revision of the app to allow for non-pigmented lesions led to a 15 percentage point increase in sensitivity for the detection of melanoma when applied to the original pigmented lesion dataset (88%, 95% confidence interval 70% to 98%); however, specificity dropped by 3 percentage points (79%, 70% to 86%).33”

• cites [33] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The app named SkinScan in Chadwick 2014 is a predecessor of SkinVision and not the CE marked TeleSkin skinScan app.

“The app named SkinScan in Chadwick 201423 is a predecessor of SkinVision and not the CE marked TeleSkin skinScan app.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Studies evaluated six different named apps.

“Studies evaluated six different named apps.”

May combine two claims.

Because of scarcity of data and poor quality of studies, no meta-analysis was performed.

“Because of scarcity of data and poor quality of studies, we did not perform a meta-analysis.”

May combine two claims.

For moderate risk lesions, one app recommends lesion monitoring (Mole Detective) and two recommend consulting a doctor (SkinVision and Dr Mole), although with less urgency than implied for a high risk result.

“For moderate risk lesions, one app recommends lesion monitoring (Mole Detective) and two recommend consulting a doctor (SkinVision and Dr Mole), although with less urgency than implied for a high risk result.”

May combine two claims.

Other sources of variability included varying definitions of the target condition, different app versions, consideration of results of a short user questionnaire, and mode of image upload.

“Other sources of variability included varying definitions of the target condition (any malignant or premalignant lesion in one study,33 and melanoma only in five studies); different app versions (adaptations to improve the performance of apps for non-pigmented lesions and apps for different mobile phone platforms); consideration of results of a short user questionnaire; and mode of image upload (d”

Probably not a substantial claim: may not pass S3 (load-bearing).

After exclusions, the mean number of included lesions was 91 (median 108).

“After exclusions, the mean number of included lesions was 91 (median 108).”

May combine two claims.

When interpretation of SkinVision was varied to consider high and moderate results as test positive, sensitivity increased (between 17 and 57 percentage points) but specificity fell by 11-45 percentage points.

“When the interpretation of SkinVision was varied to consider high and moderate results from the app as test positive, sensitivity increased (between 17 and 57 percentage points) but at a considerable cost to specificity (falling by 11-45 percentage points).33”

• cites [33] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

For the SkinScan app, sensitivity was low regardless of whether moderate risk was combined with the low or high risk category (0% or 20% respectively), with corresponding specificities of 100% and 60%.

“Sensitivity was low regardless of whether moderate risk was combined with the low or high risk category (0% or 20% respectively), with corresponding specificities of 100% and 60% (fig 2).”

• cites [23] Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Two studies assessed one and three apps without disclosing their names.

“Two studies assessed one31 and three34 apps without disclosing their names.”

May combine two claims.

All but two of the identifiable apps report lesion recommendations as high, moderate, or low risk; SpotMole and skinScan do not feature a moderate risk result.

“All but two of the identifiable apps report lesion recommendations as high, moderate, or low risk (table 1); SpotMole and skinScan do not feature a moderate risk result.”

May combine two claims.

Four studies reported exclusion criteria, which included difficult to diagnose conditions, poor image quality, and unequivocal results obtained from the apps.

“Four studies reported exclusion criteria,23 24 28 34 which included difficult to diagnose conditions; poor image quality; and unequivocal results obtained from the apps.”

May combine two claims.

Sample sizes ranged from 15 to 199 lesions, with up to 45% of lesion images reported as unevaluable.

“Sample sizes ranged from 1523 to 199 lesions,30 with up to 45%30 of lesion images reported as unevaluable.”

May combine two claims.

Most diagnostic accuracy studies (n=5) aimed to differentiate between melanomas and benign lesions; one study included other types of skin cancer and premalignant lesions as the target condition.

“Most diagnostic accuracy studies (n=5) aimed to differentiate between melanomas and benign lesions23 28 31 32 34; one study33 included other types of skin cancer and premalignant lesions as the target condition.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Three studies reported between five and 10 attempts to obtain an adequate image for each lesion, and one study analysed a minimum of three images for each lesion.

“Three studies reported between five30 and 1023 24 attempts to obtain an adequate image for each lesion, and one study28 analysed a minimum of three images for each lesion.”

May combine two claims.

Two studies included any type of skin lesion, and all other studies restricted inclusion to pigmented or melanocytic lesions only.

“Two studies included any type of skin lesion,30 33 and all other studies restricted inclusion to pigmented or melanocytic lesions only.”

May combine two claims.

Five of the six studies aimed to detect melanoma only, and one aimed to differentiate between malignant or premalignant lesions and benign lesions.

“Five of the six studies aimed to detect melanoma only, and one33 aimed to differentiate between malignant (including melanoma, basal cell carcinoma, and squamous cell carcinoma) or premalignant lesions and benign lesions.”

May combine two claims.

When participant responses to in-app questions about lesion characteristics and symptoms were included, sensitivity increased further to 80% (63% to 92%) and specificity to 78% (67% to 87%).

“When participant responses to in-app questions about lesion characteristics and symptoms were included, sensitivity increased further to 80% (63% to 92%) and specificity to 78% (67% to 87%).33”

• cites [33] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

More than a third of studies (30/80, 37.5%) were excluded on the basis of the index test.

“We excluded more than a third of studies (30/80, 37.5%) on the basis of the index test.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Only the TeleSkin skinScan and SkinVision apps are currently available; MelApp and Mole Detective were withdrawn from the market after American Federal Trade Commission investigations; and Dr Mole and Spotmole appear to be no longer available.

“Only the TeleSkin skinScan and SkinVision apps are currently available; MelApp and Mole Detective were withdrawn from the market after American Federal Trade Commission investigations17 18; and Dr Mole and Spotmole appear to be no longer available.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The search identified 418 unique records, of which 64 were selected for full text assessment along with 16 studies identified for the original review.

“The search identified 418 unique records of which 64 were selected for full text assessment along with 16 studies identified for the original review”

May combine two claims.

The review shows small improvements over time in diagnostic accuracy of one currently available app (SkinVision) and a stark lack of valid evidence for the other app (SkinScan).

“Our review shows small improvements over time in the diagnostic accuracy of one currently available app (SkinVision) and a stark lack of valid evidence for the other app (SkinScan).”

May combine two claims.

Evaluations of apps with unknown availability and of those now withdrawn from the market because of "deceptive claims" were particularly small and with highly variable results.

“Evaluations of apps with unknown availability and of those now withdrawn from the market because of “deceptive claims” were particularly small and with highly variable results.”

May combine two claims.

When high and moderate risk results were considered as test positive, sensitivities of the SkinVision app ranged from 57% to 78%, and specificities from 27% to 50%.

“When high and moderate risk results were considered as test positive, sensitivities ranged from 57% to 78%, and specificities from 27% to 50% (fig 2).”

• cites [24] Ngoo, 2018, Efficacy of smartphone applications in high-risk pigmented lesions, Australasian J Dermatol sourceNot checked yet

  Not checked in the free run.

• cites [29] sourceNot checked yet

  Not checked in the free run.

• cites [30] Chung, 2018, Automated analysis of skin cancer app is unreliable, Ned Tijdschr Dermatol VenereolNot checked yet

  Not checked in the free run.

May combine two claims.

App recommendations commonly disagreed with histopathological results or clinical assessment, with some apps unable to identify any cases of melanoma.

“Additionally app recommendations commonly disagreed with histopathological results or clinical assessment, with some apps unable to identify any cases of melanoma.23”

• cites [23] Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Only two CE marked apps are known to be currently available for download in various parts of the world.

“Only two CE marked apps are known to be currently available for download in various parts of the world.”

May combine two claims.

For apps with uncertain availability or withdrawn apps, MelApp had low sensitivities (25-50%) and Mole Detective, Dr Mole, and SpotMole had low specificities (20-60%).

“with either low sensitivities (25-50% for MelApp) or specificities (20-60% for Mole Detective, Dr Mole, and SpotMole).”

• cites [23] Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84. sourceNot checked yet

  Not checked in the free run.

• cites [24] Ngoo, 2018, Efficacy of smartphone applications in high-risk pigmented lesions, Australasian J Dermatol sourceNot checked yet

  Not checked in the free run.

• cites [32] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

When apps failed to return a risk assessment, images were either excluded a priori from the studies, excluded from analysis, or were not reported.

“When apps failed to return a risk assessment, images were either excluded a priori from the studies,23 excluded from analysis,24 28 30 31 32 34 or were not reported.29”

• cites [23] Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84. sourceNot checked yet

  Not checked in the free run.

• cites [24] Ngoo, 2018, Efficacy of smartphone applications in high-risk pigmented lesions, Australasian J Dermatol sourceNot checked yet

  Not checked in the free run.

• cites [28] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The SkinVision app was verified against expert recommendations for further investigation or intervention, presumably using the original version of the app.

“The app was verified against expert recommendations for further investigation or intervention, presumably using the original version of the app.24 29 30”

• cites [24] Ngoo, 2018, Efficacy of smartphone applications in high-risk pigmented lesions, Australasian J Dermatol sourceNot checked yet

  Not checked in the free run.

• cites [29] sourceNot checked yet

  Not checked in the free run.

• cites [30] Chung, 2018, Automated analysis of skin cancer app is unreliable, Ned Tijdschr Dermatol VenereolNot checked yet

  Not checked in the free run.

May combine two claims.

When only high risk results were considered as test positive, between 25% (95% CI 3% to 65%) and 67% (30% to 93%) of lesions that the dermatologist considered to require further investigation were picked up by the SkinVision app, with specificities of 90% and 61%, respectively.

“When only high risk results were considered as test positive, between 25% (95% confidence interval 3% to 65%)29 and 67% (30% to 93%)30 of lesions that the dermatologist considered to require further investigation were picked up by the app (with specificities of 90% and 61%, respectively).”

• cites [29] sourceNot checked yet

  Not checked in the free run.

• cites [30] Chung, 2018, Automated analysis of skin cancer app is unreliable, Ned Tijdschr Dermatol VenereolNot checked yet

  Not checked in the free run.

May combine two claims.

Selective participant recruitment, inadequate reference standards, differential verification, and high rates of unevaluable images were particular problems in the identified studies.

“Selective participant recruitment, inadequate reference standards, differential verification, and high rates of unevaluable images were particular problems.”

May combine two claims.

Studies were based on images of suspicious skin lesions that had undergone excision or biopsy, and were further selected to only include conditions identified by the apps; for example, they excluded lesions with clinical and histological features similar to melanoma, or restricted inclusion to melanocytic lesions which are more likely to be recognised by apps.

“We found studies were based on images of suspicious skin lesions that had undergone excision or biopsy, and were further selected to only include conditions identified by the apps; for example, they excluded lesions with clinical and histological features similar to melanoma,34 or restricted inclusion to melanocytic lesions which are more likely to be recognised by apps.23 28 29 31 32”

• cites [34] sourceNot checked yet

  Not checked in the free run.

• cites [23] Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical applications for melanoma risk assessment: false assurance or valuable tool? 47th Hawaii International Conference on System Sciences; 6-9 January, 2014;2675-84. sourceNot checked yet

  Not checked in the free run.

• cites [28] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Smartphone algorithm based apps for skin cancer all include disclaimers that the results should only be used as a guide and cannot replace healthcare advice.

“Smartphone algorithm based apps for skin cancer all include disclaimers that the results should only be used as a guide and cannot replace healthcare advice.”

May combine two claims.

Studies evaluated apps in selected groups of lesions, using images taken by experts rather than by app users, and many did not identify whether low risk lesions were truly benign.

“Studies evaluated apps in selected groups of lesions, using images taken by experts rather than by app users, and many did not identify whether low risk lesions were truly benign”

May combine two claims.

The review included studies that verified app findings by using expert recommendations for investigation or intervention, in addition to studies that used histology with or without follow-up.

“We included studies that verified app findings by using expert recommendations for investigation or intervention, in addition to studies that used histology with or without follow-up.”

May combine two claims.

This review identified nine eligible studies that evaluated apps for risk stratification of skin lesions, and showed variable and unreliable test accuracy for six different apps.

“This review identified nine eligible studies that evaluated apps for risk stratification of skin lesions, and showed variable and unreliable test accuracy for six different apps”

May combine two claims.

Studies included in this review did not specify algorithm versions used for risk assessment and we do not know whether three studies of the SkinVision app considered the same app version or different versions.

“Studies included in this review did not specify algorithm versions used for risk assessment and we do not know whether three studies of the SkinVision app29 30 33 considered the same app version or different versions.”

• cites [29] sourceNot checked yet

  Not checked in the free run.

• cites [30] Chung, 2018, Automated analysis of skin cancer app is unreliable, Ned Tijdschr Dermatol VenereolNot checked yet

  Not checked in the free run.

• cites [33] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Another 90 melanomas in the SkinVision study were identified from users of the app who had uploaded histology results after a high risk rating by a dermatologist and a high or moderate risk recommendation from the app, which will overestimate accuracy if more easily identifiable melanomas were included.

“Another 90 melanomas were identified from users of the app who had uploaded histology results after a high risk rating by a dermatologist and a high or moderate risk recommendation from the app (which will overestimate accuracy if more easily identifiable melanomas were included).41”

• cites [41] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

These patients are not represented in the reported studies and thus we did not evaluate this risk.

“These patients are not represented in the reported studies and thus we did not evaluate this risk.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CE marking has been applied to allow distribution of two algorithm based apps in Europe, one of which is also available in Australia and New Zealand.

“CE (Conformit Europenne) marking has been applied to allow distribution of two algorithm based apps in Europe,15 16 one of which is also available in Australia and New Zealand.16”

• cites [15] SkinVision. https://www.skinvision.com/about.Not checked yet

  Not checked in the free run.

• cites [16] SkinScan. https://teleskin.org/skinscan.html.Not checked yet

  Not checked in the free run.

May combine two claims.

A previous systematic review with a search date of December 2016 identified six studies that reported on diagnosis of melanoma, but accuracy seems to have been overestimated because it included findings from both development and validation studies.

“A previous systematic review with a search date of December 2016 examined the accuracy of mobile health apps for multiple conditions. This review identified six studies that reported on the diagnosis of melanoma, but accuracy seems to have been overestimated because it included findings from both development and validation studies.9”

• cites [9] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The current regulatory process for awarding the CE marking for algorithm based apps does not provide adequate protection to the public.

“The current regulatory process for awarding the CE marking for algorithm based apps does not provide adequate protection to the public.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Studies reduced the number of non-evaluable images by attempting up to 10 image submissions for each lesion.

“Studies reduced the number of non-evaluable images by attempting up to 10 image submissions for each lesion.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two authors independently extracted data using a prespecified data extraction form and assessed study quality.

“Two authors independently extracted data by using a prespecified data extraction form and assessed study quality.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two reviewers independently screened titles and abstracts and full texts, with discrepancies resolved by consensus or a third reviewer.

“Two reviewers independently screened the titles and abstracts of all retrieved records, and subsequently all full text publications. Discrepancies were resolved by consensus or discussion with a third reviewer.”

May combine two claims.

Five studies used expert diagnosis to confirm the final diagnosis for at least some lesions, with no confirmation by the preferred reference standard of histopathology or lesion follow-up.

“Five studies used expert diagnosis to confirm the final diagnosis for at least some lesions, with no confirmation by the preferred reference standard of histopathology or lesion follow-up.24 29 30 32 33”

May combine two claims.

One prospective study recruited patients from the general population who attended a national skin cancer day held at three university medical centres; one recruited patients who attended follow-up screening; and seven recruited only patients selected for excision of suspicious lesions or assessment of skin problems by dermatologists.

“One prospective study recruited patients from the general population who attended a national skin cancer day held at three university medical centres30; one recruited patients who attended follow-up screening28; and seven recruited only patients selected for excision of suspicious lesions or assessment of skin problems by dermatologists.23 24 29 31 32 33 34”

May combine two claims.

We had high (eight of nine) or unclear (one of nine) concerns about the application of the index test.

“We had high (eight of nine) or unclear (one of nine) concerns about the application of the index test.”

May combine two claims.

Two studies used previously acquired images of excised lesions obtained from dermatology databases, which raised concerns that the results were unlikely to be representative of real life use.

“Two studies used previously acquired images of excised lesions obtained from dermatology databases,23 34 which raised concerns that the results of the studies were unlikely to be representative of real life use.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Apps with inbuilt algorithms that make a medical claim are now classified as medical devices that require regulatory approval.

“Apps with inbuilt algorithms that make a medical claim are now classified as medical devices that require regulatory approval.”

• cites [13] FDA, 2015Not checked yet

  Not checked in the free run.

• cites [14] European Commission. Medical devices: guidance document. Qualification and classification of stand alone software. MEDDEV 2.1/6 Eur Union 2016. https://ec.europa.eu/docsroom/documents/17921Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

One study considered the results of a minimum of three images for each lesion in the final risk assessment.

“One study considered the results of a minimum of three images for each lesion in the final risk assessment.28”

Probably not a substantial claim: may not pass S3 (load-bearing).

The American Federal Trade Commission has fined the marketers of two apps (MelApp and Mole Detective) for deceptively claiming the apps accurately analysed melanoma risk.

“Further, the American Federal Trade Commission has fined the marketers of two apps (MelApp17 and Mole Detective18) for “deceptively claiming the apps accurately analysed melanoma risk.””

• cites [17] USA Federal Trade Commission. https://www.ftc.gov/system/files/documents/cases/complaint.pdf. 2015.Not checked yet

  Not checked in the free run.

• cites [18] USA Federal Trade Commission. https://www.ftc.gov/system/files/documents/cases/150223avromcmpt.pdf. 2015.Not checked yet

  Not checked in the free run.

May combine two claims.

Image quality was likely to be higher than in the real life setting because archived images were chosen on the basis of image quality or a clinician or researcher prospectively acquired images under optimised conditions using a single smartphone camera rather than participants using their own devices.

“Image quality was likely to be higher than in the real life setting for two reasons: archived images were chosen on the basis of image quality; or a clinician or researcher prospectively acquired images by using a standard protocol under optimised conditions and using a single smartphone camera rather than by participants using their own individual devices.”

May combine two claims.

Only two studies included skin lesions as selected by study participants; two did not report lesion selection; and in five, the clinician performed lesion selection.

“Only two studies included skin lesions as selected by study participants (up to two for each participant)29 30; two did not report lesion selection31 32; and in five,23 24 28 33 34 the clinician performed lesion selection.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The review was conducted according to methods detailed in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy and reported according to the PRISMA extension for diagnostic test accuracy studies.

“We conducted our review according to methods detailed in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy20 and report our findings according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for diagnostic test accuracy studies statement recommendations.21”

• cites [20] Deeks JJ, Wisniewski S, Davenport C. Chapter 4: Guide to the contents of a Cochrane diagnostic test accuracy protocol. In: Deeks JJ, Bossuyt PM, Gatsonis C, eds. Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0.0. https://methods.cochrane.org/sdt/handbook-dta-reviews 2013.Not checked yet

  Not checked in the free run.

• cites [21] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The lesion selection process was unclear in some studies or convenience sampling was used in others.

“The lesion selection process was otherwise unclear29 30 32 or convenience sampling was used.23 24”

Classified as probably not a substantial claim.

Study quality was assessed using QUADAS-2.

“These aspects of study quality were assessed using the Quality Assessment of Diagnostic Accuracy Studies 2 tool (QUADAS-222; online supplementary appendix 3).”

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The lack of clarity in smartphone app recommendations could leave concerned users uncertain as to the best course of action.

“Thirdly, the lack of clarity in smartphone app recommendations could leave concerned users uncertain as to the best course of action.”

May combine two claims.

Performance of the SkinVision app is likely to be poor because studies were small and overall of poor methodological quality, and did not evaluate the apps as they would be used in practice by the people who would use them.

“However, performance is likely to be poor because studies were small and overall of poor methodological quality, and did not evaluate the apps as they would be used in practice by the people who would use them.”

May combine two claims.

Study results are not applicable to people with amelanotic melanomas (accounting for 2-8% of all melanomas), or to identify other more common forms of skin cancers such as cutaneous squamous cell carcinoma.

“Study results are also not applicable to people with amelanotic melanomas (accounting for 2-8% of all melanomas),40 or to identify other more common forms of skin cancers such as cutaneous squamous cell carcinoma.”

• cites [40] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

These variable results could partly be caused by smaller sample sizes.

“These variable results could partly be caused by smaller sample sizes, with either low sensitivities (25-50% for MelApp) or specificities (20-60% for Mole Detective, Dr Mole, and SpotMole).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Reactions to a moderate risk result will probably depend on how risk averse people are, which will most likely result in variable decisions, with a risk of people failing to present to a specialist with a potentially malignant skin lesion.

“Reactions to a moderate risk result will probably depend on how risk averse people are, which will most likely result in variable decisions, with a risk of people failing to present to a specialist with a potentially malignant skin lesion.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FDA already has a stricter assessment process to evaluate mobile apps by taking a wider perspective of harm where "functionality could pose a risk to a patient's safety if the mobile app were to not function as intended".

“The FDA already has a stricter assessment process to evaluate mobile apps by taking a wider perspective of harm where “functionality could pose a risk to a patient’s safety if the mobile app were to not function as intended”.13”

• cites [13] FDA, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Smartphone cameras are much more likely to be used in suboptimal conditions by the general population, which results in variable image quality.

“Smartphone cameras are much more likely to be used in suboptimal conditions by the general population, which results in variable image quality.”

• AssumptionRests on an assumption: Smartphone cameras are much more likely to be used in suboptimal conditions by the general population, which results in variable image quality.

May combine two claims.

Two algorithm based apps have obtained CE marking and are currently being marketed with claims that they can "detect skin cancer at an early stage" or "track moles over time with the aim of catching melanoma at an earlier stage of the disease".

“Despite the limitations of the evidence base, two algorithm based apps have obtained the CE marking and are currently being marketed with claims that they can “detect skin cancer at an early stage”15 or “track moles over time with the aim of catching melanoma at an earlier stage of the disease”.16”

• cites [15] SkinVision. https://www.skinvision.com/about.Not checked yet

  Not checked in the free run.

• cites [16] SkinScan. https://teleskin.org/skinscan.html.Not checked yet

  Not checked in the free run.

May combine two claims.

These image quality problems resulted in image exclusion and potential overestimation of diagnostic performance of skin cancer apps.

“These problems resulted in image exclusion and potential overestimation of diagnostic performance of skin cancer apps.”

May combine two claims.

In a hypothetical population of 1000 adults in which 3% have a melanoma, using a sensitivity of 88% and a specificity of 79% as observed by Thissen and colleagues, four of 30 melanomas would not be picked up as high risk, and more than 200 people would be given false positive results.

“Therefore, in a hypothetical population of 1000 adults in which 3% have a melanoma, four of 30 melanomas would not be picked up as high risk, and more than 200 people would be given false positive results (by using a sensitivity of 88% and a specificity of 79%, as observed by Thissen and colleagues33).”

• cites [33] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Manufacturers can apply CE marking to class 1 devices as long as they have shown compliance with the "essential requirements" as outlined in the Directive, and without necessarily being subject to independent inspection by notified bodies such as the Medicines and Healthcare products Regulatory Agency in the United Kingdom.

“Manufacturers can apply CE marking to class 1 devices as long as they have shown compliance with the “essential requirements” as outlined in the Directive,35 and without necessarily being subject to independent inspection by notified bodies such as the Medicines and Healthcare products Regulatory Agency in the United Kingdom.”

• cites [35] DIRECTIVE 2007/47/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 5 September 2007 amending Council Directive 90/385/EEC on the approximation of the laws of the Member States relating to active implantable medical devices, Council Directive 93/42/EEC concerning medical devices and Directive 98/8/EC concerning the placing of biocidal products on the market. 2007. p. 22-55.Not checked yet

  Not checked in the free run.

May combine two claims.

Identified studies of test accuracy have many weaknesses and do not provide adequate evidence to support implementation of current apps.

“Identified studies of test accuracy have many weaknesses and do not provide adequate evidence to support implementation of current apps.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Under the new Medical Device Regulations, which come into full force by May 2020, smartphone apps could be in higher device classes and will be subject to inspection by notified bodies.

“Under the new Medical Device Regulations,36 37 which come into full force by May 2020, smartphone apps could be in higher device classes and will be subject to inspection by notified bodies.”

• cites [36] REGULATION (EU) 2017/745 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. 2017. p. 1-175.Not checked yet

  Not checked in the free run.

• cites [37] Medicines and Healthcare products Regulatory Agency. Guidance: Medical device stand-alone software including apps (including IVDMDs). 2018. https://www.gov.uk/government/publications/medical-devices-software-applications-appsNot checked yet

  Not checked in the free run.

May combine two claims.

A considerable number of users will receive an inappropriate high risk result that could cause unnecessary worry and a burden on primary care and dermatology services.

“A considerable number of users will receive an inappropriate high risk result that could cause unnecessary worry and a burden on primary care and dermatology services.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The potential benefit of smartphone apps lies in their availability and use by people outside the healthcare system to evaluate lesions that cause them concern.

“Finally, the potential benefit of smartphone apps lies in their availability and use by people outside the healthcare system to evaluate lesions that cause them concern.”

May combine two claims.

Despite the increasing availability of skin cancer apps, the lack of evidence and considerable limitations in the studies ultimately highlight concerns about the safety of algorithm based smartphone apps at present.

“Despite the increasing availability of skin cancer apps, the lack of evidence and considerable limitations in the studies ultimately highlight concerns about the safety of algorithm based smartphone apps at present.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The strengths of this review are that it used a comprehensive electronic literature search with stringent systematic review methods that included independent duplicate data extraction and quality assessment of studies, attempted contact with authors, and a clear analysis structure.

“The strengths of this review are that it used a comprehensive electronic literature search with stringent systematic review methods that included independent duplicate data extraction and quality assessment of studies, attempted contact with authors, and a clear analysis structure.”

May combine two claims.

Studies that used histology and follow-up are more reliable because the app assessments are evaluated against the true final diagnosis of study lesions; low sensitivities or specificities reflect the apps' inability to identify melanomas or other skin cancers as high risk.

“Studies that used histology and follow-up are more reliable because the app assessments are evaluated against the true final diagnosis of study lesions; low sensitivities or specificities reflect the apps’ inability to identify melanomas or other skin cancers as high risk.”

Classified as probably not a substantial claim.

A Cochrane review of only two studies that tried to validate algorithm based skin apps suggested that there is a high chance of skin cancers being missed.

“A Cochrane review of only two studies that tried to validate algorithm based skin apps suggested that there is a high chance of skin cancers being missed”

• cites [19] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Smartphones and dedicated skin cancer apps can have other roles; for example, assisting in skin self-examination, tracking the evolution of suspicious lesions in people more at risk of developing skin cancer, or when used for store and forward teledermatology.

“Smartphones and dedicated skin cancer apps can have other roles; for example, assisting in skin self-examination, tracking the evolution of suspicious lesions in people more at risk of developing skin cancer,47 48 or when used for store and forward teledermatology.49 50”

• cites [47] sourceNot checked yet

  Not checked in the free run.

• cites [48] sourceNot checked yet

  Not checked in the free run.

• cites [49] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The review found poor and variable performance of algorithm based smartphone apps, which indicates that these apps have not yet shown sufficient promise to recommend their use.

“Nevertheless, our review found poor and variable performance of algorithm based smartphone apps, which indicates that these apps have not yet shown sufficient promise to recommend their use.”

May combine two claims.

Studies that verify findings against expert recommendations can only provide an idea of the level of agreement between the algorithm's risk assessment and a dermatologist's clinical management decision; agreement between the two does not necessarily mean the risk assessment made was the correct one for the lesion concerned.

“Studies that verify findings against expert recommendations can only provide an idea of the level of agreement between the algorithm’s risk assessment and a dermatologist’s clinical management decision; agreement between the two does not necessarily mean the risk assessment made was the correct one for the lesion concerned.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Concern exists about the impact of false reassurances that algorithm based apps could give users with potentially malignant skin lesions, especially if they are dissuaded from seeking healthcare advice.

“Concern exists about the impact of false reassurances that algorithm based apps could give users with potentially malignant skin lesions, especially if they are dissuaded from seeking healthcare advice.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

These apps attempt to evade any responsibility for negative outcomes experienced by users.

“Therefore, these apps attempt to evade any responsibility for negative outcomes experienced by users.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In March 2019, the SkinVision app was selected to join the UK NHS Innovation Accelerator as a possible new technology to support earlier diagnosis and prevention of cancer.

“Subsequently, in March 2019,44 this app was selected to join the UK NHS Innovation Accelerator as a possible new technology to support earlier diagnosis and prevention of cancer.”

• cites [44] SkinVision. https://www.skinvision.com/news/skinvision-to-join-nhs-innovation-accelerator.Not checked yet

  Not checked in the free run.

May combine two claims.

The SkinVision study used some images and data collected by real users of the app on their own phones, but the selection of malignant and benign lesions from several different sources is likely to have introduced bias.

“The study has used some images and data collected by real users of the app on their own phones, however the selection of malignant and benign lesions from several different sources is likely to have introduced bias.”

• cites [41] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Investment in algorithm based skin cancer apps is ongoing, with the company behind SkinVision announcing an investment of US$7.6m (£5.8m; €6.8m) in 2018.

“Investment in algorithm based skin cancer apps is ongoing, with the company behind SkinVision announcing an investment of US$7.6m (£5.8m; €6.8m) in 2018.43”

• cites [43] SkinVision. https://www.skinvision.com/news/pr-funding-2018. 2018.Not checked yet

  Not checked in the free run.