Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma

May combine two claims.

For another six melanomas reviewed by the dermatologist through the app, the lesions were not labelled high risk; the dermatologist could not decide whether they were atypical or typical.

“Another 6 melanomas examined by the dermatologist via the application were not classified as high risk; instead the dermatologist was not able to classify the lesion as either 'atypical' (possibly a melanoma) or 'typical' (definitely not a melanoma).”

May combine two claims.

For the four AI-based apps that classified lesion images as melanoma (one app) or as high-risk/problematic (three apps), sensitivity ranged from 7% (95% CI 2% to 16%) to 73% (95% CI 52% to 88%), and specificity ranged from 37% (95% CI 29% to 46%) to 94% (95% CI 87% to 97%).

“sensitivities ranged f rom 7% (95% CI 2% to 16%) to 73% (95% CI 52% to 88%) and specif icities f rom 37% (95% CI 29% to 46%) to 94% (95% CI 87% to 97%).”

May combine two claims.

The one app using store-and-forward review of lesion images by a dermatologist achieved sensitivity 98% (95% CI 90% to 100%) and specificity 30% (95% CI 22% to 40%); the dermatologist missed one melanoma.

“The single application using store-and-f orward review of lesion im ages by a derm atologist had a sensitivity of 98% (95% CI 90% to 100%) and specif icity of 30% (95% CI 22% to 40%); the derm atologist m issed one m elanom a”

May combine two claims.

In three of the four app evaluations, at least one melanoma was deemed unevaluable; the dermatologist reviewing store-and-forward images excluded the most melanomas, 6 of 60 assessed.

“At least one m elanom a was classed as unevaluable in three of the f our application evaluations, the highest num ber of m elanom as excluded by the derm atologist evaluating the store-and-f orward im ages (6/ 60 m elanom as assessed)”

May combine two claims.

The review's evidence base comprised two studies covering 332 lesions, of which 86 were melanomas, each assessed by at least one smartphone app.

“The review included two studies with 332 lesions, including 86 melanomas, analysed by at least one smartphone application.”

May combine two claims.

We included two studies with a total of 332 lesions, 86 of which were melanomas.

“We included two studies with a total of 332 lesions, 86 of which were melanomas (Summary of findings).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Maier 2015 excluded an additional 31 lesions as test failures, including 13 due to two-point differences and 18 tie-cases.

“Study authors excluded an additional 31 lesions they considered as test failures for the purposes of this review (see Findings), including 13 due to”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wolf 2013 evaluated four different smartphone applications, numbered one to four to avoid consumer bias.

“Wolf 2013 evaluated four different smartphone applications, providing no names to avoid consumer bias; they numbered the applications one to four to allow an assessment of accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not identify any other systematic reviews of smartphone applications during the preparation of this review.

“We did not identify any other systematic reviews of smartphone applications during the preparation of this review.”

May combine two claims.

Wolf 2013 used archived photographs of lesions rather than images taken using a phone and did not report the real names of the applications used.

“Wolf 2013 used archived photographs of lesions rather than images taken using a phone and did not report the real names of the applications used.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The studies did not report the number of participants with lesions included in the studies.

“The studies did not report the number of participants with lesions included in the studies.”

May combine two claims.

Study participants were people who had skin lesions already scheduled for excision in a dermatology clinic setting rather than smartphone users with concerns about a new or changing mole or skin lesion, and dermatologists were likely to have taken the photographs used with the applications in the clinic setting in both studies.

“Study participants were people who had skin lesions already scheduled for excision in a dermatology clinic setting rather than smartphone users with concerns about a new or changing mole or skin lesion, and dermatologists were likely to have taken the photographs used with the applications in the clinic setting in both studies.”

May combine two claims.

Three of the four applications classed at least one melanoma as unevaluable, with the highest number of excluded melanomas (6/60 melanomas assessed) resulting from dermatologist evaluation of the store-and-forward images.

“Three of the four applications classed at least one melanoma as unevaluable, with the highest number of excluded melanomas (6/60 melanomas assessed) resulting from dermatologist evaluation of the store-and-forward images.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Wolf 2013 excluded lesions that were uncommon or had an equivocal diagnosis and lesions with moderate or high grade atypia.

“The study excluded lesions that were uncommon or had an equivocal diagnosis (such as 'melanoma cannot be ruled out' or 'atypical melanocytic proliferation') and lesions with moderate or high grade atypia.”

May combine two claims.

Total numbers of included lesions were 188 in Wolf 2013 and 144 in Maier 2015.

“total numbers of included lesions were 188 in Wolf 2013 and 144 in Maier 2015.”

May combine two claims.

The test failure rates were: 3% (App 1), 2% (App 2), 6% (App 3) and 15% (App 4).

“The test failure rates were: 3% (App 1), 2% (App 2), 6% (App 3) and 15% (App 4; designated by the dermatologist as 'send another photograph' or 'unable to categorise') (Table 1).”

May combine two claims.

Three of the four applications classed at least one melanoma as unevaluable, with the dermatologist conducting the assessment for the store-and-forward application (App 4) missing 6 (10%) melanomas for this reason.

“Three of the four applications classed at least one melanoma as unevaluable, with the dermatologist conducting the assessment for the store-and-forward application (App 4) missing 6 (10%) melanomas for this reason.”

May combine two claims.

The prevalence of melanoma was 18% in Maier 2015 and 35% in Wolf 2013.

“The prevalence of melanoma was 18% in Maier 2015 and 35% in Wolf 2013.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A total of 34,517 unique references were screened for inclusion.

“We screened a total of 34,517 unique references for inclusion.”

May combine two claims.

Kassianos 2015 identified 39 applications, and we were only able to identify test accuracy evaluations for five.

“It is notable that Kassianos 2015 identified 39 applications, and we were only able to identify test accuracy evaluations for five.”

• cites Kassianos 2015 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

For App 2 in Wolf 2013 (185 lesions and 58 melanoma cases), sensitivity was 69% (95% CI 55% to 80%) and specificity was 37% (95% CI 29% to 46%).

“For App 2 in Wolf 2013 (185 lesions and 58 melanoma cases), the resulting sensitivity was 69% (95% CI 55% to 80%), and specificity was 37% (95% CI 29% to 46%).”

May combine two claims.

The single application using store-and-forward review of lesion images by a dermatologist had a sensitivity of 98% (95% CI 90% to 100%) and specificity of 30% (95% CI 22% to 40%); the dermatologist missed one melanoma.

“The single application using store-and-forward review of lesion images by a dermatologist had a sensitivity of 98% (95% CI 90% to100%) and specificity of 30% (95% CI 22% to 40%); the dermatologist missed one melanoma.”

May combine two claims.

Sensitivities were around 70% for App 1 in Wolf 2013 (70%, 95% CI 57% to 81%) and for the SkinVision app in Maier 2015 (73%, 95% CI 52% to 88%) but only 7% (95% CI 2% to 16%) for App 3 in Wolf 2013.

“Sensitivities were around 70% for App 1 in Wolf 2013 (70%, 95% CI 57% to 81%) and for the SkinVision app in Maier 2015 (73%, 95% CI 52% to 88%) but only 7% (95% CI 2% to 16%) for App 3 in Wolf 2013.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Three applications in Wolf 2013 were artificial intelligence-based classifications of lesions as 'problematic' versus 'okay' (App 1), 'melanoma' versus 'not melanoma' (App 2) and 'high risk' versus 'medium/low-risk' (App 3).

“Three applications were artificial intelligence-based classifications of lesions as 'problematic' versus 'okay' (App 1), 'melanoma' versus 'not melanoma' (App 2) and 'high risk' versus 'medium/low-risk' (App 3) as part of the assessment of the images.”

May combine two claims.

This review reports on two cohorts of lesions published in two studies that provide six datasets.

“This review reports on two cohorts of lesions published in two studies that provide six datasets (Wolf 2013; Maier 2015).”

May combine two claims.

Both of the studies used photographs of skin lesions that were scheduled for excision in a dermatology clinic setting, and clinicians were the ones who took the photographs instead of people using their own smartphones, potentially leading to the acquisition of higher quality images.

“Both of the studies used photographs of skin lesions that were scheduled for excision in a dermatology clinic setting, and clinicians were the ones who took the photographs instead of people using their own smartphones, potentially leading to the acquisition of higher quality images.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Maier 2015 excluded 20/195 lesions (10%) due to poor image quality or incomplete imaging.

“The study excluded 20/195 lesions (10%) due to poor image quality or incomplete imaging.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Maier 2015 conducted a prospective case series of patients with melanocytic skin lesions seen routinely at the department of dermatology for skin cancer screening.

“Maier 2015 conducted a prospective case series of patients with melanocytic skin lesions seen routinely at the department of dermatology for skin cancer screening.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two studies were derivation studies that did not separate data for training and test sets.

“two studies were derivation studies that did not separate data for training and test sets (Ramlakhan 2011; Wadhawan 2011)”

May combine two claims.

Due to scarcity of data and the poor quality of studies, no meta-analysis was undertaken for this review.

“Due to scarcity of data and the poor quality of studies, we did not undertake a meta-analysis for this review.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two studies included fewer than five melanoma cases.

“Two studies included fewer than five melanoma cases (Massone 2007; Robson 2012)”

Probably not a substantial claim: may not pass S3 (load-bearing).

App 4 in Wolf 2013 used a store-and-forward approach with remote lesion assessment by a qualified dermatologist.

“App 4 used a store-and-forward approach with remote lesion assessment by a qualified dermatologist.”

May combine two claims.

The corresponding specificities for the three applications were 39% (95% CI 31% to 49%; Wolf 2013; 182 lesions and 60 melanomas), 83% (95% CI 75% to 89%; Maier 2015; 144 lesions and 26 melanomas), and 94% (95% CI 87% to 97%; Wolf 2013; 170 lesions and 59 melanomas).

“The corresponding specificities for the three applications were 39% (95% CI 31% to 49%; Wolf 2013; 182 lesions and 60 melanomas), 83% (95% CI 75% to 89%; Maier 2015; 144 lesions and 26 melanomas), and 94% (95% CI 87% to 97%; Wolf 2013; 170 lesions and 59 melanomas).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sixteen studies were considered potentially eligible for this review, and two publications were ultimately included.

“We considered 16 studies to be potentially eligible for this review of smartphone applications and ultimately included two publications.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Maier 2015 did use smartphone images; however, the details around the imaging process were unclear.

“Maier 2015 did use smartphone images; however, the details around the imaging process were unclear.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of the screened references, 1051 full-text papers were reviewed for eligibility and 203 publications were included in at least one of the suite of reviews.

“Of these, we reviewed 1051 full-text papers for eligibility and included 203 publications in at least one of the suite of reviews of tests to assist in the diagnosis of melanoma or keratinocyte skin cancer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wolf 2013 retrospectively evaluated photographs of lesions selected from a dermatology database of lesions scheduled for excision using a case-control type design.

“Wolf 2013 retrospectively evaluated photographs of lesions selected from their dermatology database of lesions that had been scheduled for excision using a case-control type design.”

May combine two claims.

The applications successfully analysed a total of 332 lesions, including 86 melanomas.

“The applications successfully analysed a total of 332 lesions, including 86 melanomas.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Maier 2015 evaluated an automated risk assessment algorithm using the SkinVision App, which relied on fractal image analysis of three images per lesion.

“Maier 2015 evaluated an automated risk assessment algorithm using the SkinVision App; this relied on fractal image analysis of three images per lesion.”

May combine two claims.

Maier 2015 reported the accuracy of face-to-face clinical diagnosis of the same lesions by a dermatologist; sensitivity was 85% (95% CI 65% to 96%) and specificity 97% (95% CI 93% to 99%; 144 lesions and 26 melanomas).

“The sensitivity of the face-to-face assessment was 85% (95% CI 65% to 96%) and specificity, 97% (95% CI 93% to 99%; Figure 8; 144 lesions and 26 melanomas).”

May combine two claims.

Across the five different applications assessed, sensitivities for the detection of invasive melanoma or atypical intraepidermal melanocytic variants ranged from 7% (95% CI 2% to 16%) to 98% (95% CI 90% to 100%) and specificities from 30% (95% CI 22% to 40%) to 94% (95% CI 87% to 97%).

“Across the five different applications that the two studies assessed, sensitivities for the detection of invasive melanoma or atypical intraepidermal melanocytic variants ranged from 7% (95% confidence interval (CI) 2% to 16%) to 98% (95% CI 90% to 100%) and specificities from 30% (95% CI 22% to 40%) to 94% (95% CI 87% to 97%; Figure 7).”

May combine two claims.

The final application, App 4 in Wolf 2013, was a store-and-forward system, with a dermatologist classifying lesion images as atypical or typical; sensitivity was 98% (95% CI 90% to 100%) and specificity 30% (95% CI 22% to 40%; 159 lesions and 54 melanoma cases).

“The final application, App 4 in Wolf 2013, was a store-and-forward system, with a dermatologist classifying lesion images as atypical or typical.The sensitivity for this application was 98% (95% CI 90% to 100%) and specificity 30% (95% CI 22% to 40%; 159 lesions and 54 melanoma cases).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, the third application using the high/medium-risk versus low-risk threshold achieved a sensitivity of 0.54 (95% CI 0.41 to 0.67).

“Wolf 2013 32 43 27 68 0.54 [ 0.41, 0.67 ] 0.61 [ 0.52, 0.70 ]”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 2 could not evaluate 3 of 185 lesions (2%).

“Application 2 185 (58) 3 (2%) 2 (3%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 3 had 1 unevaluable melanoma (2%).

“Application 3 170 (59) 12 (6%) 1 (2%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 3 could not evaluate 12 of 170 lesions (6%).

“Application 3 170 (59) 12 (6%) 1 (2%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 4 could not evaluate 29 of 159 lesions (15%).

“Application 4 159 (54) 29 (15%) 6 (10%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 2 had 2 unevaluable melanomas (3%).

“Application 2 185 (58) 3 (2%) 2 (3%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 1 could not evaluate 6 of 188 lesions (3%).

“Application 1 182 (60) 6 (3%) 0 (0%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 1 had no unevaluable melanomas (0 of 60; 0%).

“Application 1 182 (60) 6 (3%) 0 (0%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, the third application using the high-risk versus medium/low-risk threshold achieved a specificity of 0.94 (95% CI 0.87 to 0.97).

“Wolf 2013 4 7 55 104 0.07 [ 0.02, 0.16 ] 0.94 [ 0.87, 0.97 ]”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, the third application using the high-risk versus medium/low-risk threshold achieved a sensitivity of 0.07 (95% CI 0.02 to 0.16).

“Wolf 2013 4 7 55 104 0.07 [ 0.02, 0.16 ] 0.94 [ 0.87, 0.97 ]”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, Application 4 had 6 unevaluable melanomas (10%).

“Application 4 159 (54) 29 (15%) 6 (10%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Maier 2015, SkinVision could not evaluate 31 of 175 lesions (18%).

“SkinVision 144 (26) 31 (18%) Not reported (≤ 14)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, the third application using the high/medium-risk versus low-risk threshold achieved a specificity of 0.61 (95% CI 0.52 to 0.70).

“Wolf 2013 32 43 27 68 0.54 [ 0.41, 0.67 ] 0.61 [ 0.52, 0.70 ]”

Probably not a substantial claim: may not pass S3 (load-bearing).

At least one author screened titles and abstracts, discussing and resolving any queries by consensus.

“At least one author (JDi or NC) screened titles and abstracts, discussing and resolving any queries by consensus.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A pilot screen of 539 MEDLINE references showed good agreement (89% with a kappa of 0.77) between screeners.

“A pilot screen of 539 MEDLINE references showed good agreement (89% with a kappa of 0.77) between screeners.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Exclusions caused by test failures were not reported (1/2), or the final diagnoses of those exclusions were not described (2/2).

“Exclusions due to test f ailures were not reported (1/ 2) or their f inal diagnoses were not described (2/ 2)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reference lists of all included papers were checked, and subject experts within the author team reviewed the final list of included studies.

“We have checked the reference lists of all included papers, and subject experts within the author team reviewed the final list of included studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Electronic citation searching was not performed.

“We did not perform electronic citation searching.”

May combine two claims.

Because doctors, not smartphone users, took the photographs in the included studies, the applicability of the results is seriously limited.

“The photographs used were taken by doctors rather than by smartphone users, which seriously impacts the applicability of results.”

• AssumptionRests on an assumption: The prevalence of melanoma in the included studies is much higher than that observed in community settings.

May combine two claims.

Melanoma status was determined histologically, a method the review considers likely to have been dependable for confirming true melanoma.

“The final diagnosis of melanoma was made by histology, which is likely to have been a reliable method for deciding whether patients really had melanoma*.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Relevant systematic reviews identified by the searches were screened for their included primary studies, and any missed by the searches were included.

“We have screened relevant systematic reviews identified by the searches for their included primary studies, and we included any missed by our searches.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Test interpretation was blinded to the reference standard and pre-specified for AI-based diagnosis (2/2).

“Test interpretation was blinded to ref erence standard and pre-specif ied f or artif icial intelligence-based diagnosis (2/ 2).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The timing between index and reference standards was not reported.

“Tim ing of index and ref erence standards not reported.”

May combine two claims.

Store-and-forward image apps might help identify people with potentially malignant lesions in a timely way by encouraging early contact for suspicious skin lesions, though they carry resource and workload consequences.

“Store-and-forward image applications could have a potential role in the timely identification of people with potentially malignant lesions by facilitating early engagement of those with suspicious skin lesions, but they have resource and workload implications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No additional limits were used in the search.

“No additional limits were used.”

Classified as probably not a substantial claim.

The search strategy was designed to capture studies evaluating tests for the diagnosis or staging of skin cancer.

“The search strategy was designed to capture studies evaluating tests for the diagnosis or staging of skin cancer.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both a clinical reviewer and a methodologist reviewer applied inclusion criteria to all full text articles, resolving disagreements by consensus or by consultation with a third party.

“Both a clinical reviewer (from a team of 12 clinician reviewers) and a methodologist reviewer (JDi or NC) applied inclusion criteria (Appendix 4) to all full text articles, resolving disagreements by consensus or by consultation with a third party (JDe, CD, HW or RM).”

Probably not a substantial claim: may not pass S3 (load-bearing).

One clinical and one methodological reviewer independently extracted data using a piloted data extraction form.

“One clinical (as detailed above) and one methodological reviewer (JDi, NC or LFR) independently extracted data concerning details of the study design, participants, index test(s) or test combinations and criteria for index test positivity, reference standards, and data required to populate a 2 × 2 diagnostic contingency table for each index test using a piloted data extraction form.”

Classified as probably not a substantial claim.

A comprehensive search for published and unpublished studies was conducted.

“The Information Specialist (SB) carried out a comprehensive search for published and unpublished studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Disagreements in data extraction were resolved by consensus or in consultation with a third party.

“We resolved disagreements by consensus or in consultation with a third party (JDe, CD, HW or RM).”

Probably not a substantial claim: may not pass S3 (load-bearing).

No date limits were applied to the search.

“We applied no date limits.”

Classified as probably not a substantial claim.

For each analysis, only one dataset per study was included to avoid multiple counting of lesions.

“For each analysis, we included only one dataset per study to avoid multiple counting of lesions.”

May combine two claims.

These participants would have already presented to a doctor with concerns about a particular lesion and therefore are likely to represent a more severe spectrum of abnormality, which will artificially increase the sensitivity of the test in comparison to use by smartphone users in general.

“These participants would have already presented to a doctor with concerns about a particular lesion and therefore are likely represent a more severe spectrum of abnormality, which will artificially increase the sensitivity of the test in comparison to use by smartphone users in general.”

May combine two claims.

Both studies were assessed as being at low risk of bias in the index test domain, with the artificial intelligence-based assessment made without knowledge of the histological diagnosis.

“We assessed both studies as being at low risk of bias in the index test domain, with the artificial intelligence-based assessment made without knowledge of the histological diagnosis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The strengths of this review include an in-depth and comprehensive electronic literature search, systematic review methods including double extraction of papers by both clinicians and methodologists, and contact with authors to allow study inclusion or clarify data.

“The strengths of this review include an in-depth and comprehensive electronic literature search, systematic review methods including double extraction of papers by both clinicians and methodologists, and contact with authors to allow study inclusion or clarify data.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

One clinical and one methodological reviewer independently assessed quality for the remaining studies, resolving any disagreement by consensus or in consultation with a third party.

“One clinical and one methodological reviewer (JDi, NC or LFR) independently assessed quality for the remaining studies, resolving any disagreement by consensus or in consultation with a third party where necessary (JDe, CD, HW or RM).”

May combine two claims.

All studies reported the use of an acceptable reference standard; however, Maier 2015 did not report blinding of histology to the index test result.

“All studies reported the use of an acceptable reference standard; however, Maier 2015 did not report blinding of histology to the index test result.”

May combine two claims.

One study did not report blinding of the reference standard to the lesion images, and one did not mention interpretation by an experienced histopathologist.

“One study did not report blinding of the reference standard to the lesion images, and one did not mention interpretation by an experienced histopathologist.”

May combine two claims.

Studies were blinded for index test interpretation and used pre-specified test thresholds and adequate reference standards.

“Studies were blinded for index test interpretation and used pre-specified test thresholds and adequate reference standards.”

May combine two claims.

Most studies include very few patients with multiple lesions, and any potential impact of correlations of test errors on findings is likely to be very small.

“Although there is a theoretical possibility of correlations of test errors when multiple lesions are included from the same patients, most studies include very few patients with multiple lesions, and any potential impact on findings is likely to be very small, particularly in comparison with other concerns regarding risk of bias and applicability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Heterogeneity between studies was examined by visually inspecting forest plots of sensitivity and specificity and summary ROC plots.

“We examined heterogeneity between studies by visually inspecting the forest plots of sensitivity and specificity and summary receiver operating characteristics (ROC) plots.”

May combine two claims.

The main concerns for the review are the clinical applicability of the findings and exclusion of unevaluable test results, with likely overestimation of sensitivity.

“The main concerns for the review are the clinical applicability of the findings and exclusion of unevaluable test results, with likely overestimation of sensitivity.”

Classified as probably not a substantial claim.

Initial treatment in skin cancer is directed to the lesion and not systemically to a patient.

“as initial treatment in skin cancer is directed to the lesion and not systemically to a patient”

May combine two claims.

With a prevalence of melanoma ranging between 18% and 37% for these evaluations, the number of melanomas missed was between 7 and 55.

“With a prevalence of melanoma ranging between 18% and 37% for these evaluations, the number of melanomas missed was between 7 and 55.”

May combine two claims.

Both studies caused high concern about applicability of the index tests, which were not used as intended in practice.

“However, both studies caused high concern about applicability of the index tests, which were not used as intended in practice.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No sensitivity analyses were performed.

“We did not perform any sensitivity analyses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All had a pre-specified threshold.

“All had a pre-specified threshold.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The modified QUADAS-2 tool was piloted on five full-text articles.

“We piloted the modified QUADAS-2 tool on five full-text articles.”

May combine two claims.

This is not a representative spectrum of lesions that would be observed in daily life but rather a highly selected sample of participants.

“This is not a representative spectrum of lesions that would be observed in daily life but rather a highly selected sample of participants.”

May combine two claims.

One study also excluded equivocal lesions and those with moderate or high-grade atypia, both of which could potentially be more likely to produce unevaluable results or be misclassified by the application.

“One study also excluded equivocal lesions and those with moderate or high-grade atypia, both of which could potentially be more likely to produce unevaluable results or be misclassified by the application.”

Classified as probably not a substantial claim.

Risk of bias and applicability of included studies were assessed using the QUADAS-2 checklist, tailored to the topic of skin cancer diagnosis.

“We assessed risk of bias and applicability of included studies using the QUADAS-2 checklist (Whiting 2011), tailored to the topic of skin cancer diagnosis (see Appendix 5).”

• cites Whiting 2011 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Both studies caused high concern regarding included participants and setting, due to unclear reporting of patient samples and whether they included multiple lesions per patient.

“Similarly, both studies caused high concern regarding included participants and setting, due to un-clear reporting of patient samples and whether they included multiple lesions per patient.”

Classified as probably not a substantial claim. May combine two claims.

No tests to detect publication bias were performed due to uncertainty about the determinants of publication bias for diagnostic accuracy studies and the inadequacy of tests for detecting funnel plot asymmetry.

“Due to uncertainty about the determinants of publication bias for diagnostic accuracy studies and the inadequacy of tests for detecting funnel plot asymmetry (Deeks 2005), we did not perform any tests to detect publication bias.”

• cites Deeks 2005 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We planned a clear analysis structure to allow estimation of test accuracy in different study populations and undertook a detailed and replicable analysis of methodologic quality.

“We planned a clear analysis structure to allow estimation of test accuracy in different study populations and undertook a detailed and replicable analysis of methodologic quality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Wolf 2013, 202 of 390 lesion images were excluded before analysis.

“Excluded participants: 202/390 lesion images excluded due to poor image quality, containing identifiable patient information or features, or lacking sufficient clinical or histological information.”

Classified as probably not a substantial claim.

The review excluded studies that used cross-validation approaches such as leave-one-out cross-validation.

“used cross-validation approaches such as 'leave-one-out' cross-validation (Efron 1983);”

• cites Efron 1983 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Studies that use previously acquired images may be especially prone to information bias.

“however, studies using previously acquired images could be particularly susceptible to information bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review team judged studies that excluded particular lesion types, sites, or lesions due to image quality or observer disagreement to be at high risk of bias.

“In terms of exclusions, we considered studies that excluded particular lesion types (e.g.lentigo maligna), particular lesion sites, or other lesions on the basis of image quality or lack of observer agreement (e.g. on histopathology) to be at high risk of bias.”

• cites Whiting 2011 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The review team classified studies that separately sampled malignant and benign lesions as case-control designs.

“We considered studies that separately sampled malignant and benign lesions to have used a case-control design; and those that supplemented a series of suspicious lesions with additional malignant or benign lesions to be at unclear risk of bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Not all patients in Maier 2015 were included in the analysis.

“Were all patients included in the analysis?No”

May combine two claims.

Available data have limited applicability in practice due to selective participant recruitment from secondary referral settings and the use of images not acquired by the intended users of the smartphone applications.

“Available data have limited applicability in practice due to selective participant recruitment from secondary referral settings and the use of images not acquired by the intended users of the smartphone applications (i.e.members of the public).”

• AssumptionRests on an assumption: Smartphone applications for skin lesion triage are intended to be used by members of the public in a community setting.

May combine two claims.

The review could not examine heterogeneity or carry out sensitivity analyses because of a lack of data.

“For the same reason, we could not investigate heterogeneity or perform sensitivity analyses.”

Classified as probably not a substantial claim.

In prospective studies and retrospective studies using the original index test interpretation, the diagnosis is naturally interpreted and recorded before the reference standard result is known.

“For prospective studies and retrospective studies that used the original index test interpretation, the diagnosis will by nature be interpreted and recorded before the result of the reference standard is known;”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Maier 2015, 20 of 195 lesions (10%) were excluded because of poor image quality.

“Excluded participants: 20 lesions (10%) excluded due to poor image quality (significant amount of hair, lesion out of focus or multiple lesions in the focus).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Study results can be reproduced only when the diagnostic threshold is reported in enough detail.

“Study results can only be reproduced if the diagnostic threshold is described in sufficient detail.”

May combine two claims.

Smartphone applications using artificial intelligence-based analysis have not yet demonstrated sufficient promise in terms of accuracy, and they are associated with a high likelihood of missing melanomas.

“Smartphone applications using artificial intelligence-based analysis have not yet demonstrated sufficient promise in terms of accuracy, and they are associated with a high likelihood of missing melanomas.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review team considered restricting study populations to particular lesion types, such as melanocytic or nodular lesions, or by size, to raise high concern about applicability.

“In judging the applicability of patient populations to the review question, we considered restriction to particular lesion populations, such as melanocytic, nodular, high risk or restrictions by size to be of high concern for applicability.”

• cites Whiting 2011 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The review included derivation studies of new mobile phone applications only when they evaluated the new approach using a separate independent test set of participants or images.

“We included studies developing new mobile phone applications (i.e.derivation studies) if they used a separate independent 'test set' of participants or images to evaluate the new approach.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review was unable to supplement its database searches with searches of annual meeting abstracts because of the volume of retrieved evidence and time constraints.

“We planned to supplement the database searches by searching the annual meetings of appropriate organisations (e.g.British Association of Dermatologists Annual Meeting, American Academy of Dermatology Annual Meeting, European Academy of Dermatology and Venereology Meeting, Society for Melanoma Research Congress, World Congress of Dermatology, European Association of Dermato Oncology)”

• Also stated in: § Acknowledgements · plain

  “however, due to the volume of evidence retrieved from database searches and time restrictions we were unable to do this.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review excluded studies that evaluated the accuracy of individual lesion characteristics or morphological features without an overall diagnosis of malignancy.

“evaluated the accuracy of the presence or absence of individual lesion characteristics or morphological features, with no overall diagnosis of malignancy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Maier 2015, an additional 31 lesions were excluded as unevaluable (13 for two-point differences and 18 tie cases).

“An additional 31 were excluded as unevaluable (13 lesions (6%) based on two-point-differences and 18 (9%) tie cases with an equal number of results in different risk classes)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review team classified studies that added malignant or benign lesions to a series of suspicious lesions as at unclear risk of bias.

“We considered studies that separately sampled malignant and benign lesions to have used a case-control design; and those that supplemented a series of suspicious lesions with additional malignant or benign lesions to be at unclear risk of bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The index test in Wolf 2013 was not applied and interpreted in a clinically applicable manner.

“Was the test applied and interpreted in a clinically applicable manner?No”

Classified as probably not a substantial claim.

The review could not restrict its analysis to per-patient data because of a lack of data.

“In terms of analysis, we could not restrict analysis to per patient data due to lack of data.”

Classified as probably not a substantial claim.

The review excluded studies that derived an equation or algorithm from multiple diagnostic features via a statistical model and lacked a separate test set.

“used a statistical model to produce a data-driven equation or algorithm based on multiple diagnostic features, with no separate test set;”

Probably not a substantial claim: may not pass S3 (load-bearing).

Not all patients in Wolf 2013 were included in the analysis.

“Were all patients included in the analysis?No”

Classified as probably not a substantial claim.

The review team considered studies with a high ratio of lesions to participants (more than 5% difference) to be less representative than studies in a more general participant population.

“We considered studies that include a high number of lesions in relation to the number of participants in the study to be less representative than studies conducted in a more general population of participants (i.e. if the difference between the number of included lesions and number of included participants is greater than 5%).”

May combine two claims.

Applications based on store-and-forward images could have a potential role in the timely presentation of people with potentially malignant lesions by facilitating active self-management health practices and early engagement of those with suspicious skin lesions; however, there are resource and workload implications with a store-and-forward approach.

“Applications based on store-and-forward images could have a potential role in the timely presentation of people with potentially malignant lesions by facilitating active self-management health practices and early engagement of those with suspicious skin lesions; however, there are resource and workload implications with a store-and-forward approach.”

Classified as probably not a substantial claim.

Because skin cancer diagnosis is primarily lesion-based, participants with multiple lesions may contribute disproportionately to test accuracy estimates, especially if they are at high risk of skin cancer.

“Given that diagnosis of skin cancer is primarily lesion-based, there is the potential for study participants with multiple lesions to contribute disproportionately to estimates of test accuracy, especially if they are at particular risk of having skin cancer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wolf 2013 did not avoid a case-control design.

“Was a case-control design avoided?No”

Classified as probably not a substantial claim.

Conference abstracts alone lack the detail needed to fully assess study methods or methodological quality.

“studies available only as conference abstracts do not allow a comprehensive assessment of study methods or methodological quality.”

May combine two claims.

Most false-negative melanomas are detected within three months of an initial negative index test, with a small number detected up to 12 months later.

“This interval was chosen based on a study showing that most false-negative melanomas will be diagnosed within three months of the initial negative index test, although a small number will be diagnosed up to 12 months subsequently (Altamura 2008).”

• cites Altamura 2008 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Wolf 2013 did not avoid inappropriate exclusions.

“Did the study avoid inappropriate exclusions?No”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because melanoma test interpretation can be subjective, blinding the index test interpretation to the reference standard result is a key way to reduce bias.

“Given the potential for subjective differences in test interpretation for melanoma, the interpretation of the index test blinded to the result of the reference standard is a key means of reducing bias.”