Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients
Open accessPublished version of a preprint
The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; 21 of the 35 sources its claims cite were checked for support.
Abstract
Background: Digital technologies have become a common starting point for health-related information-seeking. Web- or app-based symptom checkers aim to provide rapid and accurate condition suggestions and triage advice but have not yet been investigated for mental disorders in routine health care settings. Objective: This study aims to test the diagnostic performance of a widely available symptom checker in the context of formal diagnosis of mental disorders when compared with therapists’ diagnoses based on structured clinical interviews. Methods: Adult patients from an outpatient psychotherapy clinic used the app-based symptom checker Ada–check your health (ADA; Ada Health GmbH) at intake. Accuracy was assessed as the agreement of the first and 1 of the first 5 condition suggestions of ADA with at least one of the interview-based therapist diagnoses. In addition, sensitivity, specificity, and interrater reliabilities (Gwet first-order agreement coefficient [AC1]) were calculated for the 3 most prevalent disorder categories. Self-reported usability (assessed using the System Usability Scale) and acceptance of ADA (assessed using an adapted feedback questionnaire) were evaluated. Results: A total of 49 patients (30/49, 61% women; mean age 33.41, SD 12.79 years) were included in this study. Across all patients, the interview-based diagnoses matched ADA’s first condition suggestion in 51% (25/49; 95% CI 37.5-64.4) of cases and 1 of the first 5 condition suggestions in 69% (34/49; 95% CI 55.4-80.6) of cases. Within the main disorder categories, the accuracy of ADA’s first condition suggestion was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders. Interrater reliabilities ranged from low (AC1=0.15 for anxiety disorders) to good (AC1=0.76 for somatoform and associated disorders). The usability of ADA was rated as high in the System Usability Scale (mean 81.51, SD 11.82, score range 0-100). Approximately 71% (35/49) of participants would have preferred a face-to-face over an app-based diagnostic. Conclusions: Overall, our findings suggest that a widely available symptom checker used in the formal diagnosis of mental disorders could provide clinicians with a list of condition suggestions with moderate-to-good accuracy. However, diagnostic performance was heterogeneous between disorder categories and included low interrater reliability. Although symptom checkers have some potential to complement the diagnostic process as a screening tool, the diagnostic performance should be tested in larger samples and in comparison with further diagnostic instruments. ©Severin Hennemann, Sebastian Kuhn, Michael Witthöft, Stefanie M Jungmann.
Overview
CoChat Summary
## TL;DR
In 49 psychotherapy outpatients, the app-based symptom checker ADA matched therapists’ interview-based diagnoses in 51% of cases for its first suggestion and 69% for one of its first 5 suggestions. Usability was rated high (SUS mean 81.51/100), but diagnostic performance varied by disorder category and 71% of participants preferred face-to-face diagnosis.
## Key findings
- Across all 49 patients, ADA’s first condition suggestion matched at least one therapist diagnosis in 51% of cases (25/49; 95% CI 37.5-64.4), and 1 of the first 5 suggestions matched in 69% of cases (34/49; 95% CI 55.4-80.6).
- For the 3 most prevalent disorder categories, ADA’s first-suggestion accuracy was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders; interrater reliability (AC1) ranged from 0.15 for anxiety disorders to 0.76 for somatoform and associated disorders.
- Sensitivity was highest for affective disorders (0.65-0.71) and lowest for somatoform and associated disorders (0.22-0.29); the paper also reports first-suggestion accuracy for severe vs mild/moderate depression of 44% (10/23) and top-5 accuracy of 61% (14/23).
- Usability was rated high on the SUS (mean 81.51, SD 11.82), but 71% (35/49) of participants said they would have preferred a face-to-face interview.
## Methods
- Observational, comparative, prospective study in adult outpatients at the psychotherapy outpatient clinic of the University of Mainz, Germany; participants were recruited consecutively from August 2019 to December 2020.
- Included 49 adults with sufficient German; excluded patients with acute suicidality, self-indicated acute mental/physical states preventing app use, or no therapist diagnosis. Diagnoses were made by 42 experienced therapists using the German version of the SCID.
- Patients used the app ADA–check your health on a tablet at intake; analyses compared ADA’s first and first 5 condition suggestions with therapist diagnoses, and calculated agreement, sensitivity, specificity, and Gwet AC1 for the 3 most prevalent disorder categories.
## Limitations
- The paper states that therapists’ diagnoses used additional information beyond the diagnostic interview (eg, anamnesis, medical records, questionnaires) that was not available to ADA, creating an informational imbalance.
- The authors note the study was a pilot with a rather small sample, and that pandemic restrictions further impeded recruitment; they call for larger multicenter studies.
- The diagnostic spectrum was limited, with substance abuse disorders, eating disorders, and PTSD underrepresented; this is stated in the paper.
- Evident from the text: the study did not assess the correctness of interview-based diagnoses independently, and it compared ADA’s suggestions for one symptomatology with therapists’ final diagnoses, which limits direct comparability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 Valid · 4 Not valid · 6 Not checked · 60 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57].”
§ Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients > Discussion > Principal Findings To our knowledge, this comparative study is the first to independently investigate the diagnostic accuracy of a popular symptom checker (ADA) as a screening tool for mental disorders compared with validated formal diagnoses in real-world patients. Our results show that, in approximately half of all investigated cases (25/49, 51%), ADA’s first listed condition suggestion was correctly aligned with any of the interview-based expert diagnoses. This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57]. Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35]. When compared with a study by Barriga et al [58], who investigated the accuracy of another symptom checker (Mediktor) in real patients in an emergency care unit, the accuracy for the first listed condition suggestions was in a comparable range (51% vs 42.9%). In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58]. However, our findings can only be compared with the accuracy from previous studies to a limited extent. These studies included only 1 potentially correct diagnosis per case as opposed to multiple diagnoses per case in our study. The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62]. However, the different measures of agreement must be considered here. Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34]. This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents. However, our study was designed differently as we did not use standardized vignettes, and therapist diagnoses were not checked by independent raters. An interesting future study design would be to directly compare the expert and consumer-level use of symptom checkers and explore differences in diagnostic performance. However, we provide preliminary evidence that no expert knowledge or user experience may be needed to yield performance rates comparable with those of health professionals using symptom checkers. As our participants were all novices in the use of ADA, we could not test the potential beneficial effect of familiarity on diagnostic accuracy. Future studies could, for example, include a test run where participants enter information from a standardized vignette to familiarize themselves with the symptom checker. Within the most prevalent subcategories of mental disorders in our sample, we observed considerable differences in performance statistics. For somatoform and associated disorders, accuracy, specificity, and interrater reliabilities were highest and could be considered acceptable. This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35]. Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63]. However, the base rate (<10%) was lowest across disorder categories, which in turn may have inflated specificity and interrater reliability. For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [66-68]. However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance. Furthermore, the participants rated the usability of ADA as high, which is in line with data from a previous study in primary care patients [30]. However, self-selection of study participation could have positively biased usability ratings. Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18]. This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69]. As symptom checkers are more likely to complement rather than substitute diagnostic processes, it would be interesting to also investigate patients’ and health professionals’ views on the combination of traditional and digital diagnostic procedures, for example, whether symptom checkers would be preferred as a first or second opinion in differential diagnoses or as assistance in clinical decision-making. In this regard, we did not confront the patients or therapists directly with the condition suggestions to not influence the diagnostic process. However, for clinical implementation, it would be interesting to study how symptom checkers used early in the patient journey preempt the diagnostic process and medical decisions. Further studies could also investigate the trust of users in the diagnostic and triage suggestions of symptom checkers compared with other sources of health information (eg, the internet and health professionals).
What the sources say
- Supportedcites [34] source
The Accuracy of diagnosis section reports that the correct diagnosis was listed first in 34% of evaluations across symptom checkers, within the cited 26%–36% range. The What This Study Adds section says the audit used vignettes covering a range of conditions. The comparison with ADA is the citing authors’ own finding, not a result attributed to this source. (LLM, conf=0.89)
- Supportedcites [36] source
The Results (Table 2) report that symptom checkers listed the correct diagnosis first for 34.0% of clinical vignettes, within the cited 26%–36% range. The Methods describe vignettes covering conditions of differing acuity and prevalence. This paper supplies one of the prior comparison rates; the claim does not attribute ADA’s own accuracy to it. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [57] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance.”
§ Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients > Discussion > Principal Findings To our knowledge, this comparative study is the first to independently investigate the diagnostic accuracy of a popular symptom checker (ADA) as a screening tool for mental disorders compared with validated formal diagnoses in real-world patients. Our results show that, in approximately half of all investigated cases (25/49, 51%), ADA’s first listed condition suggestion was correctly aligned with any of the interview-based expert diagnoses. This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57]. Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35]. When compared with a study by Barriga et al [58], who investigated the accuracy of another symptom checker (Mediktor) in real patients in an emergency care unit, the accuracy for the first listed condition suggestions was in a comparable range (51% vs 42.9%). In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58]. However, our findings can only be compared with the accuracy from previous studies to a limited extent. These studies included only 1 potentially correct diagnosis per case as opposed to multiple diagnoses per case in our study. The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62]. However, the different measures of agreement must be considered here. Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34]. This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents. However, our study was designed differently as we did not use standardized vignettes, and therapist diagnoses were not checked by independent raters. An interesting future study design would be to directly compare the expert and consumer-level use of symptom checkers and explore differences in diagnostic performance. However, we provide preliminary evidence that no expert knowledge or user experience may be needed to yield performance rates comparable with those of health professionals using symptom checkers. As our participants were all novices in the use of ADA, we could not test the potential beneficial effect of familiarity on diagnostic accuracy. Future studies could, for example, include a test run where participants enter information from a standardized vignette to familiarize themselves with the symptom checker. Within the most prevalent subcategories of mental disorders in our sample, we observed considerable differences in performance statistics. For somatoform and associated disorders, accuracy, specificity, and interrater reliabilities were highest and could be considered acceptable. This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35]. Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63]. However, the base rate (<10%) was lowest across disorder categories, which in turn may have inflated specificity and interrater reliability. For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [66-68]. However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance. Furthermore, the participants rated the usability of ADA as high, which is in line with data from a previous study in primary care patients [30]. However, self-selection of study participation could have positively biased usability ratings. Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18]. This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69]. As symptom checkers are more likely to complement rather than substitute diagnostic processes, it would be interesting to also investigate patients’ and health professionals’ views on the combination of traditional and digital diagnostic procedures, for example, whether symptom checkers would be preferred as a first or second opinion in differential diagnoses or as assistance in clinical decision-making. In this regard, we did not confront the patients or therapists directly with the condition suggestions to not influence the diagnostic process. However, for clinical implementation, it would be interesting to study how symptom checkers used early in the patient journey preempt the diagnostic process and medical decisions. Further studies could also investigate the trust of users in the diagnostic and triage suggestions of symptom checkers compared with other sources of health information (eg, the internet and health professionals).
What the sources say
- Supportedcites [39] source
The Discussion’s “Limitations and Research Perspectives” section explicitly says the user groups had a very small sample size. The Methods’ “Participants” section lists two people in each of the three groups. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58].”
§ Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients > Discussion > Principal Findings To our knowledge, this comparative study is the first to independently investigate the diagnostic accuracy of a popular symptom checker (ADA) as a screening tool for mental disorders compared with validated formal diagnoses in real-world patients. Our results show that, in approximately half of all investigated cases (25/49, 51%), ADA’s first listed condition suggestion was correctly aligned with any of the interview-based expert diagnoses. This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57]. Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35]. When compared with a study by Barriga et al [58], who investigated the accuracy of another symptom checker (Mediktor) in real patients in an emergency care unit, the accuracy for the first listed condition suggestions was in a comparable range (51% vs 42.9%). In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58]. However, our findings can only be compared with the accuracy from previous studies to a limited extent. These studies included only 1 potentially correct diagnosis per case as opposed to multiple diagnoses per case in our study. The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62]. However, the different measures of agreement must be considered here. Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34]. This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents. However, our study was designed differently as we did not use standardized vignettes, and therapist diagnoses were not checked by independent raters. An interesting future study design would be to directly compare the expert and consumer-level use of symptom checkers and explore differences in diagnostic performance. However, we provide preliminary evidence that no expert knowledge or user experience may be needed to yield performance rates comparable with those of health professionals using symptom checkers. As our participants were all novices in the use of ADA, we could not test the potential beneficial effect of familiarity on diagnostic accuracy. Future studies could, for example, include a test run where participants enter information from a standardized vignette to familiarize themselves with the symptom checker. Within the most prevalent subcategories of mental disorders in our sample, we observed considerable differences in performance statistics. For somatoform and associated disorders, accuracy, specificity, and interrater reliabilities were highest and could be considered acceptable. This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35]. Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63]. However, the base rate (<10%) was lowest across disorder categories, which in turn may have inflated specificity and interrater reliability. For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [66-68]. However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance. Furthermore, the participants rated the usability of ADA as high, which is in line with data from a previous study in primary care patients [30]. However, self-selection of study participation could have positively biased usability ratings. Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18]. This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69]. As symptom checkers are more likely to complement rather than substitute diagnostic processes, it would be interesting to also investigate patients’ and health professionals’ views on the combination of traditional and digital diagnostic procedures, for example, whether symptom checkers would be preferred as a first or second opinion in differential diagnoses or as assistance in clinical decision-making. In this regard, we did not confront the patients or therapists directly with the condition suggestions to not influence the diagnostic process. However, for clinical implementation, it would be interesting to study how symptom checkers used early in the patient journey preempt the diagnostic process and medical decisions. Further studies could also investigate the trust of users in the diagnostic and triage suggestions of symptom checkers compared with other sources of health information (eg, the internet and health professionals).
What the sources say
- Could not verifycites [34] sourceExperimental — accuracy not yet measured
The Accuracy of diagnosis section reports results for 23 symptom checkers using standardized vignettes, including an 84% result for the best performer within the top 20 diagnoses. It does not identify that result as Ada’s top-five performance or report Ada rates of 77%–84% or 91.3% among emergency-care patients. Searches for Ada and those rates found no passage addressing the attributed comparison; page numbers were unavailable in our copy. (LLM, conf=0.77)
- Partly supportedcites [35] sourceExperimental — accuracy not yet measured
The Methods define top-5 accuracy (M5), and the Discussion > Principal findings reports Ada’s top-5 performance in a vignette study. The passages do not give an Ada top-5 rate of 77%–84%, so they do not verify the numerical range attributed to this source. The 91.3% emergency-care figure has a separate citation in the citing sentence. (LLM, conf=0.78)
- Not matchedcites [58] Moreno Barriga, E, 2017, Emergencias
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34].”
§ Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients > Discussion > Principal Findings To our knowledge, this comparative study is the first to independently investigate the diagnostic accuracy of a popular symptom checker (ADA) as a screening tool for mental disorders compared with validated formal diagnoses in real-world patients. Our results show that, in approximately half of all investigated cases (25/49, 51%), ADA’s first listed condition suggestion was correctly aligned with any of the interview-based expert diagnoses. This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57]. Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35]. When compared with a study by Barriga et al [58], who investigated the accuracy of another symptom checker (Mediktor) in real patients in an emergency care unit, the accuracy for the first listed condition suggestions was in a comparable range (51% vs 42.9%). In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58]. However, our findings can only be compared with the accuracy from previous studies to a limited extent. These studies included only 1 potentially correct diagnosis per case as opposed to multiple diagnoses per case in our study. The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62]. However, the different measures of agreement must be considered here. Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34]. This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents. However, our study was designed differently as we did not use standardized vignettes, and therapist diagnoses were not checked by independent raters. An interesting future study design would be to directly compare the expert and consumer-level use of symptom checkers and explore differences in diagnostic performance. However, we provide preliminary evidence that no expert knowledge or user experience may be needed to yield performance rates comparable with those of health professionals using symptom checkers. As our participants were all novices in the use of ADA, we could not test the potential beneficial effect of familiarity on diagnostic accuracy. Future studies could, for example, include a test run where participants enter information from a standardized vignette to familiarize themselves with the symptom checker. Within the most prevalent subcategories of mental disorders in our sample, we observed considerable differences in performance statistics. For somatoform and associated disorders, accuracy, specificity, and interrater reliabilities were highest and could be considered acceptable. This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35]. Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63]. However, the base rate (<10%) was lowest across disorder categories, which in turn may have inflated specificity and interrater reliability. For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [66-68]. However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance. Furthermore, the participants rated the usability of ADA as high, which is in line with data from a previous study in primary care patients [30]. However, self-selection of study participation could have positively biased usability ratings. Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18]. This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69]. As symptom checkers are more likely to complement rather than substitute diagnostic processes, it would be interesting to also investigate patients’ and health professionals’ views on the combination of traditional and digital diagnostic procedures, for example, whether symptom checkers would be preferred as a first or second opinion in differential diagnoses or as assistance in clinical decision-making. In this regard, we did not confront the patients or therapists directly with the condition suggestions to not influence the diagnostic process. However, for clinical implementation, it would be interesting to study how symptom checkers used early in the patient journey preempt the diagnostic process and medical decisions. Further studies could also investigate the trust of users in the diagnostic and triage suggestions of symptom checkers compared with other sources of health information (eg, the internet and health professionals).
What the sources say
- Could not verifycites [34] sourceExperimental — accuracy not yet measured
The assessing diagnosis and triage results section describes vignette entry into symptom checkers by an author without clinical training, not by medical experts. The strengths and limitations section says the study only indirectly assesses performance with actual patients. Our copy provides no ADA-specific results or comparison with patients using ADA (searched: ADA and expert-entered case vignettes). (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62].”
§ Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients > Discussion > Principal Findings To our knowledge, this comparative study is the first to independently investigate the diagnostic accuracy of a popular symptom checker (ADA) as a screening tool for mental disorders compared with validated formal diagnoses in real-world patients. Our results show that, in approximately half of all investigated cases (25/49, 51%), ADA’s first listed condition suggestion was correctly aligned with any of the interview-based expert diagnoses. This transdiagnostic accuracy was higher than the average rates of symptom checkers from previous comparative studies (26%-36%) that used case vignettes of various health conditions [34,36,57]. Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35]. When compared with a study by Barriga et al [58], who investigated the accuracy of another symptom checker (Mediktor) in real patients in an emergency care unit, the accuracy for the first listed condition suggestions was in a comparable range (51% vs 42.9%). In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58]. However, our findings can only be compared with the accuracy from previous studies to a limited extent. These studies included only 1 potentially correct diagnosis per case as opposed to multiple diagnoses per case in our study. The transdiagnostic accuracy of ADA could be considered lower when compared with sensitivities of self-report screenings for mental disorders that range between 0.72 and 0.90 according to previous studies [59-62]. However, the different measures of agreement must be considered here. Interestingly, the transdiagnostic performance of ADA when used by patients is comparable with that of studies in which medical experts used ADA to enter information based on case vignettes [34]. This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents. However, our study was designed differently as we did not use standardized vignettes, and therapist diagnoses were not checked by independent raters. An interesting future study design would be to directly compare the expert and consumer-level use of symptom checkers and explore differences in diagnostic performance. However, we provide preliminary evidence that no expert knowledge or user experience may be needed to yield performance rates comparable with those of health professionals using symptom checkers. As our participants were all novices in the use of ADA, we could not test the potential beneficial effect of familiarity on diagnostic accuracy. Future studies could, for example, include a test run where participants enter information from a standardized vignette to familiarize themselves with the symptom checker. Within the most prevalent subcategories of mental disorders in our sample, we observed considerable differences in performance statistics. For somatoform and associated disorders, accuracy, specificity, and interrater reliabilities were highest and could be considered acceptable. This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35]. Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63]. However, the base rate (<10%) was lowest across disorder categories, which in turn may have inflated specificity and interrater reliability. For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [66-68]. However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance. Furthermore, the participants rated the usability of ADA as high, which is in line with data from a previous study in primary care patients [30]. However, self-selection of study participation could have positively biased usability ratings. Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18]. This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69]. As symptom checkers are more likely to complement rather than substitute diagnostic processes, it would be interesting to also investigate patients’ and health professionals’ views on the combination of traditional and digital diagnostic procedures, for example, whether symptom checkers would be preferred as a first or second opinion in differential diagnoses or as assistance in clinical decision-making. In this regard, we did not confront the patients or therapists directly with the condition suggestions to not influence the diagnostic process. However, for clinical implementation, it would be interesting to study how symptom checkers used early in the patient journey preempt the diagnostic process and medical decisions. Further studies could also investigate the trust of users in the diagnostic and triage suggestions of symptom checkers compared with other sources of health information (eg, the internet and health professionals).
What the sources say
- Not checked — no full text of the sourcecites [59] source
- Not checked — no full text of the sourcecites [60] source
- Not checked — no full text of the sourcecites [61] source
- Partly supportedcites [62] sourceExperimental — accuracy not yet measured
The Discussion (Principal Results; page not provided) reports sensitivities for the paper’s web screening questionnaire, supporting the general comparison with mental-disorder screening. But its reported range is 0.72–1.00, not 0.72–0.90; it gives 0.63–0.95 for other online questionnaires. The comparison with ADA is the citing authors’ own framing, not a finding attributed to this source. (LLM, conf=0.88)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
31 more claims · 1 with their cited sources checked
Create a free account to see all 83 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 22
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One such method could be smart sensing, with the goal of achieving scalable, precise, and time-effective detection of symptoms (27–29).”
From Acceptance of smart sensing, its determinants, and the efficacy of an acceptance-facilitating intervention in people with diabetes: results from a randomized controlled trial (2024) · cites this paper as [29]
Experimental — accuracy not yet measured
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“For the 3 most prevalent disorder categories in our sample (according to the interview-based diagnoses), we calculated accuracy based on contingency tables as the sum of true positives and true negatives divided by the total number of cases [51], as well as sensitivity and specificity.”
MethodBundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz
(2013) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/103.1437-1588 · Cited by 11,796
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668 · cited 2×
“81 indicate very good strength of agreement [54].”
BackgroundPractical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“The usability of the symptom checker was assessed using the 10-item, unidimensional System Usability Scale (SUS) [46], a widely used, reliable scale [47].”
MethodTHE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
eHealth Literacy: Essential Skills for Consumer Health in a Networked World.
Norman, C. D., & Skinner, H. A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e9 · Cited by 2,643 · cited 1×
“Selecting, interpreting, and using web-based health information requires sufficient eHealth literacy [16]; however, this can be unevenly distributed across age, socioeconomic, or educational groups, which has been termed “digital divide” [17].”
BackgroundThe System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958
Plummer, F., Manea, L., Trepel, D., & McMillan, D. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.11.005 · Cited by 1,882 · cited 1×
“For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [6…”
BackgroundEysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002) · JAMA · doi:10.1001/jama.287.20.2691 · Cited by 1,871 · cited 1×
“Although digital health resources can ideally increase access to health care and empower patients to engage in health behavior [8], the information provided is mostly unregulated and can also contain confusing or unsubstantiated facts and recommendations [9].”
BackgroundThe economic cost of brain disorders in Europe
Olesen, J., Gustavsson, A., Svensson, M., Wittchen, H. ‐., Jönsson, B., on behalf of the CDBE2010 study group, & the European Brain Council (2012) · European Journal of Neurology · doi:10.1111/j.1468-1331.2011.03590.x · Cited by 1,481
Global Burden of Disease and the Impact of Mental and Addictive Disorders
Rehm, J., & Shield, K. D. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-0997-0 · Cited by 1,388
Functional somatic syndromes: one or many?
Wessely, S., Nimnuan, C., & Sharpe, M. (1999) · The Lancet · doi:10.1016/s0140-6736(98)08320-2 · Cited by 1,211 · cited 1×
“Beyond this, the unifying classification of functional somatic syndromes (eg, irritable bowel syndrome and fibromyalgia) as somatoform disorders is subject to ongoing controversial debate [49,63].”
BackgroundLobbestael, J., Leurgans, M., & Arntz, A. (2011) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.693 · Cited by 1,124
Undertreatment of people with major depressive disorder in 21 countries
Thornicroft, G., Chatterji, S., Evans-Lacko, S., Gruber, M., Sampson, N., Aguilar-Gaxiola, S., Al-Hamzawi, A., Alonso, J., Andrade, L., Borges, G., et al. (2017) · British Journal of Psychiatry · doi:10.1192/bjp.bp.116.188078 · Cited by 1,121 · cited 1×
“At the public health level, symptom checkers have some potential to reduce underdiagnosis and undertreatment of mental disorders [76] and may ideally contribute to reducing chronicity and treatment delay as they represent a low-threshold, multilingual diagnostic instrument.”
BackgroundInternet Health Information Seeking and the Patient-Physician Relationship: A Systematic Review
Tan, S. S.-L., & Goonawardene, N. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.5729 · Cited by 1,101 · cited 1×
“As such, digital health resources have already become an important factor in the therapist-patient relationship [79] as more patients use digital resources for diagnostic and treatment purposes.”
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041 · cited 1×
“The AC1 is less prone to overcorrection for chance agreement and less sensitive to low base rates compared with other coefficients such as the Cohen κ [52,53].”
BackgroundMünster, R. D. (1999) · Zeitschrift für Klinische Psychologie und Psychotherapie · doi:10.1026//0084-5345.28.1.68 · Cited by 1,010
A review of depression and suicide risk assessment using speech analysis
Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., & Quatieri, T. F. (2015) · Speech Communication · doi:10.1016/j.specom.2015.03.004 · Cited by 1,003 · cited 1×
“The advancement of passive mobile sensing through smartphones or wearables (eg, mobility pattern, facial expression, and speech analysis [81,82]) may allow for in situ, fine-grained digital phenotyping even without this active user input.”
BackgroundVilagut, G., Forero, C. G., Barbaglia, G., & Alonso, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155431 · Cited by 982
eHealth Literacy: Extending the Digital Divide to the Realm of Health Information
Neter, E., & Brainin, E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1619 · Cited by 843 · cited 1×
“Selecting, interpreting, and using web-based health information requires sufficient eHealth literacy [16]; however, this can be unevenly distributed across age, socioeconomic, or educational groups, which has been termed “digital divide” [17].”
BackgroundGriffiths, F., Lindenmeyer, A., Powell, J., Lowe, P., & Thorogood, M. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e10 · Cited by 820 · cited 1×
“For example, individuals who feel stigmatized or ashamed by mental health issues (eg, obsessive-compulsive symptoms and sexual dysfunctions) could benefit from anonymity and low-threshold information [3,4].”
Background(2007) · PubMed Central · Cited by 623
Internet use and stigmatized illness
Berger, M., Wagner, T. H., & Baker, L. C. (2005) · Social Science & Medicine · doi:10.1016/j.socscimed.2005.03.025 · Cited by 547 · cited 1×
“For example, individuals who feel stigmatized or ashamed by mental health issues (eg, obsessive-compulsive symptoms and sexual dysfunctions) could benefit from anonymity and low-threshold information [3,4].”
BackgroundEvaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 14×
“This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35].”
BackgroundResultInfluentialWang, J., Wu, X., Lai, W., Long, E., Zhang, X., Li, W., Zhu, Y., Chen, C., Zhong, X., Liu, Z., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017173 · Cited by 486
The Symptom Check-List-90-R (SCL-90-R): A German validation study
Schmitz, N., Hartkamp, N., Kiuse, J., Franke, G. H., Reister, G., & Tress, W. (2000) · Quality of Life Research · doi:10.1023/a:1008931926181 · Cited by 430
Reducing the treatment gap for mental disorders: a WPA survey
PATEL, V., MAJ, M., FLISHER, A. J., DE SILVA, M. J., KOSCHORKE, M., PRINCE, M., WPA Zonal and Member Society Representatives, Tempier, R., Riba, M., Sanchez, M., et al. (2010) · World Psychiatry · doi:10.1002/j.2051-5545.2010.tb00305.x · Cited by 420
Physicians’ perceptions of chatbots in health Care: cross-sectional web-based survey
Palanica, A., Flaschner, P., Thommandram, A., Li, M., & Fossat, Y. (2019) · Journal of Medical Internet Research · doi:10.2196/12887 · Cited by 384 · cited 2×
“In contrast to patients’ rather positive perspectives on the usability and utility of symptom checkers, health professionals seem to be more skeptical [25], and symptom checkers have had an inferior performance compared with professional diagnoses in previous studies [32].”
BackgroundResultZimmerman, M., & Mattia, J. I. (2001) · Archives of General Psychiatry · doi:10.1001/archpsyc.58.8.787 · Cited by 375
Mental health monitoring with multimodal sensing and machine learning: A survey
Garcia-Ceja, E., Riegler, M., Nordgreen, T., Jakobsen, P., Oedegaard, K. J., & Tørresen, J. (2018) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2018.09.003 · Cited by 356 · cited 1×
“The advancement of passive mobile sensing through smartphones or wearables (eg, mobility pattern, facial expression, and speech analysis [81,82]) may allow for in situ, fine-grained digital phenotyping even without this active user input.”
BackgroundHigh prevalence of mental disorders in primary care
Ansseau, M., Dierick, M., Buntinkx, F., Cnockaert, P., De Smedt, J., Van Den Haute, M., & Vander Mijnsbrugge, D. (2004) · Journal of Affective Disorders · doi:10.1016/s0165-0327(02)00219-7 · Cited by 352 · cited 1×
“However, the most common mental disorders were frequent in our sample and resembled prevalence rates in medical settings [70].”
BackgroundFink, P., & Schröder, A. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2010.02.004 · Cited by 334 · cited 3×
“Functional somatic syndromes (eg, fibromyalgia and irritable bowel syndrome) were associated with somatoform disorders (F45) [49].”
BackgroundInfluentialMental health: there’s an app for that
Anthes, E. (2016) · Nature · doi:10.1038/532020a · Cited by 333 · cited 1×
“Mental health topics are among the most popular search queries [1], and it is estimated that approximately one-third of all health apps worldwide target mental health issues [2].”
BackgroundA paradigm shift in psychiatric classification: the Hierarchical Taxonomy Of Psychopathology (HiTOP)
Kotov, R., Krueger, R. F., & Watson, D. (2018) · World Psychiatry · doi:10.1002/wps.20478 · Cited by 318 · cited 1×
“Generally, one could also criticize the exclusive categorical diagnostic approach of this study, which has been challenged recently by a strictly empirical and dimensional understanding and taxonomy of psychopathology such as the Hierarchical Taxonomy of Psychopathology [75], and dimensional self-report instruments wo…”
BackgroundArtificial Intelligence in Psychological Practice: Current and Future Applications and Implications
Luxton, D. D. (2014) · Professional Psychology Research and Practice · doi:10.1037/a0034559 · Cited by 297 · cited 1×
“The mostly algorithm-based programs typically operate with chatbots to simulate a dialogue-like human interaction [24].”
BackgroundAssociations of eHealth Literacy With Health Behavior Among Adult Internet Users
Mitsutake, S., Shibata, A., Ishii, K., & Oka, K. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5413 · Cited by 271 · cited 1×
“In addition, symptom checkers could make the diagnostic process less intuitive and controllable, and vulnerable patient groups, less educated people, or older people are probably less likely to take advantage of this resource at the public health level, thus increasing the “digital divide” [77,78].”
Internet information-seeking in mental health: Population survey
Powell, J., & Clarke, A. (2006) · British Journal of Psychiatry · doi:10.1192/bjp.bp.105.017319 · Cited by 253 · cited 1×
“Users typically rate the internet “higher as a source to use than a source to trust” [19], particularly when compared with personal medical information (eg, from health professionals).”
BackgroundChambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · cited 3×
“In contrast to patients’ rather positive perspectives on the usability and utility of symptom checkers, health professionals seem to be more skeptical [25], and symptom checkers have had an inferior performance compared with professional diagnoses in previous studies [32].”
BackgroundMethodResultPhilippi, P., Baumeister, H., Apolinário-Hagen, J., Ebert, D. D., Hennemann, S., Kott, L., Lin, J., Messner, E.-M., & Terhorst, Y. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100459 · Cited by 200 · cited 1×
“This could be critical regarding the reshaping of diagnostic practice as acceptance represents a crucial premise for the implementation of health resources [69].”
BackgroundSelf-citationEastin, M. S., & Guinsler, N. M. (2006) · CyberPsychology & Behavior · doi:10.1089/cpb.2006.9.494 · Cited by 183 · cited 1×
“In addition, digital health information may lead to increased illness anxiety [20], which in turn increases unnecessary health care use and costs [21,22].”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · cited 8×
“This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35].”
BackgroundMethodResultInfluentialSafety of patient-facing digital symptom checkers
Fraser, H., Coiera, E., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 180 · cited 1×
“However, various methodological concerns regarding this study have been raised, such as sensitivity to outliers [38].”
Backgroundvon Glischinski, M., von Brachel, R., & Hirschfeld, G. H. F. (2018) · Quality of Life Research · doi:10.1007/s11136-018-2050-x · Cited by 172 · cited 1×
“For affective and anxiety disorders, performance was lower than one would expect given that these disorder categories have a high prevalence in the general as well as clinical populations [64,65] and when compared with higher sensitivities of self-report screenings, particularly those observed for anxiety disorders [6…”
BackgroundA Brief Web-Based Screening Questionnaire for Common Mental Disorders: Development and Validation
Donker, T., Straten, A. V., Marks, I., & Cuijpers, P. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1134 · Cited by 166
Meyer, A. N. D., Giardina, T. D., Spitzmueller, C., Shahid, U., Scott, T. M. T., & Singh, H. (2020) · Journal of Medical Internet Research · doi:10.2196/14679 · Cited by 161 · cited 1×
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
BackgroundClassification of Somatization and Functional Somatic Symptoms in Primary Care
Fink, P., Rosendal, M., & Olesen, F. (2005) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2005.01682.x · Cited by 159
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 157
Lilienfeld, S. O., Sauvigné, K. C., Lynn, S. J., Cautin, R. L., Latzman, R. D., & Waldman, I. D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01100 · Cited by 154 · cited 1×
“Previous studies have demonstrated a large variation in interrater reliabilities of diagnoses based on SCIDs that can range from substantial to even low agreement [71-73], which may challenge the validity of this as a gold standard in diagnosis [74].”
BackgroundWittchen, H. ‐., Höfler, M., Gander, F., Pfister, H., Storz, S., Üstün, B., Müller, N., & Kessler, R. C. (1999) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.57 · Cited by 149
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 5×
“According to a review by Semigran et al [36], 84.”
MethodResultInfluentialDigital Divide – Soziale Unterschiede in der Nutzung digitaler Gesundheitsangebote
Cornejo Müller, A., Wachtler, B., & Lampert, T. (2020) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-019-03081-y · Cited by 134 · cited 1×
“In addition, symptom checkers could make the diagnostic process less intuitive and controllable, and vulnerable patient groups, less educated people, or older people are probably less likely to take advantage of this resource at the public health level, thus increasing the “digital divide” [77,78].”
BackgroundJungmann, S. M., Klan, T., Kuhn, S., & Jungmann, F. (2019) · JMIR Formative Research · doi:10.2196/13863 · Cited by 127 · cited 6×
“A recent pilot study by Jungmann et al [39] investigated the performance and dependency on expert knowledge of the symptom checker ADA in diagnosing mental disorders in adults and adolescents.”
BackgroundMethodResultInfluentialSelf-citationAccuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · cited 8×
“Most recently, a study by Ceney et al [33] yielded comparable average performance rates (51%, range 22.2%-84”
BackgroundResultInfluentialMiller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117 · cited 3×
“This particular app was selected for various reasons: (1) the diagnostic coverage is wide [35], including mental disorders, and ADA has shown acceptable diagnostic performance in this diagnostic spectrum recently [39]; (2) it is free of charge and widely available (>10 million users and 7 languages) for Android- and i…”
BackgroundResultMorbilidad psiquiátrica en pacientes mayores de 18 años
King, M. (n.d.) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 114
Hill, M. G., Sim, M., & Mills, B. (2020) · Medical Journal of Australia · doi:10.5694/mja2.50600 · Cited by 114 · cited 3×
“Here, evidence shows that, although most symptom checkers seem to provide safe triage advice [33], they are somewhat more risk-averse [57] than health professionals, which could increase health care use and costs.”
BackgroundMethodResultSander, L. B., Schorndanner, J., Terhorst, Y., Spanhel, K., Pryss, R., Baumeister, H., & Messner, E.-M. (2020) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1701788 · Cited by 102 · cited 2×
“Eventually, we performed an independent scientific evaluation of a commercially available product, which seems important given the plethora of health apps that have not been scientifically reviewed [14,15].”
BackgroundMethodHealthcare non-adherence decisions and internet health information
Weaver, J. B., Thompson, N. J., Weaver, S. S., & Hopkins, G. L. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2009.05.011 · Cited by 102 · cited 1×
“This could promote incorrect self-diagnosis and problematic health decisions [10].”
BackgroundEvaluating Confidence Interval Methods for Binomial Proportions in Clustered Surveys
Dean, N. E., & Pagano, M. (2015) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smv024 · Cited by 93 · cited 1×
“Accuracy was calculated as the percentage of agreement along with the 95% CI for binomial distributions with the Agresti-Coull method [50].”
MethodBaumann, E., Czerwinski, F., Rosset, M., Seelig, M., & Suhr, R. (2020) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-020-03192-x · Cited by 92 · cited 2×
“Concerning acceptability, almost three-fourths of our participants (35/49, 71%) preferred face-to-face diagnostics by a health professional over the symptom checker, which is comparable with preference ratings from the German general population [18].”
BackgroundBeyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis
Millenson, M. L., Baldwin, J. L., Zipperer, L., & Singh, H. (2018) · Diagnosis · doi:10.1515/dx-2018-0009 · Cited by 91
Terhorst, Y., Rathner, E.-M., Baumeister, H., & Sander, L. (2018) · Verhaltenstherapie · doi:10.1159/000481692 · Cited by 84 · cited 2×
“Eventually, we performed an independent scientific evaluation of a commercially available product, which seems important given the plethora of health apps that have not been scientifically reviewed [14,15].”
BackgroundMethodIncrease in the prevalence of health anxiety in medical clinics: Possible cyberchondria
Tyrer, P., Cooper, S., Tyrer, H., Wang, D., & Bassett, P. (2019) · International Journal of Social Psychiatry · doi:10.1177/0020764019866231 · Cited by 80 · cited 1×
“In addition, digital health information may lead to increased illness anxiety [20], which in turn increases unnecessary health care use and costs [21,22].”
BackgroundCheniaux, E., Landeira-Fernandez, J., & Versiani, M. (2009) · Psychopathology · doi:10.1159/000228838 · Cited by 79 · cited 1×
“Previous studies have demonstrated a large variation in interrater reliabilities of diagnoses based on SCIDs that can range from substantial to even low agreement [71-73], which may challenge the validity of this as a gold standard in diagnosis [74].”
BackgroundDiagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study
Munsch, N., Martin, A., Gruarin, S., Nateqi, J., Abdarahmane, I., Weingartner-Ortner, R., & Knapp, B. (2020) · Journal of Medical Internet Research · doi:10.2196/21299 · Cited by 78 · cited 1×
“General diagnostic and triage advice of specific symptom checkers has been studied for a broad range of general and specialized health problems [26], for example, ophthalmologic [27] or viral diseases [28,29].”
BackgroundThe Quality of Mental Health Information Commonly Searched for on the Internet
Grohol, J. M., Slimowicz, J., & Granda, R. (2014) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2013.0258 · Cited by 69 · cited 2×
“A study by Grohol et al [11] on the quality of web-based mental health information revealed that 67.5% of 440 investigated websites contained information of good or better quality.”
BackgroundInfluentialAssociation of Use of Online Symptom Checkers With Patients’ Plans for Seeking Care
Winn, A. N., Somai, M., Fergestrom, N., & Crotty, B. H. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.18561 · Cited by 67 · cited 1×
“This would also allow for a better understanding of adequate medical help-seeking, which seems to be positively associated with the triage advice of symptom checkers [83].”
BackgroundDo Dr. Google and Health Apps Have (Comparable) Side Effects? An Experimental Study
Jungmann, S. M., Brand, S., Kolb, J., & Witthöft, M. (2020) · Clinical Psychological Science · doi:10.1177/2167702619894904 · Cited by 47
Self-citationIpser, J., Dewing, S., & Stein, D. J. (2007) · Current Psychiatry Reports · doi:10.1007/s11920-007-0037-3 · Cited by 46 · cited 1×
“For anxiety disorders, another study found only a poor-to-moderate quality of internet-based information [12].”
BackgroundFactors in Patient Empowerment: A Survey of an Online Patient Research Network
Chiauzzi, E. J., DasMahapatra, P., Cochin, E., Bunce, M. M., Khoury, R. R., & Dave, P. (2016) · Patient · doi:10.1007/s40271-016-0171-2 · Cited by 44
Should You Search the Internet for Information About Your Acute Symptom?
North, F., Ward, W. J., Varkey, P., & Tulledge-Scheitel, S. M. (2012) · Telemedicine and e-Health · doi:10.1089/tmj.2011.0127 · Cited by 40 · cited 1×
“In addition, many websites also showed a lack of or inadequate information regarding a rough classification of symptoms and possible health care professionals or services to contact [13].”
BackgroundAccuracy of a Popular Online Symptom Checker for Ophthalmic Diagnoses
Shen, C., Nguyen, M., Gregor, A., Isaza, G., & Beattie, A. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.0571 · Cited by 40 · cited 1×
“General diagnostic and triage advice of specific symptom checkers has been studied for a broad range of general and specialized health problems [26], for example, ophthalmologic [27] or viral diseases [28,29].”
BackgroundValidity of routine clinical DSM-IV diagnoses (Axis I/II) in inpatients with mental disorders
Andreas, S., Theisen, P., Mestel, R., Koch, U., & Schulz, H. (2009) · Psychiatry Research · doi:10.1016/j.psychres.2008.09.009 · Cited by 34 · cited 1×
“Previous studies have demonstrated a large variation in interrater reliabilities of diagnoses based on SCIDs that can range from substantial to even low agreement [71-73], which may challenge the validity of this as a gold standard in diagnosis [74].”
BackgroundOnline symptom checker diagnostic and triage accuracy for HIV and hepatitis C
Berry, A. C., Cash, B. D., Wang, B., Mulekar, M. S., Haneghan, A. B. V., Yuquimpo, K., Swaney, A., Marshall, M. C., & Green, W. K. (2019) · Epidemiology and Infection · doi:10.1017/s0950268819000268 · Cited by 28 · cited 1×
“General diagnostic and triage advice of specific symptom checkers has been studied for a broad range of general and specialized health problems [26], for example, ophthalmologic [27] or viral diseases [28,29].”
BackgroundPatients' feedback after computer-assisted diagnostic interviews for mental disorders
Hoyer, J., Ruhl, U., Scholz, D., & Wittchen, H.-U. (2006) · Psychotherapy Research · doi:10.1080/10503300500485540 · Cited by 26 · cited 3×
“Item descriptions for the feedback questionnaire (adapted from Hoyer et al [48]).”
BackgroundMethodInfluentialWangler, J., & Janský, M. (2020) · Wiener Medizinische Wochenschrift · doi:10.1007/s10354-020-00777-8 · Cited by 19 · cited 1×
“In this regard, health professionals are also facing new challenges (eg, biased expectations and less trust in medical advice) with internet-informed patients [23].”
BackgroundAre Lay People Good at Recognising the Symptoms of Schizophrenia?
Erritty, P., & Wydell, T. N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0052913 · Cited by 18
A Novel Diagnostic Decision Support System for Medical Professionals: Prospective Feasibility Study
Timiliotis, J., Blümke, B., Serfözö, P. D., Gilbert, S., Ondrésik, M., Türk, E., Hirsch, M. C., & Eckstein, J. (2022) · JMIR Formative Research · doi:10.2196/29943 · Cited by 13 · cited 1×
“For this purpose, ADA is available at the consumer level as a self-assessment app [42], whereas a prototype diagnostic decision support system for health professionals has been developed as well [43].”
Background
Research works printed but not linked: 7
- Wang, PS, 2007, World Psychiatry, 6, 177
- Beck, AT, 1996, Manual for the beck depression inventory-II
- Wittchen, HU, 1997, SKID I: Strukturiertes Klinisches Interview für DSM-IV, 1
- HoffmannHAda health: our approach to assess Ada’s diagnostic performanceAda2021-07-16https://www.itu.int/en/ITU-T/Workshops-and-Seminars/20180925/Documents/3_Henry%20Hoffmann.pdf
- Bangor, A, 2009, J Usability Stud, 4, 114
- Moreno Barriga, E, 2017, Emergencias, 29, 391
- RizzoAALucasGGratchJStratouGMorencyLPShillingRHartholtASchererSClinical interviewing by a virtual human agent with automatic behavior analysisProceedings of the 11th international conference on disability, virtual reality and associated technologies2016ICDVRAT'16September 22-26, 2016Los Angeles5764
Cited by
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99
“In addition, the accuracy of advice does not only vary considerably between symptom checkers but also within a given symptom checker, as it may prove more reliable in appraising some categories of diseases than others [48].”
BackgroundHindelang, M., Sitaru, S., & Zink, A. (2024) · JMIR Medical Informatics · doi:10.2196/56628 · Cited by 79
Dialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 86
“While the previous study was based on textbook diagnoses, the recent comparative study [14] aimed at testing the diagnostic performance of the same health application Ada on real-world patient cases.”
MethodKopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 69
“…advice (e.g., mistaking myocardial infarction for a panic attack) 18,44 or assessing the psychological harm that could result from overcautious recommendations (e.g., high-urgency advice may be safe for patients from a systems point of view, but cause unnecessary fears in individual users) 45,46 .”
Sarkar, S., Gaur, M., Chen, L. K., Garg, M., & Srivastava, B. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1229805 · Cited by 38
Metadata
- Authors
- Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M.
- Affiliations
- Johannes Gutenberg University Mainz (DE)
- Corresponding author
- Hennemann, Severin
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2022-01-31
- Volume
- 9
- Issue
- 1
- Pages
- e32832
- ISSN
- 2368-7959 (ISSN-L)
- PMC
- PMC8844983
- OpenAlex
- W3212889536
- Cited by
- 31
- References
- 80
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 9 · 2023: 7 · 2022: 7
- FWCI
- 5.16
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Mental Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- mental illness, diagnostic accuracy, structured clinical interview, somatoform disorders, mood disorders, anxiety disorders, usability, patient acceptance, digital health tools, triage, patient-reported outcomes, mental health screening, inter-rater reliability, sensitivity and specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.