Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; 21 of the 35 sources its claims cite were checked for support.

Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients

Open access
Hennemann, Severin, Kuhn, Sebastian, Witthöft, Michael, Jungmann, Stefanie M · 2022 · JMIR Mental Health

Published version of a preprint

Abstract

Background: Digital technologies have become a common starting point for health-related information-seeking. Web- or app-based symptom checkers aim to provide rapid and accurate condition suggestions and triage advice but have not yet been investigated for mental disorders in routine health care settings. Objective: This study aims to test the diagnostic performance of a widely available symptom checker in the context of formal diagnosis of mental disorders when compared with therapists’ diagnoses based on structured clinical interviews. Methods: Adult patients from an outpatient psychotherapy clinic used the app-based symptom checker Ada–check your health (ADA; Ada Health GmbH) at intake. Accuracy was assessed as the agreement of the first and 1 of the first 5 condition suggestions of ADA with at least one of the interview-based therapist diagnoses. In addition, sensitivity, specificity, and interrater reliabilities (Gwet first-order agreement coefficient [AC1]) were calculated for the 3 most prevalent disorder categories. Self-reported usability (assessed using the System Usability Scale) and acceptance of ADA (assessed using an adapted feedback questionnaire) were evaluated. Results: A total of 49 patients (30/49, 61% women; mean age 33.41, SD 12.79 years) were included in this study. Across all patients, the interview-based diagnoses matched ADA’s first condition suggestion in 51% (25/49; 95% CI 37.5-64.4) of cases and 1 of the first 5 condition suggestions in 69% (34/49; 95% CI 55.4-80.6) of cases. Within the main disorder categories, the accuracy of ADA’s first condition suggestion was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders. Interrater reliabilities ranged from low (AC1=0.15 for anxiety disorders) to good (AC1=0.76 for somatoform and associated disorders). The usability of ADA was rated as high in the System Usability Scale (mean 81.51, SD 11.82, score range 0-100). Approximately 71% (35/49) of participants would have preferred a face-to-face over an app-based diagnostic. Conclusions: Overall, our findings suggest that a widely available symptom checker used in the formal diagnosis of mental disorders could provide clinicians with a list of condition suggestions with moderate-to-good accuracy. However, diagnostic performance was heterogeneous between disorder categories and included low interrater reliability. Although symptom checkers have some potential to complement the diagnostic process as a screening tool, the diagnostic performance should be tested in larger samples and in comparison with further diagnostic instruments. ©Severin Hennemann, Sebastian Kuhn, Michael Witthöft, Stefanie M Jungmann.

Overview

CoChat Summary

## TL;DR

In 49 psychotherapy outpatients, the app-based symptom checker ADA matched therapists’ interview-based diagnoses in 51% of cases for its first suggestion and 69% for one of its first 5 suggestions. Usability was rated high (SUS mean 81.51/100), but diagnostic performance varied by disorder category and 71% of participants preferred face-to-face diagnosis.

## Key findings

  • Across all 49 patients, ADA’s first condition suggestion matched at least one therapist diagnosis in 51% of cases (25/49; 95% CI 37.5-64.4), and 1 of the first 5 suggestions matched in 69% of cases (34/49; 95% CI 55.4-80.6).
  • For the 3 most prevalent disorder categories, ADA’s first-suggestion accuracy was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders; interrater reliability (AC1) ranged from 0.15 for anxiety disorders to 0.76 for somatoform and associated disorders.
  • Sensitivity was highest for affective disorders (0.65-0.71) and lowest for somatoform and associated disorders (0.22-0.29); the paper also reports first-suggestion accuracy for severe vs mild/moderate depression of 44% (10/23) and top-5 accuracy of 61% (14/23).
  • Usability was rated high on the SUS (mean 81.51, SD 11.82), but 71% (35/49) of participants said they would have preferred a face-to-face interview.

## Methods

  • Observational, comparative, prospective study in adult outpatients at the psychotherapy outpatient clinic of the University of Mainz, Germany; participants were recruited consecutively from August 2019 to December 2020.
  • Included 49 adults with sufficient German; excluded patients with acute suicidality, self-indicated acute mental/physical states preventing app use, or no therapist diagnosis. Diagnoses were made by 42 experienced therapists using the German version of the SCID.
  • Patients used the app ADA–check your health on a tablet at intake; analyses compared ADA’s first and first 5 condition suggestions with therapist diagnoses, and calculated agreement, sensitivity, specificity, and Gwet AC1 for the 3 most prevalent disorder categories.

## Limitations

  • The paper states that therapists’ diagnoses used additional information beyond the diagnostic interview (eg, anamnesis, medical records, questionnaires) that was not available to ADA, creating an informational imbalance.
  • The authors note the study was a pilot with a rather small sample, and that pandemic restrictions further impeded recruitment; they call for larger multicenter studies.
  • The diagnostic spectrum was limited, with substance abuse disorders, eating disorders, and PTSD underrepresented; this is stated in the paper.
  • Evident from the text: the study did not assess the correctness of interview-based diagnoses independently, and it compared ADA’s suggestions for one symptomatology with therapists’ final diagnoses, which limits direct comparability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/32832

Cites

81 of 81 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

13 Valid · 4 Not valid · 6 Not checked · 60 Own results and interpretations

31 more claims · 1 with their cited sources checked

Create a free account to see all 83 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 34
  • Probably not a substantial claim. 22
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 80 references

Research works printed but not linked: 7

  1. Wang, PS, 2007, World Psychiatry, 6, 177
  2. Beck, AT, 1996, Manual for the beck depression inventory-II
  3. Wittchen, HU, 1997, SKID I: Strukturiertes Klinisches Interview für DSM-IV, 1
  4. HoffmannHAda health: our approach to assess Ada’s diagnostic performanceAda2021-07-16https://www.itu.int/en/ITU-T/Workshops-and-Seminars/20180925/Documents/3_Henry%20Hoffmann.pdf
  5. Bangor, A, 2009, J Usability Stud, 4, 114
  6. Moreno Barriga, E, 2017, Emergencias, 29, 391
  7. RizzoAALucasGGratchJStratouGMorencyLPShillingRHartholtASchererSClinical interviewing by a virtual human agent with automatic behavior analysisProceedings of the 11th international conference on disability, virtual reality and associated technologies2016ICDVRAT'16September 22-26, 2016Los Angeles5764

Cited by

See all 31

Metadata

Authors
Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M.
Affiliations
Johannes Gutenberg University Mainz (DE)
Corresponding author
Hennemann, Severin
Venue
JMIR Mental Health
Published
2022-01-31
Volume
9
Issue
1
Pages
e32832
ISSN
2368-7959 (ISSN-L)
OpenAlex
W3212889536
Cited by
31
References
80
Citations by year
2026: 2 · 2025: 6 · 2024: 9 · 2023: 7 · 2022: 7
FWCI
5.16
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Mental Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
mental illness, diagnostic accuracy, structured clinical interview, somatoform disorders, mood disorders, anxiety disorders, usability, patient acceptance, digital health tools, triage, patient-reported outcomes, mental health screening, inter-rater reliability, sensitivity and specificity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients. https://app.cochat.ai/papers/10.2196/32832
The paper
Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022). Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients. JMIR Mental Health. https://doi.org/10.2196/32832

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.