Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users
Open accessAbstract
BACKGROUND: Health apps for the screening and diagnosis of mental disorders have emerged in recent years on various levels (eg, patients, practitioners, and public health system). However, the diagnostic quality of these apps has not been (sufficiently) tested so far. OBJECTIVE: The objective of this pilot study was to investigate the diagnostic quality of a health app for a broad spectrum of mental disorders and its dependency on expert knowledge. METHODS: Two psychotherapists, two psychology students, and two laypersons each read 20 case vignettes with a broad spectrum of mental disorders. They used a health app (Ada-Your Health Guide) to get a diagnosis by entering the symptoms. Interrater reliabilities were computed between the diagnoses of the case vignettes and the results of the app for each user group. RESULTS: Overall, there was a moderate diagnostic agreement (kappa=0.64) between the results of the app and the case vignettes for mental disorders in adulthood and a low diagnostic agreement (kappa=0.40) for mental disorders in childhood and adolescence. When psychotherapists applied the app, there was a good diagnostic agreement (kappa=0.78) regarding mental disorders in adulthood. The diagnostic agreement was moderate (kappa=0.55/0.60) for students and laypersons. For mental disorders in childhood and adolescence, a moderate diagnostic quality was found when psychotherapists (kappa=0.53) and students (kappa=0.41) used the app, whereas the quality was low for laypersons (kappa=0.29). On average, the app required 34 questions to be answered and 7 min to complete. CONCLUSIONS: The health app investigated here can represent an efficient diagnostic screening or help function for mental disorders in adulthood and has the potential to support especially diagnosticians in their work in various ways. The results of this pilot study provide a first indication that the diagnostic accuracy is user dependent and improvements in the app are needed especially for mental disorders in childhood and adolescence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Experts react differently to systems than lay people.
“We fnd this, albeit understandable due to limited resources and recruitment possibilities, problematic as experts react diferently to systems than lay people (see, e.g., [59]).”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [59]
Previous findings by Jungmann et al. demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents.
“This is in contrast to previous findings by Jungmann et al [39], who demonstrated lower performance rates of ADA in laypeople compared with health professionals with regard to correctly identifying mental disorders from case vignettes of adults and adolescents.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [39]
The study had a small sample size.
“However, with regard to the small sample size, and as the diagnostic coding scheme [39] could be considered relatively liberal for some disorders, replication in a larger sample and with more fine-grained diagnostic coding seems warranted to obtain a more robust estimation of diagnostic performance.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [39]
12 of the 17 included studies had sample sizes smaller than 100.
“12 of the 17 included studies had sample sizes smaller than 100 [28–31,33–35,37–39,41,42].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
The sample sizes included in the studies ranged from 6 to 354, but the number analysed was often lower than the recruited sample.
“The sample sizes included in the studies ranged from 6 [39] to 354 [40], but the number analysed was often lower than the recruited sample.”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
In some studies, AI/ML was used to provide diagnostic decision support.
“In other studies, AI/ML was used to provide diagnostic decision support [39]”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
One study provided diagnostic support from a broad spectrum of mental disorders.
“and one provided diagnostic support from a broad spectrum of mental disorders [39].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
5 of the 17 studies did not specify what machine learning algorithm was used.
“5 of the 17 studies did not specify what machine learning algorithm was used and are thus not included in Table 5 [38–40,42,43].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
The sample sizes of the studies were relatively low, ranging from 6 to 354, with the latter being a retrospective descriptive study of real-world users, not recruited participants.
“The sample sizes of the studies were relatively low, ranging from 6 [39] to 354 [40], with the latter being a retrospective descriptive study of real-world users, not recruited participants.”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz
(2013) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/103.1437-1588 · Cited by 11,769 · see citing works
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · see citing works
Lobbestael, J., Leurgans, M., & Arntz, A. (2011) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.693 · Cited by 1,124 · see citing works
Mental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2016) · JMIR Mental Health · doi:10.2196/mental.4984 · Cited by 886 · see citing works
Griffiths, F., Lindenmeyer, A., Powell, J., Lowe, P., & Thorogood, M. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e10 · Cited by 820 · cited 1× · see citing works
“A systematic literature review showed that especially people who felt stigmatized by their problem or ashamed of it (eg, encopresis and eating disorders) use electronic mental health (e–mental health) [8].”
BackgroundMental Health Mobile Apps for Preadolescents and Adolescents: A Systematic Review.
Grist, R., Porter, J., & Stallard, P. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7332 · Cited by 518 · see citing works
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
McMullan, R. D., Berle, D., Arnáez, S., & Starcevic, V. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.11.037 · Cited by 436 · see citing works
Baig, M. M., GholamHosseini, H., Moqeem, A. A., Mirza, F., & Lindén, M. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0760-1 · Cited by 351 · see citing works
Mental health: there’s an app for that
Anthes, E. (2016) · Nature · doi:10.1038/532020a · Cited by 333 · see citing works
Muse, K., McManus, F., Leung, C., Meghreblian, B., & Williams, J. M. G. (2012) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2011.11.005 · Cited by 303 · see citing works
Van Ameringen, M., Turna, J., Khalesi, Z., Pullia, K., & Patterson, B. (2017) · Depression and Anxiety · doi:10.1002/da.22657 · Cited by 234 · cited 3× · see citing works
“There are countless health apps that assess subjective data, such as apps used for assessments (eg, Web-based questionnaires) or tracking (eg, monitoring mood or medication via diaries) [23].”
BackgroundMethodTheme issue on e-mental health: A growing field in internet research
Riper, H., Andersson, G., Christensen, H., Cuijpers, P., Lange, A., & Eysenbach, G. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1713 · Cited by 200 · see citing works
Artificial Intelligence-Assisted Online Social Therapy for Youth Mental Health
D'Alfonso, S., Santesteban-Echarri, O., Rice, S., Wadley, G., Lederman, R., Miles, C., Gleeson, J., & Alvarez-Jimenez, M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00796 · Cited by 192 · cited 1× · see citing works
“As young people use new media every day (97% daily internet consumption), and mental health problems at this age are usually experienced as stigmatizing and shameful, the youth are considered particularly accessible to health apps [10].”
BackgroundUsing Smartphones to Monitor Bipolar Disorder Symptoms: A Pilot Study
Beiwinkel, T., Kindermann, S., Maier, A., Kerl, C., Moock, J., Barbian, G., & Rössler, W. (2016) · JMIR Mental Health · doi:10.2196/mental.4560 · Cited by 191 · see citing works
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182 · see citing works
Dogan, E., Sander, C., Wagner, X., Hegerl, U., & Kohls, E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7006 · Cited by 177 · see citing works
Overdiagnosis of mental disorders in children and adolescents (in developed countries)
Merten, E. C., Cwik, J. C., Margraf, J., & Schneider, S. (2017) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-016-0140-5 · Cited by 159 · see citing works
The Ethical Use of Mobile Health Technology in Clinical Psychiatry
Torous, J. B., & Roberts, L. W. (2016) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000596 · Cited by 147 · see citing works
Fann, J. R., Berry, D. L., Wolpin, S. E., Austin‐Seymour, M., Bush, N. E., Halpenny, B., Lober, W. B., & McCorkle, R. C. (2008) · Psycho-Oncology · doi:10.1002/pon.1368 · Cited by 142 · see citing works
Understanding the quality, effectiveness and attributes of top-rated smartphone health apps
Wisniewski, H., Liu, G., Henson, P., Vaidyam, A., Hajratalli, N. K., Onnela, J.-P., & Torous, J. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2018-300069 · Cited by 112 · see citing works
Computerization and the future of primary care: A survey of general practitioners in the UK
Blease, C. R., Bernstein, M. H., Gaab, J., Kaptchuk, T. J., Kossowsky, J., Mandl, K. D., Davis, R. B., & DesRoches, C. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207418 · Cited by 92 · see citing works
Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis
Millenson, M. L., Baldwin, J. L., Zipperer, L., & Singh, H. (2018) · Diagnosis · doi:10.1515/dx-2018-0009 · Cited by 91 · see citing works
Behavioral Screening Measures Delivered With a Smartphone App
Bush, N. E., Skopp, N. A., Smolenski, D. J., Crumpton, R. M., & Fairall, J. M. (2013) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000039 · Cited by 83 · see citing works
Smartphone Applications for Mental Health
Radovic, A., Vona, P. L., Santostefano, A. M., Ciaravino, S., Miller, E., & Stein, B. D. (2016) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2015.0619 · Cited by 78 · see citing works
Self-triage for acute primary care via a smartphone application: Practical, safe and efficient?
Verzantvoort, N. C. M., Teunis, T., Verheij, T. J., & van der Velden, A. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199284 · Cited by 77 · see citing works
Predicting Mood Changes in Bipolar Disorder Through Heartbeat Nonlinear Dynamics
Valenza, G., Nardelli, M., Lanatà, A., Gentili, C., Bertschy, G., Kosel, M., & Scilingo, E. P. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2554546 · Cited by 69 · see citing works
Security Recommendations for mHealth Apps: Elaboration of a Developer’s Guide
Morera, E. P., de la Torre Díez, I., Garcia-Zapirain, B., López-Coronado, M., & Arambarri, J. (2016) · Journal of Medical Systems · doi:10.1007/s10916-016-0513-6 · Cited by 66 · see citing works
Hung, S., Li, M.-S., Chen, Y.-L., Chiang, J.-H., Chen, Y.-Y., & Hung, G. C.-L. (2016) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2016.08.003 · Cited by 46 · see citing works
The apps attempting to transfer NHS 111 online
Armstrong, S. (2018) · BMJ · doi:10.1136/bmj.k156 · Cited by 36 · see citing works
Rosis, S. D., & Nuti, S. (2017) · The International Journal of Health Planning and Management · doi:10.1002/hpm.2443 · Cited by 34 · see citing works
Albrecht, U., Afshar, K., Illiger, K., Becker, S., Hartz, T., Breil, B., Wichelhaus, D. P., & von Jan, U. (2017) · Digital Health · doi:10.1177/2055207617695135 · Cited by 33 · see citing works
Albrecht, U., Hillebrand, U. C., & von Jan, U. (2018) · JMIR mhealth and uhealth · doi:10.2196/10394 · Cited by 33 · see citing works
Mobile App Tools for Identifying and Managing Mental Health Disorders in Primary Care
Magee, J. C., Adut, S., Brazill, K., & Warnick, S. (2018) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-018-0154-0 · Cited by 25 · see citing works
E-Health in Diagnostik und Therapie psychischer Störungen
Lüttke, S., Hautzinger, M., & Fuhr, K. (2018) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-017-2684-9 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [7]Löcherer R Apolináro-Hagen J Efficacy and acceptance of webbased programs for self-help for facilitation of psycological health and handling of stress during academic studies 2019-02-11 Wirksamkeit und Akzeptanz von webbasierten Selbsthilfeprogrammen zur Förderung psychischer Gesundheit und zur Stressbewältigung im Studium: Ein Scoping-Review der aktuellen Forschungsliteratur. [Efficacy and acceptance of webbased programs for self-help for facilitation of psycological health and handling of stress during academic studies.] https://www.e-beratungsjournal.net/ausgabe_0117/Loecherer_Apolinario-Hagen.pdf
- [9]Döpfner M Schürmann S Lehmkuhl G Wackelpeter und Trotzkopf. Hilfen für Eltern bei ADHS-Symptomen, hyperkinetischem und oppositionellem Verhalten. Mit Online-Materialien. 4., überarb. Aufl. [Jello-Pete and stubbornness: Help for Parents with ADHS-symptoms, hyperkinetic and opposing behavior. With online materials. 4th, revised edition.] 2011 Weinheim, Basel Beltz
- [35]Hoffmann H ITU: Committed to connecting the world 2019-05-10 Ada Health Our Approach to Assess Ada’s Diagnostic Performance. https://www.itu.int/en/ITU-T/Workshops-and-Seminars/20180925/Documents/3_Henry%20Hoffmann.pdf
- [39]Freyberger H Dilling H Fallbuch Psychiatrie: Kasuistiken zum Kapitel V(F) der ICD-10. 2 Auflage. [Case Book Psychiatry: Casuistics on Chapter V(F) of the ICD-10. 2nd edition.] 2014 Bern Huber
- [40]Poustka F van Goor-Lambo G Fallbuch Kinder- und Jugendpsychiatrie. Erfassung und Bewertung belastender Lebensumstände von Kindern nach Kapitel V (F) der ICD-10. [Case Book Child and Adolescent Psychiatry. Recording and evaluation of stressful life circumstances of children according to Chapter V (F) of the ICD-10.] 2008 Göttingen Hogrefe
- [48]Field A Discovering Statistics Using IBM SPSS Statistics. Fourth Edition 2013 Thousand Oaks, California, United States SAGE
- [52]Razzaki S Baker A Perov Y Middleton K Baxter J Mullarkey D Sangar D Taliercio M Butt M Majeed A DoRosario A Mahoney M Johri S arXiv 2018 2019-09-27 A Comparative Study of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosi. https://marketing-assets.babylonhealth.com/press/BabylonJune2018Paper_Version1.4.2.pdf
Cited by
Sorted by citations
Artificially intelligent chatbots in digital mental health interventions: a review
Boucher, E. M., Harake, N. R., Ward, H. E., Stoeckl, S. E., Vargas, J., Minkel, J., Parks, A. C., & Zilca, R. (2021) · Expert Review of Medical Devices · doi:10.1080/17434440.2021.2013200 · Cited by 440 · see citing works
Fahim, Y. A., Hasani, I. W., Kabba, S., & Ragab, W. M. (2025) · European Journal of Medical Research · doi:10.1186/s40001-025-03196-w · Cited by 329 · see citing works
Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 164 · see citing works
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146 · see citing works
Chatbots to Support Young Adults’ Mental Health: An Exploratory Study of Acceptability
Koulouri, T., Macredie, R. D., & Olakitan, D. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3485874 · Cited by 127 · see citing works
Large language models: a survey of their development, capabilities, and applications
Yadagiri, A., & Pakray, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02310-4 · Cited by 135 · see citing works
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99 · see citing works
Security, privacy, and healthcare-related conversational agents: a scoping review
May, R., & Denecke, K. (2021) · Informatics for Health and Social Care · doi:10.1080/17538157.2021.1983578 · Cited by 91 · see citing works
Chung, K., Cho, H. Y., & Park, J. Y. (2020) · JMIR Medical Informatics · doi:10.2196/18607 · Cited by 94 · see citing works
Huq, S. M., Maskeliūnas, R., & Damaševičius, R. (2022) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2022.2146768 · Cited by 85 · see citing works
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74 · see citing works
Vaidyam, A. N., Linggonegoro, D. W., & Torous, J. B. (2020) · The Canadian Journal of Psychiatry · doi:10.1177/0706743720966429 · Cited by 83 · see citing works
Examining the effect of explanation on satisfaction and trust in AI diagnostic systems
Alam, L., & Mueller, S. T. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01542-6 · Cited by 82 · see citing works
Hindelang, M., Sitaru, S., & Zink, A. (2024) · JMIR Medical Informatics · doi:10.2196/56628 · Cited by 79 · see citing works
Evaluation of chatbot-delivered interventions for self-management of depression: Content analysis.
Martinengo, L., Lum, E., & Car, J. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.09.028 · Cited by 67 · see citing works
Artificial intelligence and machine learning in mobile apps for mental health: A scoping review
Milne-Ives, M., Selby, E., Inkster, B., Lam, C., & Meinert, E. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000079 · Cited by 68 · see citing works
van Bussel, M. J. P., Odekerken–Schröder, G. J., Ou, C., Swart, R. R., & Jacobs, M. J. G. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08189-7 · Cited by 60 · see citing works
“The self-diagnosis health app Ada was an efficient diagnostic screening or help function for mental disorders in adulthood with the potential to support diagnosticians [45].”
BackgroundDialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 71 · see citing works
Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy
Graf, M., Knitza, J., Leipe, J., Krusche, M., Welcker, M., Kühn, S., Mücke, J., Hueber, A. J., Hornig, J., Klemm, P., et al. (2022) · Rheumatology International · doi:10.1007/s00296-022-05202-4 · Cited by 59 · see citing works
Validity of Chatbot Use for Mental Health Assessment: Experimental Study
Schick, A., Feine, J., Morana, S., Maedche, A., & Reininghaus, U. (2022) · JMIR mhealth and uhealth · doi:10.2196/28082 · Cited by 71 · see citing works
The Intersection of ChatGPT, Clinical Medicine, and Medical Education
Wong, R. S.-Y., Ming, L. C., & Ali, R. A. R. (2023) · JMIR Medical Education · doi:10.2196/47274 · Cited by 60 · see citing works
Ćirković, A. (2020) · Journal of Medical Internet Research · doi:10.2196/18097 · Cited by 52 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
Jui, S., Deo, R., Barua, P., Devi, A., Soar, J., & Acharya, U. R. (2023) · IEEE Access · doi:10.1109/access.2023.3294473 · Cited by 45 · see citing works
Trends & Methods in Chatbot Evaluation
Casas, J., Tricot, M.-O., Khaled, O. A., Mugellini, E., & Cudré-Mauroux, P. (2020) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3395035.3425319 · Cited by 47 · see citing works
Chow, J. C. L., & Li, K. (2025) · JMIR Cancer · doi:10.2196/66633 · Cited by 38 · see citing works
Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A. (2022) · JMIR Human Factors · doi:10.2196/35219 · Cited by 37 · see citing works
El-Osta, A. A. R., Webber, I., Alaa, A., Bagkeris, E., Mian, S., Sharabiani, M. T. A., & Majeed, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053566 · Cited by 30 · see citing works
Knitza, J., Tascilar, K., Gruber, E., Kaletta, H., Hagen, M., Liphardt, A.-M., Schenker, H., Krusche, M., Wacker, J., Kleyer, A., et al. (2021) · Arthritis Research & Therapy · doi:10.1186/s13075-021-02616-6 · Cited by 38 · see citing works
Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31 · see citing works
“A recent pilot study by Jungmann et al [39] investigated the performance and dependency on expert knowledge of the symptom checker ADA in diagnosing mental disorders in adults and adolescents.”
BackgroundMethodResultInfluentialSelf-citationChallenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review
Ilicki, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279636 · Cited by 33 · see citing works
Hi, Can I Help? Exploring How to Design a Mental Health Chatbot for Youths
Høiland, C. G., Følstad, A., & Karahasanović, A. (2020) · Human Technology · doi:10.17011/ht/urn.202008245640 · Cited by 32 · see citing works
Evaluation Metrics for Health Chatbots: A Delphi Study
Denecke, K., Abd-Alrazaq, A. A., Househ, M. S., & Warren, J. (2021) · Methods of Information in Medicine · doi:10.1055/s-0041-1736664 · Cited by 32 · see citing works
Damij, N., & Bhattacharya, S. D. (2022) · IEEE Technology and Engineering Management Conference (TEMSCON) · doi:10.1109/temsconeurope54743.2022.9801962 · Cited by 34 · see citing works
Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · Journal of Medical Internet Research · doi:10.2196/37408 · Cited by 30 · see citing works
Telepsychiatry in an Era of Digital Mental Health Startups
Achtyes, E. D., Glenn, T., Monteith, S., Geddes, J. R., Whybrow, P. C., Martini, J., & Bauer, M. (2023) · Current Psychiatry Reports · doi:10.1007/s11920-023-01425-9 · Cited by 28 · see citing works
Use of artificial intelligence in sports medicine: a report of 5 fictional cases
Rigamonti, L., Estel, K., Gehlen, T., Wolfarth, B., Lawrence, J. B., & Back, D. A. (2021) · BMC Sports Science Medicine and Rehabilitation · doi:10.1186/s13102-021-00243-x · Cited by 26 · see citing works
Snoozy: A Chatbot-Based Sleep Diary for Children Aged Eight to Twelve
Aarts, T., Markopoulos, P., Giling, L., Văcărețu, T., & Pillen, S. (2022) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3501712.3529718 · Cited by 26 · see citing works
Knitza, J., Taşçılar, K., Fuchs, F., Mohn, J., Kühn, S., Bohr, D., Muehlensiepen, F., Bergmann, C., Labinsky, H., Morf, H., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55542 · Cited by 24 · see citing works
Mehl, A., Bergey, F., Cawley, C., & Gilsdorf, A. (2020) · JMIR mhealth and uhealth · doi:10.2196/21364 · Cited by 24 · see citing works
Faqar-Uz-Zaman, S. F., Anantharajah, L., Baumartz, P., Sobotta, P., Filmann, N., Zmuc, D., Wagner, M., Detemble, C., Sliwinski, S., Marschall, U., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005614 · Cited by 25 · see citing works
Desenvolvimento de uma tecnologia digital de apoio à terapia cognitivo-comportamental
do Pinho, J. C. G., Ferreira, A., & Alves, A. G. (2023) · Revista Psicologia Diversidade e Saúde · doi:10.17267/2317-3394rpds.2023.e4511 · Cited by 19 · see citing works
Linardon, J. (2025) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-025-00344-1 · Cited by 20 · see citing works
An In-Depth Exploration of AI and Humanoid Robotics' Role in Contemporary Healthcare
Barua, D. R. (2024) · Advances in medical technologies and clinical practice book series · doi:10.4018/979-8-3693-2238-3.ch003 · Cited by 20 · see citing works
Artificial Intelligence in Health Promotion and Disease Reduction: Rapid Review
Yousefi, F., Naye, F., Ouellet, S., Yameogo, A. R., Sasseville, M., Bergeron, F., Ozkan, M., Cousineau, M., Amil, S., Rhéaume, C., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/70381 · Cited by 26 · see citing works
AI-enabled clinical decision support tools for mental healthcare: A product review
Kleine, A., Kokje, E., Hummelsberger, P., Lermer, E., Schaffernak, I., & Gaube, S. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103052 · Cited by 19 · see citing works
Knevel, R., Knitza, J., Hensvold, A. H., Cîrciumaru, A., Bruce, T., Evans, S., Maarseveen, T. D., Maurits, M. P., de Voorde, L. J. J. B.-V., Simón, D., et al. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.774945 · Cited by 16 · see citing works
Empowering Preventive Care with GECA Chatbot
Maia, E., Vieira, P. N. D. A., & Praça, I. (2023) · Healthcare · doi:10.3390/healthcare11182532 · Cited by 18 · see citing works
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2024) · JMIR AI · doi:10.2196/46875 · Cited by 15 · see citing works
Albrink, K., Joos, C., Schröder, D., Müller, F., Hummers, E., & Noack, E. M. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01928-0 · Cited by 14 · see citing works
Künstliche Intelligenz in der Orthopädie und Unfallchirurgie
der DGOU, A. D., Tjardes, T., Heller, R. A., Pförringer, D., Lohmann, R., & Back, D. A. (2020) · Der Chirurg · doi:10.1007/s00104-019-01091-9 · Cited by 14 · see citing works
Vijayarani, M., & Balamurugan, G. (2019) · Indian Journal of Psychiatric Nursing · doi:10.4103/iopn.iopn_34_19 · Cited by 13 · see citing works
Nelson, J., Kaplan, J., Simerly, G., Nutter, N., Edson-Heussi, A., Woodham, B., & Broman‐Fulks, J. J. (2025) · Current Psychology · doi:10.1007/s12144-025-07320-1 · Cited by 17 · see citing works
Novel Blended Learning on Artificial Intelligence for Medical Students: Qualitative Interview Study
Oftring, Z. S., Deutsch, K., Tolks, D., Jungmann, F., & Kühn, S. (2025) · JMIR Medical Education · doi:10.2196/65220 · Cited by 12 · see citing works
Self-citationKuhail, M. A., Al-Shamaileh, O., Farooq, S., Shahin, H., Abdelzaher, F., & Thomas, J. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/9942295 · Cited by 11 · see citing works
A conversational agent framework for mental health screening: design, implementation, and usability
Boian, R., Bucur, A.-M., Todea, D., Luca, A., Rebedea, T. E., & Podina, I. R. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2332934 · Cited by 11 · see citing works
Characteristics of Acute Childhood Illness Apps for Parents: Environmental Scan
Benoit, J. R. A., Hartling, L., Chan, M. C., & Scott, S. D. (2021) · Journal of Medical Internet Research · doi:10.2196/29441 · Cited by 12 · see citing works
Orji, J., Chan, G., Ndulue, C., & Orji, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-58226-4_18 · Cited by 10 · see citing works
Faqar-Uz-Zaman, S. F., Filmann, N., Mahkovic, D., Wagner, M., Detemble, C., Kippke, U., Marschall, U., Anantharajah, L., Baumartz, P., Sobotta, P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-041396 · Cited by 10 · see citing works
Shania, M., Handayani, P. W., & Asih, S. R. (2023) · JMIR Formative Research · doi:10.2196/48913 · Cited by 15 · see citing works
The Evolving Role of Artificial Intelligence in Gastrointestinal Histopathology: An Update
Codipilly, D. C., Faghani, S., Hagan, C., Lewis, J. S., Erickson, B. J., & Iyer, P. G. (2023) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2023.11.044 · Cited by 10 · see citing works
Meczner, A., Cohen, N., Qureshi, A., Reza, M., Sutaria, S., Blount, E., Bagyura, Z., & Malak, T. (2024) · JMIR Formative Research · doi:10.2196/49907 · Cited by 10 · see citing works
Peven, K., Wickham, A. P., Wilks, O., Kaplan, Y. C., Marhol, A., Ahmed, S., Bamford, R., Cunningham, A. C., Prentice, C., Meczner, A., et al. (2023) · JMIR mHealth and uHealth · doi:10.2196/46718 · Cited by 9 · see citing works
Contrasting rule and machine learning based digital self triage systems in the USA
Naved, B. A., & Luo, Y. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01367-3 · Cited by 9 · see citing works
Vague Language in Online Medical Consultation
He, L., & Smit, E. S. (2021) · European Journal of Health Communication · doi:10.47368/ejhc.2021.001 · Cited by 9 · see citing works
Can I Trust This Chatbot? Assessing User Privacy in AI ‐Healthcare Chatbot Applications
Yener, R., Chen, G., Gumusel, E., & Bashir, M. N. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1299 · Cited by 7 · see citing works
Nasrallah, M., Moghrabi, A. E., Kayal, M., Matar, D., Alwan, I., & Fakhouy, M. (2025) · International Journal of Developmental Disabilities · doi:10.1080/20473869.2025.2456800 · Cited by 7 · see citing works
Herausforderungen der Digitalisierung in der Traumaversorgung
Trentzsch, H., Osterhoff, G., Heller, R. A., Nienaber, U., Lazarovici, M., Digitalisierung, A., (DGOU), D. D. G. F. O. U. U., & Sektion Notfall‑, I. U. S. (. D. D. G. F. U. (. (2020) · Der Unfallchirurg · doi:10.1007/s00113-020-00859-7 · Cited by 6 · see citing works
Potential and pitfalls of conversational agents in health care
Denecke, K. (2023) · Nature Reviews Disease Primers · doi:10.1038/s41572-023-00482-x · Cited by 6 · see citing works
Ciesla, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-51004-5_5 · Cited by 6 · see citing works
Ökonomische Aspekte der Digitalisierung in Orthopädie und Unfallchirurgie
Pförringer, D., Matusiewicz, D., Tsitsilonis, S., Gehlen, T., & Back, D. A. (2020) · Der Unfallchirurg · doi:10.1007/s00113-020-00891-7 · Cited by 6 · see citing works
Vignapiano, A., Monaco, F., Landi, S., Steardo, L., Mancuso, C., Pagano, C., Petrillo, G., Marenna, A., Piacente, M., Leo, S., et al. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1512818 · Cited by 5 · see citing works
Avey: An Accurate AI Algorithm for Self-DiagnosisPreprint
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2022) · medRxiv · doi:10.1101/2022.03.08.22272076 · Cited by 5 · see citing works
Triage-Bot: An Assistive Triage Framework
Sharma, D., Rashno, E., Zulkernine, F. H., ElKhodary, E., Beninger, M., Almeida, R., Tao, J., Alaca, F., & Elgazzar, K. (2024) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00033 · Cited by 4 · see citing works
Evaluating digital triage symptom checker with historical triage-related adverse events
Ilicki, J., Edman, S., Stalfors, J., & Molin, C. J. (2025) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2025.2563517 · Cited by 4 · see citing works
Orji, J., Chan, G., & Orji, R. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2494278 · Cited by 3 · see citing works
Adhantoro, M. S., Gunawan, D., Prayitno, H. J., Riyanti, R. F., Purnomo, E., & Jufriansah, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1446590 · Cited by 9 · see citing works
Crowdsourcing smartphone data for biomedical research: Ethical and legal questions
Lang, M., McKibbin, K., Shabani, M., Borry, P., Gautrais, V., Verbeke, K., & Zawati, M. H. (2023) · Digital Health · doi:10.1177/20552076231204428 · Cited by 4 · see citing works
Evaluating privacy policies of AI-powered mHealth iOS applications
Javed, Y., & Bhojanam, S. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf130 · Cited by 3 · see citing works
Abbas, S. A., Yusifzada, I., & Athar, S. (2025) · Cureus · doi:10.7759/cureus.80935 · Cited by 4 · see citing works
Smartphone-based devices for healthcare monitoring and management
Konnaiyan, K. R., & Zhdanov, A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30300-5.00019-1 · Cited by 2 · see citing works
Orji, J., Chan, G., & Orji, R. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103600 · Cited by 2 · see citing works
Depiction of conversational agents as health professionals: a scoping review
MacNeill, A. L., MacNeill, L., Yi, S., Goudreau, A., Luke, A., & Doucet, S. A. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00029 · Cited by 2 · see citing works
Detecting Depressive Symptoms: Activity Tracking and Chatbot Intervention
Shaker, R., Ibrahim, N., & Abdennadher, S. (2024) · SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health · doi:10.1109/segah61285.2024.10639580 · Cited by 1 · see citing works
Lin, Y.-C., Chen, Y.-M., Chuang, C. F., Yuan, C. W., Bi, N., Chang, H., Huang, M., & You, C.-W. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720128 · Cited by 1 · see citing works
Adla, Y. A., & Hajj, A. E. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03694-9 · Cited by 1 · see citing works
Technical Innovations in Pneumology: E-Health, Screening, Diagnostics, and Therapy
Herkenrath, S. D., Dreher, M., Gompelmann, D., Held, M., Koczulla, A. R., Köhnlein, T., Rohde, G. G. U., Wälscher, J., Watz, H., & Randerath, W. J. (2021) · Respiration · doi:10.1159/000516335 · Cited by 1 · see citing works
Eeles, E. M., Worthy, P., Teodorczuk, A., Kaur, P., & Dissanayaka, N. N. (2022) · Delirium · doi:10.56392/001c.37365 · Cited by 1 · see citing works
Randerath, W. J., Dreher, M., Gompelmann, D., Held, M., Koczulla, A. R., Köhnlein, T., Rohde, G. G. U., Wälscher, J., Watz, H., & Steinkamp, G. (2020) · Pneumologie · doi:10.1055/a-1186-7333 · Cited by 1 · see citing works
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · doi:10.2196/preprints.42481 · Cited by 1 · see citing works
Knevel, R., Knitza, J., Hensvold, A. H., Cîrciumaru, A., Bruce, T., Evans, S., Maarseveen, T. D., Maurits, M. P., de Voorde, L. J. J. B.-V., Simón, D., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-386435/v1
Research on the Application of Deep Learning Technology in Intelligent Dialogue Robots
Zeng, X., Liang, D., Liang, Z., Chen, G., & Li, Y. (2020) · International Conference on Information Technology and Computer Application (ITCA) · doi:10.1109/itca52113.2020.00031
van Bussel, M. V., Odekerken-Schröder, G., Ou, C., Swart, R., & Jacobs, M. (2022) · doi:10.2196/preprints.36535
Regli, S. F., Gashi, F., & Denecke, K. (2021) · doi:10.2196/preprints.34489
Guerrero, D. A. R., Beltrán Saavedra, D. P., & Proaño Navas, M. A. (2022) · doi:10.11144/javeriana.10554.61844
"MindScape Jr: Cognitive well-being analysis system for young minds"
Pawar, V. E., Kadam, S. S., Joshi, A., Sawant, C., Bhoir, A., & Patil, L. (2023) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit59222.2023.10205923
Freyer, O., Bergey, F., Cotte, F., Gilbert, S., & Mehl, A. (2023) · doi:10.2196/preprints.52486
- Authors
- Jungmann, S. M., Klan, T., Kuhn, S., & Jungmann, F.
- Affiliations
- Johannes Gutenberg University Mainz (DE)
- Corresponding author
- Jungmann, Stefanie Maria
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-10-29
- Volume
- 3
- Issue
- 4
- Pages
- e13863
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC6914276
- OpenAlex
- W2982702361
- Cited by
- 127
- References
- 35
- Citations by year
- 2026: 10 · 2025: 20 · 2024: 16 · 2023: 18 · 2022: 24 · 2021: 25 · 2020: 13 · 2019: 1
- FWCI
- 17.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Gutenberg Open Science (publishedVersion, cc-by)
- Gutenberg Open Science (cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- chatbots, psychiatric diagnosis, diagnostic accuracy, health apps, symptom-based screening, inter-rater reliability, psychotherapists, case vignettes, adult mental disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
