The DOI matches its Crossref record; 28 of the 30 references it cites were found in a registry or an index.
How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs
Open accessPublished version of a preprint
Abstract
Objectives: To compare breadth of condition coverage, accuracy of suggested conditions and appropriateness of urgency advice of eight popular symptom assessment apps. Design: Vignettes study. Setting: 200 primary care vignettes. Intervention/comparator: For eight apps and seven general practitioners (GPs): breadth of coverage and condition-suggestion and urgency advice accuracy measured against the vignettes’ gold-standard. Primary outcome measures: (1) Proportion of conditions ‘covered’ by an app, that is, not excluded because the user was too young/old or pregnant, or not modelled; (2) proportion of vignettes with the correct primary diagnosis among the top 3 conditions suggested; (3) proportion of ‘safe’ urgency advice (ie, at gold standard level, more conservative, or no more than one level less conservative). Results: Condition-suggestion coverage was highly variable, with some apps not offering a suggestion for many users: in alphabetical order, Ada: 99.0%; Babylon: 51.5%; Buoy: 88.5%; K Health: 74.5%; Mediktor: 80.5%; Symptomate: 61.5%; Your.MD: 64.5%; WebMD: 93.0%. Top-3 suggestion accuracy was GPs (average): 82.1%±5.2%; Ada: 70.5%; Babylon: 32.0%; Buoy: 43.0%; K Health: 36.0%; Mediktor: 36.0%; Symptomate: 27.5%; WebMD: 35.5%; Your.MD: 23.5%. Some apps excluded certain user demographics or conditions and their performance was generally greater with the exclusion of corresponding vignettes. For safe urgency advice, tested GPs had an average of 97.0%±2.5%. For the vignettes with advice provided, only three apps had safety performance within 1 SD of the GPs—Ada: 97.0%; Babylon: 95.1%; Symptomate: 97.8%. One app had a safety performance within 2 SDs of GPs—Your.MD: 92.6%. Three apps had a safety performance outside 2 SDs of GPs—Buoy: 80.0% (p<0.001); K Health: 81.3% (p<0.001); Mediktor: 87.3% (p=1.3×10 -3 ). Conclusions: The utility of digital symptom assessment apps relies on coverage, accuracy and safety. While no digital tool outperformed GPs, some came close, and the nature of iterative improvements to software offers scalable improvements to care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On this matter, evidence indicates that the use of algorithms over other methods, the inclusion of demographic information [57], or more rigorous questioning [35] could explain the differences between symptom checkers’ diagnostic performances.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As previous studies have shown considerable differences between symptom checkers’ diagnostic accuracy [33,35], future studies could compare various symptom checkers for the formal diagnosis of mental disorders.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476
Controlling the false discovery rate in behavior genetics research
Benjamini, Y., Drai, D., Elmer, G., Kafkafi, N., & Golani, I. (2001) · Behavioural Brain Research · doi:10.1016/s0166-4328(01)00297-2 · Cited by 4,146
COVID-19: what is next for public health?
Heymann, D. L., & Shindo, N. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30374-3 · Cited by 1,146
Video consultations for covid-19
Greenhalgh, T., Wherton, J., Shaw, S., & Morrison, C. (2020) · BMJ · doi:10.1136/bmj.m998 · Cited by 866 · cited 3×
“Other situations, such as the inability to search for certain symptoms, providing no condition-suggestions/urgency advice for certain input symptoms, or, excluding comorbidities, mental health or pregnancy are more problematic and can raise concerns about the safety and benefits of the app for…”
InfluentialEvans, S. C., Roberts, M. C., Keeley, J. W., Blossom, J. B., Amaro, C. M., Garcia, A. M., Stough, C. O., Canter, K. S., Robles, R., & Reed, G. M. (2015) · International Journal of Clinical and Health Psychology · doi:10.1016/j.ijchp.2014.12.001 · Cited by 757
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489
Clinical Vignette-Based Surveys: A Tool for Assessing Physician Practice Variation
Veloski, J., Tai, S., Evans, A., & Nash, D. (2005) · American Journal of Medical Quality · doi:10.1177/1062860605274520 · Cited by 396
Performance of Four Computer-Based Diagnostic Systems
Berner, E. S., Webster, G. D., Shugerman, A. A., Jackson, J. R., Algina, J. J., Baker, A. L., Ball, E. V., Cobbs, C. G., Dennis, V. W., Frenkel, E. P., et al. (1994) · New England Journal of Medicine · doi:10.1056/nejm199406233302506 · Cited by 320
Fisher’s exact approach for post hoc analysis of a chi-squared test
Shan, G., & Gerstenberger, S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0188709 · Cited by 234 · cited 1×
“MD and for this reason a subanalysis of the 150 vignettes for which full verification was possible for these apps is provided in online supplemental table 6 and 7.”
Research works printed but not linked: 1
- 10.2307/2276774 · DOI · Paper page
Cited by
Sorted by citations
The promise of digital healthcare technologies
Yeung, A. W. K., Torkamani, A., Butte, A. J., Glicksberg, B. S., Schuller, B. W., Rodríguez, B., Ting, D. S. W., Bates, D. W., Schaden, E., Peng, H., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1196596 · Cited by 284
“On the other hand, GPs and healthcare professionals need to learn computer science skills to master AI/ML programs and fully utilize their potentials.”
InfluentialThe Advent of Generative Language Models in Medical Education
Karabacak, M., Ozkara, B. B., Margetis, K., Wintermark, M., & Bisdas, S. (2023) · JMIR Medical Education · doi:10.2196/48163 · Cited by 208
“Moreover, GLMs’ enhanced comprehension of complex medical terminology and context might enable AI-powered health companions such as Ada Health to provide more precise diagnostic suggestions and individualized health advice to both clinicians and patients [9].”
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 213
“[1,16,18,19,21–24]”
BackgroundMethodResultInfluentialFraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128
“As has been seen with the use of search engines such as Google [36] and existing SCs [7,20], a substantial proportion of the population are likely to use such tools.”
BackgroundMethodResultInfluentialSelf-citationAccuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125
“[15]) and one published at the end of last year [16] examined multiple presentations and conditions; the others focused on single condition studies such as those examining system performance for cervical myelopathy, inflammatory arthritis, HIV / Hepatitis C and ophthalmic conditions [17–20].”
BackgroundResultDavis, R., Eppler, M., Ayo-Ajibola, O., Loh-Doyle, J. C., Nabhani, J., Samplaski, M., Gill, I., & Cacciamani, G. E. (2023) · Journal of Urology · doi:10.1097/ju.0000000000003615 · Cited by 119
“to assess ChatGPT output quality drawing from Quality Evaluation Scoring Tool (QUEST) and DISCERN, 2 validated tools that assess quality of web-based information,(15,16) and from Gilbert et al.(17)”
Miller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117
“In a preprint from our group (currently undergoing peer review), we evaluated the performance of 8 popular symptom checkers against one another and 7 human GP raters, as well as a gold-standard diagnostic suggestion using 200 clinical vignettes [31].”
MethodSelf-citationAdvancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 121
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99
“Although other relevant studies competitively comparing symptom checkers’ accuracy have been published since 2015 [24,32,33], their chosen methodology (eg, sampling of apps and definition of triage levels) hinders a direct comparison with the data from the study by Semigran et al [23].”
BackgroundResultTriage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review
Sasco, E. R., El-Osta, A. A. R., Alaa, A., Webber, I., Karki, M., Asmar, M. L. E., Purohit, K., Painter, A., & Hayhoe, B. W. J. (2023) · Journal of Medical Internet Research · doi:10.2196/43803 · Cited by 85
“Gilbert et al [17] c Attributed to the vignette • GPs as additional comparator • Semigran et al [15] c”
InfluentialEvaluation of chatbot-delivered interventions for self-management of depression: Content analysis.
Martinengo, L., Lum, E., & Car, J. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.09.028 · Cited by 67
Knitza, J., Mohn, J., Bergmann, C., Kampylafka, E. I., Hagen, M., Bohr, D., Morf, H., Araujo, E. G., Englbrecht, M., Simón, D., et al. (2021) · Arthritis Research & Therapy · doi:10.1186/s13075-021-02498-8 · Cited by 65
“Ada covers a great variety of different conditions [19] and is not limited to IRDs, whereas Rheport is exclusively meant for the triage of new suspected IRD patients.”
BackgroundResultDialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 73
“MDw [21].”
Aboueid, S., Meyer, S. B., Wallace, J. R., Mahajan, S., & Chaurasia, A. K. (2020) · JMIR Public Health and Surveillance · doi:10.2196/22637 · Cited by 62
“The scarce literature on symptom checker accuracy suggests that the quality of diagnostic and triage advice differs based on the digital platform used [12] with those enabled by artificial intelligence having a higher percentage of listing the correct diagnosis first [13].”
MethodComparison of physician and artificial intelligence-based symptom checker diagnostic accuracy
Graf, M., Knitza, J., Leipe, J., Krusche, M., Welcker, M., Kühn, S., Mücke, J., Hueber, A. J., Hornig, J., Klemm, P., et al. (2022) · Rheumatology International · doi:10.1007/s00296-022-05202-4 · Cited by 59
“1 3 diagnostic accuracy (71%), although being inferior to GP diagnostic accuracy (82%) [15].”
BackgroundMartinengo, L., Jabir, A. I., Goh, W. W. T., Lo, N. Y. W., Ho, M. R., Kowatsch, T., Atun, R. A., Michie, S., & Car, L. T. (2022) · Journal of Medical Internet Research · doi:10.2196/39243 · Cited by 54
“…needs, such as providing timely information [5], supporting mental health disorder management [6,7], assisting with triage in clinical settings [8,9], supporting chronic disease self-management, or delivering lifestyle change interventions, such as physical activity [10] and dietary changes,…”
Harada, Y., Katsukura, S., Kawamura, R., & Shimizu, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18042086 · Cited by 44
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 57
“%) studies analyzed the self-triage accuracy of SAAs 2,4,7,15,16,20,29,33 – 39 , four (21”
InfluentialCotte, F., Mueller, T., Gilbert, S., Blümke, B., Multmeier, J., Hirsch, M. C., Wicks, P. J., Wolanski, J., Tutschkow, D., Schade‐Brittinger, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/32340 · Cited by 41
“Although vignette studies testing SAAs have been conducted [21-24], not many studies have explored them in a prospective ED setting, which is another area of interest for such apps besides the at-home setting.”
BackgroundSelf-citationMahlknecht, A., Engl, A., Piccoliori, G., & Wiedermann, C. J. (2023) · BMC Primary Care · doi:10.1186/s12875-023-02143-0 · Cited by 41
“…of different symptom checkers with physicians’ appraisals and showed that GPs performed consistently better than the evaluated symptom checkers on diagnostic accuracy, while the triage advice (appropriateness and safety of urgency) of some apps was nearly comparable to that of the GPs [24].”
ResultKnitza, J., Muehlensiepen, F., Ignatyev, Y., Fuchs, F., Mohn, J., Simón, D., Kleyer, A., Fagni, F., Boeltz, S., Morf, H., et al. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.844669 · Cited by 38
“Rheport for example is an online self-referral tool, designed to optimize rheumatology referrals, being based on a fixed questionnaire (9, 10), whereas Ada is an artificial intelligence (AI) and chatbot-based symptom checker coveringmultiple diagnoses (8, 10).”
BackgroundMethodInfluentialDelshad, S. D., Dontaraju, V. S., & Chengat, V. (2021) · Cureus · doi:10.7759/cureus.16956 · Cited by 37
“demonstrated that three available AI-based applications provided safe triage advice when compared to general practitioners for 200 clinical vignettes [11,16].”
BackgroundScheder-Bieschin, J., Blümke, B., de Buijzer, E., Cotte, F., Echterdiek, F., Nacsa, J., Ondrésik, M., Ott, M., Paul, G., Schilling, T., et al. (2021) · JMIR Formative Research · doi:10.2196/28199 · Cited by 40
“Its underlying technology has the highest condition-suggestion accuracy among tools of its class and has the same quality of safe urgency advice as general practitioners [11].”
BackgroundSelf-citationEl-Osta, A. A. R., Webber, I., Alaa, A., Bagkeris, E., Mian, S., Sharabiani, M. T. A., & Majeed, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053566 · Cited by 30
“26 However, it remains uncertain if clinical vignettes are ideal to investigate the accuracy and safety of OSC generally.”
InfluentialKnitza, J., Tascilar, K., Gruber, E., Kaletta, H., Hagen, M., Liphardt, A.-M., Schenker, H., Krusche, M., Wacker, J., Kleyer, A., et al. (2021) · Arthritis Research & Therapy · doi:10.1186/s13075-021-02616-6 · Cited by 38
“Previous studies reported favorable results for the Ada symptom checker, suggesting that rare diseases could be diagnosed earlier [18, 19], mental disorders could efficiently be screened [20], and Ada have the best overall diagnostic accuracy and condition coverage [21].”
ResultHennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31
“This may resemble the accuracy of ADA, particularly in detecting somatic medical conditions, which has been the focus of previous studies [34,35].”
BackgroundMethodResultInfluentialChallenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review
Ilicki, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279636 · Cited by 33
“This can be an issue as systems and/or vignettes can have a geographical bias [10], in e.”
BackgroundInfluentialFraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/38364 · Cited by 32
“In addition, the vignettes may be less challenging to diagnose than real patient histories collected in an emergency situation, as illustrated in a recent study that included vignettes created using actual patient presentations to an urgent health care hotline [13].”
BackgroundMethodInfluentialSelf-citationOnline Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · Journal of Medical Internet Research · doi:10.2196/37408 · Cited by 30
“Most published OSC comparative studies use disease-based vignettes authored by clinicians [4,5,11,12,16].”
InfluentialNovel technical and privacy-preserving technology for artificial intelligence in ophthalmology
Lim, J. S., Hong, M., Lam, W. S. T., Zhang, Z., Teo, Z. L., Liu, Y., Ng, W. Y., Foo, L. L., & Ting, D. S. W. (2022) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000000846 · Cited by 31
Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Ben‐Shabat, N., Sharvit, G., Meimis, B., Joya, D. B., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, ., & Amital, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104897 · Cited by 28
“For these reasons most of the studies conducted so far to address diagnostic accuracy have used similar methodology [24 – 27].”
ResultKnitza, J., Taşçılar, K., Fuchs, F., Mohn, J., Kühn, S., Bohr, D., Muehlensiepen, F., Bergmann, C., Labinsky, H., Morf, H., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55542 · Cited by 24
“Multiple case-vignette studies showcased its high diagnostic accuracy [5,12], and more than 30 million symptom assessments have been completed in 130 countries [13].”
Mehl, A., Bergey, F., Cawley, C., & Gilsdorf, A. (2020) · JMIR mhealth and uhealth · doi:10.2196/21364 · Cited by 26
Self-citationFaqar-Uz-Zaman, S. F., Anantharajah, L., Baumartz, P., Sobotta, P., Filmann, N., Zmuc, D., Wagner, M., Detemble, C., Sliwinski, S., Marschall, U., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005614 · Cited by 25
“8,9,19 – 23 Gilbert et al 10 recently evaluated the performance of eight digital symptom assessment apps, showing that Ada achieved a higher accuracy (71”
BackgroundUse of Artificial Intelligence in Clinical Neurology
Hillis, J. M., & Bizzo, B. C. (2022) · Seminars in Neurology · doi:10.1055/s-0041-1742180 · Cited by 21
Kopka, M., Feufel, M. A., Berner, E. S., & Schmieding, M. L. (2023) · Digital Health · doi:10.1177/20552076231194929 · Cited by 23
“This contrasts the current trend in the literature, where studies base their assessment of symptom checkers on a growing quantity of case vignettes, without scrutinizing whether they are suitable for this purpose.(1,9,12,37,38) Another factor to be considered when creating a set of case vignettes is the prevalence of…”
MethodResultTriage Errors in Primary and Pre–Primary Care
Nguyen, H., Meczner, A., Burslam-Dawe, K., & Hayhoe, B. W. J. (2022) · Journal of Medical Internet Research · doi:10.2196/37209 · Cited by 21
“Triage advice provided by symptom checkers is found to be more risk averse than that provided by health care professionals [30,32], with 85% of the users advised to see their doctor in one study [33].”
BackgroundMontazeri, M., Multmeier, J., Novorol, C., Upadhyay, S., Wicks, P. J., & Gilbert, S. (2021) · JMIR Formative Research · doi:10.2196/26402 · Cited by 19
Self-citationGilbert, S., Fenech, M. E., Upadhyay, S., Wicks, P. J., & Novorol, C. (2021) · Australian Journal of Primary Health · doi:10.1071/py21032 · Cited by 17
Self-citationGellert, G. A., Kabat-Karabon, A., Gellert, G. L., Rasławska-Socha, J., Górski, S., Price, T., Kuszczyński, K., Marcjasz, N., Palczewski, M., Jaszczak, J., et al. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1362246 · Cited by 17
“For example, since the analysis conducted by 2019 Gilbert et al., (9) there have been 29 releases of the Symptomate medical content model and 17 updates to core functionality, including new epidemiological models and triage algorithms, each improving accuracy and safety.”
BackgroundMethodInfluentialHarada, Y., Sakamoto, T., Sugimoto, S., & Shimizu, T. (2024) · JMIR Formative Research · doi:10.2196/53985 · Cited by 17
“Concerns have arisen regarding the low diagnostic accuracy of current symptom checker output, which often lags behind that of physicians [12,17,23].”
BackgroundKnevel, R., Knitza, J., Hensvold, A. H., Cîrciumaru, A., Bruce, T., Evans, S., Maarseveen, T. D., Maurits, M. P., de Voorde, L. J. J. B.-V., Simón, D., et al. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.774945 · Cited by 16
“In these studies, the DDSS that were evaluated often use patient-vignettes instead of true data (17, 19).”
MethodWetzel, A.-J., Koch, R., Preiser, C., Mueller, R., Klemmt, M., Ranisch, R., Ehni, H.-J., Wiesing, U., Rieger, M. A., Henking, T., et al. (2022) · JMIR Research Protocols · doi:10.2196/34026 · Cited by 15
“Some SCAs exclude specific user groups for symptom analysis, for example, pregnant women, children, older adults, and patients with specific comorbidities [14].”
BackgroundApp Store Governance: The Implications and Limitations of Duopolistic Dominance
Cowls, J., & Morley, J. (2022) · Digital ethics lab yearbook · doi:10.1007/978-3-031-09846-8_6 · Cited by 15
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2024) · JMIR AI · doi:10.2196/46875 · Cited by 21
“Besides, as in the studies by Gilbert et al [28], Baker et al [38], and Kannan et al [49], for each tested gold-standard vignette, we used recall (or sensitivity in medical parlance) as a measure of the percentage of relevant diseases that are returned in the symptom checker’s or doctor’s…”
BackgroundMethodInfluentialDigital health for allergen immunotherapy
Dramburg, S., Matricardi, P. M., Pfaar, O., & Klimek, L. (2022) · Allergologie select · doi:10.5414/alx02301e · Cited by 13
“While general guidance of users via mobile applications analyzing the clinical history and giving (mostly probabilistic) estimations of a potential pre-diagnosis can guide patients to contact a healthcare professional instead of self-medicating [30], symptom diaries enable the precise and prospective collection of sym…”
BackgroundKnitza, J., Hasanaj, R., Beyer, J., Ganzer, F., Slagman, A. C., Bolanaki, M., Napierala, H., Schmieding, M. L., Al-Zaher, N., Orlemann, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/56514 · Cited by 12
“In total, 2 of the most promising SCs according to recent studies [14-16] available in multiple languages, including German, are Ada and Symptoma.”
Influential“That’s just Future Medicine” - a qualitative study on users’ experiences of symptom checker apps
Müller, R., Klemmt, M., Koch, R., Ehni, H., Henking, T., Langmann, E., Wiesing, U., & Ranisch, R. (2024) · BMC Medical Ethics · doi:10.1186/s12910-024-01011-5 · Cited by 13
“One study by Gilbert et al. compared the extent of disease coverage, the accuracy of the suggested diseases, and the appropriateness of the recommendations from diverse SCAs with general practitioners (GPs) [11].”
BackgroundZawati, M. H., & Lang, M. (2024) · Journal of Medical Internet Research · doi:10.2196/50344 · Cited by 14
“At least 1 study suggests that Ada’s rate of condition suggestion accuracy is consistently just over 70%, as compared with a general practitioner’s accuracy of just over 82% [31].”
BackgroundKopka, M., Napierala, H., Privoznik, M., Sapunova, D., Zhang, S., & Feufel, M. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-83844-z · Cited by 14
North, F., Jensen, T. B., Stroebel, R. J., Nelson, E. M., Johnson, B. J., Thompson, M. C., Pecina, J. L., & Crum, B. A. (2023) · Health Services Research and Managerial Epidemiology · doi:10.1177/23333928231168121 · Cited by 11
“16,21 Vignettes are designed to recreate how a patient might interact with an online symptom checker.”
BackgroundCharacteristics of Acute Childhood Illness Apps for Parents: Environmental Scan
Benoit, J. R. A., Hartling, L., Chan, M. C., & Scott, S. D. (2021) · Journal of Medical Internet Research · doi:10.2196/29441 · Cited by 12
“As one recent investigation into symptom checkers concluded, apps must be assessed on the accuracy of their results in the context of coverage of available conditions and patient populations if these apps do not give results for some subgroups (eg, children) [67].”
Pawelczyk, J., Kraus, M., Voigtlaender, S., Siebenlist, S., & Rupp, M. (2025) · HSS Journal® The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251341321 · Cited by 13
“One product demonstrated 70% diagnostic accuracy for inflammatory rheumatic diseases, outperforming physicians (54%, p = .002) and providing urgency advice on par with general practitioners [18,19].”
Faqar-Uz-Zaman, S. F., Filmann, N., Mahkovic, D., Wagner, M., Detemble, C., Kippke, U., Marschall, U., Anantharajah, L., Baumartz, P., Sobotta, P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-041396 · Cited by 10
Meczner, A., Cohen, N., Qureshi, A., Reza, M., Sutaria, S., Blount, E., Bagyura, Z., & Malak, T. (2024) · JMIR Formative Research · doi:10.2196/49907 · Cited by 10
“One possible reason for the lack of data on intertester variability is that many studies have only used 1 sole tester for each test case [14,22,31,32].”
BackgroundPeven, K., Wickham, A. P., Wilks, O., Kaplan, Y. C., Marhol, A., Ahmed, S., Bamford, R., Cunningham, A. C., Prentice, C., Meczner, A., et al. (2023) · JMIR mHealth and uHealth · doi:10.2196/46718 · Cited by 9
“The multicondition symptom checkers evaluated by Gilbert et al [68] and Schmieding et al [69] were assessed using vignette cases that covered both common and less-common conditions seen in primary care practice, conditions that affect all body systems, and conditions that have a range of urgency…”
BackgroundMethodSelf-citationThe Utility and Limitations of Artificial Intelligence-Powered Chatbots in Healthcare
Hayat, J., Lari, M., Alherz, M., & Lari, A. (2024) · Cureus · doi:10.7759/cureus.73127 · Cited by 11
Technology-supported self-triage decision making
Kopka, M., Wang, S. M., Kunz, S., Schmid, C., & Feufel, M. A. (2025) · npj Health Systems · doi:10.1038/s44401-024-00008-x · Cited by 11
Irwin, P., Rehman, S. U., Fealy, S. M., Kornhaber, R. A., Matheson, A., & Cleary, M. L. (2025) · Contemporary Nurse · doi:10.1080/10376178.2025.2459701 · Cited by 9
“Apps have been developed to assist in both acute and chronic physical and mental health conditions (Gilbert et al., 2020; Sohail, 2023) with accuracy rates of up to 93% and 23.5% reduction in assessment times of mental health clinical cases (Sohail, 2023).”
InfluentialVuillaume, L. A., Turpinier, J., Cipolat, L., Arnaud-Dépil-Duval, Dumontier, T., Peschanski, N., Kieffer, Y., Barbat, B., Riquier, T., Dinot, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0277568 · Cited by 9
Visualisation of Information Using Patient Journey Maps for a Mobile Health Application
Lee, B., Lee, J., Cho, Y., Shin, Y., Oh, C., Park, H., & Kim, H. K. (2023) · Applied Sciences · doi:10.3390/app13106067 · Cited by 9
Schmude, M., Salim, N., Azadzoy, H., Bane, M., Millen, E., O’Donnell, L., Bode, P., Türk, E., Vaidya, R., & Gilbert, S. (2022) · JMIR Research Protocols · doi:10.2196/34298 · Cited by 9
“The accuracy and comprehensiveness of the Ada platform has been validated in several studies of the Ada SAA [31-33].”
InfluentialSelf-citationPreiser, C., Radionova, N., Ög, E., Koch, R., Klemmt, M., Mueller, R., Ranisch, R., Joos, S., & Rieger, M. A. (2024) · JMIR Human Factors · doi:10.2196/57360 · Cited by 8
“Others emphasize techno-dystopian visions of a technology that outperforms or replaces humans [11,83] and becomes an uncontrollable participant in health care systems [16].”
van der Linden, S., Sevastjanova, R., Funk, M., & El‐Assady, M. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3564275 · Cited by 10
Rathbone, A. L., Neill, S., Prime, S. J., Thomas, N., & Everett, K. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105504 · Cited by 8
Liu, V., Kaila, M., & Koskela, T. (2024) · JMIR Human Factors · doi:10.2196/55099 · Cited by 7
“We set the safe symptom assessment based on a previous study, as at most one level of urgency less urgent than the assessment of a nurse, and the performance of the ESC as 97% safe accuracy across all completed assessments [23].”
BackgroundMethodResultInfluentialSähköisten oirearvioiden käytettävyys
Liu, V., Sellgren, L., Kaila, M., & Koskela, T. (2021) · Finnish Journal of eHealth and eWelfare · doi:10.23996/fjhw.97020 · Cited by 7
“Aiemmissa tutkimuksissa on arvioitu sähköisten oirearvioiden diagnostiikan tarkkuutta ja kykyä tehdä hoidon tarpeen arviota (HTA) hyödyntäen kliinisiä tapausvinjettejä, eli potilastapausesimerkkejä [3-5].”
BackgroundInfluentialNoori, S., Khasnavis, S., DeCroce-Movson, E., Blay-Tofey, M., & Vitiello, E. M. (2023) · JMIR Formative Research · doi:10.2196/41573 · Cited by 6
“[11,12].”
Haase, L., Sedlmayr, B., Sedlmayr, M., Monett, D., & Winter, J. (2025) · BMC Veterinary Research · doi:10.1186/s12917-025-04658-3 · Cited by 6
“Symptom checkers like the Ada Health app present themselves as close to the efficiency of general practitioners in evaluating urgency and suggesting diagnoses [9].”
Integrating statistical design and inference: A roadmap for robust and trustworthy medical AI
Wei, Q., Cui, M., Liu, Z., Liu, Z., Feng, G., Li, Y., Christiani, D. C., Li, L., Wang, J., Hao, Y., et al. (2025) · The Innovation Medicine · doi:10.59717/j.xinn-med.2025.100145 · Cited by 6
Promoting xenomorphic patient-facing AIs: The case against anthropomorphism in medical AIs
Milford, S. R., Herger, E., Eichinger, J., & Elger, B. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02046-7 · Cited by 6
“While results are mixed, there are indications that in certain circumstances chatbots have demonstrated diagnostic accuracy akin to human doctors 4 – 6 .”
Shared decision making between patients with Fabry disease and physicians in Japan: An online survey
Inagaki, N., Tsuchiya, M., Otani, K., & Nakayama, T. (2022) · Molecular Genetics and Metabolism Reports · doi:10.1016/j.ymgmr.2022.100899 · Cited by 7
“Recently, digital tools were developed for recording daily symptoms in patients with chronic disease [19]; such practical solutions should also be applied to Fabry disease management to enhance communication between physicians and patients about the QoL-related symptoms that cannot be evaluated or monitored using labo…”
BackgroundGilbert, A., Diep, A. N., Boufraioua, M., Pétré, B., Donneau, A., & Ghuysen, A. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08571-5 · Cited by 6
International dimensions of clinical decision support systems
Magrabi, F., Cresswell, K. M., & Fraser, H. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00013-9 · Cited by 6
Self-citationMohan, S., & Sharmil, H. (2023) · PubMed Central · doi:10.30476/ijcbnm.2023.98777.2264 · Cited by 6
“22 The overall result of E-Midwife’s usability testing indicated that it had good usability.”
BackgroundPotential and pitfalls of conversational agents in health care
Denecke, K. (2023) · Nature Reviews Disease Primers · doi:10.1038/s41572-023-00482-x · Cited by 6
“In another study, the appropriateness of urgency advice of Ada achieved a performance of 97% measured against the vignettes’ gold standard 5 .”
BackgroundLiu, V., Koskela, T., & Kaila, M. (2023) · JMIR Research Protocols · doi:10.2196/41423 · Cited by 6
Cawley, C., Bergey, F., Mehl, A., Finckh, A., & Gilsdorf, A. (2021) · JMIR Public Health and Surveillance · doi:10.2196/26523 · Cited by 6
Self-citationBarr, K. R., Hawker, P. J., Winata, T., Wang, S., Smead, M., Ignatius, H., Kohlhoff, J. R., Schmied, V. A., Jalaludin, B. B., Lawson, K. D., et al. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11397-y · Cited by 5
Gellert, G. A., Garber, L., Kabat-Karabon, A., Kuszczyński, K., Price, T. J., McLean, E. J., Trybucka, K., Nichols, M. W., Pike, J., Powers, M. J., et al. (2024) · International Journal of Healthcare · doi:10.5430/ijh.v10n1p41 · Cited by 5
Martinengo, L., Lo, N. Y. W., Goh, W. W. T., & Car, L. T. (2021) · JMIR Research Protocols · doi:10.2196/30166 · Cited by 5
Avey: An Accurate AI Algorithm for Self-DiagnosisPreprint
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2022) · medRxiv · doi:10.1101/2022.03.08.22272076 · Cited by 5
Evaluating digital triage symptom checker with historical triage-related adverse events
Ilicki, J., Edman, S., Stalfors, J., & Molin, C. J. (2025) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2025.2563517 · Cited by 4
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Acharya, A., Daniels, E., Normahani, P., Matin, R., Markar, S. R., Sounderajah, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000558 · Cited by 5
“The observed primary diagnostic sensitivity for detecting MI in our study was higher than previous vignette studies examining symptom checkers for other common medical conditions [8,30,31].”
ResultReal-World Evaluation of a Chest Pain Digital Triage Platform at the Point of Care
Hourizadeh, J., Zeltser, R., Makaryus, A., & Druz, R. (2023) · Southern medical journal (Birmingham, Ala. Print) · doi:10.14423/smj.0000000000001620 · Cited by 4
European regulation of digital respiratory healthcare
Gilbert, S. (2023) · European Respiratory Society eBooks · doi:10.1183/2312508x.10000923 · Cited by 5
Self-citationLowe, C., Sephton, R., Marsh, W., & Morrissey, D. (2024) · JMIR Formative Research · doi:10.2196/56715 · Cited by 4
Lowe, C., Atherton, L., Lloyd, P. A., Waters, A., & Morrissey, D. (2025) · Journal of Medical Internet Research · doi:10.2196/67269 · Cited by 4
Gellert, G. A., Palczewski, M., Marecka, M., Paczkowska, K., Suwińska, A., Jaszczak, J., Gellert, G. L., Orzechowski, P. M., Price, T., & Górski, S. (2024) · Medinformatics · doi:10.47852/bonviewmedin42023535 · Cited by 4
Pankow, A., Meißner-Bendzko, N., Kaufeld, J. K., Fouquette, L., Cotte, F., Gilbert, S., Türk, E., Das, A. M., Terkamp, C., Burmester, G.-R., et al. (2025) · JMIR AI · doi:10.2196/55001 · Cited by 3
“M scores represent the degree to which the model’s diagnostic output aligns with the correct diagnosis, serving as a metric to assess the model’s accuracy in identifying the correct diseases, also referred to as conditions in the context of SCs [11].”
InfluentialSelf-citationMillen, E., Salim, N., Azadzoy, H., Bane, M., O’Donnell, L., Schmude, M., Bode, P., Türk, E., Vaidya, R., & Gilbert, S. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055915 · Cited by 4
Self-citationShah, A. B., Oyegun, E., Hampton, W. B., Neri, A., Maddox, N., Raso, D., Sandhu, P. K., Patel, A., Koonin, L. M., Lee, L., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/39054 · Cited by 3
“However, the existing literature on symptom checkers is mostly limited to small validation studies [5-7].”
Sellin, J., Pantel, J. T., Börsch, N., Conrad, R., & Mücke, M. (2024) · Der Schmerz · doi:10.1007/s00482-023-00777-8 · Cited by 3
Gilbert, S., Fenech, M. T., Idris, A., & Türk, E. (2021) · Journal of Medical Internet Research · doi:10.2196/26514 · Cited by 3
Self-citationKopka, M., Napierala, H., Privoznik, M., Sapunova, D., Zhang, S., & Feufel, M. A. (2024) · medRxiv · doi:10.1101/2024.04.02.24305193 · Cited by 3
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2024) · medRxiv · doi:10.1101/2024.09.13.24313657 · Cited by 3
Gehlen, T., Joost, T., Solbrig, P., Stahnke, K., Zahn, R., Jahn, M., Amini, D. A., & Back, D. A. (2025) · Diagnostics · doi:10.3390/diagnostics15020221 · Cited by 2
“The above-mentioned apps delivered significantly better results in the study by Gilbert et al., in which an accuracy of 99% was achieved for Ada Health, for example [27].”
Concordance between an artificial intelligence self-triage programme and physical triage
Wempe, M., Holleman, F., Schinkel, M., & Gorzeman, M. P. (2026) · Emergency Medicine Journal · doi:10.1136/emermed-2024-214212 · Cited by 2
“4 5 9–11 13–15 Since developments in this field are evolving rapidly, further investigation is warranted.”
Patient-targeted mobile applications in healthcare
Karia, J., Mohamed, R., & Petrushkin, H. (2023) · British Journal of Hospital Medicine · doi:10.12968/hmed.2023.0158 · Cited by 2
Digital Allergology and Allergen-Immunotherapy in Germany
Klimek, L., Walter, U., Becker, S., Casper, I., Kianfar, R., Röseler, S., Schareina, A., Wrede, H., & Dramburg, S. (2025) · Health informatics · doi:10.1007/978-3-031-71021-6_20 · Cited by 2
Metadata
- Authors
- Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., Richter, C., Türk, E., Upadhyay, S., Virani, V., Vona, N., Wicks, P., & Novorol, C.
- Affiliations
- Brown University (US)
- Corresponding author
- Gilbert, Stephen
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2020-12
- Volume
- 10
- Issue
- 12
- Pages
- e040269
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC7745523
- OpenAlex
- W3112185662
- Cited by
- 183
- References
- 30
- Citations by year
- 2026: 20 · 2025: 25 · 2024: 31 · 2023: 37 · 2022: 40 · 2021: 27 · 2020: 3
- FWCI
- 27.92
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- symptom checkers, clinical vignettes, general practitioners, Ada, Babylon, buoys, WebMD, primary care, diagnostic accuracy, patient safety, user demographics, mobile health apps, telemedicine, primary diagnosis
- MeSH terms
- Humans, Primary Health Care, General Practitioners, Symptom Assessment, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
