The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review
Open accessAbstract
Objectives: In England, the NHS111 service provides assessment and triage by telephone for urgent health problems. A digital version of this service has recently been introduced. We aimed to systematically review the evidence on digital and online symptom checkers and similar services. Design: Systematic review. Data sources: We searched Medline, Embase, the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Health Management Information Consortium, Web of Science and ACM Digital Library up to April 2018, supplemented by phrase searches for known symptom checkers and citation searching of key studies. Eligibility criteria: Studies of any design that evaluated a digital or online symptom checker or health assessment service for people seeking advice about an urgent health problem. Data extraction and synthesis: Data extraction and quality assessment (using the Cochrane Collaboration version of QUADAS for diagnostic accuracy studies and the National Heart, Lung and Blood Institute tool for observational studies) were done by one reviewer with a sample checked for accuracy and consistency. We performed a narrative synthesis of the included studies structured around pre-defined research questions and key outcomes. Results: We included 29 publications (27 studies). Evidence on patient safety was weak. Diagnostic accuracy varied between different systems but was generally low. Algorithm-based triage tended to be more risk averse than that of health professionals. There was very limited evidence on patients’ compliance with online triage advice. Study participants generally expressed high levels of satisfaction, although in mainly uncontrolled studies. Younger and more highly educated people were more likely to use these services. Conclusions: The English ‘digital 111’ service has been implemented against a background of uncertainty around the likely impact on important outcomes. The health system may need to respond to short-term changes and/or shifts in demand. The popularity of online and digital services with younger and more educated people has implications for health equity. PROSPERO registration number: CRD42018093564.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“We considered some major limitations of previous studies [32] given that we collected real-world patient data, which comes closer to the current intended laypeople-oriented application of symptom checkers.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548
The effects of integrated care: a systematic review of UK and international evidence
Baxter, S. K., Johnson, M., Chambers, D., Sutton, A., Goyder, E., & Booth, A. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3161-3 · Cited by 745 · cited 2×
“Overall strength of the evidence base for key outcomes was assessed using an adaptation of the method described by Baxter et al.(11) This involves classifying evidence as ‘stronger’, ‘weaker’, ‘conflicting’ or ‘insufficient’ based on study numbers and design.”
MethodSelf-citationEvaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 5×
“Studies either evaluated a single system or multiple systems.(6) 21 We found only one study of a symptom checker specifically intended for the assessment of children’s symptoms, a development of the Strategy for Off-Site Rapid Triage system for influenza-like illness.”
BackgroundInfluentialElectronic healthcare: a study of people's readiness and attitude toward performing self‐diagnosis
Lanseng, E. J., & Andreassen, T. W. (2007) · International Journal of Service Industry Management · doi:10.1108/09564230710778155 · Cited by 177
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 3×
“In addition, the students’ complaints were generally common and uncomplicated, a scenario in which symptom checkers performed relatively well in the study by Semigran et al.(21)”
BackgroundResultInfluentialImpact of the urgent care telephone service NHS 111 pilot sites: a controlled before and after study
Turner, J., O’Cathain, A., Knowles, E., & Nicholl, J. P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003451 · Cited by 114 · cited 1×
“For example, an evaluation of the NHS111 telephone service at four pilot sites and three control sites found that in its first year the service was not successful in reducing 999 emergency calls or in shifting patients from emergency to urgent care.(5) A recent study of 23 symptom checker algorithms providing diagnost…”
MethodSelf-citationCowie, J., Calveley, E., Bowers, G., & Bowers, J. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15050896 · Cited by 89
Nijland, N., van Gemert-Pijnen, J. E. W. C., Boer, H., Steehouder, M. F., & Seydel, E. R. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.06.002 · Cited by 84 · cited 1×
“Patients with low to medium levels of education tended to be motivated toward indirect e-consultation (which involves contact with a health professional via email), mainly to reduce uncertainty.(27) Evidence from included studies suggests that users of e-consultation were more likely to be young,(7) 24–26 employed(20)…”
BackgroundSelf-triage for acute primary care via a smartphone application: Practical, safe and efficient?
Verzantvoort, N. C. M., Teunis, T., Verheij, T. J., & van der Velden, A. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199284 · Cited by 77
Older Adult Experience of Online Diagnosis: Results From a Scenario-Based Think-Aloud Protocol
Luger, T. M., Houston, T. K., & Suls, J. M. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2924 · Cited by 73
Research works printed but not linked: 2
- Reitsma JB , Rutjes AWS , Whiting P , et al ; Chapter 9: Assessing methodological quality. In: Deeks JJ , Bossuyt PM , Gatsonis C , eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 100: The Cochrane Collaboration, 2009.
- Luger, 2011, Online health information and intentions to seek healthcare, Psychosomatic Medicine, 73
Cited by
Sorted by citations
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 213
“[13] The diagnostic accuracy was found to be variable between different platforms and was generally low.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183
An introduction to digital determinants of health
Chidambaram, S., Jain, B., Jain, U., Mwavu, R., Baru, R., Thomas, B., Greaves, F., Jayakumar, S., Jain, P., Rojo, M., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000346 · Cited by 175
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167
“Previous work has investigated the diagnostic and triage accuracy of symptom checkers (Semigran et al., 2015; Millenson et al., 2018; Chambers et al., 2019; Yu et al., 2019).”
Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 164
“However, symptom checkers often use some proprietary diagnostic algorithms, such as branching logic and Bayesian inference, that are not optimal for processing complicated and sometimes ambiguous user inputs, compromising not only user experience but also diagnostic accuracy [47,49].”
BackgroundChenais, G., Lagarde, E., & Gil-Jardiné, C. (2023) · Journal of Medical Internet Research · doi:10.2196/40031 · Cited by 142
“Multiple proprietary self-sorting apps”
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125
“Several previous studies [13, 14] into the effectiveness of algorithmic performance have found deficiencies in the diagnostic capabilities and a cautious approach to triage.”
BackgroundMiller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117
“More recently, internet-enabled options have emerged such as using a search engine to look up symptoms (Dr Google), high-quality online resources such as NHS Choices [4], symptom checkers [5,6], telehealth consultations by phone or videocall [7], Minute Clinics [8] that can be booked via smartphone, and peer-to-peer n…”
BackgroundAdvancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 121
Martin, A. G., Nateqi, J., Gruarin, S., Munsch, N., Abdarahmane, I., Zobel, M., & Knapp, B. (2020) · Scientific Reports · doi:10.1038/s41598-020-75912-x · Cited by 103
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99
“A key source of these and other reviews [30,31] on symptom checkers is a study by Semigran et al [23] published in 2015.”
BackgroundExploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86
“Very little research has been undertaken to investigate how real-world users actually use these symptom checkers, as identified by recent review articles [1, 8, 46], with one exception [59] testing the Technology Trust Model on users of one OSC and another [85] examining the user experience of OSCs.”
BackgroundMorse, K., Ostberg, N. P., Jones, V. G., & Chan, A. S. (2020) · Journal of Medical Internet Research · doi:10.2196/20549 · Cited by 86
“Symptom checker triage recommendations have been theorized to reduce unnecessary clinic and emergency room visits [5], and a recent study showed that completing a web-based symptom assessment reduced the urgency of the care that patients intended to seek [6].”
BackgroundTriage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review
Sasco, E. R., El-Osta, A. A. R., Alaa, A., Webber, I., Karki, M., Asmar, M. L. E., Purohit, K., Painter, A., & Hayhoe, B. W. J. (2023) · Journal of Medical Internet Research · doi:10.2196/43803 · Cited by 85
“The 2019 systematic review by Chambers et al [12] included any type of publication, including gray literature, but was limited to studies relating to urgent health issues only.”
InfluentialAI in healthcare: A narrative review
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.26997.2 · Cited by 69
Woolley, K. E., Bright, D., Ayres, T., Morgan, F. M., Little, K. E., & Davies, A. R. (2023) · Journal of Medical Internet Research · doi:10.2196/44181 · Cited by 69
“However, 18% (4/22) of reviews reported a possible association between the use of digital health care and education showing that individuals with higher levels of education were more likely to use digital health services than individuals with a lower education level [18,21,29,35].”
InfluentialKnitza, J., Mohn, J., Bergmann, C., Kampylafka, E. I., Hagen, M., Bohr, D., Morf, H., Araujo, E. G., Englbrecht, M., Simón, D., et al. (2021) · Arthritis Research & Therapy · doi:10.1186/s13075-021-02498-8 · Cited by 65
“The results of this study are in line with previous SC analyses [12, 17, 18].”
BackgroundResultJones, O. T., Calanzani, N., Saji, S., Duffy, S. W., Emery, J., Hamilton, W., Singh, H., de Wit, N. J., & Walter, F. M. (2021) · Journal of Medical Internet Research · doi:10.2196/23483 · Cited by 65
“This included manually searching commercial research archives and networks (eg, arXiv [28], Google [29], Microsoft [30], and IBM [31]), reviewing the computer-based technologies identified in 3 recent reviews [19-21], and manually searching for further technologies mentioned in the text or references of the studies an…”
BackgroundMethodDialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 73
“Research shows that although symptom check technologies are highly adopted by most users due to their ease, their diagnostic performance varies significantly between other similar tools and was considered low to moderate at best [19].”
BackgroundJanda, M., Horsham, C., Vagenas, D., Loescher, L. J., Gillespie, N. A., Koh, U., Curiel‐Lewandrowski, C. N., Hofmann‐Wellenhof, R., Halpern, A. C., Whiteman, D. C., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30001-7 · Cited by 65
Carmona, K. A., Chittamuru, D., Kravitz, R. L., Ramondt, S., & Ramírez, A. S. (2022) · Journal of Medical Internet Research · doi:10.2196/36322 · Cited by 54
“Consistent with data on internet-based health-seeking behaviors more generally [49], most Buoy users were middle-aged (or younger), female, and highly educated.”
BackgroundDarley, S., Coulson, T., Peek, N., Moschogianis, S., van der Veer, S. N., Wong, D. C., & Brown, B. C. (2022) · Journal of Medical Internet Research · doi:10.2196/37436 · Cited by 53
“Although symptom checkers [29,42] and patient portals [30,43,44] have been well studied, only a small number of evidence syntheses directly relevant to OCs have been published: a systematic review of 57 articles on delivering “e-consultation” in primary care largely focused on generic stand-alone…”
Ćirković, A. (2020) · Journal of Medical Internet Research · doi:10.2196/18097 · Cited by 54
Pairon, A., Philips, H., & Verhoeven, V. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2022.1040926 · Cited by 48
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 57
“These reviews consistently report that SAA accuracy is relatively low, but note that the variability between the tools is very high 17 – 19 .”
InfluentialGalmiche, S., Rahbé, E., Fontanet, A. L., Dinh, A., Bénézit, F., Lescure, F.-X., & Denis, F. (2020) · Journal of Medical Internet Research · doi:10.2196/22924 · Cited by 47
“A recent systematic review of self-triage symptom checkers for urgent health problems suggested they led to less frequent health care use [13].”
BackgroundBenchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study
Schmieding, M. L., Mörgeli, R., Schmieding, M. A. L., Feufel, M. A., & Balzer, F. (2021) · Journal of Medical Internet Research · doi:10.2196/24475 · Cited by 42
“Despite their popularity, there is no established framework to evaluate the performance of SCs [3,4].”
BackgroundPerlman, A., Vodonos, A., Bak, P., Dreyfuss, M., Leventer‐Roberts, M., Vurembrand, Y., Jeffries, H. E., Fisher, E., Steuerman, Y., Namir, Y., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23197 · Cited by 42
“While the use of digital health technologies among seniors has been reported to be increasing [28], users of digital health applications are still predominantly younger adults [29].”
BackgroundCotte, F., Mueller, T., Gilbert, S., Blümke, B., Multmeier, J., Hirsch, M. C., Wicks, P. J., Wolanski, J., Tutschkow, D., Schade‐Brittinger, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/32340 · Cited by 41
“This observational study addressed an underresearched SAA triage topic in the ED [18].”
BackgroundMahlknecht, A., Engl, A., Piccoliori, G., & Wiedermann, C. J. (2023) · BMC Primary Care · doi:10.1186/s12875-023-02143-0 · Cited by 41
“This somewhat contradicts the perception of several patients who mentioned a potential in this regard (Supplementary Table I a), which was also confirmed by previous studies [15, 16, 25].”
BackgroundResultInfluentialCadamuro, J. (2021) · Diagnostics · doi:10.3390/diagnostics11081399 · Cited by 38
“, performed a systematic review following the same goal [70].”
BackgroundProtocol: Remote care as the ‘new normal’? Multi-site case study in UK general practice
Greenhalgh, T., Shaw, S. E., Nishio, A. A., Booth, A., Byng, R., Clarke, A., Dakin, F., Davies, R., Faulkner, S. D., Hemmings, N., et al. (2022) · NIHR Open Research · doi:10.3310/nihropenres.13289.1 · Cited by 40
Self-citationDelshad, S. D., Dontaraju, V. S., & Chengat, V. (2021) · Cureus · doi:10.7759/cureus.16956 · Cited by 37
“Prior research indicates that online symptom checkers often provide inaccurate or incomplete diagnostic or triage advice for users [8,9].”
BackgroundResultDeterminants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A. (2022) · JMIR Human Factors · doi:10.2196/35219 · Cited by 37
Scheder-Bieschin, J., Blümke, B., de Buijzer, E., Cotte, F., Echterdiek, F., Nacsa, J., Ondrésik, M., Ott, M., Paul, G., Schilling, T., et al. (2021) · JMIR Formative Research · doi:10.2196/28199 · Cited by 40
“A number of studies [9,13,14] have described a range of tools for self-assessment of urgency and triage in the general practice and acute primary care setting, but studies evaluating the handover of history and symptom information to HCPs have not been carried out with real patients.”
BackgroundEl-Osta, A. A. R., Webber, I., Alaa, A., Bagkeris, E., Mian, S., Sharabiani, M. T. A., & Majeed, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053566 · Cited by 30
“13 14 However, OSCs that have high false-negative rates may run similar risks if used by patients with high-risk disease such as cardiac ischaemia, pulmonary embolism or meningitis.”
Applying Artificial Intelligence to Gynecologic Oncology: A Review
Mysona, D., Kapp, D. S., Rohatgi, A. P., Lee, D., Mann, A. K., Tran, P. M. H., Tran, L. K. H., She, J., & Chan, J. K. (2021) · Obstetrical & Gynecological Survey · doi:10.1097/ogx.0000000000000902 · Cited by 41
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L. R., Baxter, S. K., Booth, A., & Turner, J. (2019) · Health Services and Delivery Research · doi:10.3310/hsdr07290 · Cited by 35
“Some parts of this chapter have been reproduced from Chambers et al.(1) © Author(s) (or their employer(s)) 2019.”
MethodSelf-citationLevine, D. M., & Mehrotra, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3287 · Cited by 35
Samhammer, D., Roller, R., Hummel, P., Osmanodja, B., Burchardt, A., Mayrdorfer, M., Duettmann, W., & Dabrock, P. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.1016366 · Cited by 33
“For remote health assessments, systems provide factual diagnostic suggestions in the form of symptom checkers (6).”
BackgroundHennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31
“In contrast to patients’ rather positive perspectives on the usability and utility of symptom checkers, health professionals seem to be more skeptical [25], and symptom checkers have had an inferior performance compared with professional diagnoses in previous studies [32].”
BackgroundMethodResultHo, A. (2023) · doi:10.1093/med/9780197556269.001.0001 · Cited by 35
Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review
Ilicki, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279636 · Cited by 33
“Vignette design lacks external validity Vignette content/mix not representative of real life • Vignettes not representative of physical primary care population [10, 45] • Vignettes covering limited range of conditions [12] • No out-of-sample external validation [45] • Vignettes with clear symptoms and diagnosis [1, 44…”
BackgroundInfluentialFraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/38364 · Cited by 32
“Diagnostic and triage apps designed for patients, often called symptom checkers, have become widely available to the general public over the last decade [7].”
BackgroundRadionova, N., Ög, E., Wetzel, A.-J., Rieger, M. A., & Preiser, C. (2023) · Journal of Medical Internet Research · doi:10.2196/39219 · Cited by 35
“Some authors attested that SCs have a lower accuracy in diagnostics [14,15,37,40,43] and in triage, understood as an evaluation of urgency [15,36,37,40], compared with physicians.”
InfluentialOnline Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · Journal of Medical Internet Research · doi:10.2196/37408 · Cited by 30
“Studies have demonstrated that OSCs often advise contact with health care services for conditions that can be self-managed and thus ‘overtriage’patients, which could result in an increased burden on health care systems [1,2,4,13,21-23].”
Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study
Kopka, M., Scatturin, L., Napierala, H., Fürstenau, D., Feufel, M. A., Balzer, F., & Schmieding, M. L. (2023) · Journal of Medical Internet Research · doi:10.2196/46231 · Cited by 26
“Lastly, users of SCs tend to have higher levels of formal education, which is associated with an increased likelihood of searching for diagnoses online [8,20,24-27].”
BackgroundRedesigning Primary Care: The Emergence of Artificial-Intelligence-Driven Symptom Diagnostic Tools
Wiedermann, C. J., Mahlknecht, A., Piccoliori, G., & Engl, A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13091379 · Cited by 29
“Some studies have highlighted the propensity of these tools to either misdiagnose or lean towards excessively cautious triage recommendations [9,16].”
How to conduct written online consultations with patients in primary care
Bakhai, M., & Atherton, H. (2021) · BMJ · doi:10.1136/bmj.n264 · Cited by 28
Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Ben‐Shabat, N., Sharvit, G., Meimis, B., Joya, D. B., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, ., & Amital, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104897 · Cited by 28
Burzyńska, J., Bartosiewicz, A., & Januszewicz, P. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20020978 · Cited by 30
“There are data indicating that younger people who are in worse health condition seek access to this information more often [10].”
BackgroundUnderstanding the invisible workforce: lessons for general practice from a survey of receptionists
Litchfield, I. J., Burrows, M., Gale, N. K., & Greenfield, S. (2022) · BMC Family Practice · doi:10.1186/s12875-022-01842-4 · Cited by 27
“There is also an increasing use of online symptom checkers which can provide alternative diagnoses and suggest or facilitate a course of action including making an appointment with a GP [79], though early indications are that they tend to be favoured by younger or better educated patients [79].”
BackgroundSun, Y., & Sundar, S. S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519668 · Cited by 31
“Despitethe increasing attention to the transparency issue, several questions remain: First, the current research on explanations from OSCs applies to a single message, which is presented all at once, even when it is interwoven into conversation flow (e.g., [7]).”
BackgroundFaqar-Uz-Zaman, S. F., Anantharajah, L., Baumartz, P., Sobotta, P., Filmann, N., Zmuc, D., Wagner, M., Detemble, C., Sliwinski, S., Marschall, U., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005614 · Cited by 25
“7,24 In a comprehensive audit study, Semigran et al assessed 23 commercial symptom checker apps across 770 standardized patient evaluations and found, that the diagnostic accuracy for the fi rst diagnosis was only 34% and 51.2% for the top 3 diagnoses.”
BackgroundInfluentialSiripurapu, S., Darimireddy, N. K., Chehri, A., Sridhar, B., & Paramkusam, A. V. (2023) · Electronics · doi:10.3390/electronics12030750 · Cited by 27
Kopka, M., Feufel, M. A., Berner, E. S., & Schmieding, M. L. (2023) · Digital Health · doi:10.1177/20552076231194929 · Cited by 23
“, different apps or versions of them are being assessed) and methodological differences, for example, how urgency levels are defined or on which types of complaints the apps are tested.(8) For example, some authors evaluate symptom checkers using vignettes on a broad variety of general symptoms,1,2,9–13 while others f…”
BackgroundMethodTriage Errors in Primary and Pre–Primary Care
Nguyen, H., Meczner, A., Burslam-Dawe, K., & Hayhoe, B. W. J. (2022) · Journal of Medical Internet Research · doi:10.2196/37209 · Cited by 21
“Based on limited evidence, digital symptom checkers have relatively low triage accuracy, with a mean error rate of around 50% [25,30].”
BackgroundChambers, D., Preston, L. R., Clowes, M., Cantrell, A. J., & Goyder, E. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-081934 · Cited by 21
“16”
Self-citationKujala, S., & Hörhammer, I. (2022) · Journal of Medical Internet Research · doi:10.2196/33505 · Cited by 30
“However, the impact of symptom checkers on the urgent care system and the wider scope of health care is not known [1].”
BackgroundInfluentialProposal of a novel Artificial Intelligence Distribution Service platform for healthcare
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K. M., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.36775.1 · Cited by 20
Haase, C. B., Bearman, M., Brodersen, J., Hoeyer, K., & Risør, T. (2020) · Scandinavian Journal of Public Health · doi:10.1177/1403494820980268 · Cited by 19
“The review found little evidence of cost-effectiveness and patient compliance [18].”
BackgroundGellert, G. A., Orzechowski, P. M., Price, T. J., Kabat-Karabon, A., Jaszczak, J., Marcjasz, N., Młodawska, A., Kwiecien, A. K., & Kurkiewicz, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1047291 · Cited by 18
Litchfield, I. J., Gale, N. K., Burrows, M., & Greenfield, S. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21298 · Cited by 18
“…advice and signposting to appropriate services for people with urgent health-care problems [82], however it remains under utilised by the most frequent GP attenders namely older or less well-educated patients [83] with evidence of it causing duplication of effort for patients and providers [76,82].”
BackgroundGilbert, S., Fenech, M. E., Upadhyay, S., Wicks, P. J., & Novorol, C. (2021) · Australian Journal of Primary Health · doi:10.1071/py21032 · Cited by 17
Harada, Y., Sakamoto, T., Sugimoto, S., & Shimizu, T. (2024) · JMIR Formative Research · doi:10.2196/53985 · Cited by 17
“Indeed, some symptom checkers have already been used in clinical practice [15-17] and even in national health services, such as the National Health Service 111 system in the United Kingdom [18].”
BackgroundInfluentialKubb, C., & Foran, H. M. (2022) · JMIR Pediatrics and Parenting · doi:10.2196/29618 · Cited by 19
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2024) · JMIR AI · doi:10.2196/46875 · Cited by 21
“For instance, Chambers et al [87] found in 2019 that symptom checkers were much less accurate than physicians.”
BackgroundPerformance of national COVID-19 ‘symptom checkers’: a comparative case simulation study
Mansab, F., Bhatti, S., & Goyal, D. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100187 · Cited by 15
“While self-triage has been used for some years in non-emergency conditions to varying degrees of success, 17 self-triage has never before been used in a pandemic setting and as yet the efficacy and safety has not be formally studied.”
Klinische Entscheidungsfindung mit Künstlicher Intelligenz
Samhammer, D., Beck, S., Budde, K., Burchardt, A., Faber, M., Gerndt, S., Möller, S., Osmanodja, B., Roller, R., & Dabrock, P. (2023) · Essentials · doi:10.1007/978-3-662-67008-8 · Cited by 14
Ahn, H. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037161 · Cited by 14
Kopka, M., & Feufel, M. A. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00096-7 · Cited by 13
“For users, it is crucial that symptom checkers give safe advice and prevent potential harm [18], whereas for healthcare systems overtriage could inflate costs and strain scarce resources (e.g., due to unnecessary emergency department visits) [5, 14].”
BackgroundIslam, S., Shahriyar, R., Agarwala, A., Zaman, M., Ahamed, S. A., Rahman, R., Chowdhury, M. H., Sarker, F., & Mamun, K. A. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02864-5 · Cited by 16
Leaders’ support for using online symptom checkers in Finnish primary care: Survey study
Laukka, E., Kujala, S., Gluschkoff, K., Kanste, O. I., Hörhammer, I., & Heponiemi, T. (2021) · Health Informatics Journal · doi:10.1177/14604582211052259 · Cited by 12
“10 However, there exists limited evidence on how patients react to advice provided by SCs. 9 In our study, leaders identi fi ed several ways through which SCs bene fi t patients, for example, increased accessibility to health services and a lower threshold for care.”
BackgroundInfluentialJudson, T. J., Pierce, L., Tutman, A., Mourad, M., Neinstein, A. B., Shuler, G., Gonzales, R., & Odisho, A. Y. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac161 · Cited by 12
Health Professionals’ Experiences of the Benefits and Challenges of Online Symptom Checkers
Kujala, S., Hörhammer, I., Hänninen-Ervasti, R., & Heponiemi, T. (2020) · Studies in health technology and informatics · doi:10.3233/shti200305 · Cited by 12
“Online symptom checkers and assessment services are used by patients seeking guidance on health problems [1].”
BackgroundInfluentialWetzel, A.-J., Koch, R., Koch, N. N., Klemmt, M., Müller, R., Preiser, C., Rieger, M. A., Rösel, I., Ranisch, R., Ehni, H., et al. (2024) · Digital Health · doi:10.1177/20552076241231555 · Cited by 11
Evaluation of AI-Driven LabTest Checker for Diagnostic Accuracy and Safety: Prospective Cohort Study
Szumilas, D., Ochmann, A., Zięba, K. A., Bartoszewicz, B., Kubrak, A., Makuch, S., Agrawal, S., Mazur, G. J., & Chudek, J. T. (2024) · JMIR Medical Informatics · doi:10.2196/57162 · Cited by 12
“Several studies have assessed the effectiveness and safety of AI-driven symptom checkers, tools designed to aid patients in self-diagnosing symptoms and making informed health care choices [8-10].”
Artificial Intelligence in Digital Self-Diagnosis Tools: A Narrative Overview of Reviews
Mentzou, A., Rogers, A., Carvalho, E., Daly, A., Malone, M., & Kerasidou, X. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100242 · Cited by 12
“6,19 The reviews suggest that independent research and evaluative autonomy should address con fl icts of interest by involving the disclosure of tool architecture and in-built values to allow for informed decision making.”
InfluentialChurruca, K., Ellis, L. A., Pope, C. J., MacLellan, J., Zurynski, Y. A., & Braithwaite, J. (2023) · Digital Health · doi:10.1177/20552076231181201 · Cited by 10
“They are regarded by many as overly ‘risk averse’, advising users to seek healthcare where self-care is appropriate.(5,10) One systematic review of telephone triage services found that the median for appropriate triage advice was 75%, with evidence of under-referral in the range of 0.”
BackgroundLiu, A. W., Odisho, A. Y., Brown, W., Gonzales, R., Neinstein, A. B., & Judson, T. J. (2022) · JMIR Human Factors · doi:10.2196/40064 · Cited by 11
Meczner, A., Cohen, N., Qureshi, A., Reza, M., Sutaria, S., Blount, E., Bagyura, Z., & Malak, T. (2024) · JMIR Formative Research · doi:10.2196/49907 · Cited by 10
“This may also be one of the reasons why systematic reviews on accuracy of SCs show a marked spread of results when comparing individual vignette studies or both clinical and vignette studies [2,37-39].”
ResultHertling, S. F., Loos, F. M., & Graul, I. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18137110 · Cited by 10
Irwin, P., Rehman, S. U., Fealy, S. M., Kornhaber, R. A., Matheson, A., & Cleary, M. L. (2025) · Contemporary Nurse · doi:10.1080/10376178.2025.2459701 · Cited by 9
“Research that incorporates the wider implications of the apps considers patient safety, cost and clinical effectiveness, impact on service use, and patient compliance (Chambers et al., 2019).”
Lammila-Escalera, E., Greenfield, G., Aldakhil, R., Mak, H. M., Sehgal, H., Neves, A. L., Harmon, M. J., Majeed, A., & Hayhoe, B. W. J. (2025) · Journal of Medical Internet Research · doi:10.2196/69528 · Cited by 9
“While past reviews have assessed digital triage accuracy, they did not use kiosks as a delivery method [20,25,26].”
Sexton, V., Dale, J. R., & Atherton, H. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01576-x · Cited by 9
“Findings could also feed into the design of the newly emerging patient self-triage approaches that are being adopted by care services [19, 20], for example the NHS 111Online, which allows patients to self-triage and receive care advice online [21].”
BackgroundVuillaume, L. A., Turpinier, J., Cipolat, L., Arnaud-Dépil-Duval, Dumontier, T., Peschanski, N., Kieffer, Y., Barbat, B., Riquier, T., Dinot, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0277568 · Cited by 9
“Specifically regarding the evaluation of SC, it is important to know that there is no reference method [16, 23, 24].”
BackgroundLorenzo, S. M., Astier-Peña, M. P., & Benejam, T. C. (2021) · Atención Primaria · doi:10.1016/j.aprim.2021.102227 · Cited by 9
Hertling, S. F., Hertling, D., Matziolis, G., Schleußner, E., Loos, F. M., & Graul, I. (2022) · PLoS ONE · doi:10.1371/journal.pone.0267144 · Cited by 9
“Members of the Working Group Young Forum of the German Society for orthopedics and trauma surgery (DGOU) were asked to provide feedback on the validation process’s format, completeness, clarity, and procedure [15, 17].”
MethodChambers, D., Cantrell, A. J., Baxter, S. K., Turner, J., & Booth, A. (2020) · BMC Medicine · doi:10.1186/s12916-020-01580-3 · Cited by 9
Self-citationBan, K., Greenfield, S., Burrows, M., Gale, N. K., & Litchfield, I. J. (2024) · British Journal of General Practice · doi:10.3399/bjgp.2024.0228 · Cited by 8
Preiser, C., Radionova, N., Ög, E., Koch, R., Klemmt, M., Mueller, R., Ranisch, R., Joos, S., & Rieger, M. A. (2024) · JMIR Human Factors · doi:10.2196/57360 · Cited by 8
“For the purpose of this study, we use an operational definition: symptom checkers provide lay users with preliminary diagnoses, give a first sense of urgency, and suggest a course of action [9].”
A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History
Jones, A. M., & Jones, D. R. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.727486 · Cited by 10
“In previous studies such DAs have underperformed physicians in diagnostic accuracy (Semigran et al., 2015, 2016; Millenson et al., 2018; Chambers et al., 2019; Yu et al., 2019).”
BackgroundUser Experience of Symptom Checkers: A Systematic Review
You, Y., Ma, R., & Gui, X. (2022) · PubMed Central · doi:10.48550/arxiv.2208.09100 · Cited by 8
Savolainen, K., & Kujala, S. (2024) · JMIR Human Factors · doi:10.2196/45275 · Cited by 8
“Online symptom checkers are used by people seeking health-related guidance, and these services typically provide an urgent assessment and suggest guidance based on the symptoms reported by the user [11].”
BackgroundSehgal, H. L. K., Greenfield, G., Neves, A. L., Harmon, M., Majeed, A., & Hayhoe, B. W. J. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-084506 · Cited by 7
“17 19 This systematic review seeks to address these critical issues by conducting a rigorous examination of the existing literature, aiming to evaluate the efficacy and safety of self-service kiosk implementation within EDs.”
BackgroundStrehlow, M. C., Alvarez, A., Blomkalns, A. L., Caretta‐Weyer, H. A., Gharahbaghian, L., Imler, D. L., Khan, A., Lee, M. O., Lobo, V. E., Newberry, J. A., et al. (2024) · Academic Emergency Medicine · doi:10.1111/acem.14962 · Cited by 8
Digital Health Technology Implementation: Is It Effective in a Healthy Healthcare Perspective?
Stein, A. T., Ben, Â. J., Pachito, D. V., Cazella, S. C., van Dongen, J. M., & Bosmans, J. E. (2020) · Springer eBooks · doi:10.1007/978-3-030-59467-1_9 · Cited by 9
Conversational Agents in Health Care—Reply
McGreevey, J. D., Hanson, C. W., & Koppel, R. (2020) · JAMA · doi:10.1001/jama.2020.21518 · Cited by 7
Litchfield, I. J., Gale, N. K., Greenfield, S., Shukla, D., & Burrows, M. (2025) · Frontiers in Health Services · doi:10.3389/frhs.2024.1499847 · Cited by 6
Factors Influencing the Use of Online Symptom Checkers in the United Kingdom: Cross-Sectional Study
El-Osta, A. A. R., Sasco, E. R., Al-Ammouri, M., Altalib, S., Neves, A. L., Majeed, A., & Hayhoe, B. W. J. (2025) · JMIR Formative Research · doi:10.2196/65314 · Cited by 6
“Individual factors like internet access, health technology literacy, and awareness of OSC availability also play significant roles in determining user engagement with OSCs [5,12,20], with one UK survey showing higher mean health technology literacy across users of an OSC [21].”
Metadata
- Authors
- Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J.
- Affiliations
- University of Sheffield (GB)
- Corresponding author
- Chambers, Duncan
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2019-08
- Volume
- 9
- Issue
- 8
- Pages
- e027743
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC6688675
- OpenAlex
- W2965593496
- Cited by
- 218
- References
- 31
- Citations by year
- 2026: 28 · 2025: 24 · 2024: 27 · 2023: 46 · 2022: 37 · 2021: 27 · 2020: 28 · 2019: 1
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMJ Open (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Health Services and Delivery Research Programme
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- telephone triage, diagnostic accuracy, patient safety, patient satisfaction, health equity, digital health services
- MeSH terms
- England, Health Services Accessibility, Health Services Research, Humans, Quality of Health Care, Triage, Hotlines, Telemedicine, Information Seeking Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
