Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review
Open accessAbstract
Objectives: In England, the NHS111 service provides assessment and triage by telephone for urgent health problems. A digital version of this service has recently been introduced. We aimed to systematically review the evidence on digital and online symptom checkers and similar services. Design: Systematic review. Data sources: We searched Medline, Embase, the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Health Management Information Consortium, Web of Science and ACM Digital Library up to April 2018, supplemented by phrase searches for known symptom checkers and citation searching of key studies. Eligibility criteria: Studies of any design that evaluated a digital or online symptom checker or health assessment service for people seeking advice about an urgent health problem. Data extraction and synthesis: Data extraction and quality assessment (using the Cochrane Collaboration version of QUADAS for diagnostic accuracy studies and the National Heart, Lung and Blood Institute tool for observational studies) were done by one reviewer with a sample checked for accuracy and consistency. We performed a narrative synthesis of the included studies structured around pre-defined research questions and key outcomes. Results: We included 29 publications (27 studies). Evidence on patient safety was weak. Diagnostic accuracy varied between different systems but was generally low. Algorithm-based triage tended to be more risk averse than that of health professionals. There was very limited evidence on patients’ compliance with online triage advice. Study participants generally expressed high levels of satisfaction, although in mainly uncontrolled studies. Younger and more highly educated people were more likely to use these services. Conclusions: The English ‘digital 111’ service has been implemented against a background of uncertainty around the likely impact on important outcomes. The health system may need to respond to short-term changes and/or shifts in demand. The popularity of online and digital services with younger and more educated people has implications for health equity. PROSPERO registration number: CRD42018093564.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Diagnostic performance of symptom checkers varies significantly between different symptom checkers and has been interpreted as low to moderate at best.
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
The study collected real-world patient data, which comes closer to the current intended laypeople-oriented application of symptom checkers.
“We considered some major limitations of previous studies [32] given that we collected real-world patient data, which comes closer to the current intended laypeople-oriented application of symptom checkers.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548 · see citing works
The effects of integrated care: a systematic review of UK and international evidence
Baxter, S. K., Johnson, M., Chambers, D., Sutton, A., Goyder, E., & Booth, A. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3161-3 · Cited by 694 · cited 2× · see citing works
“Overall strength of the evidence base for key outcomes was assessed using an adaptation of the method described by Baxter et al.(11) This involves classifying evidence as ‘stronger’, ‘weaker’, ‘conflicting’ or ‘insufficient’ based on study numbers and design.”
MethodSelf-citationEvaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 5× · see citing works
“Studies either evaluated a single system or multiple systems.(6) 21 We found only one study of a symptom checker specifically intended for the assessment of children’s symptoms, a development of the Strategy for Off-Site Rapid Triage system for influenza-like illness.”
BackgroundInfluentialElectronic healthcare: a study of people's readiness and attitude toward performing self‐diagnosis
Lanseng, E. J., & Andreassen, T. W. (2007) · International Journal of Service Industry Management · doi:10.1108/09564230710778155 · Cited by 172 · see citing works
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 3× · see citing works
“In addition, the students’ complaints were generally common and uncomplicated, a scenario in which symptom checkers performed relatively well in the study by Semigran et al.(21)”
BackgroundResultInfluentialImpact of the urgent care telephone service NHS 111 pilot sites: a controlled before and after study
Turner, J., O’Cathain, A., Knowles, E., & Nicholl, J. P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003451 · Cited by 114 · cited 1× · see citing works
“For example, an evaluation of the NHS111 telephone service at four pilot sites and three control sites found that in its first year the service was not successful in reducing 999 emergency calls or in shifting patients from emergency to urgent care.(5) A recent study of 23 symptom checker algorithms providing diagnost…”
MethodSelf-citationCowie, J., Calveley, E., Bowers, G., & Bowers, J. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15050896 · Cited by 89 · see citing works
Nijland, N., van Gemert-Pijnen, J. E. W. C., Boer, H., Steehouder, M. F., & Seydel, E. R. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.06.002 · Cited by 84 · cited 1× · see citing works
“Patients with low to medium levels of education tended to be motivated toward indirect e-consultation (which involves contact with a health professional via email), mainly to reduce uncertainty.(27) Evidence from included studies suggests that users of e-consultation were more likely to be young,(7) 24–26 employed(20)…”
BackgroundSelf-triage for acute primary care via a smartphone application: Practical, safe and efficient?
Verzantvoort, N. C. M., Teunis, T., Verheij, T. J., & van der Velden, A. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199284 · Cited by 77 · see citing works
Older Adult Experience of Online Diagnosis: Results From a Scenario-Based Think-Aloud Protocol
Luger, T. M., Houston, T. K., & Suls, J. M. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2924 · Cited by 73 · see citing works
Carter, M., Fletcher, E., Sansom, A., Warren, F. C., & Campbell, J. L. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-018688 · Cited by 54 · see citing works
Web-Based Self-Triage of Influenza-Like Illness During the 2009 H1N1 Influenza Pandemic
Kellermann, A. L., Isakov, A. P., Parker, R. M., Handrigan, M. T., & Foldy, S. L. (2010) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.04.005 · Cited by 45 · see citing works
Little, P. S., Stuart, B. L., Andreou, P., McDermott, L. M., Joseph, J. A., Mullee, M., Moore, M. V., Broomfield, S., Thomas, T., & Yardley, L. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009769 · Cited by 45 · see citing works
Yardley, L., Joseph, J., Michie, S., Weal, M., Wills, G., & Little, P. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1599 · Cited by 38 · see citing works
Nijland, N., Cranen, K., Boer, H., van Gemert-Pijnen, J. E. W. C., & Seydel, E. R. (2010) · Journal of Telemedicine and Telecare · doi:10.1258/jtt.2009.001004 · Cited by 37 · cited 3× · see citing works
“A similar level of agreement between algorithm and clinician (39%) was reported by Poote et al,(18) while the system evaluated by Nijland et al advised patients to visit a doctor in 85% of cases, even when the symptoms were appropriate for self-care.(20) The only studies to report clearly equal or superior accuracy of…”
BackgroundMethodInfluentialMarco-Ruiz, L., Bønes, E., de la Asunción, E., Gabarrón, E., Solis, J. C. A., Lee, E., Traver, V., Sato, K., & Bellika, J. G. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.09.002 · Cited by 35 · see citing works
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L. R., Baxter, S. K., Booth, A., & Turner, J. (2019) · Health Services and Delivery Research · doi:10.3310/hsdr07290 · Cited by 35 · cited 2× · see citing works
“See the full report for further details.(9)”
BackgroundSelf-citationA study of automated self-assessment in a primary care student health centre setting
Poote, A., French, D., Dale, J. R., & powell, J. A. (2014) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x14529246 · Cited by 27 · see citing works
Internet Websites for Chest Pain Symptoms Demonstrate Highly Variable Content and Quality
Joury, A., Alshathri, M., Alkhunaizi, M., Jaleesah, N., & Pines, J. M. (2016) · Academic Emergency Medicine · doi:10.1111/acem.13039 · Cited by 27 · see citing works
Sorting out symptoms: design and evaluation of the 'babylon check' automated triage systemPreprint
Middleton, K., Butt, M., Hammerla, N., Hamblin, S., Mehta, K., & Parsa, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02041 · Cited by 19 · cited 1× · see citing works
“Two reports with some overlap of content evaluated the ‘babylon check’ app.(17) 23 Studies were conducted in the USA, UK or other European countries.”
BackgroundWeb-Based Triage in a College Health Setting
Sole, M. L., Stuart, P. L., & Deichen, M. (2006) · Journal of American College Health · doi:10.3200/jach.54.5.289-294 · Cited by 15 · cited 1× · see citing works
“The only exception to the rule was an evaluation carried out at a student health centre.(19) Using data from 59 participants who used the 24/7 WebMed system and who were subsequently treated at the health centre, the study found good agreement between chief complaint, 24/7 WebMed classification and provider diagnosis…”
MethodAre e-Health Web Users Looking for Different Symptom Information Than Callers to Triage Centers?
North, F., Varkey, P., Laing, B. Y., Cha, S. S., & Tulledge‐Scheitel, S. M. (2011) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2010.0120 · Cited by 14 · see citing works
Facilitating patient self-management through telephony and web technologies in seasonal influenza
Nagykaldi, Z. J., Calmbach, W. L., DeAlleaume, L., Temte, J. L., Mold, J. W., & Ryan, J. (2010) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v18i1.749 · Cited by 14 · see citing works
Backman, A.-S., Lagerlund, M., Svensson, T., Blomqvist, P., & Adami, J. (2012) · Emergency Medicine Journal · doi:10.1136/emermed-2011-200464 · Cited by 14 · see citing works
Symptom checkers versus doctors: A prospective, head‐to‐head comparison for cough
Berry, A. C., Berry, N. A., Wang, B., Mulekar, M. S., Melvin, A., Battiola, R. J., Bulacan, F. K., & Berry, B. B. (2019) · The Clinical Respiratory Journal · doi:10.1111/crj.13135 · Cited by 12 · see citing works
Berry, A. C., Cash, B. D., Mulekar, M. S., Wang, B., Melvin, A., & Berry, B. B. (2017) · Gastroenterology · doi:10.1016/s0016-5085(17)32937-2 · Cited by 9 · see citing works
Limitations of Study on Symptom Checkers
Fraser, H., Clamp, S. E., & Wilson, C. J. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.0333 · Cited by 9 · see citing works
Limitations of Study on Symptom Checkers—Reply
Mehrotra, A., Semigran, H. L., Levine, D. M., & Nundy, S. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.0351 · Cited by 4 · see citing works
Feasibility of Web-Based Self-Triage by Parents of Children With Influenza-Like Illness
Price, R. A., Fagbuyi, D. B., Harris, R., Hanfling, D., Place, F., Taylor, T. B., & Kellermann, A. L. (2012) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2013.1573 · Cited by 2 · see citing works
Berry, A. C., Berry, B. B., Nakshabendi, R., Kanar, O., Avalos, D. J., Myers, T., & Cash, B. D. (2016) · Gastroenterology · doi:10.1016/s0016-5085(16)32869-4 · Cited by 2 · see citing works
Berry, A. C., Berry, N. A., Wang, B., Mulekar, M. S., Melvin, A., & Berry, B. B. (2017) · The American Journal of Gastroenterology · doi:10.14309/00000434-201710001-00353 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [10]Reitsma JB, Rutjes AWS, Whiting P, et al; Chapter 9: Assessing methodological quality In: Deeks JJ, Bossuyt PM, Gatsonis C, eds Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 100: The Cochrane Collaboration, 2009.
- [31]Luger TM, Suls J Online health information and intentions to seek healthcare. Psychosomatic Medicine 2011; 73: A59.
Cited by
Sorted by citations
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 189 · see citing works
“[13] The diagnostic accuracy was found to be variable between different platforms and was generally low.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · see citing works
An introduction to digital determinants of health
Chidambaram, S., Jain, B., Jain, U., Mwavu, R., Baru, R., Thomas, B., Greaves, F., Jayakumar, S., Jain, P., Rojo, M., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000346 · Cited by 175 · see citing works
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167 · see citing works
“Previous work has investigated the diagnostic and triage accuracy of symptom checkers (Semigran et al., 2015; Millenson et al., 2018; Chambers et al., 2019; Yu et al., 2019).”
Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 164 · see citing works
“However, symptom checkers often use some proprietary diagnostic algorithms, such as branching logic and Bayesian inference, that are not optimal for processing complicated and sometimes ambiguous user inputs, compromising not only user experience but also diagnostic accuracy [47,49].”
BackgroundChenais, G., Lagarde, E., & Gil-Jardiné, C. (2023) · Journal of Medical Internet Research · doi:10.2196/40031 · Cited by 142 · see citing works
“Multiple proprietary self-sorting apps”
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · see citing works
“Several previous studies [13, 14] into the effectiveness of algorithmic performance have found deficiencies in the diagnostic capabilities and a cautious approach to triage.”
BackgroundMiller, S., Gilbert, S., Virani, V., & Wicks, P. (2020) · JMIR Human Factors · doi:10.2196/19713 · Cited by 117 · see citing works
“More recently, internet-enabled options have emerged such as using a search engine to look up symptoms (Dr Google), high-quality online resources such as NHS Choices [4], symptom checkers [5,6], telehealth consultations by phone or videocall [7], Minute Clinics [8] that can be booked via smartphone, and peer-to-peer n…”
BackgroundAdvancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 112 · see citing works
Martin, A. G., Nateqi, J., Gruarin, S., Munsch, N., Abdarahmane, I., Zobel, M., & Knapp, B. (2020) · Scientific Reports · doi:10.1038/s41598-020-75912-x · Cited by 103 · see citing works
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99 · see citing works
“A key source of these and other reviews [30,31] on symptom checkers is a study by Semigran et al [23] published in 2015.”
BackgroundExploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86 · see citing works
“Very little research has been undertaken to investigate how real-world users actually use these symptom checkers, as identified by recent review articles [1, 8, 46], with one exception [59] testing the Technology Trust Model on users of one OSC and another [85] examining the user experience of OSCs.”
BackgroundMorse, K., Ostberg, N. P., Jones, V. G., & Chan, A. S. (2020) · Journal of Medical Internet Research · doi:10.2196/20549 · Cited by 86 · see citing works
“Symptom checker triage recommendations have been theorized to reduce unnecessary clinic and emergency room visits [5], and a recent study showed that completing a web-based symptom assessment reduced the urgency of the care that patients intended to seek [6].”
BackgroundTriage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review
Sasco, E. R., El-Osta, A. A. R., Alaa, A., Webber, I., Karki, M., Asmar, M. L. E., Purohit, K., Painter, A., & Hayhoe, B. W. J. (2023) · Journal of Medical Internet Research · doi:10.2196/43803 · Cited by 76 · see citing works
“The 2019 systematic review by Chambers et al [12] included any type of publication, including gray literature, but was limited to studies relating to urgent health issues only.”
InfluentialAI in healthcare: A narrative review
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.26997.2 · Cited by 69 · see citing works
Woolley, K. E., Bright, D., Ayres, T., Morgan, F. M., Little, K. E., & Davies, A. R. (2023) · Journal of Medical Internet Research · doi:10.2196/44181 · Cited by 69 · see citing works
“However, 18% (4/22) of reviews reported a possible association between the use of digital health care and education showing that individuals with higher levels of education were more likely to use digital health services than individuals with a lower education level [18,21,29,35].”
InfluentialKnitza, J., Mohn, J., Bergmann, C., Kampylafka, E. I., Hagen, M., Bohr, D., Morf, H., Araujo, E. G., Englbrecht, M., Simón, D., et al. (2021) · Arthritis Research & Therapy · doi:10.1186/s13075-021-02498-8 · Cited by 65 · see citing works
“The results of this study are in line with previous SC analyses [12, 17, 18].”
BackgroundResultJones, O. T., Calanzani, N., Saji, S., Duffy, S. W., Emery, J., Hamilton, W., Singh, H., de Wit, N. J., & Walter, F. M. (2021) · Journal of Medical Internet Research · doi:10.2196/23483 · Cited by 65 · see citing works
“This included manually searching commercial research archives and networks (eg, arXiv [28], Google [29], Microsoft [30], and IBM [31]), reviewing the computer-based technologies identified in 3 recent reviews [19-21], and manually searching for further technologies mentioned in the text or references of the studies an…”
BackgroundMethodDialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 71 · see citing works
“Research shows that although symptom check technologies are highly adopted by most users due to their ease, their diagnostic performance varies significantly between other similar tools and was considered low to moderate at best [19].”
BackgroundJanda, M., Horsham, C., Vagenas, D., Loescher, L. J., Gillespie, N. A., Koh, U., Curiel‐Lewandrowski, C. N., Hofmann‐Wellenhof, R., Halpern, A. C., Whiteman, D. C., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30001-7 · Cited by 63 · see citing works
Carmona, K. A., Chittamuru, D., Kravitz, R. L., Ramondt, S., & Ramírez, A. S. (2022) · Journal of Medical Internet Research · doi:10.2196/36322 · Cited by 54 · see citing works
“Consistent with data on internet-based health-seeking behaviors more generally [49], most Buoy users were middle-aged (or younger), female, and highly educated.”
BackgroundDarley, S., Coulson, T., Peek, N., Moschogianis, S., van der Veer, S. N., Wong, D. C., & Brown, B. C. (2022) · Journal of Medical Internet Research · doi:10.2196/37436 · Cited by 53 · see citing works
“Although symptom checkers [29,42] and patient portals [30,43,44] have been well studied, only a small number of evidence syntheses directly relevant to OCs have been published: a systematic review of 57 articles on delivering “e-consultation” in primary care largely focused on generic stand-alone…”
Ćirković, A. (2020) · Journal of Medical Internet Research · doi:10.2196/18097 · Cited by 52 · see citing works
Pairon, A., Philips, H., & Verhoeven, V. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2022.1040926 · Cited by 48 · see citing works
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 54 · see citing works
“These reviews consistently report that SAA accuracy is relatively low, but note that the variability between the tools is very high 17 – 19 .”
InfluentialGalmiche, S., Rahbé, E., Fontanet, A. L., Dinh, A., Bénézit, F., Lescure, F.-X., & Denis, F. (2020) · Journal of Medical Internet Research · doi:10.2196/22924 · Cited by 47 · see citing works
“A recent systematic review of self-triage symptom checkers for urgent health problems suggested they led to less frequent health care use [13].”
BackgroundBenchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study
Schmieding, M. L., Mörgeli, R., Schmieding, M. A. L., Feufel, M. A., & Balzer, F. (2021) · Journal of Medical Internet Research · doi:10.2196/24475 · Cited by 44 · see citing works
“Despite their popularity, there is no established framework to evaluate the performance of SCs [3,4].”
BackgroundPerlman, A., Vodonos, A., Bak, P., Dreyfuss, M., Leventer‐Roberts, M., Vurembrand, Y., Jeffries, H. E., Fisher, E., Steuerman, Y., Namir, Y., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23197 · Cited by 42 · see citing works
“While the use of digital health technologies among seniors has been reported to be increasing [28], users of digital health applications are still predominantly younger adults [29].”
BackgroundCotte, F., Mueller, T., Gilbert, S., Blümke, B., Multmeier, J., Hirsch, M. C., Wicks, P. J., Wolanski, J., Tutschkow, D., Schade‐Brittinger, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/32340 · Cited by 41 · see citing works
“This observational study addressed an underresearched SAA triage topic in the ED [18].”
BackgroundMahlknecht, A., Engl, A., Piccoliori, G., & Wiedermann, C. J. (2023) · BMC Primary Care · doi:10.1186/s12875-023-02143-0 · Cited by 41 · see citing works
“This somewhat contradicts the perception of several patients who mentioned a potential in this regard (Supplementary Table I a), which was also confirmed by previous studies [15, 16, 25].”
BackgroundResultInfluentialCadamuro, J. (2021) · Diagnostics · doi:10.3390/diagnostics11081399 · Cited by 38 · see citing works
“, performed a systematic review following the same goal [70].”
BackgroundProtocol: Remote care as the ‘new normal’? Multi-site case study in UK general practice
Greenhalgh, T., Shaw, S. E., Nishio, A. A., Booth, A., Byng, R., Clarke, A., Dakin, F., Davies, R., Faulkner, S. D., Hemmings, N., et al. (2022) · NIHR Open Research · doi:10.3310/nihropenres.13289.1 · Cited by 38 · see citing works
Self-citationDelshad, S. D., Dontaraju, V. S., & Chengat, V. (2021) · Cureus · doi:10.7759/cureus.16956 · Cited by 37 · see citing works
“Prior research indicates that online symptom checkers often provide inaccurate or incomplete diagnostic or triage advice for users [8,9].”
BackgroundResultDeterminants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A. (2022) · JMIR Human Factors · doi:10.2196/35219 · Cited by 37 · see citing works
Scheder-Bieschin, J., Blümke, B., de Buijzer, E., Cotte, F., Echterdiek, F., Nacsa, J., Ondrésik, M., Ott, M., Paul, G., Schilling, T., et al. (2021) · JMIR Formative Research · doi:10.2196/28199 · Cited by 36 · see citing works
“A number of studies [9,13,14] have described a range of tools for self-assessment of urgency and triage in the general practice and acute primary care setting, but studies evaluating the handover of history and symptom information to HCPs have not been carried out with real patients.”
BackgroundEl-Osta, A. A. R., Webber, I., Alaa, A., Bagkeris, E., Mian, S., Sharabiani, M. T. A., & Majeed, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053566 · Cited by 30 · see citing works
“13 14 However, OSCs that have high false-negative rates may run similar risks if used by patients with high-risk disease such as cardiac ischaemia, pulmonary embolism or meningitis.”
Applying Artificial Intelligence to Gynecologic Oncology: A Review
Mysona, D., Kapp, D. S., Rohatgi, A. P., Lee, D., Mann, A. K., Tran, P. M. H., Tran, L. K. H., She, J., & Chan, J. K. (2021) · Obstetrical & Gynecological Survey · doi:10.1097/ogx.0000000000000902 · Cited by 42 · see citing works
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L. R., Baxter, S. K., Booth, A., & Turner, J. (2019) · Health Services and Delivery Research · doi:10.3310/hsdr07290 · Cited by 35 · see citing works
“Some parts of this chapter have been reproduced from Chambers et al.(1) © Author(s) (or their employer(s)) 2019.”
MethodSelf-citationLevine, D. M., & Mehrotra, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3287 · Cited by 35 · see citing works
Samhammer, D., Roller, R., Hummel, P., Osmanodja, B., Burchardt, A., Mayrdorfer, M., Duettmann, W., & Dabrock, P. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.1016366 · Cited by 33 · see citing works
“For remote health assessments, systems provide factual diagnostic suggestions in the form of symptom checkers (6).”
BackgroundHennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31 · see citing works
“In contrast to patients’ rather positive perspectives on the usability and utility of symptom checkers, health professionals seem to be more skeptical [25], and symptom checkers have had an inferior performance compared with professional diagnoses in previous studies [32].”
BackgroundMethodResultHo, A. (2023) · doi:10.1093/med/9780197556269.001.0001 · Cited by 33 · see citing works
Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review
Ilicki, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279636 · Cited by 33 · see citing works
“Vignette design lacks external validity Vignette content/mix not representative of real life • Vignettes not representative of physical primary care population [10, 45] • Vignettes covering limited range of conditions [12] • No out-of-sample external validation [45] • Vignettes with clear symptoms and diagnosis [1, 44…”
BackgroundInfluentialFraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/38364 · Cited by 32 · see citing works
“Diagnostic and triage apps designed for patients, often called symptom checkers, have become widely available to the general public over the last decade [7].”
BackgroundRadionova, N., Ög, E., Wetzel, A.-J., Rieger, M. A., & Preiser, C. (2023) · Journal of Medical Internet Research · doi:10.2196/39219 · Cited by 31 · see citing works
“Some authors attested that SCs have a lower accuracy in diagnostics [14,15,37,40,43] and in triage, understood as an evaluation of urgency [15,36,37,40], compared with physicians.”
InfluentialOnline Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · Journal of Medical Internet Research · doi:10.2196/37408 · Cited by 30 · see citing works
“Studies have demonstrated that OSCs often advise contact with health care services for conditions that can be self-managed and thus ‘overtriage’patients, which could result in an increased burden on health care systems [1,2,4,13,21-23].”
Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study
Kopka, M., Scatturin, L., Napierala, H., Fürstenau, D., Feufel, M. A., Balzer, F., & Schmieding, M. L. (2023) · Journal of Medical Internet Research · doi:10.2196/46231 · Cited by 26 · see citing works
“Lastly, users of SCs tend to have higher levels of formal education, which is associated with an increased likelihood of searching for diagnoses online [8,20,24-27].”
BackgroundRedesigning Primary Care: The Emergence of Artificial-Intelligence-Driven Symptom Diagnostic Tools
Wiedermann, C. J., Mahlknecht, A., Piccoliori, G., & Engl, A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13091379 · Cited by 29 · see citing works
“Some studies have highlighted the propensity of these tools to either misdiagnose or lean towards excessively cautious triage recommendations [9,16].”
How to conduct written online consultations with patients in primary care
Bakhai, M., & Atherton, H. (2021) · BMJ · doi:10.1136/bmj.n264 · Cited by 28 · see citing works
Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Ben‐Shabat, N., Sharvit, G., Meimis, B., Joya, D. B., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, ., & Amital, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104897 · Cited by 28 · see citing works
Burzyńska, J., Bartosiewicz, A., & Januszewicz, P. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20020978 · Cited by 28 · see citing works
“There are data indicating that younger people who are in worse health condition seek access to this information more often [10].”
BackgroundUnderstanding the invisible workforce: lessons for general practice from a survey of receptionists
Litchfield, I. J., Burrows, M., Gale, N. K., & Greenfield, S. (2022) · BMC Family Practice · doi:10.1186/s12875-022-01842-4 · Cited by 27 · see citing works
“There is also an increasing use of online symptom checkers which can provide alternative diagnoses and suggest or facilitate a course of action including making an appointment with a GP [79], though early indications are that they tend to be favoured by younger or better educated patients [79].”
BackgroundSun, Y., & Sundar, S. S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519668 · Cited by 28 · see citing works
“Despitethe increasing attention to the transparency issue, several questions remain: First, the current research on explanations from OSCs applies to a single message, which is presented all at once, even when it is interwoven into conversation flow (e.g., [7]).”
BackgroundFaqar-Uz-Zaman, S. F., Anantharajah, L., Baumartz, P., Sobotta, P., Filmann, N., Zmuc, D., Wagner, M., Detemble, C., Sliwinski, S., Marschall, U., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005614 · Cited by 25 · see citing works
“7,24 In a comprehensive audit study, Semigran et al assessed 23 commercial symptom checker apps across 770 standardized patient evaluations and found, that the diagnostic accuracy for the fi rst diagnosis was only 34% and 51.2% for the top 3 diagnoses.”
BackgroundInfluentialSiripurapu, S., Darimireddy, N. K., Chehri, A., Sridhar, B., & Paramkusam, A. V. (2023) · Electronics · doi:10.3390/electronics12030750 · Cited by 26 · see citing works
Kopka, M., Feufel, M. A., Berner, E. S., & Schmieding, M. L. (2023) · Digital Health · doi:10.1177/20552076231194929 · Cited by 23 · see citing works
“, different apps or versions of them are being assessed) and methodological differences, for example, how urgency levels are defined or on which types of complaints the apps are tested.(8) For example, some authors evaluate symptom checkers using vignettes on a broad variety of general symptoms,1,2,9–13 while others f…”
BackgroundMethodTriage Errors in Primary and Pre–Primary Care
Nguyen, H., Meczner, A., Burslam-Dawe, K., & Hayhoe, B. W. J. (2022) · Journal of Medical Internet Research · doi:10.2196/37209 · Cited by 21 · see citing works
“Based on limited evidence, digital symptom checkers have relatively low triage accuracy, with a mean error rate of around 50% [25,30].”
BackgroundChambers, D., Preston, L. R., Clowes, M., Cantrell, A. J., & Goyder, E. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-081934 · Cited by 21 · see citing works
“16”
Self-citationKujala, S., & Hörhammer, I. (2022) · Journal of Medical Internet Research · doi:10.2196/33505 · Cited by 21 · see citing works
“However, the impact of symptom checkers on the urgent care system and the wider scope of health care is not known [1].”
BackgroundInfluentialProposal of a novel Artificial Intelligence Distribution Service platform for healthcare
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K. M., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.36775.1 · Cited by 20 · see citing works
Haase, C. B., Bearman, M., Brodersen, J., Hoeyer, K., & Risør, T. (2020) · Scandinavian Journal of Public Health · doi:10.1177/1403494820980268 · Cited by 19 · see citing works
“The review found little evidence of cost-effectiveness and patient compliance [18].”
BackgroundAI in healthcare: A narrative review
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K. M., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.26997.1 · Cited by 89 · see citing works
Gellert, G. A., Orzechowski, P. M., Price, T. J., Kabat-Karabon, A., Jaszczak, J., Marcjasz, N., Młodawska, A., Kwiecien, A. K., & Kurkiewicz, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1047291 · Cited by 18 · see citing works
Litchfield, I. J., Gale, N. K., Burrows, M., & Greenfield, S. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21298 · Cited by 18 · see citing works
“…advice and signposting to appropriate services for people with urgent health-care problems [82], however it remains under utilised by the most frequent GP attenders namely older or less well-educated patients [83] with evidence of it causing duplication of effort for patients and providers [76,82].”
BackgroundGilbert, S., Fenech, M. E., Upadhyay, S., Wicks, P. J., & Novorol, C. (2021) · Australian Journal of Primary Health · doi:10.1071/py21032 · Cited by 17 · see citing works
Harada, Y., Sakamoto, T., Sugimoto, S., & Shimizu, T. (2024) · JMIR Formative Research · doi:10.2196/53985 · Cited by 17 · see citing works
“Indeed, some symptom checkers have already been used in clinical practice [15-17] and even in national health services, such as the National Health Service 111 system in the United Kingdom [18].”
BackgroundInfluentialKubb, C., & Foran, H. M. (2022) · JMIR Pediatrics and Parenting · doi:10.2196/29618 · Cited by 19 · see citing works
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2024) · JMIR AI · doi:10.2196/46875 · Cited by 15 · see citing works
“For instance, Chambers et al [87] found in 2019 that symptom checkers were much less accurate than physicians.”
BackgroundPerformance of national COVID-19 ‘symptom checkers’: a comparative case simulation study
Mansab, F., Bhatti, S., & Goyal, D. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100187 · Cited by 15 · see citing works
“While self-triage has been used for some years in non-emergency conditions to varying degrees of success, 17 self-triage has never before been used in a pandemic setting and as yet the efficacy and safety has not be formally studied.”
Klinische Entscheidungsfindung mit Künstlicher Intelligenz
Samhammer, D., Beck, S., Budde, K., Burchardt, A., Faber, M., Gerndt, S., Möller, S., Osmanodja, B., Roller, R., & Dabrock, P. (2023) · Essentials · doi:10.1007/978-3-662-67008-8 · Cited by 14 · see citing works
Ahn, H. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037161 · Cited by 14 · see citing works
Kopka, M., & Feufel, M. A. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00096-7 · Cited by 13 · see citing works
“For users, it is crucial that symptom checkers give safe advice and prevent potential harm [18], whereas for healthcare systems overtriage could inflate costs and strain scarce resources (e.g., due to unnecessary emergency department visits) [5, 14].”
BackgroundIslam, S., Shahriyar, R., Agarwala, A., Zaman, M., Ahamed, S. A., Rahman, R., Chowdhury, M. H., Sarker, F., & Mamun, K. A. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02864-5 · Cited by 13 · see citing works
Leaders’ support for using online symptom checkers in Finnish primary care: Survey study
Laukka, E., Kujala, S., Gluschkoff, K., Kanste, O. I., Hörhammer, I., & Heponiemi, T. (2021) · Health Informatics Journal · doi:10.1177/14604582211052259 · Cited by 12 · see citing works
“10 However, there exists limited evidence on how patients react to advice provided by SCs. 9 In our study, leaders identi fi ed several ways through which SCs bene fi t patients, for example, increased accessibility to health services and a lower threshold for care.”
BackgroundInfluentialJudson, T. J., Pierce, L., Tutman, A., Mourad, M., Neinstein, A. B., Shuler, G., Gonzales, R., & Odisho, A. Y. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac161 · Cited by 12 · see citing works
Health Professionals’ Experiences of the Benefits and Challenges of Online Symptom Checkers
Kujala, S., Hörhammer, I., Hänninen-Ervasti, R., & Heponiemi, T. (2020) · Studies in health technology and informatics · doi:10.3233/shti200305 · Cited by 12 · see citing works
“Online symptom checkers and assessment services are used by patients seeking guidance on health problems [1].”
BackgroundInfluentialWetzel, A.-J., Koch, R., Koch, N. N., Klemmt, M., Müller, R., Preiser, C., Rieger, M. A., Rösel, I., Ranisch, R., Ehni, H., et al. (2024) · Digital Health · doi:10.1177/20552076241231555 · Cited by 11 · see citing works
Evaluation of AI-Driven LabTest Checker for Diagnostic Accuracy and Safety: Prospective Cohort Study
Szumilas, D., Ochmann, A., Zięba, K. A., Bartoszewicz, B., Kubrak, A., Makuch, S., Agrawal, S., Mazur, G. J., & Chudek, J. T. (2024) · JMIR Medical Informatics · doi:10.2196/57162 · Cited by 12 · see citing works
“Several studies have assessed the effectiveness and safety of AI-driven symptom checkers, tools designed to aid patients in self-diagnosing symptoms and making informed health care choices [8-10].”
Artificial Intelligence in Digital Self-Diagnosis Tools: A Narrative Overview of Reviews
Mentzou, A., Rogers, A., Carvalho, E., Daly, A., Malone, M., & Kerasidou, X. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100242 · Cited by 12 · see citing works
“6,19 The reviews suggest that independent research and evaluative autonomy should address con fl icts of interest by involving the disclosure of tool architecture and in-built values to allow for informed decision making.”
InfluentialChurruca, K., Ellis, L. A., Pope, C. J., MacLellan, J., Zurynski, Y. A., & Braithwaite, J. (2023) · Digital Health · doi:10.1177/20552076231181201 · Cited by 10 · see citing works
“They are regarded by many as overly ‘risk averse’, advising users to seek healthcare where self-care is appropriate.(5,10) One systematic review of telephone triage services found that the median for appropriate triage advice was 75%, with evidence of under-referral in the range of 0.”
BackgroundLiu, A. W., Odisho, A. Y., Brown, W., Gonzales, R., Neinstein, A. B., & Judson, T. J. (2022) · JMIR Human Factors · doi:10.2196/40064 · Cited by 11 · see citing works
Meczner, A., Cohen, N., Qureshi, A., Reza, M., Sutaria, S., Blount, E., Bagyura, Z., & Malak, T. (2024) · JMIR Formative Research · doi:10.2196/49907 · Cited by 10 · see citing works
“This may also be one of the reasons why systematic reviews on accuracy of SCs show a marked spread of results when comparing individual vignette studies or both clinical and vignette studies [2,37-39].”
ResultHertling, S. F., Loos, F. M., & Graul, I. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18137110 · Cited by 10 · see citing works
Irwin, P., Rehman, S. U., Fealy, S. M., Kornhaber, R. A., Matheson, A., & Cleary, M. L. (2025) · Contemporary Nurse · doi:10.1080/10376178.2025.2459701 · Cited by 9 · see citing works
“Research that incorporates the wider implications of the apps considers patient safety, cost and clinical effectiveness, impact on service use, and patient compliance (Chambers et al., 2019).”
Lammila-Escalera, E., Greenfield, G., Aldakhil, R., Mak, H. M., Sehgal, H., Neves, A. L., Harmon, M. J., Majeed, A., & Hayhoe, B. W. J. (2025) · Journal of Medical Internet Research · doi:10.2196/69528 · Cited by 9 · see citing works
“While past reviews have assessed digital triage accuracy, they did not use kiosks as a delivery method [20,25,26].”
Sexton, V., Dale, J. R., & Atherton, H. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01576-x · Cited by 9 · see citing works
“Findings could also feed into the design of the newly emerging patient self-triage approaches that are being adopted by care services [19, 20], for example the NHS 111Online, which allows patients to self-triage and receive care advice online [21].”
BackgroundVuillaume, L. A., Turpinier, J., Cipolat, L., Arnaud-Dépil-Duval, Dumontier, T., Peschanski, N., Kieffer, Y., Barbat, B., Riquier, T., Dinot, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0277568 · Cited by 9 · see citing works
“Specifically regarding the evaluation of SC, it is important to know that there is no reference method [16, 23, 24].”
BackgroundLorenzo, S. M., Astier-Peña, M. P., & Benejam, T. C. (2021) · Atención Primaria · doi:10.1016/j.aprim.2021.102227 · Cited by 9 · see citing works
Hertling, S. F., Hertling, D., Matziolis, G., Schleußner, E., Loos, F. M., & Graul, I. (2022) · PLoS ONE · doi:10.1371/journal.pone.0267144 · Cited by 9 · see citing works
“Members of the Working Group Young Forum of the German Society for orthopedics and trauma surgery (DGOU) were asked to provide feedback on the validation process’s format, completeness, clarity, and procedure [15, 17].”
MethodChambers, D., Cantrell, A. J., Baxter, S. K., Turner, J., & Booth, A. (2020) · BMC Medicine · doi:10.1186/s12916-020-01580-3 · Cited by 9 · see citing works
Self-citationBan, K., Greenfield, S., Burrows, M., Gale, N. K., & Litchfield, I. J. (2024) · British Journal of General Practice · doi:10.3399/bjgp.2024.0228 · Cited by 8 · see citing works
Preiser, C., Radionova, N., Ög, E., Koch, R., Klemmt, M., Mueller, R., Ranisch, R., Joos, S., & Rieger, M. A. (2024) · JMIR Human Factors · doi:10.2196/57360 · Cited by 8 · see citing works
“For the purpose of this study, we use an operational definition: symptom checkers provide lay users with preliminary diagnoses, give a first sense of urgency, and suggest a course of action [9].”
A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History
Jones, A. M., & Jones, D. R. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.727486 · Cited by 10 · see citing works
“In previous studies such DAs have underperformed physicians in diagnostic accuracy (Semigran et al., 2015, 2016; Millenson et al., 2018; Chambers et al., 2019; Yu et al., 2019).”
BackgroundUser Experience of Symptom Checkers: A Systematic Review
You, Y., Ma, R., & Gui, X. (2022) · PubMed Central · doi:10.48550/arxiv.2208.09100 · Cited by 8 · see citing works
Savolainen, K., & Kujala, S. (2024) · JMIR Human Factors · doi:10.2196/45275 · Cited by 8 · see citing works
“Online symptom checkers are used by people seeking health-related guidance, and these services typically provide an urgent assessment and suggest guidance based on the symptoms reported by the user [11].”
BackgroundSehgal, H. L. K., Greenfield, G., Neves, A. L., Harmon, M., Majeed, A., & Hayhoe, B. W. J. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-084506 · Cited by 7 · see citing works
“17 19 This systematic review seeks to address these critical issues by conducting a rigorous examination of the existing literature, aiming to evaluate the efficacy and safety of self-service kiosk implementation within EDs.”
BackgroundStrehlow, M. C., Alvarez, A., Blomkalns, A. L., Caretta‐Weyer, H. A., Gharahbaghian, L., Imler, D. L., Khan, A., Lee, M. O., Lobo, V. E., Newberry, J. A., et al. (2024) · Academic Emergency Medicine · doi:10.1111/acem.14962 · Cited by 7 · see citing works
Digital Health Technology Implementation: Is It Effective in a Healthy Healthcare Perspective?
Stein, A. T., Ben, Â. J., Pachito, D. V., Cazella, S. C., van Dongen, J. M., & Bosmans, J. E. (2020) · Springer eBooks · doi:10.1007/978-3-030-59467-1_9 · Cited by 8 · see citing works
Conversational Agents in Health Care—Reply
McGreevey, J. D., Hanson, C. W., & Koppel, R. (2020) · JAMA · doi:10.1001/jama.2020.21518 · Cited by 7 · see citing works
Litchfield, I. J., Gale, N. K., Greenfield, S., Shukla, D., & Burrows, M. (2025) · Frontiers in Health Services · doi:10.3389/frhs.2024.1499847 · Cited by 6 · see citing works
- Authors
- Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J.
- Affiliations
- University of Sheffield (GB)
- Corresponding author
- Chambers, Duncan
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2019-08
- Volume
- 9
- Issue
- 8
- Pages
- e027743
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC6688675
- OpenAlex
- W2965593496
- Cited by
- 218
- References
- 31
- Citations by year
- 2026: 28 · 2025: 24 · 2024: 27 · 2023: 46 · 2022: 37 · 2021: 27 · 2020: 28 · 2019: 1
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMJ Open (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Health Services and Delivery Research Programme
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- telephone triage, diagnostic accuracy, patient safety, patient satisfaction, health equity, digital health services
- MeSH terms
- England, Health Services Accessibility, Health Services Research, Humans, Quality of Health Care, Triage, Hotlines, Telemedicine, Information Seeking Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
