The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review
Open accessAbstract
Objectives: In England, the NHS111 service provides assessment and triage by telephone for urgent health problems. A digital version of this service has recently been introduced. We aimed to systematically review the evidence on digital and online symptom checkers and similar services. Design: Systematic review. Data sources: We searched Medline, Embase, the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Health Management Information Consortium, Web of Science and ACM Digital Library up to April 2018, supplemented by phrase searches for known symptom checkers and citation searching of key studies. Eligibility criteria: Studies of any design that evaluated a digital or online symptom checker or health assessment service for people seeking advice about an urgent health problem. Data extraction and synthesis: Data extraction and quality assessment (using the Cochrane Collaboration version of QUADAS for diagnostic accuracy studies and the National Heart, Lung and Blood Institute tool for observational studies) were done by one reviewer with a sample checked for accuracy and consistency. We performed a narrative synthesis of the included studies structured around pre-defined research questions and key outcomes. Results: We included 29 publications (27 studies). Evidence on patient safety was weak. Diagnostic accuracy varied between different systems but was generally low. Algorithm-based triage tended to be more risk averse than that of health professionals. There was very limited evidence on patients’ compliance with online triage advice. Study participants generally expressed high levels of satisfaction, although in mainly uncontrolled studies. Younger and more highly educated people were more likely to use these services. Conclusions: The English ‘digital 111’ service has been implemented against a background of uncertainty around the likely impact on important outcomes. The health system may need to respond to short-term changes and/or shifts in demand. The popularity of online and digital services with younger and more educated people has implications for health equity. PROSPERO registration number: CRD42018093564.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“We considered some major limitations of previous studies [32] given that we collected real-world patient data, which comes closer to the current intended laypeople-oriented application of symptom checkers.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548
The effects of integrated care: a systematic review of UK and international evidence
Baxter, S. K., Johnson, M., Chambers, D., Sutton, A., Goyder, E., & Booth, A. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3161-3 · Cited by 745 · cited 2×
“Overall strength of the evidence base for key outcomes was assessed using an adaptation of the method described by Baxter et al.(11) This involves classifying evidence as ‘stronger’, ‘weaker’, ‘conflicting’ or ‘insufficient’ based on study numbers and design.”
MethodSelf-citationEvaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 5×
“Studies either evaluated a single system or multiple systems.(6) 21 We found only one study of a symptom checker specifically intended for the assessment of children’s symptoms, a development of the Strategy for Off-Site Rapid Triage system for influenza-like illness.”
BackgroundInfluentialElectronic healthcare: a study of people's readiness and attitude toward performing self‐diagnosis
Lanseng, E. J., & Andreassen, T. W. (2007) · International Journal of Service Industry Management · doi:10.1108/09564230710778155 · Cited by 177
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 3×
“In addition, the students’ complaints were generally common and uncomplicated, a scenario in which symptom checkers performed relatively well in the study by Semigran et al.(21)”
BackgroundResultInfluentialImpact of the urgent care telephone service NHS 111 pilot sites: a controlled before and after study
Turner, J., O’Cathain, A., Knowles, E., & Nicholl, J. P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003451 · Cited by 114 · cited 1×
“For example, an evaluation of the NHS111 telephone service at four pilot sites and three control sites found that in its first year the service was not successful in reducing 999 emergency calls or in shifting patients from emergency to urgent care.(5) A recent study of 23 symptom checker algorithms providing diagnost…”
MethodSelf-citationCowie, J., Calveley, E., Bowers, G., & Bowers, J. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15050896 · Cited by 89
Nijland, N., van Gemert-Pijnen, J. E. W. C., Boer, H., Steehouder, M. F., & Seydel, E. R. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.06.002 · Cited by 84 · cited 1×
“Patients with low to medium levels of education tended to be motivated toward indirect e-consultation (which involves contact with a health professional via email), mainly to reduce uncertainty.(27) Evidence from included studies suggests that users of e-consultation were more likely to be young,(7) 24–26 employed(20)…”
BackgroundSelf-triage for acute primary care via a smartphone application: Practical, safe and efficient?
Verzantvoort, N. C. M., Teunis, T., Verheij, T. J., & van der Velden, A. W. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199284 · Cited by 77
Older Adult Experience of Online Diagnosis: Results From a Scenario-Based Think-Aloud Protocol
Luger, T. M., Houston, T. K., & Suls, J. M. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2924 · Cited by 73
Research works printed but not linked: 2
- Reitsma JB , Rutjes AWS , Whiting P , et al ; Chapter 9: Assessing methodological quality. In: Deeks JJ , Bossuyt PM , Gatsonis C , eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 100: The Cochrane Collaboration, 2009.
- Luger, 2011, Online health information and intentions to seek healthcare, Psychosomatic Medicine, 73
Cited by
Sorted by citations
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 213
“[13] The diagnostic accuracy was found to be variable between different platforms and was generally low.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183
An introduction to digital determinants of health
Chidambaram, S., Jain, B., Jain, U., Mwavu, R., Baru, R., Thomas, B., Greaves, F., Jayakumar, S., Jain, P., Rojo, M., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000346 · Cited by 175
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167
“Previous work has investigated the diagnostic and triage accuracy of symptom checkers (Semigran et al., 2015; Millenson et al., 2018; Chambers et al., 2019; Yu et al., 2019).”
Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 164
“However, symptom checkers often use some proprietary diagnostic algorithms, such as branching logic and Bayesian inference, that are not optimal for processing complicated and sometimes ambiguous user inputs, compromising not only user experience but also diagnostic accuracy [47,49].”
Background
Metadata
- Authors
- Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J.
- Affiliations
- University of Sheffield (GB)
- Corresponding author
- Chambers, Duncan
- ORCID
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2019-08
- Volume
- 9
- Issue
- 8
- Pages
- e027743
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC6688675
- OpenAlex
- W2965593496
- Cited by
- 218
- References
- 31
- Citations by year
- 2026: 28 · 2025: 24 · 2024: 27 · 2023: 46 · 2022: 37 · 2021: 27 · 2020: 28 · 2019: 1
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMJ Open (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Health Services and Delivery Research Programme
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- telephone triage, diagnostic accuracy, patient safety, patient satisfaction, health equity, digital health services
- MeSH terms
- England, Health Services Accessibility, Health Services Research, Humans, Quality of Health Care, Triage, Hotlines, Telemedicine, Information Seeking Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
