Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 180

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Safety of patient-facing digital symptom checkers

Open access
Fraser, Hamish, Coiera, Enrico, Wong, David·2018·The Lancet

Abstract

Misdiagnosis by physicians occurs in approximately 5% of outpatients. Computerised diagnostic decision support (CDDS) programmes can help, and interest in this area has increased alongside advances in artificial intelligence and wider availability of clinical data. Originally designed for doctors, CDDS called symptom checkers are designed to directly assist patients by creating differential diagnoses and advising on the need for further care. The health technology company Babylon recently claimed that their Babylon Diagnostic and Triage System outperformed the average human doctor on a subset of the Royal College of General Practitioners exam. They supported this claim with an internal evaluation study, the results of which were met with scepticism because of methodological concerns. In particular, data in the trials were entered by doctors, not the intended lay users, and no statistical significance testing was performed. Comparisons between the Babylon Diagnostic and Triage System and seven doctors were sensitive to outliers; poor performance of just one doctor skewed results in favour of the Babylon Diagnostic and Triage System. Qualitative assessment of diagnosis appropriateness made by three clinicians exhibited high levels of disagreement. Comparison to historical results from a study by Semigran and colleagues produced high scores for the Babylon Diagnostic and Triage System but was potentially biased by unblinded selection of a subset of 30 of 45 test cases. The detailed analysis is shown in the appendix. Babylon is commended for releasing a fairly detailed description of the system development and the three evaluation studies. This is an important first step in determining its performance and safety. Overall, these results suggest that the Babylon Diagnostic and Triage System potentially showed some improvement compared to the average symptom checkers in the Semigran study. However methodological issues mean that any performance improvement is not proven. It is not possible to determine how well the Babylon Diagnostic and Triage System would perform on a broader randomised set of cases or with data entered by patients instead of doctors. Babylon’s study does not offer convincing evidence that its Babylon Diagnostic and Triage System can perform better than doctors in any realistic situation, and there is a possibility that it might perform significantly worse. If this study is the only evidence for the performance of the Babylon Diagnostic and Triage System, then it appears to be early in stage 2 of the STEAD framework (preclinical). Further clinical evaluation is necessary to ensure confidence in patient safety. Similar concerns with the perfo r m­ ance of other CDDS for patients have been reported. Wolf and colleagues showed a high false negative rate in three of four systems designed to detect melanomas from images, which if used in the real world could falsely reassure patients and put their lives at risk. Symptom checkers with significant false negative rates could create similar dangers if used by patients presenting with high risk diseases such as cardiac ischaemia, pulmonary embolism, or meningitis. These cases highlight the urgent need for guidelines on robust evaluation of CDDS directed at patients for safety, efficacy, effectiveness, and cost. Such guidelines should form the basis of a regulatory framework, as there is currently minimal regulatory oversight of these technologies. Without such structure, commercial entities have little incentive to develop Published Online November 6, 2018 http://dx.doi.org/10.1016/ S0140­6736(18)32819­8

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s0140-6736(18)32819-8

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Cited by

See all 180

Metadata

Authors
Fraser, H., Coiera, E., & Wong, D.
Venue
The Lancet
Publisher
Elsevier BV
Published
2018-11
Volume
392
Issue
10161
Pages
2263-2264
ISSN
  • 0099-5355 (ISSN-L)
  • 0140-6736
OpenAlex
W2899560923
Cited by
180
References
6
FWCI
2.96
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Safety of patient-facing digital symptom checkers. https://app.cochat.ai/papers/10.1016/s0140-6736(18)32819-8
The paper
Fraser, H., Coiera, E., & Wong, D. (2018). Safety of patient-facing digital symptom checkers. The Lancet. https://doi.org/10.1016/s0140-6736(18)32819-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.