Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Background: Pressure on the US health care system has been increasing due to a combination of aging populations, rising health care expenditures, and most recently, the COVID-19 pandemic. Responses to this pressure are hindered in part by reliance on a limited supply of highly trained health care professionals, creating a need for scalable technological solutions. Digital symptom checkers are artificial intelligence–supported software tools that use a conversational “chatbot” format to support rapid diagnosis and consistent triage. The COVID-19 pandemic has brought new attention to these tools due to the need to avoid face-to-face contact and preserve urgent care capacity. However, evidence-based deployment of these chatbots requires an understanding of user demographics and associated triage recommendations generated by a large general population. Objective: In this study, we evaluate the user demographics and levels of triage acuity provided by a symptom checker chatbot deployed in partnership with a large integrated health system in the United States. Methods: This population-based descriptive study included all web-based symptom assessments completed on the website and patient portal of the Sutter Health system (24 hospitals in Northern California) from April 24, 2019, to February 1, 2020. User demographics were compared to relevant US Census population data. Results: A total of 26,646 symptom assessments were completed during the study period. Most assessments (17,816/26,646, 66.9%) were completed by female users. The mean user age was 34.3 years (SD 14.4 years), compared to a median age of 37.3 years of the general population. The most common initial symptom was abdominal pain (2060/26,646, 7.7%). A substantial number of assessments (12,357/26,646, 46.4%) were completed outside of typical physician office hours. Most users were advised to seek medical care on the same day (7299/26,646, 27.4%) or within 2-3 days (6301/26,646, 23.6%). Over a quarter of the assessments indicated a high degree of urgency (7723/26,646, 29.0%). Conclusions: Users of the symptom checker chatbot were broadly representative of our patient population, although they skewed toward younger and female users. The triage recommendations were comparable to those of nurse-staffed telephone triage lines. Although the emergence of COVID-19 has increased the interest in remote medical assessment tools, it is important to take an evidence-based approach to their deployment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 1×
“However, little is known about aggregate triage recommendations generated by a symptom checker used in larger populations, and a number of recent reviews have called for more research to be shared [5,7,8].”
BackgroundFactors associated with increases in US health care spending, 1996-2013.
Dieleman, J. L., Squires, E., Bui, A. L., Campbell, M., Chapin, A., Hamavid, H., Horst, C., Li, Z., Matyasz, T., Reynolds, A., et al. (2017) · JAMA · doi:10.1001/jama.2017.15927 · Cited by 347
The Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332 · cited 1×
“One potential approach is a digital symptom checker, which is an artificial intelligence (AI)–supported software tool that uses a conversational “chatbot” format to ask questions about a patient’s symptoms and returns a list of likely diagnoses to support self-diagnosis and appropriate triage [3].”
MethodChambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · cited 2×
“Symptom checker triage recommendations have been theorized to reduce unnecessary clinic and emergency room visits [5], and a recent study showed that completing a web-based symptom assessment reduced the urgency of the care that patients intended to seek [6].”
Background
Cited by
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · Journal of Medical Internet Research · doi:10.2196/56930 · Cited by 305
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302
Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A. (2022) · JMIR Human Factors · doi:10.2196/35219 · Cited by 37
Siira, E., Tyskbo, D., & Nygren, J. (2024) · BMC Primary Care · doi:10.1186/s12875-024-02516-z · Cited by 29
Designing for Control in Nurse-AI Collaboration During Emergency Medical Calls
Berge, A., Guribye, F., Fotland, S.-L. S., Fonnes, G., Johansen, I. H., & Trattner, C. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596110 · Cited by 20
Metadata
- Authors
- Morse, K. E., Ostberg, N. P., Jones, V. G., & Chan, A. S.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-11-30
- Volume
- 22
- Issue
- 11
- Pages
- e20549
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W3109894209
- Cited by
- 85
- References
- 4
- FWCI
- 2.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.