Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The burden on healthcare systems is mounting continuously owing to population growth and aging, overuse of medical services, and the recent COVID-19 pandemic. This overload is also causing reduced healthcare quality and outcomes. One solution gaining momentum is the integration of intelligent self-assessment tools, known as symptom-checkers, into healthcare-providers' systems. To the best of our knowledge, no study so far has investigated the data-gathering capabilities of these tools, which represent a crucial resource for simulating doctors' skills in medical-interviews. OBJECTIVES: The goal of this study was to evaluate the data-gathering function of currently available chatbot symptom-checkers. METHODS: We evaluated 8 symptom-checkers using 28 clinical vignettes from the repository of MSD-Manual case studies. The mean number of predefined pertinent findings for each case was 31.8 ± 6.8. The vignettes were entered into the platforms by 3 medical students who simulated the role of the patient. For each conversation, we obtained the number of pertinent findings retrieved and the number of questions asked. We then calculated the recall-rates (pertinent-findings retrieved out of all predefined pertinent-findings), and efficiency-rates (pertinent-findings retrieved out of the number of questions asked) of data-gathering, and compared them between the platforms. RESULTS: The overall recall rate for all symptom-checkers was 0.32(2,280/7,112;95 %CI 0.31-0.33) for all pertinent findings, 0.37(1,110/2,992;95 %CI 0.35-0.39) for present findings, and 0.28(1140/4120;95 %CI 0.26-0.29) for absent findings. Among the symptom-checkers, Kahun platform had the highest recall rate with 0.51(450/889;95 %CI 0.47-0.54). Out of 4,877 questions asked overall, 2,280 findings were gathered, yielding an efficiency rate of 0.46(95 %CI 0.45-0.48) across all platforms. Kahun was the most efficient tool 0.74 (95 %CI 0.70-0.77) without a statistically significant difference from Your.MD 0.69(95 %CI 0.65-0.73). CONCLUSION: The data-gathering performance of currently available symptom checkers is questionable. From among the tools available, Kahun demonstrated the best overall performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Effect of Emergency Department Crowding on Clinically Oriented Outcomes
Bernstein, S. L., Aronsky, D., Duseja, R., Epstein, S., Handel, D., Hwang, U., McCarthy, M., John McConnell, K., Pines, J. M., Rathlev, N., et al. (2009) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2008.00295.x · Cited by 1,147 · cited 1×
“These, in turn, are causing reduced quality of medical care and worsened patients ’ clinical outcomes [8 – 12].”
BackgroundUse of Telehealth During the COVID-19 Pandemic: Scoping Review
Doraiswamy, S., Abraham, A., Mamtani, R., & Cheema, S. (2020) · Journal of Medical Internet Research · doi:10.2196/24087 · Cited by 698 · cited 2×
“These implications are especially relevant for telemedicine and home-care practices, which are developing and transforming owing to technological advancements and the effects of the COVID-19 pandemic [17,51].”
BackgroundMethodInequality and the health-care system in the USA
Dickman, S. L., Himmelstein, D. U., & Woolhandler, S. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30398-7 · Cited by 656
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 1×
“Although symptom checkers have been investigated in terms of their accuracy for diagnosis and triage [14,23 – 28], no study had investigated their data-gathering function, which is a cornerstone skill for performing an adequate medical interview [29 – 31].”
BackgroundFactors associated with increases in US health care spending, 1996-2013.
Dieleman, J. L., Squires, E., Bui, A. L., Campbell, M., Chapin, A., Hamavid, H., Horst, C., Li, Z., Matyasz, T., Reynolds, A., et al. (2017) · JAMA · doi:10.1001/jama.2017.15927 · Cited by 347 · cited 1×
“The burden on healthcare systems is mounting worldwide owing to an increase in population growth and aging, the rise in disease incidence, the overuse of medical services, and the recent COVID-19 pandemic [1 – 3].”
BackgroundThe Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332 · cited 1×
“Most advanced symptom checkers use a conversational “ chatbot ” format to collect information from the patient and then return an output of likely diagnoses and triage advice [16].”
BackgroundChambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218
Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · cited 1×
“For these reasons most of the studies conducted so far to address diagnostic accuracy have used similar methodology [24 – 27].”
ResultComparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 1×
“For these reasons most of the studies conducted so far to address diagnostic accuracy have used similar methodology [24 – 27].”
ResultAccuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · cited 3×
“One patient-oriented approach gaining popularity is the use of intelligent digital self-assessment tools, known as symptom checkers [14,15].”
BackgroundMethod
Cited by
Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges
Zhang, P., & Kamel Boulos, M. N. (2023) · Future Internet · doi:10.3390/fi15090286 · Cited by 411
“Ben-Shabat et al. demonstrated Kahun’s superior performance compared to a selected set of similar AI symptom checkers [33].”
ResultRoles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · Journal of Medical Internet Research · doi:10.2196/56930 · Cited by 305
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302
Niu, B., & Mvondo, G. F. N. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103562 · Cited by 178
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74
Metadata
- Authors
- Ben-Shabat, N., Sharvit, G., Meimis, B., Ben Joya, D., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, A., & Amital, H.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2022-12
- Volume
- 168
- Pages
- 104897
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W4307815809
- Cited by
- 27
- References
- 11
- FWCI
- 1.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study. https://app.cochat.ai/papers/10.1016/j.ijmedinf.2022.104897
- The paper
- Ben-Shabat, N., Sharvit, G., Meimis, B., Ben Joya, D., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, A., & Amital, H. (2022). Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2022.104897
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.