Diagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Background: A large number of web-based COVID-19 symptom checkers and chatbots have been developed; however, anecdotal evidence suggests that their conclusions are highly variable. To our knowledge, no study has evaluated the accuracy of COVID-19 symptom checkers in a statistically rigorous manner. Objective: The aim of this study is to evaluate and compare the diagnostic accuracies of web-based COVID-19 symptom checkers. Methods: We identified 10 web-based COVID-19 symptom checkers, all of which were included in the study. We evaluated the COVID-19 symptom checkers by assessing 50 COVID-19 case reports alongside 410 non–COVID-19 control cases. A bootstrapping method was used to counter the unbalanced sample sizes and obtain confidence intervals (CIs). Results are reported as sensitivity, specificity, F1 score, and Matthews correlation coefficient (MCC). Results: The classification task between COVID-19–positive and COVID-19–negative for “high risk” cases among the 460 test cases yielded (sorted by F1 score): Symptoma (F1=0.92, MCC=0.85), Infermedica (F1=0.80, MCC=0.61), US Centers for Disease Control and Prevention (CDC) (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Cleveland Clinic (F1=0.40, MCC=0.07), Providence (F1=0.40, MCC=0.05), Apple (F1=0.29, MCC=-0.10), Docyet (F1=0.27, MCC=0.29), Ada (F1=0.24, MCC=0.27) and Your.MD (F1=0.24, MCC=0.27). For “high risk” and “medium risk” combined the performance was: Symptoma (F1=0.91, MCC=0.83) Infermedica (F1=0.80, MCC=0.61), Cleveland Clinic (F1=0.76, MCC=0.47), Providence (F1=0.75, MCC=0.45), Your.MD (F1=0.72, MCC=0.33), CDC (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Apple (F1=0.70, MCC=0.25), Ada (F1=0.42, MCC=0.03), and Docyet (F1=0.27, MCC=0.29). Conclusions: We found that the number of correctly assessed COVID-19 and control cases varies considerably between symptom checkers, with different symptom checkers showing different strengths with respect to sensitivity and specificity. A good balance between sensitivity and specificity was only achieved by two symptom checkers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chu, D. K., Akl, E. A., Duda, S., Solo, K., Yaacoub, S., Schünemann, H. J., Chu, D. K., Akl, E. A., El-harakeh, A., Bognanni, A., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)31142-9 · Cited by 3,843
Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period
Kissler, S. M., Tedijanto, C., Goldstein, E., Grad, Y. H., & Lipsitch, M. (2020) · Science · doi:10.1126/science.abb5793 · Cited by 2,611 · cited 1×
“The importance of social distancing has been highlighted in the COVID-19 pandemic [4,5], the 2009 H1N1 influenza pandemic [6], and the 1918-1919 influenza pandemic [7] and is reviewed in [8].”
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 3×
“Interactive web sources developed to provide web-based diagnoses are sometimes referred to as symptom checkers or chatbots [2,3].”
Ahmed, F., Zviedrite, N., & Uzicanin, A. (2018) · BMC Public Health · doi:10.1186/s12889-018-5446-1 · Cited by 416 · cited 1×
“The importance of social distancing has been highlighted in the COVID-19 pandemic [4,5], the 2009 H1N1 influenza pandemic [6], and the 1918-1919 influenza pandemic [7] and is reviewed in [8].”
Cited by
ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60
Self-citationMahlknecht, A., Engl, A., Piccoliori, G., & Wiedermann, C. J. (2023) · BMC Primary Care · doi:10.1186/s12875-023-02143-0 · Cited by 41
Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31
“General diagnostic and triage advice of specific symptom checkers has been studied for a broad range of general and specialized health problems [26], for example, ophthalmologic [27] or viral diseases [28,29].”
BackgroundKremer, P., Schiebisch, H., Lechner, F., Haase, I., Cirkel, L., Kötter, I., Kuhn, S., Krusche, M., & Knitza, J. (2025) · EULAR Rheumatology Open · doi:10.1016/j.ero.2025.04.007 · Cited by 17
Peven, K., Wickham, A. P., Wilks, O., Kaplan, Y. C., Marhol, A., Ahmed, S., Bamford, R., Cunningham, A. C., Prentice, C., Meczner, A., et al. (2023) · JMIR mHealth and uHealth · doi:10.2196/46718 · Cited by 9
Metadata
- Authors
- Munsch, N., Martin, A., Gruarin, S., Nateqi, J., Abdarahmane, I., Weingartner-Ortner, R., & Knapp, B.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-10-06
- Volume
- 22
- Issue
- 10
- Pages
- e21299
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W3089650935
- Cited by
- 78
- References
- 5
- FWCI
- 3.87
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Diagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study. https://app.cochat.ai/papers/10.2196/21299
- The paper
- Munsch, N., Martin, A., Gruarin, S., Nateqi, J., Abdarahmane, I., Weingartner-Ortner, R., & Knapp, B. (2020). Diagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study. Journal of Medical Internet Research. https://doi.org/10.2196/21299
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.