Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Reproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions. OBJECTIVE: This study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools. METHODS: Independent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value. RESULTS: A total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS. CONCLUSIONS: The single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Prevalence and Features of the Polycystic Ovary Syndrome in an Unselected Population
Azziz, R., Woods, K. S., Reyna, R., Key, T. J., Knochenhauer, E. S., & Yildiz, B. O. (2004) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2003-032046 · Cited by 2,712
Azziz, R., Carmina, E., Chen, Z., Dunaif, A., Laven, J. S. E., Legro, R. S., Lizneva, D., Natterson-Horowtiz, B., Teede, H. J., & Yildiz, B. O. (2016) · Nature Reviews Disease Primers · doi:10.1038/nrdp.2016.57 · Cited by 2,005
Teede, H. J., Misso, M. L., Costello, M. F., Dokras, A., Laven, J., Moran, L., Piltonen, T., Norman, R. J., & the International PCOS Network (2018) · Clinical Endocrinology · doi:10.1111/cen.13795 · Cited by 1,823
ESHRE guideline: endometriosis
Becker, C. M., Bokor, A., Heikinheimo, O., Horne, A., Jansen, F., Kiesel, L., King, K., Kvaskoff, M., Nap, A., Petersen, K., et al. (2022) · Human Reproduction Open · doi:10.1093/hropen/hoac009 · Cited by 1,712 · cited 1×
“…the accuracy (agreement between clinician and symptom checker) of 3 symptom checkers for endometriosis, uterine fibroids, and PCOS developed using current medical guidelines (Monash, European Society of Human Reproduction and Embryology, and American Academy of Family Physicians) [24,59,60].”
MethodEskenazi, B., & Warner, M. L. (1997) · Obstetrics and Gynecology Clinics of North America · doi:10.1016/s0889-8545(05)70302-8 · Cited by 1,617
Teede, H. J., Misso, M. L., Costello, M. F., Dokras, A., Laven, J., Moran, L., Piltonen, T., Norman, R. J., International PCOS Network, Andersen, M., et al. (2018) · Human Reproduction · doi:10.1093/humrep/dey256 · Cited by 1,593
ESHRE guideline for the diagnosis and treatment of endometriosis
Kennedy, S., Bergqvist, A., Chapron, C., D’Hooghe, T., Dunselman, G., Greb, R., Hummelshoj, L., Prentice, A., & Saridogan, E. (2005) · Human Reproduction · doi:10.1093/humrep/dei135 · Cited by 1,584 · cited 1×
“Endometriosis is a condition where endometrial tissue is found outside of the uterus and is typically characterized by painful periods, abnormal bleeding, and chronic pelvic pain, among other symptoms [5,14,15].”
BackgroundTeede, H., Deeks, A., & Moran, L. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-41 · Cited by 1,565
Bozdag, G., Mumusoglu, S., Zengin, D., Karabulut, E., & Yildiz, B. O. (2016) · Human Reproduction · doi:10.1093/humrep/dew218 · Cited by 1,526
Criteria, Prevalence, and Phenotypes of Polycystic Ovary Syndrome
Lizneva, D., Suturina, L., Walker, W., Brakta, S., Gavrilova-Jordan, L., & Azziz, R. (2016) · Fertility and Sterility · doi:10.1016/j.fertnstert.2016.05.003 · Cited by 1,358
Research works printed but not linked: 3
- Rawson, JM, 1991, J Reprod Med, 36, 513
- De La Cruz, MSD, 2017, Am Fam Physician, 95, 100
- Pedersen, SD, 2007, Can Fam Physician, 53, 1042
Cited by
Polycystic Ovary Syndrome and the Internet of Things: A Scoping Review
Graca, S., Alloh, F., Lagojda, L., Dallaway, A., Kyrou, I., Randeva, H. S., & Kite, C. (2024) · Healthcare · doi:10.3390/healthcare12161671 · Cited by 14
Digitale Gesundheits-Apps: Wie gelingt die Integration in die medizinische Versorgung
Widmer, A. (2024) · Die Innere Medizin · doi:10.1007/s00108-024-01774-4 · Cited by 12
Wright, P. J., Burts, C., Harmon, C., & Corbett, C. F. (2025) · JMIR Infodemiology · doi:10.2196/68469 · Cited by 8
“…chronic health conditions [66], engagement diminished over time as women with PCOS sought more common ground than the illness experience, such as life stage (eg, student, mother, and retiree) and, therefore, overall health goal (eg, fertility vs risk reduction of comorbidities) [14,23,26,31,40].”
Wickham, A. P., Hewings-Martin, Y., Goddard, F. G. B., Rodgers, A. K., Cunningham, A. C., Prentice, C., Wilks, O., Kaplan, Y. C., Marhol, A., Meczner, A., et al. (2024) · JMIR Formative Research · doi:10.2196/65469 · Cited by 7
Self-citationC-PATH: Conversational Patient Assistance and Triage in Healthcare System
Shi, Q. F., Han, Q., & Soares, C. (2025) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh67620.2025.00016 · Cited by 7
Metadata
- Authors
- Peven, K., Wickham, A. P., Wilks, O., Kaplan, Y. C., Marhol, A., Ahmed, S., Bamford, R., Cunningham, A. C., Prentice, C., Meczner, A., Fenech, M., Gilbert, S., Klepchukova, A., Ponzo, S., & Zhaunova, L.
- Affiliations
- Else Kröner Fresenius Center for Digital Health (DE)
- Corresponding author
- Wickham, Aidan P
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2023-12-05
- Volume
- 11
- Pages
- e46718
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC10731551
- OpenAlex
- W4388478893
- Cited by
- 9
- References
- 78
- Citations by year
- 2026: 4 · 2025: 1 · 2024: 4
- FWCI
- 3.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- endometriosis, uterine fibroids, polycystic ovary syndrome, clinical vignettes, diagnostic accuracy, symptom assessment, delayed diagnosis, general practitioners, women's health, menstrual cycle tracking apps, health equity
- MeSH terms
- Endometriosis, Female, Humans, Leiomyoma, Prevalence, Reproductive Health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.