CoChat
Sign Up
Cited by 3Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareAI in cancer detection

Natural Language Processing as Consultation Service Platform or Clinical Decision Support System in Gynecologic Oncology: A Systematic Review

Open access
Rosati, Andrea, Criscione, Massimo, Lilli, Livia, Petrillo, Marco, Capobianco, Giampiero, Patarnello, Stefano, Fagotti, Anna·2026·International Journal of Gynecological Cancer
DOI 10.1016/j.ijgc.2026.104558
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: Natural language processing is emerging as a key application of artificial intelligence in oncology. This systematic review aims to evaluate the performance and methodological frameworks of natural language processing systems in gynecologic oncology. METHODS: We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020 guidelines. MEDLINE, EMBASE, and Web of Science were searched for studies published between January 2015 and February 2025. Outcomes were synthesized across 3 research questions: the accuracy of natural language processing systems used as consultation service platforms; the accuracy of natural language processing systems used as clinical decision support systems; and the benchmarking methodologies applied, including their associated methodological outcomes. Consultation service platforms deliver general medical information, whereas clinical decision support systems provide recommendations that are integrated into the patient's clinical workflow. RESULTS: This review analyzed 12 retrospective studies. Consultation service platforms were less accurate than clinicians (60% vs 86.7%) and rated lower in response quality (2.96/5 vs 4.2/5) but outperformed guideline-based answers (1.54/2 vs 1.38/2). In cervical cancer, ChatGPT surpassed experts (7.0 vs 6.1). ChatGPT-4 showed a concordance of 70% with the National Comprehensive Cancer Network and 60% with the European Society of Gynaecological Oncology guidelines in clinical decision support tasks, with an overall recommendation accuracy of 75%. IBM Watson achieved a 72.8% concordance with guidelines. Prompting was applied from 100% to 37.5% across studies. Qualitative benchmarking varied across studies: 83.3% used clinical guidelines and 37.5% of consultation service platforms studies used expert answers. Four- or 5-point scales and binary scoring were used to assess consultation service platforms and clinical decision support systems, respectively. CONCLUSIONS: Clinicians remain superior in complex reasoning, but natural language processing systems demonstrate robust performance in guideline-driven tasks, with advantages in speed, readability, and reproducibility. However, performance declined in nuanced scenarios and among under-represented patient sub-groups. Large language models currently play a supportive rather than substitutive role in gynecologic oncology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijgc.2026.104558

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. Achiam OJ. GPT-4 Technical Report. arXiv. Accessed February 26, 2025. https:// arxiv.org/abs/2303.08774 · arXiv
  2. Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4): 1234-1240. https://doi.org/10.1093/bioinformatics/btz682. · DOI · Paper page
  3. Wu C, Lin W, Zhang X, Zhang Y, Xie W, Wang Y. PMC-LLaMA: toward building open- source language models for medicine. J Am Med Inform Assoc. 2024;31(9): 1833-1843. https://doi.org/10.1093/jamia/ocae045. · DOI · Paper page
  4. Merlino DJ, Brufau SR, Saieed G, et al. Comparative assessment of otolaryngology knowledge among large language models. Laryngoscope. 2025;135(2):629-634. https://doi.org/10.1002/lary.31781. · DOI
  5. Temsah A, Alhasan K, Altamimi I, et al. DeepSeek in healthcare: revealing opportunities and steering challenges of a new open-source artificial intelligence frontier. Cureus. 2025;17(2):e79221. https://doi.org/10.7759/cureus.79221. · DOI · Paper page
  6. Vaid A, Duong SQ, Lampert J, et al. Local large language models for privacy- preserving accelerated review of historic echocardiogram reports. J Am Med Inform Assoc. 2024;31(9):2097-2102. https://doi.org/10.1093/jamia/ocae085. · DOI
  7. Zhong J, Sehgal K, Hickey K, et al. A local large language model pipeline automatically risk stratifies pancreatic cysts for population health management from serial radiology reports. Gastroenterology. 2024;166:S-687. https://doi.org/10.1016/ s0016-5085(24)02038-9. · DOI
  8. Yue L, Yang L. Clinical experience with IBM Watson for Oncology (WFO) for multiple types of cancer patients in China. Ann Oncol. 2017;28:x162. https://doi.org/ 10.1093/annonc/mdx676.024. · DOI
  9. Tavabi N, Pruneski J, Golchin S, et al. Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline. Artif Intell Med. 2024;151:102847. https://doi.org/10.1016/ j.artmed.2024.102847. · DOI
  10. Zeng K, Pan Z, Xu Y, Qu Y. An ensemble learning strategy for eligibility criteria text classification for clinical trial recruitment: algorithm development and validation. JMIR Med Inform. 2020;8(7):e17832. https://doi.org/10.2196/17832. · DOI
  11. Das R, Maheswari K, Siddiqui S, et al. Improved precision oncology question- answering using agentic LLM. medRxiv. February 26, 2025. https://www.medrxiv. org/content/10.1101/2024.09.20.24314076v1 · DOI
  12. Kim AR, Park HA. A question answering chatbot for gastric cancer patients after curative gastrectomy: development and evaluation of user experience and performance. Comput Inform Nurs. 2024;42(11):829-839. https://doi.org/10.1097/ CIN.0000000000001153. · DOI
  13. Xu Q, Xie W, Liao B, et al. Interpretability of clinical decision support systems based on artificial intelligence from technological and medical perspective: a systematic review. J Healthc Eng. 2023;2023:9919269. https://doi.org/10.1155/2023/ 9919269. · DOI
  14. Ingraham NE, Jones EK, King S, et al. Re-aiming equity evaluation in clinical decision support: a scoping review of equity assessments in surgical decision support systems. Ann Surg. 2023;277(3):359-364. https://doi.org/10.1097/ SLA.0000000000005661. · DOI
  15. Omar M, Agbareia R, Glicksberg BS, Nadkarni GN, Klang E. Benchmarking the confidence of large language models in clinical questions. medRxiv. February 26, 2025. https://www.medrxiv.org/content/10.1101/2024.08.11.24311810v2 · DOI
  16. Erdat EC, Kavak EE. Benchmarking LLM chatbots' oncological knowledge with the Turkish Society of Medical Oncology's annual board examination questions. BMC Cancer. 2025;25(1):197. https://doi.org/10.1186/s12885-025-13596-0. · DOI
  17. Haddaway NR, Page MJ, Pritchard CC, McGuinness LA. PRISMA2020: an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Syst Rev. 2022;18(2):e1230. https://doi.org/10.1002/cl2.1230. · DOI · Paper page
  18. Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid-Based Healthc. 2015;13(3):179-187. https://doi.org/10.1097/XEB.0000000000000062. · DOI · Paper page
  19. Lewin S, Booth A, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(suppl 1):2. https://doi.org/10.1186/s13012-017-0688-3. · DOI · Paper page
  20. Anastasio MK, Peters P, Foote J, et al. The doc versus the bot: a pilot study to assess the quality and accuracy of physician and chatbot responses to clinical questions in gynecologic oncology. Gynecol Oncol Rep. 2024;55:101477. https://doi.org/ 10.1016/j.gore.2024.101477. · DOI
  21. Finch L, Broach V, Feinberg J, et al. ChatGPT compared to national guidelines for management of ovarian cancer: did ChatGPT get it right? -a Memorial Sloan Kettering Cancer Center Team Ovary study. Gynecol Oncol. 2024;189:75-79. https://doi.org/10.1016/j.ygyno.2024.07.007. · DOI
  22. Ye Z, Zhang B, Zhang K, et al. An assessment of ChatGPT's responses to frequently asked questions about cervical and breast cancer. BMC Womens Health. 2024; 24(1):482. https://doi.org/10.1186/s12905-024-03320-8. · DOI
  23. Hermann CE, Patel JM, Boyd L, Growdon WB, Aviki E, Stasenko M. Let's chat about cervical cancer: assessing the accuracy of ChatGPT responses to cervical cancer questions. Gynecol Oncol. 2023;179:164-168. https://doi.org/10.1016/ j.ygyno.2023.11.008. · DOI
  24. Patel JM, Hermann CE, Growdon WB, Aviki E, Stasenko M. ChatGPT accurately performs genetic counseling for gynecologic cancers. Gynecol Oncol. 2024;183: 115-119. https://doi.org/10.1016/j.ygyno.2024.04.006. · DOI
  25. Yurtcu E, Ozvural S, Keyif B. Analyzing the performance of ChatGPT in answering inquiries about cervical cancer. Int J Gynaecol Obstet. 2025;168(2):502-507. https://doi.org/10.1002/ijgo.15861. · DOI
  26. Kuerbanjiang W, Peng S, Jiamaliding Y, Yi Y. Performance evaluation of large language models in cervical cancer management based on a standardized questionnaire: comparative study. J Med Internet Res. 2025;27:e63626. https:// doi.org/10.2196/63626. · DOI
  27. Piazza D, Martorana F, Curaba A, et al. The consistency and quality of ChatGPT responses compared to clinical guidelines for ovarian cancer: A Delphi approach. Curr Oncol. 2024;31(5):2796-2804. https://doi.org/10.3390/ curroncol31050212. · DOI
  28. Levin G, Gotlieb W, Ramirez P, Meyer R, Brezinov Y. ChatGPT in a gynaecologic oncology multidisciplinary team tumour board: a feasibility study. BJOG. 2025; 132(1):99-101. https://doi.org/10.1111/1471-0528.17929. · DOI
  29. Ebner F, Hartkopf A, Veselinovic K, et al. A comparison of ChatGPT and multidisciplinary team meeting treatment recommendations in 10 consecutive cervical cancer patients. Cureus. 2024;16(8):e67458. https://doi.org/10.7759/ cureus.67458. · DOI
  30. Zou FW, Tang YF, Liu CY, Ma JA, Hu CH. Concordance study between IBM Watson for oncology and real clinical practice for cervical cancer patients in China: a retrospective analysis. Front Genet. 2020;11:200. https://doi.org/10.3389/ fgene.2020.00200. · DOI
  31. Krückel A, Brückner L, Psilopatis I, Fasching PA, Beckmann MW, Emons J. Evaluation of ChatGPT's potential in tailoring gynecological cancer therapies. Vivo. 2024;38(4):1649-1659. https://doi.org/10.21873/invivo.13614. · DOI
  32. Benary M, Wang XD, Schmidt M, et al. Leveraging large language models for decision support in personalized oncology. JAMA Netw Open. 2023;6(11):e2343689. https://doi.org/10.1001/jamanetworkopen.2023.43689. · DOI · Paper page
  33. Liu C, Liu X, Wu F, Xie M, Feng Y, Hu C. Using artificial intelligence (Watson for oncology) for treatment recommendations amongst Chinese patients with lung cancer: feasibility study. J Med Internet Res. 2018;20(9):e11087. https://doi.org/ 10.2196/11087. · DOI
  34. Pugliese N, Wai-Sun Wong V, Schattenberg JM, et al. Accuracy, reliability, and comprehensibility of ChatGPT-generated medical responses for patients with nonalcoholic fatty liver disease. Clin Gastroenterol Hepatol. 2024;22(4):886-889.e5. https://doi.org/10.1016/j.cgh.2023.08.033. · DOI
  35. Tan RSYC, Lin Q, Low GH, et al. Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting. J Am Med Inform Assoc. 2023;30(10):1657-1664. https://doi.org/10.1093/jamia/ ocad133. · DOI · Paper page
  36. Gulati V, Roy SG, Moawad A, et al. Transcending language barriers: can ChatGPT be the key to enhancing multilingual accessibility in health care? J Am Coll Radiol. 2024;21(12):1888-1895. https://doi.org/10.1016/ j.jacr.2024.05.009. · DOI
  37. Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med Educ. 2023;9:e46885. https://doi.org/10.2196/46885. · DOI · Paper page
  38. Hincapié MA, Gallego JC, Gempeler A, Piñeros JA, Nasner D, Escobar MF. Implementation and usefulness of telemedicine during the COVID-19 pandemic: a scoping review. J Prim Care Community Health. 2020;11:2150132720980612. https://doi.org/10.1177/2150132720980612. · DOI
  39. Monteith S, Glenn T, Geddes J, Whybrow PC, Bauer M. Commercial use of emotion artificial intelligence (AI): implications for psychiatry. Curr Psychiatry Rep. 2022; 24(3):203-211. https://doi.org/10.1007/s11920-022-01330-7. · DOI

Cited by

Sorted by citations

Authors
Rosati, A., Criscione, M., Lilli, L., Petrillo, M., Capobianco, G., Patarnello, S., & Fagotti, A.
Affiliations
  • Università Cattolica del Sacro Cuore (IT)
  • University of Sassari (IT)
  • Agostino Gemelli University Polyclinic (IT)
Corresponding author
  • Rosati, Andrea
  • Criscione, Massimo
  • Lilli, Livia
  • Petrillo, Marco
  • Capobianco, Giampiero
  • Patarnello, Stefano
  • Fagotti, Anna
ORCID
Venue
International Journal of Gynecological Cancer
Publisher
Elsevier BV
Published
2026-04
Volume
36
Issue
4
Pages
104558
ISSN
1048-891X (ISSN-L)
DOI
10.1016/j.ijgc.2026.104558
OpenAlex
W7128367665
Cited by
3
References
5
Citations by year
2026: 3
FWCI
1.89
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Gynecological Cancer (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubliCatt (Università Cattolica del Sacro Cuore) (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
natural language processing, gynecologic oncology, clinical decision support systems, large language models, ChatGPT, IBM Watson, cervical cancer, benchmarking, artificial intelligence in oncology, response quality
MeSH terms
Intelligent Systems, Female, Genital Neoplasms, Female, Gynecology, Humans, Medical Oncology, Natural Language Processing, Referral and Consultation, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review