Artificial Intelligence and Gynecological Oncology: A Comparative Study of ChatGPT Omni and Gemini Pro across Repeated Intervals with Case-Scenario and Open-Ended Queries
Abstract
Introduction: Artificial intelligence (AI) models offer potential benefits in supporting clinical decision-making, diagnosis, and treatment. The study aimed to compare the performance of ChatGPT-4o (Omni) and Gemini Pro in answering clinical questions and case scenarios related to gynecological oncology and to assess the consistency of their long-term responses. Methods: A two-phase comparative analysis was conducted. 700 clinical questions (350 per model) were developed and categorized into open-ended and case-scenario questions. Three months later, the same set of questions was presented again to evaluate any changes in performance for accuracy, completeness, and guideline adherence. Results: Omni outperformed Gemini Pro across all question types (p = 0.001). Omni achieved a mean score of 5.9 for the basic open-ended questions, higher than Gemini, which had 5.1 (p = 0.001). It also maintained a clear advantage in complex, open-ended questions, scoring a mean of 5.6 than Gemini AI’s 4.2 (p = 0.001). Omni scored a mean of 5.7 for basic case scenarios, while Gemini AI lagged with a mean score of 5 (p = 0.001). Omni showed a modest improvement in complex, open-ended queries, with an increase of 0.2 points (+3.57%) (p = 0.001). Omni provided more accurate and comprehensive responses in guideline adherence than Gemini, particularly in complex cases requiring nuanced judgment and adherence to oncology protocols. Its responses aligned with the latest guidelines, including the American Society of Clinical Oncology and the National Comprehensive Cancer Network. Conclusion: Omni is a more reliable and consistent model for answering questions related to gynecological cancers than Gemini. The stability of Omni’s performance over time highlights its potential as an effective tool for clinical applications requiring high accuracy and consistency.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Kaplan et al. found that while temporal stability was modest, consistency remained suboptimal for complex clinical scenarios in gynecologic oncology questions.
“Kaplan et al., evaluating ChatGPT Omni across 700 gynecologic oncology questions at three-month intervals, found that while temporal stability was modest, consistency remained suboptimal for complex clinical scenarios [29].”
Cites
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · see citing works
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234 · see citing works
Lukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · cited 1× · see citing works
“Although Omni could recommend appropriate treatments based on molecular pro fi les for advanced solid tumors, it sometimes deviated from established clinical guidelines in complex scenarios [11, 12].”
Flagged first, then by citations
Cited by
Sorted by citations
Asoglu, H., Kose, S., Kayim, O., Abaci, A., Khatib, G., Vardar, M. A., Kilic Bagir, E., Gumurdulu, D., Kidi, M. M., Bayram, E., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15124451 · Cited by 1 · see citing works
“…represent a particularly vulnerable domain for current LLMs. Kaplan et al., evaluating ChatGPT Omni across 700 gynecologic oncology questions at three-month intervals, found that while temporal stability was modest, consistency remained suboptimal for complex clinical scenarios [29].”
- Authors
- Kaplan, S. T.
- ORCID
- Kaplan, Seckin Tuna
- Venue
- Oncology Research and Treatment
- Publisher
- S. Karger AG
- Published
- 2025-03-12
- Volume
- 48
- Issue
- 6
- Pages
- 325-331
- ISSN
- 2296-5262 (ISSN-L)
- 2296-5270
- OpenAlex
- W4408345247
- Cited by
- 3
- References
- 3
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Oncology Research and Treatment (publishedVersion)
- PubMed (publishedVersion)
- Kocaeli Üniversitesi - AVESİS (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- gynecologic oncology, GPT-4o, Gemini Pro, large language models, medical question answering, case scenarios, guideline adherence, model performance comparison, clinical decision support
- MeSH terms
- Clinical Decision-Making, Generative Artificial Intelligence, Artificial Intelligence, Female, Genital Neoplasms, Female, Gynecology, Humans, Medical Oncology, Surveys and Questionnaires, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
