Comparative analysis of the performance of the large language models DeepSeek-V3, DeepSeek-R1, open AI-O3 mini and open AI-O3 mini high in urology
Open accessAbstract
Objectives: We sought to compare how DeepSeek‑V3, DeepSeek‑R1, OpenAI o3‑mini, and OpenAI o3‑mini high handle urological questions, especially in areas such as benign prostatic enlargement, urinary stones, infections, and guideline updates. The intent was to identify how these text‑creation platforms might aid clinical practice without overlooking potential gaps in accuracy. Methods: A set of 34 routinely asked questions plus 25 queries based on newly revised guidelines was assembled. Six board‑certified urologists independently scored each system’s replies using a five‑point scale. Questions scoring below a set threshold were reintroduced to the same system, accompanied by critiques, to gauge self‑correction. Statistical analyses focused on total scores, percentage of excellent ratings, and improvements after iterative prompting. Results: Across all 59 queries (34 general plus 25 guideline-based), OpenAI o3-mini high recorded the highest median total score (22 [20–24]), significantly outperforming DeepSeek-R1, DeepSeek-V3 and OpenAI o3-mini (all pair-wise p < 0.01). DeepSeek-R1’s accuracy approached that of o3-mini high in patient-counseling items, where their excellent-answer rates were 49% and 57%, respectively. DeepSeek‑V3 achieved solid baseline correctness but made fewer successful corrections on subsequent attempts. Although OpenAI o3‑mini initially produced more concise responses, it showed a surprisingly strong capacity to revise earlier errors. Conclusion: OpenAI o3‑mini high, followed by DeepSeek‑R1, provided the most reliable answers for modern urological concerns, whereas DeepSeek‑V3 exhibited limited adaptability during re‑evaluation. Despite often briefer replies, OpenAI o3‑mini outdid DeepSeek‑V3 in self‑correction. These findings indicate that, when reviewed by a clinician, o3-mini high can serve as a rapid second-opinion tool for outpatient counselling and protocol updates, whereas DeepSeek-R1 may provide a cost-effective alternative in resource-limited settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Empirical research has primarily examined DeepSeek's technical performance, with limited attention to how students interact with the system in real learning contexts.
“empirical research has primarily examined its technical performance, with limited attention to how students interact with the system in real learning contexts [40, 50–52].”
From From adoption to interaction: examining human-GenAI engagement patterns, ethical tensions, and educational trade-offs in nursing education (2026) · cites this paper as [51]
The Methods (Study design) describe testing DeepSeek models on urological questions, and the Conclusion reports their performance scores. This paper therefore exemplifies research on technical performance, but it does not establish that empirical research has primarily focused on performance or that student interactions in real learning contexts have received limited attention. Page numbers were not available in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 46 · Cited by 24
Research works printed but not linked: 4
- [22]Liu A, Feng B, Xue B et al (2024) Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 · arXiv
- [23]Guo D, Yang D, Zhang H et al (2025) Deepseek-r1: incentivizing reasoning capability in Llms via reinforcement learning. ArXiv Preprint arXiv:2501.12948 · arXiv
- [24]Pfister R, Jud H (2025) Understanding and benchmarking artificial intelligence: openAI’s o3 is not AGI. arXiv preprint arXiv:2501.07458 · arXiv
- [25]Garrison E (2024) Memory makes computation universal, remember? arXiv preprint arXiv:2412.17794 · arXiv
- Authors
- Yan, Z., Fan, K.-Q., Zhang, Q., Wu, X., Chen, Y., Wu, X., Yu, T., Su, N., Zou, Y., Chi, H., Xia, L., & Cao, Q.
- Affiliations
- Kunming Medical University (CN)
- Panzhihua Central Hospital (CN)
- Hong Kong Baptist University (HK)
- Monash University (AU)
- Southwest Medical University (CN)
- Kunming University of Science and Technology (CN)
- Corresponding author
- Chi, Hao
- Xia, Liangjing
- Cao, Qiang
- Venue
- World Journal of Urology
- Published
- 2025-07-07
- Volume
- 43
- Issue
- 1
- Pages
- 416-416
- ISSN
- 0724-4983 (ISSN-L)
- 1433-8726
- PMC
- PMC12234633
- OpenAlex
- W4412088642
- Cited by
- 24
- References
- 46
- Citations by year
- 2026: 15 · 2025: 9
- FWCI
- 5.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- World Journal of Urology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Hong Kong Baptist University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, urology, DeepSeek-V3, DeepSeek-R1, benign prostatic enlargement, urolithiasis, urological infections, self-correction, iterative prompting, patient counseling, model comparison
- MeSH terms
- Large Language Models, Humans, Male, Surveys and Questionnaires, Urology, Practice Guidelines as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
