Performance of DeepSeek-R1, ChatGPT (GPT-o3-mini), and Gemini 2.0 Flash on German Medical Multiple-Choice Questions: Comparative Evaluation
Open accessPublished version of a preprint
Abstract
Background: Despite the transformative potential of artificial intelligence (AI)–based chatbots in medicine, their implementation is hindered by data privacy and security concerns. DeepSeek offers a conceivable solution through its capability for local offline operations. However, as of 2025, it remains unclear whether DeepSeek can achieve an accuracy comparable to that of conventional, cloud-based AI chatbots. Objective: This study aims to evaluate whether DeepSeek, an AI-based chatbot capable of offline operation, achieves answer accuracy on medical multiple-choice questions (MCQs) comparable to that of leading chatbots (ie, ChatGPT and Gemini) on German medical MCQs, thereby assessing its potential as a privacy-preserving alternative for clinical use. Methods: A total of 200 interdisciplinary MCQs from the German Progress Test Medicine were administered to ChatGPT (GPT-o3-mini), DeepSeek (DeepSeek-R1), and Gemini (Gemini 2.0 Flash). Accuracy was defined as the proportion of correctly solved questions. Overall differences among the 3 models were tested with the Cochran Q test, while pairwise comparisons were conducted using the McNemar test. Subgroup analyses were performed by medical domain (Fisher exact test) and question length (Wilcoxon rank-sum test). An a priori power analysis indicated a minimum sample size of 195 questions. Results: All 3 chatbots surpassed the conventional passing threshold of 60%, with accuracies of 96% (192/200) for DeepSeek, 94% (188/200) for Gemini, and 92.5% (185/200) for ChatGPT. The overall difference among models was not statistically significant (P=.10) nor were pairwise comparisons. However, incorrect responses were significantly associated with longer question length for DeepSeek (P=.049) and ChatGPT (P=.04) but not for Gemini. No significant differences in performance were observed across clinical versus preclinical domains or medical specialties (all P>.05). Conclusions: Overall, DeepSeek demonstrates outstanding performance on German medical MCQs comparable to the widely used chatbots ChatGPT and Gemini. Similar to ChatGPT, DeepSeek’s performance declined with increasing question length, highlighting verbosity as a persistent challenge for large language models. While DeepSeek’s offline capability and lower operational costs are advantageous, its safe and reliable application in clinical contexts requires further investigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Unsupervised or uncontrolled AI use leads to increased dependency, diminished problem-solving, and limited ability to make autonomous clinical choices.
“Studies underway to examine the use of AI with no supervision or control prove that it leads to increased dependency, diminished work in finding a solution to the issue, and limited the ability to make autonomous clinical choices in an expressionless way [16,17].”
From HOW AI TOOLS (LIKE CHATGPT/GEMINI) AFFECT CRITICAL THINKING IN NURSING STUDENTS. A SYSTEMATIC REVIEW (2026) · cites this paper as [17]
The Discussion’s Conclusions say that hallucinations and biases make expert oversight and critical appraisal indispensable. They do not report that unsupervised AI use causes dependency, diminished problem-solving, or reduced ability to make autonomous clinical choices. Searches for those outcomes and for supervision found no passage addressing the claimed effects. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1× · see citing works
“In addition, these chatbots exhibit considerable linguistic fluency and empathy in response to patients’ inquiries, although answer quality can vary across medical specialties [8,9].”
Overview of artificial intelligence in medicine
Amisha, Malik, P., Pathania, M., & Rathaur, V. K. (2019) · Journal of Family Medicine and Primary Care · doi:10.4103/jfmpc.jfmpc_440_19 · Cited by 1,106 · cited 3× · see citing works
“…AI in medical MCQs. DeepSeek-R1, for example, achieved accuracies of 97% on English and Chinese licensing items [1]; 92% to 95% on the Chinese National Medical Licensing Examination, with significant and stable advantages over ChatGPT [2,6]; and more than 90% in oncology [10] and microbiology [11].”
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · cited 1× · see citing works
“In the context of large language models, such errors are commonly referred to as hallucinations , meaning the confident generation of factually incorrect or fabricated information that is not supported by the input or external knowledge [30].”
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302 · see citing works
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110 · see citing works
Self-citationRethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 67 · cited 1× · see citing works
“Many so-called open models are only open-weight , sharing parameters but withholding training and fine-tuning data, a practice termed open-washing [43].”
Flagged first, then by citations
Research works printed but not linked: 27
- [1]Abulibdeh R Celi LA Sejdić E The illusion of safety: a report to the FDA on AI healthcare product approvals PLOS Digit Health 2025 06 5 4 6 e0000866 10.1371/journal.pdig.0000866 40471897 PDIG-D-25-00085 40471897 PMC12140231 · DOI
- [3]Beam AL Drazen JM Kohane IS Leong TY Manrai AK Rubin EJ Artificial intelligence in medicine N Engl J Med 2023 03 30 388 13 1220 1 10.1056/NEJMe2206291 36988598 36988598 · DOI · Paper page
- [4]Thirunavukarasu AJ Ting DS Elangovan K Gutierrez L Tan TF Ting DS Large language models in medicine Nat Med 2023 08 17 29 8 1930 40 10.1038/s41591-023-02448-8 37460753 10.1038/s41591-023-02448-8 37460753 · DOI · Paper page
- [6]Liu J Wang C Liu S Utility of ChatGPT in clinical practice J Med Internet Res 2023 06 28 25 e48568 10.2196/48568 37379067 v25i1e48568 37379067 PMC10365580 · DOI · Paper page
- [7]Meyer A Ruthard J Streichert T Dear ChatGPT – can you teach me how to program an app for laboratory medicine? J Lab Med 2024 5 48 5 197 201 10.1515/labmed-2024-0034 · DOI
- [8]Meyer A Soleman A Riese J Streichert T Comparison of ChatGPT, Gemini, and Le Chat with physician interpretations of medical laboratory questions from an online health forum Clin Chem Lab Med 2024 11 26 62 12 2425 34 10.1515/cclm-2024-0246 38804035 cclm-2024-0246 38804035 · DOI
- [10]Temsah A Alhasan K Altamimi I Jamal A Al-Eyadhy A Malki KH Temsah MH DeepSeek in healthcare: revealing opportunities and steering challenges of a new open-source artificial intelligence frontier Cureus 2025 02 17 2 e79221 10.7759/cureus.79221 39974299 39974299 PMC11836063 · DOI · Paper page
- [12]Peng Y Malin BA Rousseau JF Wang Y Xu Z Xu X Weng C Bian J From GPT to DeepSeek: significant gaps remain in realizing AI in healthcare J Biomed Inform 2025 03 163 104791 10.1016/j.jbi.2025.104791 39938624 S1532-0464(25)00020-6 39938624 PMC12188495 · DOI
- [13]Poo M Reflections on DeepSeek's breakthrough Natl Sci Rev 2025 03 12 3 nwaf044 10.1093/nsr/nwaf044 40041025 nwaf044 40041025 PMC11879125 · DOI
- [14]Kayaalp ME Prill R Sezgin EA Cong T Królikowska A Hirschmann MT DeepSeek versus ChatGPT: multimodal artificial intelligence revolutionizing scientific discovery. From language editing to autonomous content generation-redefining innovation in research and practice Knee Surg Sports Traumatol Arthrosc 2025 05 33 5 1553 6 10.1002/ksa.12628 39936363 39936363 · DOI
- [15]Gibney E China's cheap, open AI model DeepSeek thrills scientists Nature 2025 02 638 8049 13 4 10.1038/d41586-025-00229-6 39849139 10.1038/d41586-025-00229-6 39849139 · DOI
- [17]Karay Y Schauber SK A validity argument for progress testing: examining the relation between growth trajectories obtained by progress tests and national licensing examinations using a latent growth curve approach Med Teach 2018 11 40 11 1123 9 10.1080/0142159X.2018.1472370 29950124 29950124 · DOI
- [18]Alfertshofer M Knoedler S Hoch CC Cotofana S Panayi AC Kauke-Navarro M Tullius SG Orgill DP Austen WG Pomahac B Knoedler L Analyzing question characteristics influencing ChatGPT's performance in 3000 USMLE®-style questions Med Sci Educ 2024 09 28 35 1 257 67 10.1007/s40670-024-02176-9 40144074 PMC11933601 · DOI
- [22]Wickham H Averick M Bryan J Chang W McGowan L François R Grolemund G Hayes A Henry L Hester J Kuhn M Pedersen T Miller E Bache S Müller K Ooms J Robinson D Seidel D Spinu V Takahashi K Vaughan D Wilke C Woo K Yutani H Welcome to the Tidyverse J Open Source Softw 2019 4 43 1686 10.21105/JOSS.01686 · DOI · Paper page
- [31]Tomidis Chatzimanouil MK Goppelt I Zeissig Y Sachs UJ Laass MW Metamizole-induced agranulocytosis (MIA): a mini review Mol Cell Pediatr 2023 08 17 10 1 6 10.1186/s40348-023-00160-8 37589909 10.1186/s40348-023-00160-8 37589909 PMC10435429 · DOI
- [32]Rudin D Roos NJ Duthaler U Krähenbühl S Toxicity of metamizole on differentiating HL60 cells and human neutrophil granulocytes Toxicology 2019 10 01 426 152254 10.1016/j.tox.2019.152254 31356851 S0300-483X(19)30209-4 31356851 · DOI
- [33]Rodrigues Alessi M Gomes HA Lopes de Castro M Terumy Okamoto C Performance of ChatGPT in solving questions from the progress test (Brazilian National Medical Exam): a potential artificial intelligence tool in medical practice Cureus 2024 07 16 7 e64924 10.7759/cureus.64924 39156244 39156244 PMC11330648 · DOI
- [34]Friederichs H Friederichs WJ März Maren ChatGPT in medical school: how successful is AI in progress testing? Med Educ Online 2023 12 28 1 2220920 10.1080/10872981.2023.2220920 37307503 37307503 PMC10262795 · DOI · Paper page
- [35]Liu M Okuhara T Chang X Shirabe R Nishiie Y Okada H Kiuchi T Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis J Med Internet Res 2024 07 25 26 e60807 10.2196/60807 39052324 v26i1e60807 39052324 PMC11310649 · DOI
- [36]Alfertshofer M Hoch CC Funk PF Hollmann K Wollenberg B Knoedler S Knoedler L Sailing the seven seas: a multinational comparison of ChatGPT's performance on medical licensing examinations Ann Biomed Eng 2024 06 52 6 1542 5 10.1007/s10439-023-03338-3 37553555 10.1007/s10439-023-03338-3 37553555 PMC11082010 · DOI
- [37]Mikhail D Farah A Milad J Nassrallah W Mihalache A Milad D Antaki F Balas M Popovic M Feo A Muni RH Keane PA Duval R Performance of DeepSeek-R1 in ophthalmology: an evaluation of clinical decision-making and cost-effectiveness Br J Ophthalmol 2025 08 20 109 9 976 81 10.1136/bjo-2025-327360 40701781 bjo-2025-327360 40701781 · DOI
- [38]Xu P Wu Y Jin K Chen X He M Shi D DeepSeek-R1 outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning Adv Ophthalmol Pract Res 2025 08 5 3 189 95 10.1016/j.aopr.2025.05.001 40678192 S2667-3762(25)00029-0 40678192 PMC12269606 · DOI
- [39]Mondillo G Colosimo S Perrotta A Frattolillo V Masino M Comparative evaluation of advanced AI reasoning models in pediatric clinical decision support: ChatGPT O1 vs. DeepSeek-R1 medRxiv Preprint posted online on January 28, 2025 10.1101/2025.01.27.25321169 · DOI
- [40]Smith J Daily briefing: the pros and cons of DeepSeek. Online ahead of print Nature 2025 01 30 10.1038/d41586-025-00330-w 39890911 10.1038/d41586-025-00330-w 39890911 · DOI
- [41]Meyer A Wetsch WA Steinbicker AU Streichert T Through ChatGPT's eyes: the large language model's stereotypes and what they reveal about healthcare J Med Syst 2025 02 05 49 1 20 10.1007/s10916-025-02159-2 39907718 10.1007/s10916-025-02159-2 39907718 · DOI
- [42]Arrieta A Ugarte M Valle P Parejo JA Segura S o3-mini vs DeepSeek-R1: which one is safer? ArXiv Preprint posted online on January 30, 2025 10.48550/arXiv.2501.18438 · DOI
- [45]Gabriel N Bhatia A Lost in translation: large language models in non-English content analysis ArXiv Preprint posted online on June 12, 2023 10.48550/arXiv.2306.07377 · DOI
Cited by
Sorted by citations
HOW AI TOOLS (LIKE CHATGPT/GEMINI) AFFECT CRITICAL THINKING IN NURSING STUDENTS. A SYSTEMATIC REVIEW
Din, A. L. (2026) · Journal of Medical & Health Sciences Review · doi:10.65035/650ytk03 · Cited by 1 · see citing works
- Authors
- Meyer, A., Karay, Y., Steinbicker, A. U., Streichert, T., & Overbeek, R.
- Affiliations
- University of Cologne (DE)
- University Hospital Cologne (DE)
- Corresponding author
- Meyer, Annika
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-12-18
- Volume
- 9
- Pages
- e77357
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC12757712
- OpenAlex
- W4417474482
- Cited by
- 7
- References
- 7
- Citations by year
- 2026: 7
- FWCI
- 1.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Kölner Universitäts PublikationsServer (Universität zu Köln) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Deutsche Forschungsgemeinschaft
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- DeepSeek-R1, ChatGPT, Gemini 2.0 Flash, large language models, German medical education, data privacy, question length, clinical reasoning, model comparison, medical education assessment, clinical decision support, AI hallucination
- MeSH terms
- Generative Artificial Intelligence, Artificial Intelligence, Educational Measurement, Germany, Humans, Surveys and Questionnaires
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
