Open-source versus proprietary large language models for academic writing in medicine: A controlled evaluation of DeepSeek-R1 and ChatGPT-o1 in neuroradiology

Classified as probably not a substantial claim.

Large language models have achieved a level of fluency and coherence that makes their outputs increasingly indistinguishable from those by humans.

“Large language models (LLMs) have achieved a level of fluency and coherence that makes their outputs increasingly indistinguishable from those by humans.1,2”

• cites [1] Casal JE Kessler M. Can linguists distinguish between ChatGPT/AI and human writing? A study of research ethics and academic publishing. Research Methods in Applied Linguistics 2023; 2(3): 100068. 10.1016/j.rmal.2023.100068 sourceNot checked — no full text of the source
• cites [2] Gao CA Howard FM Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ digital medicine 2023; 6(1): 75. 10.1038/s41746-023-00819-6 37100871 PMC10133283 sourceSupported

May combine two claims.

Prior studies have mostly discussed potential applications and ethical implications of LLMs for scientific writing in medicine overall and, when performance was measured, analyzed often only one (typically proprietary) LLM.

“Prior studies have mostly discussed potential applications and ethical implications of LLMs for scientific writing in medicine overall and – when performance was measured – analyzed often only one (typically proprietary) LLM.12–17”

• cites [12] Cheng HZ Sheng B Lee A, et al. Have AI-generated texts from LLM infiltrated the realm of scientific writing? A large-scale analysis of preprint platforms. bioRxiv 2024-03. 2024.Not checked yet

  Not checked in the free run.

• cites [13] Ahn S. The transformative impact of large language models on medical writing and publishing: current applications, challenges and future directions. The Korean journal of physiology & pharmacology: official journal of the Korean Physiological Society and the Korean Society of Pharmacology 2024; 28(5): 393– 401. 10.4196/kjpp.2024.28.5.393 39198220 PMC11362003Not checked yet

  Not checked in the free run.

• cites [14] Abu-Jeyyab M Alrosan S Alkhawaldeh I. Harnessing large language models in medical research and scientific writing: A closer look to the future: Llms in medical research and scientific writing. High Yield Medical Reviews 2023; 1(2). 10.59707/hymrfbya5348Not checked yet

  Not checked in the free run.

• cites [15] Ramoni D Sgura C Liberale L, et al. Artificial intelligence in scientific medical writing: legitimate and deceptive uses and ethical concerns. European Journal of Internal Medicine 2024; 127: 31– 35. 10.1016/j.ejim.2024.07.012 39048335Not checked yet

  Not checked in the free run.

• cites [16] Hwang T Aggarwal N Khan PZ, et al. Can ChatGPT assist authors with abstract writing in medical journals? Evaluating the quality of scientific abstracts generated by ChatGPT and original abstracts. PLoS One 2024; 19(2): e0297701. 10.1371/journal.pone.0297701 38354135 PMC10866463Not checked yet

  Not checked in the free run.

• cites [17] Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. MDPI, 2023, vol 11, p. 887. 10.3390/healthcare11060887 PMC10048148 36981544 sourcePartly supported

  The Discussion (§§4.1–4.2; page unavailable) reports that sources discussed ChatGPT’s potential uses and ethical concerns in scientific writing. The Results (§3.2; page unavailable) also identify editorials and letters among the included records. But the review does not establish that prior studies of scientific writing mostly took that approach, or that performance studies often tested only one, typically proprietary, LLM. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Open-source models like DeepSeek-R1 offer transparency, local deployment, and the ability to tailor the model to sensitive healthcare data, though they demand more in-house technical expertise.

“Open-source models like DeepSeek-R1 offer transparency, local deployment, and the ability to tailor the model to sensitive healthcare data, though they demand more in-house technical expertise.”

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourcePartly supported

  The Future challenges section says DeepSeek’s open weights and low resource requirements make local deployment and in-house AI training feasible for large healthcare organizations. It also flags missing training code as a transparency concern. The paper does not establish that the model can be tailored to sensitive healthcare data or that doing so demands more in-house technical expertise. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

ChatGPT can produce inaccurate or incomplete information, potentially leading to misunderstandings of complex scientific concepts.

“ChatGPT can produce inaccurate or incomplete information, potentially leading to misunderstandings of complex scientific concepts. 4”

• cites [4] Gilson A Safranek CW Huang T, et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR medical education 2023; 9(1): e45312. 10.2196/45312 36753318 PMC9947764 sourcePartly supported

  The Methods (Model Testing; page unavailable) describe ChatGPT answers that omit key information or contain factual errors, and the Results (Qualitative Breakdown of Responses; page unavailable) report information errors among incorrect answers. The paper does not establish that these errors lead to misunderstandings of complex scientific concepts; that educational consequence is an inference beyond its findings. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A recent nationwide survey in Germany revealed that nearly two-thirds of university students already use AI-based tools in their studies, with ChatGPT being most commonly used.

“A recent nationwide survey in Germany revealed that nearly two-thirds of university students already use Artificial Intelligence (AI)-based tools in their studies, with ChatGPT being most commonly used. 3”

• cites [3] Von Garrel J Mayer J. Artificial Intelligence in studies—use of ChatGPT and AI-based tools among students in Germany. Humanities and social sciences communications 2023; 10(1): 1– 9. 10.1057/s41599-023-02304-7 sourceSupported

May combine two claims.

DeepSeek-R1 respected word limits in 90% of outputs, while ChatGPT-o1 respected them in 50%.

“DeepSeek-R1 respected word limits in 90%, ChatGPT-o1 only in 50%.”

• Also stated in: § Results · plain

  “In binary assessments, DeepSeek-R1 respected the word limit 90% of the time versus 50% for ChatGPT-o1 (Figure 4).”

May combine two claims.

Reviewers identified typical AI stylistic markers in 50% of ChatGPT-o1 outputs and in 10% of DeepSeek-R1 outputs.

“In 50% of cases, reviewers identified typical AI stylistic markers in ChatGPT-o1 outputs (e.g. parenthetical clauses are inserted quite frequently), compared to only 10% for DeepSeek-R1.”

May combine two claims. The quote only partly states this claim.

DeepSeek-R1 showed statistically significant advantages over ChatGPT-o1 in Reasoning Depth, Contextual Coherence, Subtlety, and Evidence Integration using unadjusted p-values.

“Notably, DeepSeek-R1 showed statistically significant advantages in Reasoning Depth, Contextual Coherence, Subtlety, and Evidence Integration.”

May combine two claims.

DeepSeek-R1 produced more concise and better-structured text than ChatGPT-o1.

“DeepSeek-R1 produced more concise and better-structured text.”

May combine two claims.

DeepSeek-R1 respected the prescribed word limit in every instance but one, whereas ChatGPT-o1 met that limit in only half of the cases.

“it respected the prescribed word limit in every instance but one, whereas ChatGPT-o1 met that limit in only half of the cases”

May combine two claims.

In essays, both models performed comparably, with only minor stylistic differences such as phrasing, coherence, and redundancy.

“In essays, both models performed comparably, with only minor stylistic differences such as phrasing, coherence, and redundancy.”

May combine two claims.

DeepSeek-R1 displayed a substantially higher confirmed fabrication rate than ChatGPT-o1, with DOIs consistently resolving to unrelated publications.

“DeepSeek-R1 displayed a substantially higher confirmed fabrication rate, with DOIs consistently resolving to unrelated publications”

May combine two claims.

In Q&A tasks, DeepSeek-R1 more consistently demonstrated advantages in reasoning depth, nuance, and contextual integration, whereas ChatGPT-o1 showed only marginal benefits in readability and citation accuracy.

“in Q&A tasks, DeepSeek-R1 more consistently demonstrated advantages in reasoning depth, nuance, and contextual integration, whereas ChatGPT-o1 showed only marginal benefits in readability and citation accuracy.”

May combine two claims.

Per-task mean scores show DeepSeek-R1 received higher mean ratings in 5 of 10 tasks, ChatGPT-o1 in none, and 5 tasks were tied within ±0.2 mean rating points.

“Per-task mean scores (Supplementary Table S8) show a consistent directional pattern across tasks, with DeepSeek-R1 receiving higher mean ratings in 5 of 10 tasks, ChatGPT-o1 in none, and 5 tasks tied within ±0.2 mean rating points”

May combine two claims.

Across seven binary criteria, both models consistently used references on every task, with no criterion reaching significance in Holm-corrected McNemar tests.

“Across the seven binary criteria, both models consistently used references on every task with no criterion reaching significance in Holm-corrected McNemar tests (Table 3).”

May combine two claims.

In Q&A tasks, DeepSeek-R1 displayed more pronounced advantages, with Nuance and Contextual Coherence reaching the minimum achievable exact p-value at n = 5 pairs (p = 0.0625 each).

“in Q&A tasks DeepSeek-R1 displayed more pronounced advantages, with Nuance and Contextual Coherence reaching the minimum achievable exact p-value at n = 5 pairs (p = 0.0625 each).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The low performance in referencing may reflect broader deficits in analytical depth, structured reasoning, and the ability to integrate and synthesize clinical and scientific evidence, not only difficulties in citation accuracy.

“this may not only reflect difficulties in citation accuracy, but also broader deficits in analytical depth, structured reasoning, and the ability to integrate and synthesize clinical and scientific evidence.”

Classified as probably not a substantial claim. May combine two claims.

Because of the exploratory study design and limited number of paired tasks, unadjusted p-values were used as the primary inferential measure, while Holm correction was additionally explored to assess robustness against multiple testing.

“Because of the exploratory study design and limited number of paired tasks, unadjusted p-values were used as the primary inferential measure, while Holm correction was additionally explored to assess the robustness of findings against multiple testing.”

May combine two claims.

Binary criteria showed substantial overall agreement (κ = 0.746), with the exception of AI Authorship Indicator (κ = 0.105).

“Binary criteria showed substantial overall agreement (κ = 0.746), with the exception of AI Authorship Indicator (κ = 0.105).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both DeepSeek-R1 and ChatGPT-o1 employ chain-of-thought reasoning.

“Both LLMs employ “chain-of-thought”-reasoning, that guides to models to follow and “deep think” a reasoning process when dealing with complex problems.8,10”

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

• cites [10] Wang J. A tutorial on LLM reasoning: Relevant methods behind ChatGPT o1. arXiv preprint arXiv:250210867 2025.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The four criteria with statistically significant differences displayed poor agreement (ICC range 0.035-0.477), and conclusions from these comparisons are treated as exploratory.

“The four criteria with statistically significant differences (Reasoning Depth, Contextual Coherence, Subtlety, and Evidence Integration) displayed poor agreement (ICC range: 0.035–0.477) and conclusions from these comparisons are accordingly treated as exploratory throughout this discussion.”

May combine two claims.

DeepSeek-R1's public weights and modest hardware needs enable institutions to run and refine the model on-site, but undisclosed training data and code leave questions about provenance, reproducibility, and governance.

“Its public weights and modest hardware needs enable institutions to run and refine the model on-site, but undisclosed training data and code leave questions about provenance, reproducibility, and governance. 8”

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

Probably not a substantial claim: may not pass S3 (load-bearing).

One of the evaluated models (ChatGPT-o1) was also used for preliminary topic scoping during study design.

“Seventh, one of the evaluated models (ChatGPT-o1) was also used for preliminary topic scoping during study design.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tasks were limited to neuroradiology topics and two writing formats.

“First, tasks were limited to neuroradiology topics and two writing formats.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the five stylistic criteria with chance-level agreement, findings are reported descriptively only and no comparative conclusions are drawn.

“For the five stylistic criteria with chance-level agreement, findings are reported descriptively only and no comparative conclusions are drawn.”

May combine two claims.

Proprietary models demonstrate strong general performance, but their use is hindered by limitations in transparency, customizability, data privacy, and reproducibility.

“While proprietary models demonstrate strong general performance, their use is hindered by limitations in transparency, customizability, data privacy, and reproducibility.7,8”

• cites [7] Choi WC Chang CI. Advantages and Limitations of Open-Source Versus Commercial Large Language Models (LLMs): A Comparative Study of DeepSeek and OpenAI’s ChatGPT, 2025.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

May combine two claims.

With just two expert raters who co-developed the evaluation instrument, independence assumptions of ICC and Cohen's κ may be affected, individual bias cannot be excluded, and the multidimensional rating scale remains inherently subjective.

“Second, with just two expert raters who co-developed the evaluation instrument, independence assumptions of ICC and Cohen’s κ may be affected, individual bias cannot be excluded, and the multidimensional rating scale, while comprehensive, remains inherently subjective.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Joint development of the evaluation instrument by both raters may affect the independence assumptions underlying ICC and Cohen's κ.

“Joint development of the evaluation instrument by both raters may additionally affect the independence assumptions underlying ICC and Cohen’s κ.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The same two authors (S.E., J.N.) also served as raters, which may introduce confirmation and anchoring bias despite full a priori blinding.

“The same two authors (S.E., J.N.) also served as raters, which may introduce confirmation and anchoring bias despite full a priori blinding.”

May combine two claims.

DeepSeek-R1 combines supervised fine-tuning, reinforcement learning, and iterative refinement.

“DeepSeek-R1 combines supervised fine-tuning, reinforcement learning, and iterative refinement. 18”

• cites [18] Guo D Yang D Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning accessed 28 March 2025]. arXiv preprint arXiv:250112948 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Open-source development fosters reproducibility, community-driven improvements, and equitable access.

“Moreover, open-source development fosters reproducibility, community-driven improvements, and equitable access.7,8”

• cites [7] Choi WC Chang CI. Advantages and Limitations of Open-Source Versus Commercial Large Language Models (LLMs): A Comparative Study of DeepSeek and OpenAI’s ChatGPT, 2025.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On-site, open-source models can lower costs and aid under-resourced regions, but uneven access to high-performance hardware and quality data risks deepening the digital divide.

“While on-site, open-source models can lower costs and aid under-resourced regions – however, uneven access to high-performance hardware and quality data risks deepening the digital divide. 22”

• cites [22] Guo SB Shen Y Meng Y, et al. Surge in large language models exacerbates global regional healthcare inequalities. Journal of Translational Medicine 2025; 23(1): 706. 10.1186/s12967-025-06751-5 40597368 PMC12211289 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

Probably not a substantial claim: may not pass S2 (checkable).

This performance advantage, together with the transparency and adaptability inherent to open-source approaches, underscores their value for the responsible integration of LLMs into academic medicine.

“This performance advantage, together with the transparency and adaptability inherent to open-source approaches, underscores their value for the responsible integration of LLMs into academic medicine.”

Classified as probably not a substantial claim.

The fabricated references were so richly detailed that they likely exploit fluent reader bias.

“both models fabricated hallucinated references so richly detailed that they likely exploit fluent reader bias”

Classified as probably not a substantial claim. May combine two claims.

Responses were blind-scored independently by two radiologists using a 5-point Likert scale applied to 14 qualitative criteria and a binary scale applied to 7 objective criteria.

“Responses were blind-scored independently by using two complementary rating schemes: a 5-point Likert scale applied to 14 qualitative criteria and a binary (yes/no) scale applied to 7 objective criteria.”

Classified as probably not a substantial claim.

All 20 outputs were de-identified using randomized alphanumeric identifiers prior to rating.

“All 20 outputs were de-identified using randomized alphanumeric identifiers prior to rating.”

May combine two claims.

DeepSeek-R1 outperformed ChatGPT-o1 overall for both generating scientific neuroradiology essays and evidence-based answering of short Q&A settings.

“Our findings demonstrate that DeepSeek-R1 outperformed ChatGPT-o1 overall for both generating scientific neuroradiology essays and evidence-based answering of short Q&A settings”

Classified as probably not a substantial claim.

Presentation order was randomized per task.

“Presentation order was randomized per task.”

May combine two claims.

In this controlled evaluation, both tested models produced outputs with reasonable academic quality and fluent linguistic expression, but frequent inaccuracies and unreliable references highlighted clear limitations in factual reliability.

“In this controlled evaluation of academic writing tasks in neuroradiology, both tested models produced outputs with reasonable academic quality and fluent linguistic expression, but frequent inaccuracies and unreliable references highlighted clear limitations in factual reliability.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Ongoing LLM advancements could limit the relevance of our findings for subsequent model versions.

“Last, ongoing LLM advancements could limit the relevance of our findings for subsequent model versions.”

May combine two claims.

The absence of a human-authored control group means that the low detectability of AI-generated markers should not be interpreted as evidence of indistinguishability from expert human writing, but rather as inconsistent recognition of AI stylistic features within a pairwise LLM-only evaluation.

“Fifth, the absence of a human-authored control group means that the low detectability of AI-generated markers should not be interpreted as evidence of indistinguishability from expert human writing, but rather as inconsistent recognition of AI stylistic features within a pairwise LLM-only evaluation.”

May combine two claims.

Although DeepSeek-R1 outperformed ChatGPT-o1 on key metrics, both models' absolute scores indicate room for improvement, particularly in evidence integration.

“Third, although DeepSeek-R1 outperformed ChatGPT-o1 on key metrics, both models’ absolute scores indicate room for improvement, particularly in evidence integration.”

May combine two claims.

Essay-based tasks predominantly expose stylistic variation, while Q&A tasks more clearly highlight substantive differences in clinical reasoning and evidence use.

“These observations suggest that essay-based tasks predominantly expose stylistic variation, while Q&A tasks more clearly highlight substantive differences in clinical reasoning and evidence use.”

May combine two claims.

Inter-rater reliability was limited for most Likert criteria, and generalizability remains accordingly constrained.

“Inter-rater reliability was limited for most Likert criteria, and generalizability remains accordingly constrained.”

May combine two claims.

Open-source LLMs offer full access to weights and architecture, enabling local deployment, fine-tuning on institutional data, and alignment with privacy and governance requirements.

“These models offer full access to weights and architecture, enabling local deployment, fine-tuning on institutional data, and alignment with privacy and governance requirements.”

• cites [7] Choi WC Chang CI. Advantages and Limitations of Open-Source Versus Commercial Large Language Models (LLMs): A Comparative Study of DeepSeek and OpenAI’s ChatGPT, 2025.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] Peng Y Chen Q Shih G. DeepSeek is open-access and the next AI disrupter for radiology. Radiology Advances 2025; 2(1): umaf009. 10.1093/radadv/umaf009 40837590 PMC12363671 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The academic tasks covered a heterogeneous range of topics, which supports content validity but implies varying performance expectations across topics.

“Sixth, the academic tasks covered a heterogeneous range of topics, which supports content validity but implies varying performance expectations across topics.”

May combine two claims.

The failure mode of fabricated references with DOIs resolving to unrelated publications is not fully captured by holistic quality ratings and is potentially more hazardous than unverifiable citations.

“a failure mode not fully captured by holistic quality ratings and potentially more hazardous than unverifiable citations.”

May combine two claims.

The open-source model DeepSeek-R1 achieved higher overall ratings than the proprietary ChatGPT-o1, particularly in reasoning depth, coherence, subtlety, and evidence integration.

“The open-source model DeepSeek-R1 achieved higher overall ratings than the proprietary ChatGPT-o1, particularly in reasoning depth, coherence, subtlety, and evidence integration”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The evaluation framework was developed a priori by two authors and remained unmodified throughout rating.

“This evaluation framework was developed a priori by two of the authors (S.E., J.N.) and remained unmodified throughout rating.”

Classified as probably not a substantial claim. May combine two claims.

The blinding key was held by a non-rating co-author and released only after independent completion of all evaluations.

“The blinding key was held by a non-rating co-author and released only after independent completion of all evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The indistinguishability of AI-generated writing has given rise to ethical challenges, including academic dishonesty and plagiarism.

“Furthermore, the indistinguishability of AI-generated writing has given rise to ethical challenges, including academic dishonesty and plagiarism.5,6”

• cites [5] Stokel-Walker C. ChatGPT listed as author on research papers: many scientists disapprove, 2023. 36653617 sourceNot checked — no full text of the source
• cites [6] Basic Tutorials. TU Munich ruling on AI-generated essay. https://basic-tutorialscom/news/tu-munich-court-ruling-confirms-rejection-of-ai-essay (2024).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Models like DeepSeek-R1 can be fully audited, locally deployed, and iteratively fine-tuned on institution-specific datasets, showing directional gains in coherence and reasoning depth.

“Models like DeepSeek-R1 can be fully audited, locally deployed, and iteratively fine-tuned on institution-specific datasets, showing directional gains in coherence and reasoning depth.”