Performance of Large Language Models on Medical Oncology Examination Questions
Open accessAbstract
Importance: Large language models (LLMs) recently developed an unprecedented ability to answer questions. Studies of LLMs from other fields may not generalize to medical oncology, a high-stakes clinical setting requiring rapid integration of new information. Objective: To evaluate the accuracy and safety of LLM answers on medical oncology examination questions. Design, Setting, and Participants: This cross-sectional study was conducted between May 28 and October 11, 2023. The American Society of Clinical Oncology (ASCO) Oncology Self-Assessment Series on ASCO Connection, the European Society of Medical Oncology (ESMO) Examination Trial questions, and an original set of board-style medical oncology multiple-choice questions were presented to 8 LLMs. Main Outcomes and Measures: The primary outcome was the percentage of correct answers. Medical oncologists evaluated the explanations provided by the best LLM for accuracy, classified the types of errors, and estimated the likelihood and extent of potential clinical harm. Results: Proprietary LLM 2 correctly answered 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random answering). Proprietary LLM 2 outperformed an earlier version, proprietary LLM 1, which correctly answered 89 of 147 questions (60.5%; 95% CI, 52.2%-68.5%; P < .001), and the best open-source LLM, Mixtral-8x7B-v0.1, which correctly answered 87 of 147 questions (59.2%; 95% CI, 50.0%-66.4%; P < .001). The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 questions (93.9%; 95% CI, 88.7%-97.2%). Incorrect responses were most commonly associated with errors in information retrieval, particularly with recent publications, followed by erroneous reasoning and reading comprehension. If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%; 95% CI, 59.7%-94.8%) would have a medium or high likelihood of moderate to severe harm. Conclusions and Relevance: In this cross-sectional study of the performance of LLMs on medical oncology examination questions, the best LLM answered questions with remarkable performance, although errors raised safety concerns. These results demonstrated an opportunity to develop and evaluate LLMs to improve health care clinician experiences and patient care, considering the potential impact on capabilities and safety.
AI snapshot
Written from the paper’s full text.
TL;DR
In a cross-sectional evaluation of 147 medical oncology examination-style multiple-choice questions, ChatGPT-4 (proprietary LLM 2) answered 85.0% correctly, outperforming ChatGPT-3.5 and open-source LLMs. Incorrect answers were uncommon but often involved recent evidence and were rated as having potential for moderate to severe harm if acted on.
Key findings
- Proprietary LLM 2 correctly answered 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random), including 42 of 52 ASCO questions (80.8%), 66 of 75 ESMO questions (88.0%), and 17 of 20 original questions (85.0%).
- Proprietary LLM 1 correctly answered 89 of 147 questions (60.5%), and the best open-source LLM, Mixtral-8x7B-v0.1, correctly answered 87 of 147 (59.2%); BioMistral-7B DARE answered 50 of 147 (33.6%).
- Among proprietary LLM 2’s 22 incorrect answers, 13 of 22 (59.1%) contained minor errors and 9 of 22 (40.9%) had major errors; no hallucinations were observed. Incorrect answers were more common for questions requiring recent publications: 12 of 106 (11.3%) before 2018, 3 of 20 (15.0%) during 2018-2019, and 7 of 21 (33.3%) after 2019.
- Clinicians rated incorrect answers as having medium likelihood of harm in 14 of 22 (63.6%) and high likelihood in 4 of 22 (18.2%); possible harm extent was moderate in 14 of 22 (63.6%) and severe or death in 4 of 22 (18.2%). Overall, 18 of 22 incorrect answers (81.8%) would have a medium or high likelihood of moderate to severe harm if acted upon.
Methods
- Cross-sectional study conducted May 28 to October 11, 2023, using 147 examination-style multiple-choice medical oncology questions: 52 from ASCO, 75 from ESMO (75 of 80 after discarding image-based questions), and 20 original questions developed by three oncologists.
- Evaluated ChatGPT-3.5 and ChatGPT-4 (May 28, 2023 versions) plus six open-source LLMs; questions and answer options were copied and pasted into the user interface, with additional prompting techniques described in the supplement.
- Two oncologists classified prose explanations using a 4-level ordinal error scale, with a third oncologist resolving disagreements; analyses included binomial tests, McNemar tests, Fisher tests, weighted κ, and Wilcoxon rank sum tests in R version 4.3.0.
Limitations
- The authors state that classifications of error severity, cause, and potential harm were subjective, though interrater concordance was high; proprietary LLM training data are private, so inclusion of ASCO/ESMO examinations cannot be ruled out, though similar results on original questions made memorization unlikely; the incorrect-answer analysis had a small sample size; and results may not generalize to future LLMs because models change rapidly.
- Evident from the text: the study assessed multiple-choice examination-style questions rather than real clinical workflows or patient outcomes, and image-based ESMO questions were excluded because of LLM capabilities at the time.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 16 cited sources checked · 0 supported · 0 flagged
- Findingown data
The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 (93.9%) questions.
“The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 questions (93.9%; 95% CI, 88.7%-97.2%).”
Quote in the paperSubstantial claim - Findingown data
Prompt engineering did not enable proprietary LLM 2 to correctly answer any questions it had answered incorrectly without specific prompting.
“Prompt engineering did not enable proprietary LLM 2 to correctly answer any of the questions that were incorrectly answered without specific prompting.”
Quote in the paperNot a substantial claim - Findingown data
If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%) would have a medium or high likelihood of moderate to severe harm.
“If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%; 95% CI, 59.7%-94.8%) would have a medium or high likelihood of moderate to severe harm.”
Quote in the paperSubstantial claim - Findingown data
Proprietary LLM 2 incorrectly answered both questions (2/2, 100%) requiring knowledge after its September 2021 training cutoff.
“Proprietary LLM 2 was trained on data until September 2021, and incorrectly answered both questions (2 of 2 [100%]; 95% CI, 15.8%-100.0%) that required knowledge after that cut-off.”
Quote in the paperSubstantial claim- Rests on an assumption: Proprietary LLM 2 was trained on data until September 2021.
- Findingown data
Proprietary LLM 2's accuracy was similar across ASCO (42/52, 80.8%), ESMO (66/75, 88.0%), and original (17/20, 85.0%) question sets.
“Performance on each of the specific sets of questions was similar, with 42 of 52 on ASCO (80.8%; 95% CI, 67.5%-90.4%; P < .001 vs random answering), 66 of 75 on ESMO (88.0%; 95% CI, 78.4%-94.4%; P < .001), and 17 of 20 on the original questions (85.0%; 95% CI, 62.1%-96.8%; P < .001) (Figure 1).”
Quote in the paperNot a substantial claim
- Findingown data
Twelve of 22 incorrect answers (54.5%) were corrected when proprietary LLM 2 was offered a second attempt.
“Twelve of 22 incorrect answers (54.5%; 95% CI, 32.2%-75.6%) were corrected when proprietary LLM 2 was offered a second attempt.”
Quote in the paperNot a substantial claim - Findingown data
No hallucinations were observed in proprietary LLM 2's responses.
“We did not observe any hallucinations, which is when an LLM writes content that cannot be verified or conflicts with facts.34”
Quote in the paperNot a substantial claim - Findingown data
The prose explanations provided by proprietary LLM 2 were rated by clinicians as correct without errors for 123 of 147 (83.7%) questions.
“The prose provided by proprietary LLM 2 answered most questions correctly without errors when evaluated qualitatively by clinicians according to the scientific and clinical consensus (123 of 147 [83.7%]; 95% CI, 76.7%-89.3%) (Figure 2).”
Quote in the paperSubstantial claim - Interpretationown data
These results demonstrate a current gap between open-source and proprietary LLMs.
“Together, these results demonstrate a current gap between open-source and proprietary LLMs and highlight the need for research on how to pretrain or fine-tune models for specific health care tasks.”
Quote in the paperSubstantial claim - Interpretationown data
The performance of LLMs was comparable with medical oncology trainees.
“The performance of LLMs was comparable with medical oncology trainees.”
Quote in the paperSubstantial claim - Interpretationown data
LLMs performed remarkably well on medical oncology examination-style questions designed for the final stages of training before entering clinical practice.
“In this study, LLMs performed remarkably well on medical oncology examination-style questions designed for the final stages of training before entering clinical practice.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Examination-style questions are a valid proxy for LLM potential when applied to medical oncology more broadly.
- Study design and conductown data
The analysis of incorrect answers was based on a small sample size because proprietary LLM 2 made few errors.
“Third, our analysis of incorrect answers was based on a small sample size because proprietary LLM 2 made few errors.”
Quote in the paperNot a substantial claim - Interpretationown data
If acted upon in clinical practice, most incorrect answers from proprietary LLM 2 displayed a potential for harm.
“If acted upon in clinical practice, most incorrect answers from proprietary LLM 2 displayed a potential for harm.”
Quote in the paperSubstantial claim - Interpretationown data
Proprietary LLM 2's performance (85.0%) was the highest proportion reported on medical examination questions to date.
“Proprietary LLM 2 answered 85.0% of medical oncology questions correctly, the highest proportion reported on medical examination questions to date.”
Quote in the paperSubstantial claim - Interpretationcitation
Errors in retrieving recent information may be intrinsic to current LLMs because pretraining procedures weight all text equally without emphasis on recent or high-quality information.
“Errors in retrieving recent information may be intrinsic to the current LLMs because the pretraining procedures weight all text in the training data equally, without specific emphasis on more recent or high-quality information.27”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [27] Touvron H, Martin L, Stone K,. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online July 18, 2023. doi: 10.48550/arXiv.2307.09288 source
Not checked in the free run.
- Rests on an assumption: Proprietary LLM 2 was trained on data until September 2021.
- Interpretationown data
The incorrect answers raise critical safety concerns particularly relevant to medical oncology, where dynamic evidence informs decisions with high-stakes consequences.
“However, the incorrect answers raise critical safety concerns particularly relevant to medical oncology, where dynamic evidence informs decisions with high-stakes consequences.”
Quote in the paperSubstantial claim - Study design and conductown data
There was high concordance between the two oncologists' ratings of proprietary LLM 2 explanations (weighted κ 83.4%).
“There was a high concordance between clinician ratings (weighted κ 83.4%; 95% CI, 80.0%-87.0%).”
Quote in the paperNot a substantial claim - Premisecitation
Previous research in general medicine found that clinicians mostly rated responses from LLMs as having a low likelihood of harm.
“These results contrast with previous research in general medicine, where clinicians mostly rated responses from LLMs as having a low likelihood of harm.7”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [7] Singhal K, Azizi S, Tu T,. Large language models encode clinical knowledge. Nature. 2023; 620(7972): 172- 180. 2023; 620(7972): 172- 180. doi: 10.1038/s41586-023-06291-2 PMC10396962 37438534 source
Not checked in the free run.
- Classified as probably not a substantial claim. 14
- May combine two claims. 18
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataProprietary LLM 2 required a prompt to select a specific answer in 33 of 147 (22.4%) questions.
“LLMs provided answers in prose to all questions (Box), with proprietary LLM 2 requiring a prompt to select a specific answer in 33 of 147 questions (22.4%).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataMixtral-8x7B-v0.1 was the best open-source LLM, correctly answering 87 of 147 (59.2%) questions; its performance was not significantly different from proprietary LLM 1 but inferior to proprietary LLM 2.
“Mixtral-8x7B-v0.1 was best among open-source LLMs (eFigure in Supplement 1), correctly answering 87 of 147 questions (59.2%; 95% CI, 50.0%-66.4%), which was not significantly different from proprietary LLM 1 (χ21 = 0.021; P > .88) but was inferior to proprietary LLM 2 (χ21 = 23.564; P < .001).”
- Also stated in: § Discussion · plain
“On these examination questions, the best open-source LLM, Mixtral-8x7B-v0.1, was comparable with the proprietary LLM 1 but inferior to proprietary LLM 2.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
May combine two claims.
Findingown dataProprietary LLM 2 correctly answered 125 of 147 (85.0%) medical oncology examination questions, significantly better than random chance.
“Proprietary LLM 2 was most accurate among LLMs, correctly answering 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random answering).”
- Also stated in: § Discussion · plain
“The most recent version, proprietary LLM 2, correctly answered 85.0% of multiple-choice questions and provided accurate written explanations supporting the answers.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
May combine two claims.
Findingown dataIncorrect answers were more common when questions required knowledge of recent publications (P = .02); incorrect rates were 11.3% for pre-2018, 15.0% for 2018-2019, and 33.3% for after 2019.
“Incorrect answers were more common when the questions required knowledge of recent publications (Wilcoxon test P = .02) (Figure 3): 12 of 106 questions (11.3%; 95% CI, 6.0%-18.9%) requiring only information before 2018 were incorrect, compared with 3 of 20 questions during 2018 and 2019 (15.0%; 95% CI, 3.2%-37.9%), and 7 of 21 questions after 2019 (33.3%; 95% CI, 14.6%-57.0%).”
- Also stated in: § Discussion · plain
“The explanations accompanying incorrect answers from proprietary LLM 2 showed errors in information retrieval, especially with recent publications.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
May combine two claims.
Findingown dataProprietary LLM 1 correctly answered 89 of 147 (60.5%) questions, significantly better than random chance but inferior to proprietary LLM 2.
“Proprietary LLM 1 correctly answered 89 of 147 questions (60.5%; 95% CI, 39.5%-67.8%; P < .001 vs random answering) of the questions, which was inferior to proprietary LLM 2 (χ21 = 24.596; P < .001).”
- Also stated in: § Discussion · plain
“Proprietary LLM 2 substantially outperformed its earlier version, proprietary LLM 1.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenWhenever there was a high likelihood of harm, the extent was severe; and low likelihood corresponded to mild extent.
“Whenever there was a high likelihood of harm, the extent was severe, and the same for low likelihood and mild extent.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataClinicians rated the extent of possible harm from incorrect answers as mild in 4 of 22 (18.2%), moderate in 14 of 22 (63.6%), and severe or death in 4 of 22 (18.2%).
“Clinicians rated the extent of possible harm as mild in 4 of 22 questions (18.2%; 95% CI, 5.2%-40.3%), moderate in 14 of 22 questions (63.6%; 95% CI, 43.0%-85.4%), and likely to be severe or cause death in 4 of 22 questions (18.2%; 95% CI, 5.2%-40.3%).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataClinicians rated the likelihood of harm from incorrect answers as medium in 14 of 22 (63.6%) and high in 4 of 22 (18.2%).
“Among incorrect answers, clinicians considered the likelihood of the error causing patient harm in practice to be medium in 14 of 22 questions (63.6%; 95% CI, 43.0%-85.4%) and high in 4 of 22 questions (18.2%; 95% CI, 5.2%-40.3%).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data41 of 147 (27.9%) questions required knowledge of evidence published from 2018 onwards.
“From the entire question set, 41 of 147 (27.9%) required knowledge of evidence published from 2018 onwards.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAmong incorrect answers from proprietary LLM 2, 13 of 22 (59.1%) contained minor errors and 9 of 22 (40.9%) contained major errors.
“Among incorrect answers, 13 of 22 (59.1%; 95% CI, 36.4%-79.3%) contained minor errors, and 9 of 22 (40.9%; 95% CI, 20.7%-63.6%) had major errors.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataThe question set included 147 questions: 52 ASCO, 75 ESMO, and 20 original questions.
“We evaluated LLMs across 147 examination questions, including 52 from ASCO, 75 from ESMO, and 20 original questions (Table; eTable 1 in Supplement 1).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataBioMistral-7B DARE correctly answered 50 of 147 (33.6%) questions, inferior to both proprietary LLM 1 and proprietary LLM 2.
“BioMistral-7B DARE, an LLM tuned for the biomedical domain, answered 50 of 147 questions (33.6%; 95% CI, 26.0%-41.7%) correctly, inferior to both proprietary LLM 1 (χ21 = 22.737; P < .001) and proprietary LLM 2 (χ21 = 60.012; P < .001).”
- Also stated in: § Discussion · plain
“Surprisingly, BioMistral-7B DARE,30 an LLM that pretrained the Mistral foundation model on PubMed Central, underperformed.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
May combine two claims.
PremisecitationAcross other medical subspecialties, proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly, while proprietary LLM 2 answered 57% to 84% correctly.
“This improvement may explain some of the variation in performance observed across examinations from other medical subspecialties, where proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly,6,7,8,9,11,35 while proprietary LLM 2 answered 57% to 84% correctly.10,12”
Quote in the paperNot a substantial claim0 of 8 sources checked- cites [6] Kung TH, Cheatham M, Medenilla A,. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2(2): e0000198. doi: 10.1371/journal.pdig.0000198 PMC9931230 36812645 source
Not checked in the free run.
- cites [7] Singhal K, Azizi S, Tu T,. Large language models encode clinical knowledge. Nature. 2023; 620(7972): 172- 180. 2023; 620(7972): 172- 180. doi: 10.1038/s41586-023-06291-2 PMC10396962 37438534 source
Not checked in the free run.
- cites [8] Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology. 2023; 307(5): e230582. doi: 10.1148/radiol.230582 37191485 source
Not checked in the free run.
- cites [9] Beam K, Sharma P, Kumar B,. Performance of a large language model on practice questions for the neonatal board examination. JAMA Pediatr. 2023; 177(9): 977- 979. doi: 10.1001/jamapediatrics.2023.2373 PMC10352922 37459084 source
Not checked in the free run.
- cites [11] Deebel NA, Terlecki R. ChatGPT performance on the American Urological Association self-assessment study program and the potential influence of artificial intelligence in urologic training. Urology. 2023; 177: 29- 33. doi: 10.1016/j.urology.2023.05.010 37209880 source
Not checked in the free run.
- cites [35] Yalamanchili A, Sengupta B, Song J,. Quality of large language model responses to radiation oncology patient care questions. JAMA Netw Open. 2024; 7(4): e244630. doi: 10.1001/jamanetworkopen.2024.4630 PMC10988356 38564215
Not checked in the free run.
- cites [10] Mihalache A, Huang RS, Popovic MM, Muni RH. Performance of an upgraded artificial intelligence chatbot for ophthalmic knowledge assessment. JAMA Ophthalmol. 2023; 141(8): 798- 800. doi: 10.1001/jamaophthalmol.2023.2754 PMC10346493 37440220 source
Not checked in the free run.
- cites [12] Hoch CC, Wollenberg B, Lüers JC,. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023; 280(9): 4271- 4278. doi: 10.1007/s00405-023-08051-4 PMC10382366 37285018 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataClassifications of error severity, cause, and harm likelihood/extent are subjective, though ratings showed high interrater concordance.
“First, classifications such as the severity and cause of the errors and the likelihood and extent of potential harm are subjective. While our ratings showed high interrater concordance, we acknowledge that others could reasonably disagree with specific ratings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA recent study found that proprietary LLM 1 correctly answered 56.1% (583 of 1040) of questions on the ASCO Oncology Self-Assessment Series.
“Our results extend beyond a recent study36 where proprietary LLM 1 correctly answered 56.1% (583 of 1040) of questions on the ASCO Oncology Self-Assessment Series.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [36] Odabashian R, Bastin D, Jones G,. Assessment of ChatGPT-3.5′s knowledge in oncology: comparative study with ASCO-SEP benchmarks. JMIR AI. 2024; 3(1): e50442. doi: 10.2196/50442 PMC11041475 38875575
Not checked in the free run.
May combine two claims.
Interpretationown dataThese results demonstrate that LLMs contain substantial medical oncology knowledge and can provide written responses demonstrating comprehension, retrieval, and reasoning.
“These results demonstrate that LLMs contain substantial medical oncology knowledge and can provide written responses demonstrating comprehension, retrieval, and reasoning.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Examination-style questions are a valid proxy for LLM potential when applied to medical oncology more broadly.
May combine two claims.
Study design and conductown dataThe training data for proprietary LLMs is private, so contamination by ASCO/ESMO examinations cannot be ruled out; however, similar results on original questions make memorization unlikely.
“Second, the training data used to create proprietary LLM 1 and 2 is private, so we cannot rule out the possibility that training included the ASCO and ESMO examinations. However, we observed similar results on a set of original questions created for the study, making the performance observed in this study unlikely to reflect memorization during LLM training.”
Quote in the paperSubstantial claim- Rests on an assumption: The training data sources of proprietary LLMs are not publicly available and could theoretically include the ASCO and ESMO examinations.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe study followed the STROBE reporting guideline for cross-sectional studies.
“We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe safe use of LLMs in medical oncology may be restricted to low-risk settings or require intensive human oversight.
“These findings suggest that the safe use of LLMs in medical oncology may be restricted to low-risk settings or require intensive human oversight.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Examination-style questions are a valid proxy for LLM potential when applied to medical oncology more broadly.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningWhether results generalize to future LLMs is unknown because LLMs are rapidly improving and results change over time.
“Fourth, we considered popular and performant LLMs available when the manuscript was drafted, so whether results will generalize to future LLMs is unknown, considering that LLMs are rapidly improving and results from LLMs change over time.48”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThe remarkable performance of proprietary LLM 1 and 2 suggests an urgent opportunity to improve the practice of medical oncology.
“In this cross-sectional study of the performance of LLMs or examination-style questions in medical oncology, the remarkable performance of proprietary LLM 1 and 2 suggest an urgent opportunity to improve the practice of medical oncology.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe original questions' answers were justified with references to the literature.
“We justified each answer with references to the literature.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataTwo oncologists classified the prose explanations, with disagreements resolved by a third medical oncologist.
“Two oncologists (I.H. and F.B.) classified the prose that accompanied answers into a 4-level ordinal error scale: wrong with major errors, wrong with minor errors, right with minor errors, and right with no errors.”
- Also stated in: § Data Coding · plain
“Discussion with a third medical oncologist (R.G.) solved disagreements.”
Quote in the paperSubstantial claim- Also stated in: § Data Coding · plain
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationPotential applications of LLMs span the clinician experience, direct patient care, education, and research.
“Potential applications of LLMs span the clinician experience, direct patient care, education, and research.49”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [49] Haupt CE, Marks M. AI-generated medical advice-GPT and beyond. JAMA. 2023; 329(16): 1349- 1350. doi: 10.1001/jama.2023.5321 36972070
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOpen-source LLMs can be customized through pretraining or fine-tuning for specific tasks.
“Open-source LLMs can be customized through pretraining, such as with personal health information or fine-tuning for specific tasks, such as mortality prediction.39”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [39] Jiang LY, Liu XC, Nejatian NP,. Health system-scale language models are all-purpose prediction engines. Nature. 2023; 619(7969): 357- 362. doi: 10.1038/s41586-023-06160-y PMC10338337 37286606
Not checked in the free run.
May combine two claims.
PremisecitationIn 2022, the ESMO examination passing threshold was 53.2% correct answers, with the average candidate achieving 60.7% (SD 11.8%).
“According to the annual Examination Report by ESMO, in 2022, the passing threshold was 53.2% correct answers, with the average candidate achieving 60.7% (standard deviation of 11.8%).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationProposed use cases for LLMs include drafting responses to patient inbox messages, generating clinical reports, and other research and administrative activities.
“Proposed use cases include drafting responses to patient inbox messages,40 generating clinical reports from patient encounters,41 as well as other research and administrative activities.42”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [40] Garcia P, Ma SP, Shah S,. Artificial intelligence-generated draft replies to patient inbox messages. JAMA Netw Open. 2024; 7(3): e243201. doi: 10.1001/jamanetworkopen.2024.3201 PMC10955355 38506805
Not checked in the free run.
- cites [41] Giorgi J, Toma A, Xie R,. WangLab at MEDIQA-Chat 2023: clinical note generation from doctor-patient conversations using large language models. arXiv. Preprint posted online May 3, 2023. doi: 10.48550/arXiv.2305.02220
Not checked in the free run.
- cites [42] Umeton R, Kwok A, Maurya R, GPT-4 in a cancer center—institute-wide deployment challenges and lessons learned. NEJM AI. 2024; 1(4). doi: 10.1056/AIcs2300191
Not checked in the free run.
The paper relies on this without support.
The training data sources of proprietary LLMs are not publicly available and could theoretically include the ASCO and ESMO examinations.
“The training data sources used by proprietary LLMs are not publicly available and could theoretically include these examinations.”
The paper relies on this without support.
Proprietary LLM 2 was trained on data until September 2021.
“Proprietary LLM 2 was trained on data until September 2021, and incorrectly answered both questions (2 of 2 [100%]; 95% CI, 15.8%-100.0%) that required knowledge after that cut-off.”
The authors state this as an assumption.
Examination-style questions are a valid proxy for LLM potential when applied to medical oncology more broadly.
“We evaluated LLMs on examination questions as a proxy for their potential when applied to medical oncology more broadly.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Generative AI can produce incorrect or hallucinated content.
“the reality is that generative AI can produce incorrect [Longwell et al., 2024] or hallucinated content [Maleki et al., 2024].”
From How AI Impacts Skill Formation (2026) · cites this paper as Longwell et al., 2024
LLMs have attained human-level accuracy on medical QA benchmarks.
“Large Language Models (LLMs) have attained human-level accuracy on medical question-answer (QA) benchmarks.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
ASCO answer-option counts exceed the stated question count
MinorThe counts of questions with 4 and 5 answer options add to 53, but the ASCO bank is stated to contain 52 questions.
Reported 52 · recomputed 53
“The bank includes 52 questions, with 49 questions containing 4 multiple-choice options and 4 having 5 options.”
Breast-topic percentage is miscalculated
MinorFifteen of 147 questions is 10.2% to one decimal place, not the reported 10.0%.
Reported 10 · recomputed 10.204082
“| Breast | 6 (4.1) | 9 (6.1) | 0 | 15 (10.0) |”
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Springer series in statistics · doi:10.1007/978-1-4612-4380-9_16 · Cited by 18,704 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
The R Project in Statistical Computing
Ripley, B. D. (2001) · MSOR Connections · doi:10.11120/msor.2001.01010023 · Cited by 2,681 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
American Society of Clinical Oncology
Pochon, S. (2005) · American Journal of Cancer · doi:10.2165/00024669-200504040-00008 · Cited by 1,429 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · see citing works
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.10373 · Cited by 584
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507 · see citing works
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460 · see citing works
Health system-scale language models are all-purpose prediction engines
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06160-y · Cited by 460 · see citing works
Creation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
OpenChat: Advancing Open-source Language Models with Mixed-Quality DataPreprint
Wang, G., Cheng, S., Zhan, X., Li, X.-G., Song, S., & Liu, Y. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.11235 · Cited by 368 · see citing works
AI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
Trends in the approval of cancer therapies by the FDA in the twenty-first century
Scott, E., Baines, A. C., Gong, Y. U. T., Moore, R. V., Pamuk, G. E., Saber, H., Subedee, A., Thompson, M. D., Xiao, W., Pazdur, R., et al. (2023) · Nature Reviews Drug Discovery · doi:10.1038/s41573-023-00723-4 · Cited by 291 · see citing works
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282 · see citing works
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.348 · Cited by 269 · see citing works
Korngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · see citing works
Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages
Garcia, P., Ma, S. P., Shah, S. J., Smith, M., Jeong, Y., Devon-Sand, A., Tai-Seale, M., Takazawa, K., Clutter, D., Vogt, K., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.3201 · Cited by 257 · see citing works
Use of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251 · see citing works
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247 · see citing works
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 206 · see citing works
Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions
Yalamanchili, A., Sengupta, B., Song, J., Lim, S. N., Thomas, T. O., Mittal, B. B., Abazeed, M. E., & Teo, P. T. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.4630 · Cited by 128 · see citing works
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
Applications of large language models in cancer care: current evidence and future perspectives
Iannantuono, G. M., Bracken-Clarke, D., Floudas, C. S., Roselli, M., Gulley, J. L., & Karzai, F. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1268915 · Cited by 115 · see citing works
Coşkun, B., Ocakoğlu, G., Yetemen, M., & Kaygısız, O. (2023) · Urology · doi:10.1016/j.urology.2023.05.040 · Cited by 97 · see citing works
To do no harm — and the most good — with AI in health care
Goldberg, C. B., Adams, L. L., Blumenthal, D., Brennan, P. F., Brown, N., Butte, A. J., Cheatham, M., deBronkart, D., Dixon, J., Drazen, J. M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02853-7 · Cited by 72 · see citing works
Performance of an upgraded artificial intelligence chatbot for ophthalmic knowledge assessment.
Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.2754 · Cited by 67 · see citing works
Deebel, N. A., & Terlecki, R. (2023) · Urology · doi:10.1016/j.urology.2023.05.010 · Cited by 54 · see citing works
Performance of a large language model on practice questions for the neonatal board examination.
Beam, K., Sharma, P., Kumar, B., Wang, C., Brodsky, D., Martin, C. R., & Beam, A. (2023) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2023.2373 · Cited by 43 · see citing works
GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned
Umeton, R., Kwok, A., Maurya, R., Leco, D., Lenane, N., Willcox, J., Abel, G. A., Tolikas, M., & Johnson, J. M. (2024) · NEJM AI · doi:10.1056/aics2300191 · Cited by 37 · see citing works
Reyes‐Aldasoro, C. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173671 · Cited by 33 · see citing works
Quality of ChatGPT Responses to Questions Related to Pancreatic Cancer and its Surgical Care
Moazzam, Z., Cloyd, J. M., Lima, H. A., & Pawlik, T. M. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13777-w · Cited by 33 · see citing works
Giorgi, J., Toma, A., Xie, R., Chen, S., An, K., Zheng, G., & Wang, B. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.36 · Cited by 31 · see citing works
OpenChat: Advancing Open-source Language Models with Mixed-Quality DataPreprint
Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., & Liu, Y. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.32119.80804 · Cited by 29 · see citing works
Assessment of ChatGPT-3.5's Knowledge in Oncology: Comparative Study with ASCO-SEP Benchmarks
Odabashian, R., Bastin, D. J., Jones, G., Manzoor, M., Tangestaniapour, S., Assad, M., Lakhani, S., Odabashian, M., & McGee, S. F. (2023) · JMIR AI · doi:10.2196/50442 · Cited by 18 · see citing works
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024) · neural information processing systems · doi:10.52202/079017-1054 · Cited by 12 · see citing works
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language ModelsPreprint
Han, T., Kumar, A., Agarwal, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03744 · Cited by 5 · see citing works
Combining Constraint Programming Reasoning with Large Language Model PredictionsPreprint
Régin, F., MARIA, E. D., & Bonlarron, A. (2024) · arXiv (Cornell University) · doi:10.4230/lipics.cp.2024.25 · Cited by 2 · see citing works
Giorgi, J., Toma, A., Xie, R., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02220 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Model Prompting Techniques for Advancement in Clinical Medicine
Shah, K., Xu, A. Y., Sharma, Y. K., Daher, M., McDonald, C. L., Diebo, B. G., & Daniels, A. H. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13175101 · Cited by 94 · see citing works
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Using mathematical modelling and AI to improve delivery and efficacy of therapies in cancer
Harkos, C., Hadjigeorgiou, A. G., Voutouri, C., Kumar, A. S., Stylianopoulos, T., & Jain, R. K. (2025) · Nature reviews. Cancer · doi:10.1038/s41568-025-00796-w · Cited by 60 · see citing works
Natural Language Processing in medicine and ophthalmology: A review for the 21st-century clinician
Rojas-Carabali, W. A., Agrawal, R., Gutiérrez-Sinisterra, L., Baxter, S. L., Cifuentes‐González, C., Wei, Y. C., Abisheganaden, J. A., Kannapiran, P., Wong, S., Lee, B. T. K., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100084 · Cited by 55 · see citing works
Large language models in oncology: a review
Chen, D., Parsa, R., Swanson, K. E., Nuñez, J.-J., Critch, A., Bitterman, D. S., Liu, F., & Raman, S. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000759 · Cited by 52 · see citing works
Workum, J. D., Volkers, B. W. S., van de Sande, D., Arora, S., Goeijenbier, M., Gommers, D. A. M. P. J., & van Genderen, M. E. (2025) · Critical Care · doi:10.1186/s13054-025-05302-0 · Cited by 40 · see citing works
Medical accuracy of artificial intelligence chatbots in oncology: a scoping review
Chen, D. J., Avison, K., Alnassar, S., Huang, R. S., & Raman, S. (2025) · The Oncologist · doi:10.1093/oncolo/oyaf038 · Cited by 40 · see citing works
Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives
Lho, S. K., Park, S. C., Lee, H., Oh, D. Y., Kim, H., Jang, S., Jung, H. Y., Yoo, S. Y., Park, S. M., & Lee, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.11922 · Cited by 36 · see citing works
Mavrych, V., Yousef, E. M., Yaqinuddin, A., & Bolgova, O. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2534065 · Cited by 31 · see citing works
Sezgin, E., Jackson, D. I., Kocaballi, A. B., Bibart, M., Zupanec, S., Landier, W., Audino, A., Ranalli, M., & Skeens, M. (2025) · Cancer Medicine · doi:10.1002/cam4.70554 · Cited by 30 · see citing works
Causality Extraction from Medical Text Using Large Language Models (LLMs)
Gopalakrishnan, S., Garbayo, L., & Zadrozny, W. (2024) · Information · doi:10.3390/info16010013 · Cited by 25 · see citing works
Taming Unleashed Large Language Models With Blockchain for Massive Personalized Reliable Healthcare
Sun, L., Liu, D., Wang, M., Han, Y., Zhang, Y., Zhou, B., Ren, Y. C., & zhu, P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3528526 · Cited by 22 · see citing works
The pitfalls of multiple-choice questions in generative AI and medical education
Singh, S., Alyakin, A., Alber, D. A., Stryker, J., Tong, A. P., Sangwon, K. L., Goff, N. K., Paz, M. D. L., Hernandez‐Rovira, M. A., Park, K. Y., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-26036-7 · Cited by 20 · see citing works
Geantă, M., Bădescu, D., Chirca, N., Nechita, O. C., Radu, C. G., Rașcu, Ș., Radavoi, D. G., Sima, C., Toma, C. V., & Jinga, V. (2024) · Healthcare · doi:10.3390/healthcare12151548 · Cited by 18 · see citing works
Zhang, Y., Wan, X.-H., Kong, Q., Liu, H., Liu, J., Guo, J., Yang, X., Zuo, X., & Li, Y. (2025) · World Journal of Gastroenterology · doi:10.3748/wjg.v31.i6.102090 · Cited by 17 · see citing works
Wang, D., Ye, J., Li, J., Liang, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z. Y., Shi, W. H., et al. (2025) · JMIR Medical Education · doi:10.2196/70190 · Cited by 15 · see citing works
Explaining decisions without explainability? Artificial intelligence and medicolegal accountability
McCradden, M. D., & Stedman, I. (2024) · Future Healthcare Journal · doi:10.1016/j.fhj.2024.100171 · Cited by 14 · see citing works
Lafourcade, C., Kérourédan, O., Ballester, B., & Richert, R. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105764 · Cited by 13 · see citing works
Yang, K., Lee, J., Jang, J. W., Yoon, S. K., Sung, P. S., & Han, J. W. (2026) · PLoS Medicine · doi:10.1371/journal.pmed.1004855 · Cited by 11 · see citing works
Kınıkoğlu, O., & Işık, D. (2025) · Cureus · doi:10.7759/cureus.78076 · Cited by 11 · see citing works
Enhanced large language models for effective screening of depression and anxiety
Liu, J. M., Gao, M., Sabour, S., Chen, Z., Huang, M., & Lee, T. M. C. (2025) · Communications Medicine · doi:10.1038/s43856-025-01158-1 · Cited by 17 · see citing works
Pavlik, E. J., Land, J. M., Lawton, F., Swiecki‐Sikora, A. L., Ramaiah, D., & Rives, T. A. (2025) · Diagnostics · doi:10.3390/diagnostics15060735 · Cited by 10 · see citing works
Automated MRI protocoling in neuroradiology in the era of large language models
Reiner, L. N., Chelbi, M., Fetscher, L., Stöckel, J. C., Schmidt, C. G., Guseynova, S., Mohamad, F. A., Bressem, K. K., Nawabi, J., Siebert, E., et al. (2025) · La radiologia medica · doi:10.1007/s11547-025-02040-9 · Cited by 8 · see citing works
Zhao, W., Lai, H., Pan, B., Huang, J., Xia, D., Bai, C. B., Liu, J., Liu, J., Jin, Y., Shang, H., et al. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1649041 · Cited by 8 · see citing works
Jayasrilakshmi, S., & Mahapatra, A. (2026) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2026.111469 · Cited by 12 · see citing works
Lee, J. K., Choi, S., Park, S., Hwang, S., & Cho, D. (2025) · Annals of Laboratory Medicine · doi:10.3343/alm.2024.0588 · Cited by 7 · see citing works
Enhancing healthcare resource allocation through large language models
Wan, F. J., Wang, K., Wang, T., Qin, H., Fondrevelle, J., & Duclos, A. (2025) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2025.101859 · Cited by 7 · see citing works
Li, R., Mao, S., Zhu, C., Yang, Y., Tan, C., Li, L., Mu, X., Liu, H., & Yang, Y. (2025) · Journal of Medical Internet Research · doi:10.2196/72638 · Cited by 6 · see citing works
May, P., Nokodian, S., Nuernbergk, C., Knauer, M., Hefter, M., von Rose, A. B., Bassermann, F., & Jung, J. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00194 · Cited by 5 · see citing works
Asif, S., Hadi, F., Ain, Q. U., Yan, Y., Wang, V. Y., & Xu, D. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100847 · Cited by 5 · see citing works
How AI Impacts Skill Formation
Shen, J. H., & Tamkin, A. (2026) · arXiv · doi:10.48550/arxiv.2601.20245 · Cited by 4 · see citing works
“Overreliance Although much of the literature in economics on AI-enhanced productivity implicitly assumes that generations of AI are trustworthy, the reality is that generative AI can produce incorrect [Longwell et al., 2024] or hallucinated content [Maleki et al., 2024].”
Liang, J., Xing, M., Xiang, P., Wang, G., Chen, M., Fang, Q., Zhou, T., Lu, Z., Leng, X., Huang, J., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02537-1 · Cited by 4 · see citing works
Hasei, J., Nakahara, R., Takeuchi, K., Yoshida, A., Itano, T., Fujiwara, T., Nakata, E., Kunisada, T., & Ozaki, T. (2025) · Journal of Orthopaedic Science · doi:10.1016/j.jos.2025.03.005 · Cited by 4 · see citing works
Evaluating Large Language Models in Ptosis-Related inquiries: A Cross-Lingual Study
Niu, L.-H., Li, W., Qin, B., Chen, T., Dong, L., He, Y., Jiang, X., Wang, M., Ma, L., Geng, J., et al. (2025) · Translational Vision Science & Technology · doi:10.1167/tvst.14.7.9 · Cited by 3 · see citing works
Yum, J.-E. I., Naqvi, S. A. A., Zhou, B., & Riaz, I. B. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00134 · Cited by 4 · see citing works
Antiperovitch, P., Liu, I., Mokhtar, A. T., & Tang, A. (2025) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2025.04.008 · Cited by 3 · see citing works
Bitterman, J. S., D’Angelo, A. S., Holachek, A., & Eubanks, J. E. (2025) · PM&R · doi:10.1002/pmrj.13386 · Cited by 3 · see citing works
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2 · see citing works
Large-scale deep learning for metastasis detection in pathology reports
Krawczuk, P., Fox, Z., Petkov, V. I., Negoita, S., Doherty, J. A., Stroup, A. M., Schwartz, S., Penberthy, L. T., Hsu, E. R., Gounley, J. P., et al. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf070 · Cited by 2 · see citing works
Ghorbian, M., & Ghobaei-Arani, M. (2025) · Journal of Big Data · doi:10.1186/s40537-025-01304-5 · Cited by 4 · see citing works
Chen, P. Q., Gu, H., Lo, H. K. Y., Chang, W. C., Lai, C., Lai, S. H. S., Cheng, A. S. K., & Ng, P. H. F. (2025) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/80607 · Cited by 2 · see citing works
Performance of large language model in cross-specialty medical scenarios
Cui, Z., Liu, W., Tian, X., You, C., Meng, X., ZHANG, H., Gong, K., Wang, X., & Wu, J. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-07577-x · Cited by 2 · see citing works
Xin, J., & He, X. (2026) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05458-8 · Cited by 2 · see citing works
Multimodal large language models challenge NEJM image challenge
Sheng, C., Shen, S., Wang, L., Chen, J., Chen, W. J., Wang, S., & Wang, N. (2026) · Scientific Reports · doi:10.1038/s41598-026-39201-3 · Cited by 2 · see citing works
Yazarkan, Y., Sönmez, G., & Simsek, C. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106413 · Cited by 2 · see citing works
The Most Disruptive Near-Term Use of AI in Cancer Care: Patient Empowerment Through Software Agents
Nothaft, F. A., & Power, B. (2024) · AI in Precision Oncology · doi:10.1089/aipo.2024.0027 · Cited by 1 · see citing works
Performance of Large Language Models on the Radiation and Cancer Biology Practice Exam
Bertschmann, J., Xu, Y., Bayley, C., Abdellatif, A., & Lee, S. L. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.07.1347 · Cited by 1 · see citing works
Wang, C., Liu, Z., Hao, L., Chen, S., Guo, R., Wei, L., Ji, K., Wang, F., & Xu, B. (2025) · Displays · doi:10.1016/j.displa.2025.103144 · Cited by 1 · see citing works
Miyake, M., Yonemoto, N., Togo, K., Xu, L., Oguri, T., Tanaka, M., Hasegawa, Y., Izawa, Y., & Araki, K. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00249 · Cited by 1 · see citing works
AlShahwan, N., Fetyani, I. M., Beyari, M. B., Aldeligan, S. H., Beyari, M. B., Alshehri, R. S., Alburakan, A. A., Mashbari, H. N. A., Alkanhal, A., & Nouh, T. A. (2025) · Surgical Innovation · doi:10.1177/15533506251361664 · Cited by 1 · see citing works
Yokoe, T., Roversi, G., Sevivas, N., Kamei, N., Diniz, P., & Pereira, H. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70393 · Cited by 1 · see citing works
Performance of the large language model in general medicine
Tao, L., Liu, J., Lu, X., Zhao, Y., Zhang, Y., Zhu, Z., Li, T., Zhang, Z., Zhang, Y., Yan, W., et al. (2025) · Global Transitions · doi:10.1016/j.glt.2025.10.004 · Cited by 1 · see citing works
Huang, Z., Tian, L., Lin, H., Zheng, Y., Chin, M. P., Wang, H., Xu, P., & Chen, H. (2026) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205251409499 · Cited by 1 · see citing works
Arzideh, K., Hosch, R., Turki, A. T., Eryılmaz, B., Bahn, M., Schäfer, H., Idrissi-Yaghir, A., Khattab, S., Dada, A., Baba, H. A., et al. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00254 · Cited by 1 · see citing works
Elkefi, S., Scheinmann, R., Hao, D., Bhar, S., Tounsi, A., & Feiner, S. (2026) · BMC Artificial Intelligence · doi:10.1186/s44398-026-00024-x · Cited by 1 · see citing works
Karakoyun, Z. N., Yörük, M. D., Özdemir, M. E., & Koşar, M. İ. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09173-z · Cited by 1 · see citing works
Kasa, K., Yoshimaru, D., Ohta, H., Ohki, T., & Okano, H. J. (2026) · General Thoracic and Cardiovascular Surgery · doi:10.1007/s11748-026-02304-9 · Cited by 1 · see citing works
Dukes, D., Yost, C. D., Wang, R., Diagne, Y. F., Garr, L., Yost, M., Owaka, J., York, M., Kang, P., Sookram, J., et al. (2026) · Gynecologic Oncology · doi:10.1016/j.ygyno.2026.07.005 · Cited by 1 · see citing works
Wojtas, J., Pak, G., & Górski, W. J. (2026) · Cureus · doi:10.7759/cureus.112868 · Cited by 1 · see citing works
Corso, F., Zec, A., Favali, M., Ligero, M., Hilgers, L., Invernizzi, L. M., Mazzeo, L., Morales, A. M., Müller‐Franzes, G. A., Machado, I. P., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102966 · Cited by 2 · see citing works
Sohail, A., Patel, O. P., Choi, J., Venditti, J. C. S., & Wu, A. J. (2025) · medRxiv · doi:10.1101/2025.07.06.25330972 · Cited by 1 · see citing works
Evaluating advanced artificial intelligence in oncology education and clinical knowledge assessment
Ahmed, Y., Ibrahim, H., & Khayal, S. (2025) · International Journal of Research in Medical Sciences · doi:10.18203/2320-6012.ijrms20252009
Ng, P. H. F., Chen, P. Q., Gu, H., Lo, H. K. Y., Chang, W. C., Lai, C., Lai, S. H. S., & Cheng, A. S. K. (2025) · doi:10.2196/preprints.80607
Automated ICD-O-3 Coding of Real-World Pathology Reports Using Self-Hosted Large Language ModelsPreprint
Arzideh, K., Hosch, R., Turki, A. T., Erilymaz, B., Bahn, M., Schäfer, H., Idrissi-Yaghir, A., Khattab, S., Dada, A., Baba, H. A., et al. (2025) · medRxiv · doi:10.1101/2025.07.23.25332040
Bitterman, J. S., D’Angelo, A. S., Holachek, A., & Eubanks, J. E. (2025) · PM&R · doi:10.1002/pmrj.13451
Sheng, C., Shen, S., Wang, L., Chen, W. J., Wang, S., & Wang, N. (2025) · Research Square · doi:10.21203/rs.3.rs-7175790/v1
Ahmad, H. F., Sajid, J., Rasool, R. U., & Malik, A. W. (2025) · Cluster Computing · doi:10.1007/s10586-025-05793-0
Nakaura, T., Kobayashi, N., Shiraishi, K., Yoshida, N., Nagayama, Y., Uetani, H., Kidoh, M., Oda, S., Funama, Y., & Hirai, T. (2025) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001807
A Comparative Performance Study of Retrieval-Augmented Generation Systems in Gynecologic OncologyPreprint
Peng, S., Kuerbanjiang, W., Xiao, H., He, X., & Yi, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-8221196/v1
Context for Large Language Model Evaluation in Ophthalmology—Reply
Srinivasan, S., Chen, Q., & Tham, Y. C. (2026) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2025.5847
A multi-cohort evaluation of a curriculum-aligned, oncology-focused chatbot in pharmacy education
Halford, Z., & Heitz, N. (2026) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2026.102659
Yildiz, D., & Özyürek, E. B. A. (2026) · International Journal of Ophthalmology and Clinical Research · doi:10.23937/2378-346x/1410167
Xu, H., Liu, J., & Cheng, Q. (2026) · Applied Sciences · doi:10.3390/app16125827
Zhou, Y., Lin, S., Wang, X., & Hu, K. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1808714
Yasser, M., Barakat, G., Awny, S., Ezzat, M., Ahmed, M. S. A., Wael, S., Moustafa, M., & Hamdy, O. (2026) · Scientific Reports · doi:10.1038/s41598-026-60786-2
Before the Answer: Preventing Never-Skilling in Oncology Training
Baloch, Z., Leyfman, Y., & Thaker, N. G. (2026) · AI in Precision Oncology · doi:10.1177/2993091x261474371
Zhang, J., Wang, X., Zhao, Y., Cui, X., Zhao, L., Zhao, X., Zhang, H., & Zhu, Z. (2026) · Journal of Medical Internet Research · doi:10.2196/94904
Tang, Y., Liu, Y., Liu, M., Ni, H., Wang, S., & Wang, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1926969
Kırca, Ö., Çakır, M. O., Kaya, M. S., Kurt, B., Yilmaz, B., Bilge, U., & Ozdogan, M. (2026) · Biomedicines · doi:10.20944/preprints202609.0010.v1
Sebin, B., & Sebin, I. K. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1956533
Who is Responsible for Tackling Professional Deskilling?
Kärki, K. (2026) · Human Affairs · doi:10.1515/humaff-2026-0080
Holmes, J., Hao, Y., Borras-Osorio, M., Mastroleo, F., Brufau, S. R., Carducci, V., Abel, K. M., Routman, D. M., Foong, A., Muller, O. M., et al. (2026) · Meta-Radiology · doi:10.1016/j.metrad.2026.100245
- Authors
- Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C.
- Affiliations
- University Health Network (CA)
- Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
- University of Toronto (CA)
- Corresponding author
- Grant, Robert C.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2024-06-18
- Volume
- 7
- Issue
- 6
- Pages
- e2417641
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC11185976
- OpenAlex
- W4399774223
- Semantic Scholar
- e0ef8acaa0e55b326757dd5e2d33f7bdbe009f59
- Cited by
- 81
- References
- 45
- Citations by year
- 2026: 29 · 2025: 45 · 2024: 7
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, medical oncology, multiple-choice questions, board certification examination, LLM accuracy, LLM safety, reasoning errors, reading comprehension errors
- MeSH terms
- Cross-Sectional Studies, Educational Measurement, Humans, Language, Medical Oncology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
