The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.
Can large language models reason about medical questions?
Open accessAbstract
Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper evaluates whether GPT-3.5 and open-source LLMs can answer and reason about medical questions across MedQA-USMLE, MedMCQA, and PubMedQA using prompting, chain-of-thought (CoT), retrieval augmentation, and ensembling. It finds that LLMs can often reason and recall expert knowledge, and that Codex 5-shot CoT with many samples reaches passing scores on all three benchmarks.
Key findings
- Zero-shot InstructGPT outperformed the best fine-tuned BERT baselines on all three datasets: +1.4% on USMLE, +1.0% on MedMCQA, and +1.1% on PubMedQA.
- Expert annotation of 50 USMLE CoTs found that incorrect answers often contained reasoning errors (86%), insufficient/incorrect knowledge (74%), and reading-comprehension failures (50%); still, 59% of CoTs leading to incorrect answers contained at least one correct reasoning step and 65% showed correct knowledge recall.
- Codex 5-shot CoT with an ensemble of up to k=100 samples reached passing scores: USMLE 60.2%, MedMCQA 62.7%, and PubMedQA 78.2%; Llama 2 70B also passed USMLE with 62.5% accuracy.
- Codex 5-shot CoT and Llama 2 were close to well calibrated, though calibration was worse for PubMedQA; prompting also induced answer-position biases, with direct prompting overestimating C/D and underestimating A, and CoT prompting underestimating B/C while overestimating D.
Methods
- The study used three medical multiple-choice QA datasets: MedQA-USMLE, MedMCQA, and PubMedQA, with questions and reference explanations used for few-shot CoT prompts.
- Models included closed-source GPT-3.5 series (InstructGPT, text-davinci-002; Codex, code-davinci-002) and open-source models from 7B to 70B parameters, including Llama 2, Vicuna, Guanaco, Falcon, MPT, and GPT-NeoX.
- Prompting conditions included zero-shot direct, zero-shot CoT with domain-specific cues, few-shot direct/CoT, BM25 retrieval augmentation over Wikipedia, and ensemble/self-consistency sampling up to k=100 completions; a medical expert annotated 50 generated CoTs using six success/failure patterns.
Limitations
- The paper states that deploying LLMs in clinical scenarios requires more robust techniques, that it exposed only one form of bias (answer-option ordering), and that many more biases may exist, including hidden training-data biases such as gender and race.
- The paper notes that benchmark contamination in training sets is a valid concern for both proprietary and open-source LLMs, and that a large gap remains between the models and human experts despite passing scores.
- Evident from the text: the expert CoT evaluation is based on a subset of only 50 USMLE CoTs annotated by one medical expert, and the BM25 retrieval experiment is described as limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 19 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When using the direct prompt, we recorded gains on the USMLE (+1.3%) and on the MedMCQA (+2.7%) datasets, suggesting that retrieval augmentation might be beneficial.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The smaller model text-curie-002 delivered close to random performances, with a maximum accuracy of 27.9%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without BM25 grounding, InstructGPT scored +1.4% on the USMLE questions, +1.0% on the MedMCQA exam questions, and +1.1% on PubMedQA over the best BERT methods.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, the domain-specific CoT prompts #2–#5 did not significantly outperform the original CoT prompt #1.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that most of the questions answered incorrectly triggered generating CoTs that contained reasoning errors (pattern D, 86%) and that exhibited a lack of knowledge (pattern E, 74%).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The best performances on the USMLE and MedMCQA datasets were obtained by combining retrieval-augmented prompts and setting a maximum of 53.1% accuracy on the USMLE dataset and 48.8% validation accuracy on the MedMCQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, inspecting the CoTs leading to incorrect answers more closely, we found that 47% of those were inconclusive: the model could not narrow down the prediction to a single answer.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.”
§ Can large language models reason about medical questions? > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 20
- The quote only partly states this claim. 2
- Probably not a substantial claim. 10
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct prompting led to overestimating labels C and D while underestimating label A.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performances were lower for each of the considered CoT cues, except in the case of the USMLE dataset, for which half of the CoT prompts resulted in small improvements over the direct prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining the predictions of multiple prompts outperformed the single-prompt predictions, except in the case of the PubMedQA dataset, for which the direct prompt performed exceptionally well.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both prompting schemes led to biased predictive frequencies.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge.”
- Also stated in: § Contributions · plain
“A medical expert annotated a subset of CoTs generated by zero-shot InstructGPT and supports that InstructGPT, in many cases, can reason and exploit memorized expert knowledge.”
§ Can large language models reason about medical questions? > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
- Also stated in: § Contributions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting led to underestimating B and C while overestimating label D.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Best performances are obtained using Codex, outperforming the human-aligned InstructGPT, which is a fine-tuned version of Codex.”
§ Can large language models reason about medical questions? > Discussion > Scale, code pre-training, human-alignment, and few-shot learning
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 100 samples, Codex 5-shot CoT delivered unprecedented performances on the three datasets, bridging the gap with human-level performances and virtually passing the USMLE by 0.2% points.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
- Rests on an assumption: The human baseline performances reported are valid lower bounds for human performance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although Codex performed exceptionally well with 5 shots, Codex yield feeble performances with zero-shot CoT; inspecting the generated CoTs revealed lesser-quality samples (supplemental information section I).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although such prompts, taken separately, did not outperform the original zero-shot CoT prompt (see Table S2 in supplemental information section A), more specific prompts appeared to trigger alternative strategies such as working by elimination or manipulating equations (see supplemental information sections B and I).”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Codex 5-shot CoT appears to be overall calibrated, although the calibration is worse for the PubMedQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Conditioning the predictions on a knowledge base is an alternative research direction for improving language models.47,48,49”
§ Can large language models reason about medical questions? > Method > Retrieval augmentation
What the sources say
- Not checked yetcites [47] Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [48] Borgeaud, 2022, Improving language models by retrieving from trillions of tokens, PMLR
- Not checked yetcites [49] Lazaridou, 2022, Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Specialized datasets such as the MedQA-US Medical Licensing Examination (USMLE) enable assessing the capabilities of LLMs in realistic clinical scenarios requiring specialized medical knowledge, advanced reasoning capabilities, and human-level reading comprehension skills.32”
§ Can large language models reason about medical questions? > Introduction > LLMs and medical applications
What the sources say
- Not checked yetcites [32] Jin, 2021, What disease does this patient have? a large-scale open domain question answering dataset from medical exams, Appl. Sci. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“CoT prompting led to substantial improvements on many reasoning-intensive tasks,20,21,22,23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.24”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [20] Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [21] Zhou, 2022, Least-to-most Prompting Enables Complex Reasoning in Large Language Models, arXiv
- Not checked yetcites [22] Drozdov, 2022, Compositional Semantic Parsing with Large Language Models, arXiv
- Not checked yetcites [23] Nye, 2021, Show Your Work: Scratchpads for Intermediate Computation with Language Models, arXiv
- Not checked yetcites [24] Suzgun, 2022, Challenging BIG-bench Tasks and Whether Chain-of-thought Can Solve Them, arXiv
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We repeated the experiment using randomly permuted labels and observed similar patterns (see supplemental information section F).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“As an alternative to writing reference step-by-step solutions, zero-shot CoT allows for generating CoTs using a single and domain-agnostic cue: “Let’s think step by step”25”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [25] Kojima, 2022, Large language models are zero-shot reasoners, Adv. Neural Inf. Process. Syst.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Combining multiple sampled CoTs (also known as self-consistency) has also been explored in past work, and showed improvements over single-sample CoT methods.42,43”
§ Can large language models reason about medical questions? > Method > Answer likelihood
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [43] Li, 2022, On the Advance of Making Language Models Better Reasoners, arXiv
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although benchmark contamination in training sets for both proprietary and open-source LLMs is a valid concern, these results indicate that both open- and closed-source LLMs hold great potential for assisting human decision-making in medicine and beyond.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Since the completion of v.2 of this work, both GPT-4 and MedPalm 2 have achieved performance on USMLE around 85%.35,63”
§ Can large language models reason about medical questions? > Discussion > Conclusion
What the sources say
- Not checked yetcites [35] Nori, 2023, Capabilities of GPT-4 on Medical Challenge Problems, arXiv
- Not checked yetcites [63] Singhal, 2023, Towards Expert-level Medical Question Answering with Large Language Models, arXiv
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, CoTs are human readable and therefore interpretable.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation suggests that LLMs can mostly comprehend complex medical questions, can often recall expert-domain knowledge, and can often perform non-trivial reasoning steps.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
None of the sources cited for this claim have been checked yet.
In the paper
“CoTs can be combined and/or filtered using human or automated feedback.42,60”
§ Can large language models reason about medical questions? > Discussion > Generating and combining many CoTs bridges the gap with human-level performances
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [60] Cobbe, 2021, Training Verifiers to Solve Math Word Problems, arXiv
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting proved to be a powerful tool leading to better performances and more interpretable predictions.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper relies on this without support.
It was unclear how large language models perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.
“Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.”
The paper relies on this without support.
At the time of the research, studies applying large language models to the medical domain were still needed.
“At the dawn of this research work, although LLMs had been tested on large benchmarks (MMLU,31 BIG-bench18), studies applied to the medical domain were still needed.”
The paper relies on this without support.
Large language models memorize part of the knowledge embedded into the training data.
“LLMs memorize part of the knowledge embedded into the training data; nonetheless, models might fail to reuse this knowledge effectively during prediction.”
The paper relies on this without support.
A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
The paper relies on this without support.
The generated CoTs reflect the model’s reasoning process.
“The CoTs that result from that prompt not only appear to expose valid reasoning but also translate into superior zero-shot performances (see example in Figure 1).”
The paper relies on this without support.
The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
“Zero-shot InstructGPT and Codex outperformed fine-tuned BERT models on three challenging question-answering datasets (see zero-shot benchmark and supplemental information section A).”
The paper relies on this without support.
The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
The paper relies on this without support.
The human baseline performances reported are valid lower bounds for human performance.
“We report the best fine-tuned models and the human baseline.”
The paper relies on this without support.
The expert evaluation of CoTs is a valid measure of soundness and interpretability.
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) [6,7] or general reasoning capabilities [2].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
In the citing paper
“Finally, we classified the errors leading to incorrect recommendations by the LLM, leveraging a prior evaluation framework [29]. We classified each error as 1 of 4 error types: incorrect reading comprehension, incorrect recall of knowledge, incorrect reasoning step, and not enough information, as described in Table 1.”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Conflicting Codex MedMCQA accuracy
MajorThe abstract and results text report 62.7% MedMCQA accuracy for Codex 5-shot CoT, but Table 1 reports 59.7% for the same model and dataset.
“the MedMCQA dataset (62.7% ≥ 50%)”
“| Codex 5-shot CoTa | 2022 | 60.2 | 59.7 | 78.2 |”
PubMedQA passing score claimed despite none being listed
MinorThe text says Codex reached the passing score on PubMedQA by exceeding 78%, but Table 1 lists no PubMedQA passing score. It identifies 78% as the human expert score instead.
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
“| Human (passing score) | – | 60.0 | 50.0 | – | | Human (expert score) | – | 87.0 | 90.0 | 78.0 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“, 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer architecture (Vaswani et al., 2017) and its high compatibility with massively parallel computation hardware.”
BackgroundJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models…”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“This so far translated sustain gains (Kaplan et al., 2020), enabled new ways to interact with language models and made many of the past benchmarks obsolete; which triggered a community-wide effort to design a sufficiently difficult benchmarks (e.”
BackgroundDense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · arXiv · doi:10.48550/arxiv.2004.04906 · Cited by 5,311 · cited 1×
“, 2022) or dense retrievers (Karpukhin et al., 2020), (ii) using a more complete knowledge base Borgeaud et al.”
Background
Flagged first, then by citations
Research works printed but not linked: 1
- Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper AI21 Labs, 1, 9
Cited by
Sorted by citations
Items 101–200 of 352
GPT-4 versus human authors in clinically complex MCQ creation: A blinded analysis of item quality
Wu, H., Zerner, T., Lee, D., Court‐Kowalski, S., Devitt, P. G., & Palmer, E. (2025) · Medical Teacher · doi:10.1080/0142159x.2025.2505122 · Cited by 29
“The most common flaws were an incorrect DOI, the reference being an old edition of a guideline, or incorrect details, as summarized in Table 7.”
Language model and its interpretability in biomedicine: A scoping review
Lyu, D., Wang, X., Chen, Y., & Wang, F. (2024) · iScience · doi:10.1016/j.isci.2024.109334 · Cited by 17
Kwan, H. Y., Shell, J., Fahy, C., Yang, S., & Xing, Y. (2025) · Systems · doi:10.3390/systems13040281 · Cited by 18
Classifying legal interpretations using large language models
Dugac, G., & Altwicker, T. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09447-9 · Cited by 19
Bentegeac, R., Guellec, B. L., Kuchcinski, G., Amouyel, P., & Hamroun, A. (2025) · Journal of Medical Internet Research · doi:10.2196/64348 · Cited by 24
Sun, Z., Yim, W.-W., Uzuner, Ö., Xia, F., & Yetişgen, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104866 · Cited by 20
“Suarez-Lledo et al. [38] examined prevalence of health misinformation about vaccines and non-communicable diseases on social media, but their focus was primarily on the dissemination dynamics rather than on the technical methods for detection and correction.”
Wang, D., Ye, J., Li, J., Liang, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z. Y., Shi, W. H., et al. (2025) · JMIR Medical Education · doi:10.2196/70190 · Cited by 15
A comparison of chain-of-thought reasoning strategies across datasets and models
Hebenstreit, K., Praas, R., Kiesewetter, L. P., & Samwald, M. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1999 · Cited by 15
“We acknowledge that prompts which are semantically similar but differ syntactically can lead to varied outcomes, as observed in medical datasets ( Liévin, Hother & Winther, 2022 ).”
BackgroundResultBunnell, D. J., Bondy, M. J., Fromtling, L. M., Ludeman, E. M., & Gourab, K. (2025) · medRxiv · doi:10.1101/2025.04.01.25325033 · Cited by 22
Impacts of generative AI on user contributions: evidence from a coding Q &A platform
Li, X., & Kim, K. (2024) · Marketing Letters · doi:10.1007/s11002-024-09747-1 · Cited by 23
“These advanced generative intelligences have shown remarkable potential in quickly providing high-quality answers to questions that require in-depth understanding and specific domain knowledge (Liévin et al., 2023; OpenAI, 2023).”
Evaluating AI-based breastfeeding chatbots: quality, readability, and reliability analysis
Kacer, E. O. (2025) · PLOS ONE · doi:10.1371/journal.pone.0319782 · Cited by 14
Towards conversational artificial intelligence for disease management
Liévin, V., Palepu, A. K., Weng, W., Saab, K. K., Stutz, D., Cheng, Y., Kulkarni, K., Mahdavi, S. S., Barral, J. K., Webster, D. R., et al. (2026) · Nature · doi:10.1038/s41586-026-10764-5 · Cited by 53
Self-citationSpitale, G., Schneider, G., Germani, F., & Biller‐Andorno, N. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1328865 · Cited by 14
“Concerning medical applications, Beaulieu-Jones et al. (2023) attest near-human surgery knowledge, while Liévin et al. (2023) report that ChatGPT-3.”
BackgroundComparing AI and human decision-making mechanisms in daily collaborative experiments
Wang, L., Jiang, Z., Hu, C., Zhao, J., Zhu, Z., Chen, X. M., Wang, Z., Liu, T., He, G., Yin, Y., et al. (2025) · iScience · doi:10.1016/j.isci.2025.112711 · Cited by 24
“3 12 The prospect of employing LLMs to model human social interactions and decision processes is promising, potentially enabling simulations of cooperative and competitive behaviors across diverse scenarios.”
Rollwage, M., McFadyen, J., Juechems, K., Balogh, A., Pisupati, S., Mircea, M.-T., Hauser, T. U., Prichard, G., & Harper, R. J. (2026) · Nature Medicine · doi:10.1038/s41591-026-04278-w · Cited by 19
Can ChatGPT diagnose my collapsing dog?
Abani, S., Decker, S. D., Tipold, A., Nessler, J. N., & Volk, H. A. (2023) · Frontiers in Veterinary Science · doi:10.3389/fvets.2023.1245168 · Cited by 15
GPT-4 as a biomedical simulator
Schaefer, M., Reichl, S., ter Horst, R., Nicolas, A. M., Krausgruber, T., Piras, F., Stepper, P., Bock, C., & Matthias Samwald, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108796 · Cited by 13
Han, H. (2024) · Ethics & Behavior · doi:10.1080/10508422.2024.2347661 · Cited by 16
Hamane, Z., Ben-Bouazza, F.-E., & Fennan, A. (2024) · International Conference on Intelligent Systems and Computer Vision (ISCV) · doi:10.1109/iscv60512.2024.10620139 · Cited by 16
Yang, Y., Jin, Q., Zhu, Q., Wang, Z., Álvarez, F. E., Wan, N., Hou, B., & Lu, Z. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094851 · Cited by 20
Borna, S., Gomez-Cabello, C. A., Pressman, S. M., Haider, S. A., Sehgal, A., Leibovich, B. C., Cole, D., & Forte, A. J. (2024) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe14050093 · Cited by 12
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 344
“Recently, LLMs have shown the ability to encode clinical knowledge 7,8 , retrieve relevant medical texts 9 , and perform accurate single-turn medical question-answering 3,10 – 12 .”
Evaluating Emotional Detection & Classification Capabilities of GPT-2 & GPT-Neo Using Textual Data
Jain, B., Goyal, G., & Sharma, M. (2024) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence60223.2024.10463396 · Cited by 11
“5, introduced retrieval augmentation, which includes relevant Wikipedia text in the prompts, promising to enhance model performance in medical tasks [13].”
BackgroundChao, P., Chang, C.-H., Wu, J.-J., Liu, Y.-H., Shiau, J., Shih, H.-H., Lin, G.-Z., Lee, S., & Lee, T.-F. (2024) · Cancer Control · doi:10.1177/10732748241286749 · Cited by 11
Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
Ndimbo, E. V., Luo, Q., Fernando, G., Xu, Y., & Wang, B. (2025) · Applied Sciences · doi:10.3390/app15020524 · Cited by 15
Gwak, G., Hwang, U., Jung, S. H., & Kim, J. H. (2023) · Journal of Musculoskeletal Science and Technology · doi:10.29273/jmst.2023.7.1.8 · Cited by 11
Idan, D., Shitrit, I. B., Volevich, M., Binyamin, Y., Nassar, R., Nassar, M., Abelson, N., Zlotnik, A. A., & Einav, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-21689-w · Cited by 14
Survey on Efficient Large Language Models: Principles, Algorithms, Applications, and Open Issues
Cheng, J., Kang, H., Shao, Y., Li, N., Chen, P., Wang, R., Long, S., Yang, X., & Ma, L. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3628671 · Cited by 12
“L ARGE language models (LLMs), exemplified by Chat-GPT [1] and Claude [2], have demonstrated promising performance and great potential in natural language processing (NLP), including translation [3], summarization [4], question answering [5], and conversational agent [6].”
Large Language Models in Healthcare: A Review
Zou, S., & He, J. (2023) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic60498.2023.00038 · Cited by 11
“In another study, the Codex model with 175B parameters demonstrated an answering accuracy of 60.2% on the USMLE test, 59.7% on the MedMCQA validation, and 78.2% on the PubMedQA test [18] .”
BackgroundMethodInfluentialMin, D., Jin, K. N., Bang, S., Kim, M. Y., Kim, H., Jeong, W. G., Lee, H., Beck, K. S., Hwang, S. H., Kim, E. Y., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0293 · Cited by 10
Feng, H., Denny, P. C., Wuensche, B. C., Luxton-Reilly, A., & Whalley, J. L. (2024) · doi:10.1145/3680533.3697064 · Cited by 11
“…showed impressive capabilities of LLMs in many subjects, such as reading comprehension [Brown et al. 2020], law [Katz et al. 2024], medicine [Liévin et al. 2024; Nori et al. 2023], and various other academic fields [AI4Science and Quantum 2023], thus bringing opportunities and challenges in…”
A preliminary test of ChatGPT’s ESG literacy
Gao, W., & Ju, M. (2024) · Managerial Finance · doi:10.1108/mf-04-2024-0315 · Cited by 9
Benchmarking Medical LLMs on Anesthesiology: A Comprehensive Dataset in Chinese
Zhou, B., Zhan, Y., Wang, Z., Li, Y., Zhang, C., Yu, B., Ding, L., Jin, H., Liu, W., Wang, X., et al. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3502465 · Cited by 9
Nemati, A., Shalmani, M. A., Lü, Q., & Luo, J. (2025) · Future Internet · doi:10.3390/fi17040138 · Cited by 9
“Although extensive research has focused on benchmarking LLMs for general tasks such as named entity recognition [11], relation extraction [12], and question answering [12], these studies often lack a focus on real-world clinical applications.”
Sousa, S. A., Bukhari, S. S. U. H., Steagall, P. V. M., Bęczkowski, P. M., Giuliano, A., & Flay, K. J. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1616566 · Cited by 12
MedAgentSim: Self-evolving Multi-agent Simulations for Realistic Clinical Interactions
Almansoori, M., Kumar, K., & Cholakkal, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05114-1_35 · Cited by 9
Automated prompt engineering pipelines: fine-tuning LLMs for enhanced response accuracy
Hendawi, S., Kanan, T. G., Elbes, M. W., Mughaid, A. S., & AlZu’bi, S. M. F. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129689 · Cited by 9
Just how transformative will AI/ML be for immuno-oncology?
Bottomly, D., & McWeeney, S. K. (2024) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2023-007841 · Cited by 9
Patel, D., Raut, G., Zimlichman, E., Cheetirala, S. N., Nadkarni, G. N., Glicksberg, B. S., Freeman, R., Timsina, P., & Klang, E. (2023) · medRxiv · doi:10.1101/2023.08.06.23293710 · Cited by 8
Schuemie, M. J., Ostropolets, A., Zhuk, O., Korsik, U., Seo, S. I., Suchard, M. A., Hripcsak, G. M., & Ryan, P. B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01433-4 · Cited by 11
OpenAI ChatGPT: A Potential Medical Application
Chaddad, A., Jiang, Y., & He, C. (2023) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom56612.2023.10472387 · Cited by 9
OntoFact: Unveiling Fantastic Fact-Skeleton of LLMs via Ontology-Driven Reinforcement Learning
Shang, Z., Ke, W., Xiu, N., Wang, P., Liu, J., Li, Y., Luo, Z., & Ji, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29859 · Cited by 10
Infusing Multi-Hop Medical Knowledge Into Smaller Language Models for Biomedical Question Answering
Chen, J., Wei, Z., Shen, W., & Shang, R. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3547444 · Cited by 7
Surgery scheduling based on large language models
Wan, F. J., Wang, T., Wang, K., Si, Y., Fondrevelle, J., Du, S., & Duclos, A. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103151 · Cited by 8
Large language models for competence-based HRM: A case study in the aerospace industry
Barba, G., Corallo, A., Lazoi, M., & Lezzi, M. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100780 · Cited by 12
He, J., Zhang, B., Rouhizadeh, H., Chen, Y., Yang, R., Lu, J. S., Chen, X., Liu, N., & Teodoro, D. (2026) · Artificial Intelligence Review · doi:10.21203/rs.3.rs-8330917/v1 · Cited by 27
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 28
Hirosawa, T., Harada, Y., Tokumasu, K., Shiraishi, T., Suzuki, T., & Shimizu, T. (2024) · JMIR Formative Research · doi:10.2196/64844 · Cited by 7
Survey of Large Language Models for Answering Questions Across Various Fields
Shailendra, P., Ghosh, R. C., Kumar, R., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717078 · Cited by 12
“ChatGPT performance on Medical Licensing examinations is discussed by Gilson et al. [38] in United States (USMLE) for Step 1 and Step 2 examinations.”
On Prompt Sensitivity of ChatGPT in Affective Computing
Amin, M. M., & Schuller, B. W. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00028 · Cited by 18
“[11] survey the medical use of LLMs, which includes a study that experiments with several CoT prompts in the medical field [12], including CoT prompts to behave as a medical expert.”
Saowaprut, P., Wabina, R. S., Yang, J., & Siriwat, L. (2025) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2025.22.16 · Cited by 9
“Large language models (LLMs) have demonstrated remarkable capabilities in medical question-answering (QA) tasks, including interpreting complex terminology, clinical reasoning, and analyzing multi-modal data [1,2].”
InfluentialŠavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
Evaluating the Performance of Large Language Models on a Neurology Board-Style ExaminationPreprint
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · medRxiv · doi:10.1101/2023.07.13.23292598 · Cited by 7
Jiang, C., Fan, Y., Xie, J., Kuang, E., Feng, B., Zhang, K., & Fan, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713677 · Cited by 6
“LLMs, such as ChatGPT are widely used in educational scenarios, including maths [48, 81], programming [50, 58], language [17], and medicine [45, 61].”
Clinical pathway-aware large language models for reliable and transparent medical dialogue
Wu, J., Wu, X., Zheng, Y., & Yang, J. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104942 · Cited by 10
Umehara, K., Ota, J., Nishii, T., Kishimoto, R., & Ishida, T. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100064 · Cited by 6
ChatGPT usage among nursing students in Taiwan: a qualitative study
Huang, H. (2026) · BMC Nursing · doi:10.1186/s12912-026-04341-4 · Cited by 7
Kanakarajan, K. R., & Sankarasubbu, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.137 · Cited by 7
Laaber, C., Yue, T., Ali, S., Schwitalla, T., & Nygård, J. F. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613882 · Cited by 6
“, the OpenAPI schema definition, either manually or derived from documentation;7 (2) use generative models such as generative adversarial networks (GANs) or variational autoencoders (VAEs) trained on real patient records to generate valid cancer messages and cases to directly test GURI [26]; (3) use generated cancer m…”
MethodSearch for medical information for chronic rhinosinusitis through an artificial intelligence ChatBot
Yassa, A., Ayad, O., Cohen, D. A., Patel, A. M., Vengsarkar, V. A., Hegazin, M. S., Filimonov, A., Hsueh, W. D., & Eloy, J. A. (2024) · Laryngoscope Investigative Otolaryngology · doi:10.1002/lio2.70009 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“…GPT-3 (Brown et al., 2020), and ChatGPT, used as approximate stand-ins for shifts in how the community uses and therefore evaluates LMs. Common practice for evaluating autoregressive language models today diverges from the method described in the paper for all listed tasks except MMLU and MATH.”
Huang, H., & Shu, S.-H. (2025) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2025.101732 · Cited by 7
Farrokhi, M., Fallahian, A., Rahmani, E., Aghajan, A., Alipour, M., Khouzani, P. J., Hezarani, H. B., Sabzehie, H., Pirouzan, M., Pirouzan, Z., et al. (2025) · Archives of Academic Emergency Medicine · doi:10.22037/aaemj.v13i1.2712 · Cited by 7
Application prospect of large language model represented by ChatGPT in ophthalmology
Kuang, J., Zeng, M.-B., Chen, F.-L., & Huang, W. (2025) · International Journal of Ophthalmology · doi:10.18240/ijo.2025.09.21 · Cited by 5
A multi-agent approach to neurological clinical reasoning
Sorka, M., Gorenshtein, A., Aran, D., & Shelly, S. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001106 · Cited by 8
Advancing Healthcare with Large Language Models: Techniques and Application
Hu, Z., Peng, Z., Bi, Z., Shen, Q., Liu, Z., Lou, J., & Luo, X. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125540 · Cited by 5
Naderi, N., Safavi‐Naini, S. A. A., Savage, T., Khalafi, M. A., Lewis, P. R., Atf, Z., Nadkarni, G. N., & Soroush, A. (2026) · npj gut and liver. · doi:10.1038/s44355-026-00053-3 · Cited by 13
“Large Language Models (LLMs) can generate fl uent, clinically relevant answers, but they can also present incorrect information with unwarranted certainty 1,2 .”
Zeljković, I., Novak, M., Jordán, A., Lisičić, A., Blažić, T. N., Pavlović, N., & Manola, Š. (2024) · Heart Rhythm O2 · doi:10.1016/j.hroo.2024.10.005 · Cited by 7
Advancements in AI-Driven Dentistry: Tooth GenAI’s Impact on Dental Diagnosis and Treatment Planning
Shankar, A., R, M. T., Kumar, S. S., Anurag, A., Narayan, A., & P, J. A. (2024) · International Conference on Networking, Embedded and Wireless Systems (ICNEWS) · doi:10.1109/icnews60873.2024.10731122 · Cited by 5
Comparison of ChatGPT-4, Microsoft Copilot, and Google Gemini for Pediatric Ophthalmology Questions
Bahar, T. S., Öcal, O., & Yaprak, A. Ç. (2025) · Journal of Pediatric Ophthalmology & Strabismus · doi:10.3928/01913913-20250404-03 · Cited by 5
IOAgent: Democratizing Trustworthy HPC I/O Performance Diagnosis Capability via LLMs
Egersdoerfer, C., Sareen, A., Bez, J. L., Byna, S., Xu, D., & Dai, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00036 · Cited by 5
Can a Machine Ace the Test? Assessing GPT-4.0’s Precision in Plastic Surgery Board Examinations
Qurashi, A. A. A., Albalawi, I. A. S., Halawani, I. R., Asaad, A. H., Dwehji, A. M. O. A., Almusa, H. A., Alharbi, R. I., Alobaidi, H. A., Alarki, S. M. K. Z., & Aljindan, F. (2023) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000005448 · Cited by 5
“7,14 Importantly, AI is not designed to replace the role of educators or traditional teaching methods, but to enhance them.”
BackgroundExploring Prompting Approaches in Legal Textual Entailment
Bilgin, O., Fields, L., Laverghetta, A., Marji, Z., Nighojkar, A., Steinle, S., & Licato, J. (2024) · The Review of Socionetwork Strategies · doi:10.1007/s12626-023-00154-y · Cited by 5
Does Using ChatGPT Result in Human Cognitive Augmentation?
Fulbright, R., & Morrison, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61569-6_9 · Cited by 7
Han, L., Pan, S., Xing, Z., Sun, J., Yitagesu, S., Zhang, X., & Feng, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3523284 · Cited by 5
“It has been applied to text prediction tasks in specific domain, such as scientific abstracts and medical reports [83], [84].”
Guo, A., Canagasingham, A., Rasiah, K., Chalasani, V., Mundy, J. A., & Chung, A. S. J. (2025) · ANZ Journal of Surgery · doi:10.1111/ans.70186 · Cited by 4
Miao, B., Sun, Q., Wang, P., Shao, R., Ding, Y., Chen, Y., & Ying, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-16999-y · Cited by 4
Gosak, L., Štiglic, G., Tam, W., & Vrbnjak, D. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104614 · Cited by 4
Leveraging ChatGPT and explainable AI for enhancing clinical decision support
Shawi, R. E., & Jamel, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-22784-8 · Cited by 9
“Chain-of-thought (CoT) prompting, including automatic variants such as Auto-CoT, has been shown to improve reasoning in multi-step tasks and medical question answering 25,26 .”
LLM-driven collaborative framework for knowledge-enhanced cancer pain assessment and management
Liu, H., Hu, Y., Li, D., Shi, B., Niu, Y., Zhang, X., Zheng, G., Li, C., Wang, L., & Bao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02362-6 · Cited by 5
Promises and challenges of applying large language models in the healthcare domain
Wang, Q., Gong, Z., Lai, Z., Bu, L., Dahlweid, F.-M., & Sun, H. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1772274 · Cited by 4
Enhancing Medical Dialogue Summarization: A MediExtract Distillation Framework
Liu, X., Huang, M., Rusnachenko, N., Ive, J., Chang, J. M., & Zhang, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822640 · Cited by 4
“The emergence of instruction-tuned models referred to as Large Language Models (LLMs), allow significantly enhanced abstractive summarization, leading researchers to focus on improving summarization tasks in target-oriented domains [12].”
Tan, D., Huang, Y. C., Liu, M., Li, Z., Wu, X., & Huang, C. (2025) · Journal of Medical Internet Research · doi:10.2196/70733 · Cited by 4
Refining medical large language models: key insights from instruction tuning
Alqahtani, M. Q., Al-Barakati, A. A., & Alotaibi, F. S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3216 · Cited by 7
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
Farinhas, A., de Souza, J. G. C., & Martins, A. F. T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.733 · Cited by 14
Weerathunge, T., Jayalal, S., & Wijayasiriwardhane, K. (2025) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc64760.2025.10963313 · Cited by 4
“Limitations in reasoning and understanding, influenced by algorithmic biases and training data variability, further contribute to vague, biased, or contradictory responses[9], [18].”
GAGPT and Its Application to the Interactive Learning of Geometric Algebra
Wang, J., Du, P., Zhao, Z., Luo, W., Yu, Z., & Yuan, L. (2025) · Advances in Applied Clifford Algebras · doi:10.1007/s00006-025-01385-8 · Cited by 5
Generalizing Fairness to Generative Language Models via Reformulation of Non-discrimination Criteria
Sterlie, S., Weng, N., & Feragen, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92648-8_2 · Cited by 10
“The model’s answers to the medical questions are not consistently correct [21], partially due to the complexity of the medical questions, which might introduce confounding factors that could compromise the validity of the observed biases.”
BackgroundMethodInfluentialPubMed and Beyond: Biomedical Literature Search in the Age of Artificial IntelligencePreprint
Jin, Q., Leaman, R., & Lu, Z. (2023) · PubMed Central · doi:10.48550/arxiv.2307.09683 · Cited by 4
“As a result, they provide point-of-care evidence that can guide clinical decision-making.”
MODERN THOUGHTS ON THE DEVELOPMENT OF SCIENCE: IDEAS, TECHNOLOGIES AND THEORIES
International Science Group, & Vladlenov, D. (2024) · doi:10.46299/isg.p.2024.1.12 · Cited by 4
Arkangel AI: A conversational agent for real-time, evidence-based medical question-answering
Villa, M. C., Castano-Villegas, N., Llano, I., Martinez, J., Guevara, M. F., Zea, J., & Velásquez, L. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100274 · Cited by 4
Mwaniki, P., Musau, W., Isaaka, L., Wanyama, C. W., Menon, V., Denniston, A. K., Liu, X., Emmanuel-Fabula, M., Williams, G., Mateen, B. A., et al. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000081 · Cited by 2
“Large language models (LLMs) have shown impressive capabilities in several health-specific evaluation exercises, 1–3 but many of these historical benchmarking tasks have focused narrowly on multiple-choice accuracy, which fails to reflect the complexity”
Alsaad, A. K., Alanezi, S., & Melhim, L. K. B. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2589498 · Cited by 3
TRUSS-Med: Medical Question Answering via Transformers, Retrieval, and Unified State Space Models
Zhu, X., Khan, M., Mhedhbi, E., Liang, Y., & Othmani, A. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356863 · Cited by 3
“In medicine, LLMs have shown promising results in clinical decision support [2], exam question answering [3], [4], and biomedical research assistance [5].”
Wu, H., Deng, J., Qiu, X., Xu, L. L., Lu, L., Fan, M., Yu, D., Liu, C., Chen, Z., Wang, K., et al. (2026) · Frontiers in Endocrinology · doi:10.3389/fendo.2026.1667159 · Cited by 3
Dani, M., Prakash, M. J., Rosa, F., Akata, Z., & Liebe, S. (2026) · Communications Medicine · doi:10.1038/s43856-026-01653-z · Cited by 5
“Following Med-PaLM [51] and Li´evin et al.’s [32] protocol, we evaluated our models’ reasoning outputs using three categories: correct/complete, somewhat correct/complete, and incorrect/incomplete.”
Hirosawa, T., Yokose, M., Sakamoto, T., Hayashi, A., Harada, Y., Tokumasu, K., Mizuta, K., & Shimizu, T. (2026) · JMIR Formative Research · doi:10.2196/92016 · Cited by 3
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 15
Generative AI for Evidence-Based Medicine: A PICO GenAI for Synthesizing Clinical Case Reports
Mohammed, S., & Fiaidhi, J. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622271 · Cited by 3
“Moreover, important LLM model like ChatGPT reported an average of 59% in answering accurately the USMLE medical tests [24, 25].”
MethodZimmerman, K. M., Hillen, S., Wijesinghe, D. S., & Omecene, N. E. (2025) · Health Literacy and Communication Open · doi:10.1080/28355245.2025.2536511 · Cited by 4
Metadata
- Authors
- Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O.
- Affiliations
- Technical University of Denmark (DK)
- Copenhagen University Hospital (DK)
- University of Copenhagen (DK)
- Corresponding author
- Liévin, Valentin
- Venue
- Patterns
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 5
- Issue
- 3
- Pages
- 100943
- ISSN
- 2666-3899 (ISSN-L)
- PMC
- PMC10935498
- OpenAlex
- W4392359953
- Semantic Scholar
- d697b440dd0e65a05fe027e4c0ea85f62dcba033
- Cited by
- 343
- References
- 69
- Citations by year
- 2026: 95 · 2025: 108 · 2024: 79
- FWCI
- 26.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Patterns (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Danmarks Grundforskningsfond
- Novo Nordisk Fonden (NNF20OC0062606)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, medical question answering, medical reasoning, chain-of-thought prompting, retrieval augmentation, few-shot prompting, prompt engineering, open-source language models, Llama 2, GPT-3.5, MedMCQA, PubMedQA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
