The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.
Can large language models reason about medical questions?
Open accessAbstract
Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper evaluates whether GPT-3.5 and open-source LLMs can answer and reason about medical questions across MedQA-USMLE, MedMCQA, and PubMedQA using prompting, chain-of-thought (CoT), retrieval augmentation, and ensembling. It finds that LLMs can often reason and recall expert knowledge, and that Codex 5-shot CoT with many samples reaches passing scores on all three benchmarks.
Key findings
- Zero-shot InstructGPT outperformed the best fine-tuned BERT baselines on all three datasets: +1.4% on USMLE, +1.0% on MedMCQA, and +1.1% on PubMedQA.
- Expert annotation of 50 USMLE CoTs found that incorrect answers often contained reasoning errors (86%), insufficient/incorrect knowledge (74%), and reading-comprehension failures (50%); still, 59% of CoTs leading to incorrect answers contained at least one correct reasoning step and 65% showed correct knowledge recall.
- Codex 5-shot CoT with an ensemble of up to k=100 samples reached passing scores: USMLE 60.2%, MedMCQA 62.7%, and PubMedQA 78.2%; Llama 2 70B also passed USMLE with 62.5% accuracy.
- Codex 5-shot CoT and Llama 2 were close to well calibrated, though calibration was worse for PubMedQA; prompting also induced answer-position biases, with direct prompting overestimating C/D and underestimating A, and CoT prompting underestimating B/C while overestimating D.
Methods
- The study used three medical multiple-choice QA datasets: MedQA-USMLE, MedMCQA, and PubMedQA, with questions and reference explanations used for few-shot CoT prompts.
- Models included closed-source GPT-3.5 series (InstructGPT, text-davinci-002; Codex, code-davinci-002) and open-source models from 7B to 70B parameters, including Llama 2, Vicuna, Guanaco, Falcon, MPT, and GPT-NeoX.
- Prompting conditions included zero-shot direct, zero-shot CoT with domain-specific cues, few-shot direct/CoT, BM25 retrieval augmentation over Wikipedia, and ensemble/self-consistency sampling up to k=100 completions; a medical expert annotated 50 generated CoTs using six success/failure patterns.
Limitations
- The paper states that deploying LLMs in clinical scenarios requires more robust techniques, that it exposed only one form of bias (answer-option ordering), and that many more biases may exist, including hidden training-data biases such as gender and race.
- The paper notes that benchmark contamination in training sets is a valid concern for both proprietary and open-source LLMs, and that a large gap remains between the models and human experts despite passing scores.
- Evident from the text: the expert CoT evaluation is based on a subset of only 50 USMLE CoTs annotated by one medical expert, and the BM25 retrieval experiment is described as limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 2 Prompt templates In the table, we use typewriter style and brackets to represent [provided data] such as the question, additional context, or the answer and <completions> generated by GPT-3. The symbol ∅ represents an empty string.](/paper-figures/d27c38f67a2f4fd6a676a3998ef7b6a3/can-large-language-models-reason-about-medical-questions-figure-2.jpg)
Verification
Claims the paper makes
55 Own result · 11 Not checked yet · 10 Own interpretation · 5 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When using the direct prompt, we recorded gains on the USMLE (+1.3%) and on the MedMCQA (+2.7%) datasets, suggesting that retrieval augmentation might be beneficial.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The smaller model text-curie-002 delivered close to random performances, with a maximum accuracy of 27.9%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without BM25 grounding, InstructGPT scored +1.4% on the USMLE questions, +1.0% on the MedMCQA exam questions, and +1.1% on PubMedQA over the best BERT methods.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, the domain-specific CoT prompts #2–#5 did not significantly outperform the original CoT prompt #1.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that most of the questions answered incorrectly triggered generating CoTs that contained reasoning errors (pattern D, 86%) and that exhibited a lack of knowledge (pattern E, 74%).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The best performances on the USMLE and MedMCQA datasets were obtained by combining retrieval-augmented prompts and setting a maximum of 53.1% accuracy on the USMLE dataset and 48.8% validation accuracy on the MedMCQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, inspecting the CoTs leading to incorrect answers more closely, we found that 47% of those were inconclusive: the model could not narrow down the prediction to a single answer.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.”
§ Can large language models reason about medical questions? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 20
- The quote only partly states this claim. 2
- Probably not a substantial claim. 10
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct prompting led to overestimating labels C and D while underestimating label A.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performances were lower for each of the considered CoT cues, except in the case of the USMLE dataset, for which half of the CoT prompts resulted in small improvements over the direct prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining the predictions of multiple prompts outperformed the single-prompt predictions, except in the case of the PubMedQA dataset, for which the direct prompt performed exceptionally well.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both prompting schemes led to biased predictive frequencies.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge.”
- Also stated in: § Contributions · plain
“A medical expert annotated a subset of CoTs generated by zero-shot InstructGPT and supports that InstructGPT, in many cases, can reason and exploit memorized expert knowledge.”
§ Can large language models reason about medical questions? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
- Also stated in: § Contributions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting led to underestimating B and C while overestimating label D.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Best performances are obtained using Codex, outperforming the human-aligned InstructGPT, which is a fine-tuned version of Codex.”
§ Can large language models reason about medical questions? > Discussion > Scale, code pre-training, human-alignment, and few-shot learning
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 100 samples, Codex 5-shot CoT delivered unprecedented performances on the three datasets, bridging the gap with human-level performances and virtually passing the USMLE by 0.2% points.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
boosteddirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
- Rests on an assumption: The human baseline performances reported are valid lower bounds for human performance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although Codex performed exceptionally well with 5 shots, Codex yield feeble performances with zero-shot CoT; inspecting the generated CoTs revealed lesser-quality samples (supplemental information section I).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although such prompts, taken separately, did not outperform the original zero-shot CoT prompt (see Table S2 in supplemental information section A), more specific prompts appeared to trigger alternative strategies such as working by elimination or manipulating equations (see supplemental information sections B and I).”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Codex 5-shot CoT appears to be overall calibrated, although the calibration is worse for the PubMedQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
hedgedcorrelationaln = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Conditioning the predictions on a knowledge base is an alternative research direction for improving language models.47,48,49”
§ Can large language models reason about medical questions? > Method > Retrieval augmentation
What the sources say
- Not checked yetcites [47] Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [48] Borgeaud, 2022, Improving language models by retrieving from trillions of tokens, PMLR
- Not checked yetcites [49] Lazaridou, 2022, Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Specialized datasets such as the MedQA-US Medical Licensing Examination (USMLE) enable assessing the capabilities of LLMs in realistic clinical scenarios requiring specialized medical knowledge, advanced reasoning capabilities, and human-level reading comprehension skills.32”
§ Can large language models reason about medical questions? > Introduction > LLMs and medical applications
What the sources say
- Not checked yetcites [32] Jin, 2021, What disease does this patient have? a large-scale open domain question answering dataset from medical exams, Appl. Sci. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“CoT prompting led to substantial improvements on many reasoning-intensive tasks,20,21,22,23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.24”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [20] Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [21] Zhou, 2022, Least-to-most Prompting Enables Complex Reasoning in Large Language Models, arXiv
- Not checked yetcites [22] Drozdov, 2022, Compositional Semantic Parsing with Large Language Models, arXiv
- Not checked yetcites [23] Nye, 2021, Show Your Work: Scratchpads for Intermediate Computation with Language Models, arXiv
- Not checked yetcites [24] Suzgun, 2022, Challenging BIG-bench Tasks and Whether Chain-of-thought Can Solve Them, arXiv
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We repeated the experiment using randomly permuted labels and observed similar patterns (see supplemental information section F).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As an alternative to writing reference step-by-step solutions, zero-shot CoT allows for generating CoTs using a single and domain-agnostic cue: “Let’s think step by step”25”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [25] Kojima, 2022, Large language models are zero-shot reasoners, Adv. Neural Inf. Process. Syst.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Combining multiple sampled CoTs (also known as self-consistency) has also been explored in past work, and showed improvements over single-sample CoT methods.42,43”
§ Can large language models reason about medical questions? > Method > Answer likelihood
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [43] Li, 2022, On the Advance of Making Language Models Better Reasoners, arXiv
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although benchmark contamination in training sets for both proprietary and open-source LLMs is a valid concern, these results indicate that both open- and closed-source LLMs hold great potential for assisting human decision-making in medicine and beyond.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Since the completion of v.2 of this work, both GPT-4 and MedPalm 2 have achieved performance on USMLE around 85%.35,63”
§ Can large language models reason about medical questions? > Discussion > Conclusion
What the sources say
- Not checked yetcites [35] Nori, 2023, Capabilities of GPT-4 on Medical Challenge Problems, arXiv
- Not checked yetcites [63] Singhal, 2023, Towards Expert-level Medical Question Answering with Large Language Models, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, CoTs are human readable and therefore interpretable.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation suggests that LLMs can mostly comprehend complex medical questions, can often recall expert-domain knowledge, and can often perform non-trivial reasoning steps.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
None of the sources cited for this claim have been checked yet.
In the paper
“CoTs can be combined and/or filtered using human or automated feedback.42,60”
§ Can large language models reason about medical questions? > Discussion > Generating and combining many CoTs bridges the gap with human-level performances
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [60] Cobbe, 2021, Training Verifiers to Solve Math Word Problems, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting proved to be a powerful tool leading to better performances and more interpretable predictions.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
It was unclear how large language models perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.
“Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.”
The paper relies on this without support.
At the time of the research, studies applying large language models to the medical domain were still needed.
“At the dawn of this research work, although LLMs had been tested on large benchmarks (MMLU,31 BIG-bench18), studies applied to the medical domain were still needed.”
The paper relies on this without support.
Large language models memorize part of the knowledge embedded into the training data.
“LLMs memorize part of the knowledge embedded into the training data; nonetheless, models might fail to reuse this knowledge effectively during prediction.”
The paper relies on this without support.
A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
The paper relies on this without support.
The generated CoTs reflect the model’s reasoning process.
“The CoTs that result from that prompt not only appear to expose valid reasoning but also translate into superior zero-shot performances (see example in Figure 1).”
The paper relies on this without support.
The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
“Zero-shot InstructGPT and Codex outperformed fine-tuned BERT models on three challenging question-answering datasets (see zero-shot benchmark and supplemental information section A).”
The paper relies on this without support.
The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
The paper relies on this without support.
The human baseline performances reported are valid lower bounds for human performance.
“We report the best fine-tuned models and the human baseline.”
The paper relies on this without support.
The expert evaluation of CoTs is a valid measure of soundness and interpretability.
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) [6,7] or general reasoning capabilities [2].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
In the citing paper
“Finally, we classified the errors leading to incorrect recommendations by the LLM, leveraging a prior evaluation framework [29]. We classified each error as 1 of 4 error types: incorrect reading comprehension, incorrect recall of knowledge, incorrect reasoning step, and not enough information, as described in Table 1.”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
“the MedMCQA dataset (62.7% ≥ 50%)”
§ Can large language models reason about medical questions? > Method > Answer likelihood
“| Codex 5-shot CoTa | 2022 | 60.2 | 59.7 | 78.2 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
§ Can large language models reason about medical questions? > Method > Answer likelihood
“| Human (passing score) | – | 60.0 | 50.0 | – | | Human (expert score) | – | 87.0 | 90.0 | 78.0 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“, 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer architecture (Vaswani et al., 2017) and its high compatibility with massively parallel computation hardware.”
BackgroundJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models…”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“This so far translated sustain gains (Kaplan et al., 2020), enabled new ways to interact with language models and made many of the past benchmarks obsolete; which triggered a community-wide effort to design a sufficiently difficult benchmarks (e.”
BackgroundDense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · arXiv · doi:10.48550/arxiv.2004.04906 · Cited by 5,311 · cited 1×
“, 2022) or dense retrievers (Karpukhin et al., 2020), (ii) using a more complete knowledge base Borgeaud et al.”
Background
Research works printed but not linked: 1
- Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper AI21 Labs, 1, 9
Cited by
Sorted by citations
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
“[50] used Instruct GPT-3, an instruction-tuned LLM [63], and applied chain-of-thought prompting [91] on top to improve the results on the MedQA, MedMCQA, and PubMedQA datasets.”
BackgroundMethodA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“In professional domain reasoning tasks, zeroshot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved (Liévin et al., 2022).”
BackgroundToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
“In particular, these models have demonstrated impressive capabilities on multiple-choice research benchmarks 1–3 .”
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
“There has been recent interest in evaluating the capabilities of LLMs for understanding and generating natural language in medicine 9 10 .”
BackgroundA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 376
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 430
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
“…Encoder Representations from Transformers) [17] and its variants, decoder-only LMs such as the GPT (Generative Pre-trained Transformer) family [18-20] and encoder-decoder LMs such as T5 (Text-to-Text Transfer Transformer) [21] and BART (Bidirectional and AutoRegressive Transformers) [22].”
BackgroundThe shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
Matching patients to clinical trials with large language models
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z. (2024) · Nature Communications · doi:10.1038/s41467-024-53081-z · Cited by 287
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
“As prompt engineering has become a key technique for investigating and improving LLMs, Liévin et al. [128] have also applied various prompt engineering on the GPT-3.”
MethodCan ChatGPT pass the life support exams without entering the American heart association course?
Fijačko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109732 · Cited by 206
ChatGPT: The End of Online Exam Integrity?
Sušnjak, T., & McIntosh, T. R. (2024) · Education Sciences · doi:10.3390/educsci14060656 · Cited by 271
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
“Transformer-based LLMs have demonstrated close to human-level performances in medical question and answering benchmarks and summarisation tasks (7–9), and with techniques like self-consistency (9), chain of thought prompting (10), and reinforcement learning from human feedback (11) the model performance can be further…”
BackgroundPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 238
“Fortunately, we have witnessed a preliminary sign that we started to investigate large language models in medical/clinical tasks [174, 187, 270], especially BIOGPT [187], PubMedGPT 68, and Flan-PaLM [270].”
BackgroundJin, Q., Yang, Y., Chen, Q., & Lu, Z. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae075 · Cited by 176
“They also achieve state-of-the-art performance on many domain-specific tasks like biomedical question answering (Singhal et al., 2022; Liévin et al., 2022; Nori et al., 2023).”
BackgroundPrompt engineering for healthcare: Methodologies and applications
Wang, J., Shi, E., Yu, S., Wu, Z., Hu, H., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., et al. (2025) · Meta-Radiology · doi:10.1016/j.metrad.2025.100190 · Cited by 266
Retrieval-augmented generation for generative artificial intelligence in health care
Yang, R., Ning, Y., Keppo, E., Liu, M., Hong, C., Bitterman, D. S., Ong, J. C. L., Ting, D. S. W., & Liu, N. (2025) · npj Health Systems · doi:10.1038/s44401-024-00004-1 · Cited by 206
“Although some research has attempted to enhance models ’ reasoning abilities and transparency through approaches like chain of thought 34 , multi-agent discussion 35 , and post-hoc attribution 36 , there are still limitations in medical applications 37 .”
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
Jin, Q., Chen, F., Zhou, Y., Xu, Z., Cheung, J. M., Chen, R., Summers, R. M., Rousseau, J. F., Ni, P., Landsman, M. J., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01185-7 · Cited by 146
“These re fl ect the multimodal nature of the task as well as the commonly evaluated capabilities for text-only LLMs 6,8 .”
BackgroundSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodResultAl-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125
Wang, D., Feng, L., Ye, J., Zou, J., & Zheng, Y. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.43 · Cited by 127
Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology
Tan, T. F., Thirunavukarasu, A. J., Campbell, J. P., Keane, P. A., Pasquale, L. R., Abràmoff, M. D., Kalpathy–Cramer, J., Lum, F., Kim, J. E., Baxter, S. L., et al. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100394 · Cited by 124
“33 Potential additional features that can be explored include uncertainty-aware LLM applications that provide a probability score of generated responses, along with alternative recommendations when the probability score is low, as well as reporting the differential weights of input tokens that…”
BackgroundPerformance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106
Prompt Engineering in Healthcare
Patil, R., Heston, T. F., & Bhuse, V. (2024) · Electronics · doi:10.3390/electronics13152961 · Cited by 99
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.969 · Cited by 87
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“In particular, we evaluate general-purpose LLMs on answering medical QA questions that require domain knowledge in depth 1,26,38 – 41 .”
Benefits, limits, and risks of ChatGPT in medicine
Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Wang, M., Hacking, S. M., Glicksberg, B. S., Strauß, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1518049 · Cited by 88
“…application in everyday healthcare settings, proving beneficial in providing information, answering queries, and offering support to both healthcare professionals and patients (Liévin et al., 2022; Sallam, 2023; Ahn, 2023; Curtis and ChatGPT, 2023; D'Amico et al., 2023; Donato et al., 2023).”
Evaluating the ChatGPT family of models for biomedical reasoning and classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad256 · Cited by 91
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023) · arXiv · doi:10.48550/arxiv.2311.17371 · Cited by 73
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78
“Excellent LLM performance across multiple domains in medical question and answer (QA) benchmarks has been postulated, in part, to reflect emergent reasoning capabilities[10, 11].”
Hulmán, Á., Dollerup, O. L., Mortensen, J. F., Fenech, M. E., Norman, K., Støvring, H., & Hansen, T. K. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290773 · Cited by 75
Evaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68
“One study investigated if language models can effectively handle medical questions 20 , revealing that they can approximate human performance with chain-of-thought reasoning.”
Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications
Bora, A., & Cuayáhuitl, H. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6040116 · Cited by 86
The impending impacts of large language models on medical education
Ahn, S. (2023) · Korean journal of medical education · doi:10.3946/kjme.2023.253 · Cited by 64
“medical questions with a rather high level of accuracy [3,7].”
BackgroundTaylor, R. A., Sangal, R. B., Smith, M. E., Haimovich, A. D., Rodman, A., Iscoe, M. S., Pavuluri, S. K., Rose, C., Janke, A. T., Wright, D. S., et al. (2025) · Academic Emergency Medicine · doi:10.1111/acem.15066 · Cited by 62
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 69
Examining ChatGPT Performance on USMLE Sample Items and Implications for Assessment
Yaneva, V., Baldwin, P., Jurich, D. P., Swygert, K. A., & Clauser, B. E. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005549 · Cited by 62
Lahat, A., Sharif, Κ., Zoabi, N., Patt, Y. S., Sharif, Y., Fisher, L., Shani, U., Arow, M., Levin, R., & Klang, E. (2024) · Journal of Medical Internet Research · doi:10.2196/54571 · Cited by 64
“5), which managed to score approximately 60% on the USMLE (United States Medical Licensing Examination) [18].”
Performance of ChatGPT on the MCAT: The Road to Personalized and Equitable Premedical LearningPreprint
Bommineni, V. L., Bhagwagar, S. Z., Balcarcel, D. R., Davatzikos, C. A., & Boyer, D. L. (2023) · medRxiv · doi:10.1101/2023.03.05.23286533 · Cited by 62
Enhancing knowledge retrieval with in-context learning and semantic search through generative AI
Ghali, M.-K., Farrag, A., Won, D., & Jin, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113047 · Cited by 66
“Task-specific methods like chain-of-thought [13, 14, 15] emulate expert-like problem-solving.”
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53
“First, these benchmarks predominantly focus on exam-style 19,20 questions or sourcing from online conversational cases 18 , these sour-cestypicallydonotfullycapturethedepth,complexity,andstructured reasoning present in real-world clinical practice.”
Kwon, T., Ong, K. T.-I., Kang, D.-J., Moon, S., Lee, J. R., Hwang, D., Sohn, B., Sim, Y., Lee, D. H., & Yeo, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29802 · Cited by 63
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C. E., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02433-3 · Cited by 66
Self-citationChoi, W. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04832-x · Cited by 54
Su, M.-C., Lin, L.-E., Lin, L.-H., & Chen, Y. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104717 · Cited by 52
Performance of ChatGPT on registered nurse license exam in Taiwan: a descriptive study
Huang, H. (2023) · Healthcare · doi:10.3390/healthcare11212855 · Cited by 48
“It achieved a score of 60.2% in the US Medical Licensing Examination (USMLE), 57.7% in Multiple-choice medical exam questions (MedMCQA), and 78.2% in the medical reading comprehension dataset (PubMedQA) [27,29].”
BackgroundInfluentialChatGPT: Transforming Healthcare with AI
Neha, F., Bhati, D., Shukla, D. K., & Amiruzzaman, M. (2024) · AI · doi:10.3390/ai5040126 · Cited by 52
Kumah-Crystal, Y. A., Mankowitz, S. L., Embí, P. J., & Lehmann, C. U. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad104 · Cited by 47
Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success)
Shaib, C., Li, M., Joseph, S. A., Marshall, I. J., Li, J. J., & Wallace, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.119 · Cited by 58
Mind meets machine: Unravelling GPT-4’s cognitive psychology
Dhingra, S., Singh, M., Vaisakh, S. B., Malviya, N., & Gill, S. S. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2023.100139 · Cited by 62
KoSEL: Knowledge subgraph enhanced large language model for medical question answering
Zeng, Z., Cheng, Q., Hu, X., Zhuang, Y., Liu, X., He, K., & Liu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112837 · Cited by 46
Large language models provide unsafe answers to patient-posed medical questions
Draelos, R. L., Afreen, S., Blasko, B., Brazile, T. L., Chase, N., Desai, D. P., Evert, J., Gardner, H. L., Herrmann, L., House, A. V., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02428-5 · Cited by 87
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42
Mansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 46
“Liévin et al. [71] evaluated InstructGPT, Codex, and LLaMA-2 70B using zero-shot direct prompts , zero-shot Chain-of-Thought (CoT) , and few-shot ensemble prompting enhanced with domain-specific cues and BM25-based retrieval from Wikipedia (a form of RAG).”
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41
“11 The insufficiency of data and training impedes LLMs in understanding the intricate concepts 29 and encoding the necessary clinical expertise in non-English environment, 30 which is essential for accurate clinical inference.”
BackgroundUnraveling the landscape of large language models: a systematic review and future perspectives
Ding, Q., Ding, D., Wang, Y., Guan, C., & Ding, B. (2023) · Journal of Electronic Business & Digital Economics · doi:10.1108/jebde-08-2023-0015 · Cited by 51
Large language models and synthetic health data: progress and prospects
Smolyak, D., Bjarnadóttir, M. V., Crowley, K., & Agarwal, R. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae114 · Cited by 50
Can ChatGPT Guide Parents on Tympanostomy Tube Insertion?
Moise, A., Centomo-Bozzo, A., Orishchak, O., Alnoury, M. K., & Daniel, S. J. (2023) · Children · doi:10.3390/children10101634 · Cited by 35
“One of the tool’s fundamental strengths is its ability to elucidate complex medical terminology and describe complex physiology in a succinct manner [1–4].”
BackgroundOjeda, A. D., Solano-Barliza, A. D., Álvarez, D. D. O., & Boom-Cárcamo, E. (2023) · Formación universitaria · doi:10.4067/s0718-50062023000600061 · Cited by 34
AI in Dental Radiology—Improving the Efficiency of Reporting With ChatGPT: Comparative Study
Stephan, D., Bertsch, A., Burwinkel, M., Vinayahalingam, S., Al‐Nawas, B., Kämmerer, P. W., & Thiem, D. G. E. (2024) · Journal of Medical Internet Research · doi:10.2196/60684 · Cited by 39
Patel, D. D., Raut, G., Zimlichman, E., Cheetirala, S. N., Nadkarni, G. N., Glicksberg, B. S., Apakama, D. U., Bell, E., Freeman, R., Timsina, P., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-66933-x · Cited by 33
ChatGPT in pharmacometrics? Potential opportunities and limitations
Cloesmeijer, M. E., Janßen, A., Koopman, S. F., Cnossen, M. H., & Mathôt, R. A. A. (2023) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15895 · Cited by 32
EYE-Llama, an in-domain large language model for ophthalmology
Haghighi, T., Gholami, S., Sokol, J. T., Kishnani, E., Ahsaniyan, A., Rahmanian, H., Hedayati, F., Leng, T., & Alam, M. N. (2025) · iScience · doi:10.1016/j.isci.2025.112984 · Cited by 31
Kong, Z. Y., Adi, V. S. K., Segovia‐Hernández, J. G., & Sunarso, J. (2023) · Digital Chemical Engineering · doi:10.1016/j.dche.2023.100126 · Cited by 29
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1006 · Cited by 28
The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland
Nicikowski, J., Szczepański, M., Miedziaszczyk, M., & Kudliński, B. (2024) · Clinical Kidney Journal · doi:10.1093/ckj/sfae193 · Cited by 29
“( 2023) , with a 46% accuracy for the algorithm, with zero suggestions, as well as in neurology exam answers [7 ,28 ].”
BackgroundAutomating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30
“Incontrast, humanevaluationremainsthegold standard for assessing factuality and nuance but is labor-intensive and limited in scale 25 .”
Computational Politeness in Natural Language Processing: A Survey
Priya, P., Firdaus, M., & Ekbal, A. (2024) · ACM Computing Surveys · doi:10.1145/3654660 · Cited by 38
Centering health equity in large language model deployment
Singh, N. P., Lawrence, K. S., Richardson, S. I., & Mann, D. M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000367 · Cited by 29
Huang, T.-Y., Hsieh, P. H., & Chang, Y. (2024) · JMIR Medical Education · doi:10.2196/59902 · Cited by 23
“This highlights some areas where AI-assisted medical interactions can be improved [17-19].”
Fisch, U., Kliem, P. S. C., Grzonka, P., & Sutter, R. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100978 · Cited by 25
Kuerbanjiang, W., Peng, S., Jiamaliding, Y., & Yi, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/63626 · Cited by 24
A Critical Review of Methods and Challenges in Large Language Models
Asgari, R., Moradi, M., Yan, K., Colwell, D., & Samwald, M. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.061263 · Cited by 40
“Widely adopted across fields like healthcare, finance, education, and technology, LLMs demonstrate versatility and transformative impact [3,4,13,14].”
FoodSky: A food-oriented large language model that can pass the chef and dietetic examinations
Zhou, P., Min, W., Fu, C., Jin, Y., Huang, M., Li, X., Mei, S., & Jiang, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101234 · Cited by 32
Artificial intelligence agents in cancer research and oncology
Truhn, D., Azizi, S., Zou, J., Cerdá-Alberich, L., Mahmood, F., & Kather, J. N. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-025-00900-0 · Cited by 29
Wang, N., Chen, Y., & Zhang, D. (2025) · The Innovation Energy · doi:10.59717/j.xinn-energy.2025.100087 · Cited by 26
Moglia, A., Georgiou, K. E., Cerveri, P., Mainardi, L. T., Satava, R. M., & Cuschieri, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10849-5 · Cited by 27
Artificial Intelligence for Risk Assessment on Primary Prevention of Coronary Artery Disease
Chen, S., Loguercio, S., Chen, K., Lee, S. E., Park, J., Liu, S., Sadaei, H. J., & Torkamani, A. (2023) · Current Cardiovascular Risk Reports · doi:10.1007/s12170-023-00731-4 · Cited by 26
“Liévin et al. recently explored the capacity of general-purpose LLMs to reason through complex medical questions [239].”
BackgroundShukla, R., Mishra, A. K., Banerjee, N., & Verma, A. (2024) · Cureus · doi:10.7759/cureus.58232 · Cited by 31
“Recently, there has been interest in assessing large language models' (LLMs) capacity to comprehend and produce natural language in the medical field [4].”
BackgroundUranbey, Ö., Özbey, F., Kaygisiz, Ö. F., & Ayrancı, F. (2024) · Cureus · doi:10.7759/cureus.58607 · Cited by 22
“Integrating AI into healthcare systems is of paramount importance due to its capacity to enhance precision and reduce time requirements in various aspects of the healthcare ecosystem [12].”
BackgroundIlicki, J. (2023) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2023.03.006 · Cited by 20
Developing ChatGPT for biology and medicine: a complete review of biomedical question answering
Li, Q., Li, L., & Li, Y. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240004 · Cited by 27
“It surpasses other language models trained on specific domains, such as Wikipedia, news, or books, without the need for these domain-specific training datasets.”
BackgroundDeep Learning in Digital Breast Tomosynthesis: Current Status, Challenges, and Future Trends
Wang, R., Chen, F., Chen, H., Lin, C. X., Shuai, J., Wu, Y., Ma, L., Hu, X., Wu, M. H., Wang, J., et al. (2025) · MedComm · doi:10.1002/mco2.70247 · Cited by 22
“…a vast amount of medical knowledge and reasoning ability through the pretraining process, enabling them to apply existing knowledge and reasoning strategies when processing new image data, thus improving diagnostic efficiency and accuracy, even in cases of low image data availability [352–354].”
Dwyer, B., Flathers, M., Sano, A., Dempsey, A. G., Cipriani, A., Gazi, A. H., Hill, B., Gorban, C., Rodríguez, C. I., Stromeyer, C., et al. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00049-6 · Cited by 31
Lonergan, R. M., Curry, J., Dhas, K., & Simmons, B. I. (2023) · Cureus · doi:10.7759/cureus.48788 · Cited by 19
“The second gap is that existing work evaluating LLM performance on medical question-and-answer datasets tends to assess accuracy across the dataset as a whole, masking any variation present in relevant subgroups of the data, such as medical specialty [4,10].”
BackgroundMethodPrompt-based Extraction of Social Determinants of Health Using Few-shot Learning
Ramachandran, G. K., Fu, Y., Han, B., Lybarger, K., Dobbins, N., Uzuner, O., & Yetisgen, M. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.41 · Cited by 18
Streamline automated biomedical discoveries with agentic bioinformatics
Zhou, J., Jiang, J., Han, Z., Wang, Z., & Gao, X. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf505 · Cited by 28
“Recent studies have demonstrated that medical LLMs can produce clinically unsupported content with high linguistic fluency [48, 49].”
Meduri, K., Gonaygunta, H., Nadella, G. S., Pawar, P. P., & Kumar, D. (2024) · IJARCCE · doi:10.17148/ijarcce.2024.13114 · Cited by 19
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657882 · Cited by 17
“Liévin et al. [21] investigated various chain-of-thought prompting techniques on GPT-3.”
BackgroundResultInfluentialSocrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17
Synoptic reporting by summarizing cancer pathology reports using large language models
Rajaganapathy, S., Chowdhury, S., Li, X., Buchner, V., He, Z., Zhang, R., Jiang, X., Yang, P., Cerhan, J. R., & Zong, N. (2025) · npj Health Systems · doi:10.1038/s44401-025-00013-8 · Cited by 22
“These key advantages have triggered studies on evaluating the effectiveness of LLMs in the clinical setting, such as clinical text annotation 24 , medical question and answering 26 , and reasoning in the medical domain 27 — which formessentialcomponentsofsynopticreporting.”
Han, T., Jeong, W. K., & Shin, J. (2025) · ULTRASONOGRAPHY · doi:10.14366/usg.25012 · Cited by 16
Metadata
- Authors
- Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O.
- Affiliations
- Technical University of Denmark (DK)
- Copenhagen University Hospital (DK)
- University of Copenhagen (DK)
- Corresponding author
- Liévin, Valentin
- Venue
- Patterns
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 5
- Issue
- 3
- Pages
- 100943
- ISSN
- 2666-3899 (ISSN-L)
- PMC
- PMC10935498
- OpenAlex
- W4392359953
- Semantic Scholar
- d697b440dd0e65a05fe027e4c0ea85f62dcba033
- Cited by
- 343
- References
- 69
- Citations by year
- 2026: 95 · 2025: 108 · 2024: 79
- FWCI
- 26.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Patterns (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Danmarks Grundforskningsfond
- Novo Nordisk Fonden (NNF20OC0062606)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, medical question answering, medical reasoning, chain-of-thought prompting, retrieval augmentation, few-shot prompting, prompt engineering, open-source language models, Llama 2, GPT-3.5, MedMCQA, PubMedQA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
