The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.
Can large language models reason about medical questions?
Open accessAbstract
Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper evaluates whether GPT-3.5 and open-source LLMs can answer and reason about medical questions across MedQA-USMLE, MedMCQA, and PubMedQA using prompting, chain-of-thought (CoT), retrieval augmentation, and ensembling. It finds that LLMs can often reason and recall expert knowledge, and that Codex 5-shot CoT with many samples reaches passing scores on all three benchmarks.
Key findings
- Zero-shot InstructGPT outperformed the best fine-tuned BERT baselines on all three datasets: +1.4% on USMLE, +1.0% on MedMCQA, and +1.1% on PubMedQA.
- Expert annotation of 50 USMLE CoTs found that incorrect answers often contained reasoning errors (86%), insufficient/incorrect knowledge (74%), and reading-comprehension failures (50%); still, 59% of CoTs leading to incorrect answers contained at least one correct reasoning step and 65% showed correct knowledge recall.
- Codex 5-shot CoT with an ensemble of up to k=100 samples reached passing scores: USMLE 60.2%, MedMCQA 62.7%, and PubMedQA 78.2%; Llama 2 70B also passed USMLE with 62.5% accuracy.
- Codex 5-shot CoT and Llama 2 were close to well calibrated, though calibration was worse for PubMedQA; prompting also induced answer-position biases, with direct prompting overestimating C/D and underestimating A, and CoT prompting underestimating B/C while overestimating D.
Methods
- The study used three medical multiple-choice QA datasets: MedQA-USMLE, MedMCQA, and PubMedQA, with questions and reference explanations used for few-shot CoT prompts.
- Models included closed-source GPT-3.5 series (InstructGPT, text-davinci-002; Codex, code-davinci-002) and open-source models from 7B to 70B parameters, including Llama 2, Vicuna, Guanaco, Falcon, MPT, and GPT-NeoX.
- Prompting conditions included zero-shot direct, zero-shot CoT with domain-specific cues, few-shot direct/CoT, BM25 retrieval augmentation over Wikipedia, and ensemble/self-consistency sampling up to k=100 completions; a medical expert annotated 50 generated CoTs using six success/failure patterns.
Limitations
- The paper states that deploying LLMs in clinical scenarios requires more robust techniques, that it exposed only one form of bias (answer-option ordering), and that many more biases may exist, including hidden training-data biases such as gender and race.
- The paper notes that benchmark contamination in training sets is a valid concern for both proprietary and open-source LLMs, and that a large gap remains between the models and human experts despite passing scores.
- Evident from the text: the expert CoT evaluation is based on a subset of only 50 USMLE CoTs annotated by one medical expert, and the BM25 retrieval experiment is described as limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 2 Prompt templates In the table, we use typewriter style and brackets to represent [provided data] such as the question, additional context, or the answer and <completions> generated by GPT-3. The symbol ∅ represents an empty string.](/paper-figures/d27c38f67a2f4fd6a676a3998ef7b6a3/can-large-language-models-reason-about-medical-questions-figure-2.jpg)
Verification
Claims the paper makes
55 Own result · 11 Not checked yet · 10 Own interpretation · 5 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When using the direct prompt, we recorded gains on the USMLE (+1.3%) and on the MedMCQA (+2.7%) datasets, suggesting that retrieval augmentation might be beneficial.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The smaller model text-curie-002 delivered close to random performances, with a maximum accuracy of 27.9%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without BM25 grounding, InstructGPT scored +1.4% on the USMLE questions, +1.0% on the MedMCQA exam questions, and +1.1% on PubMedQA over the best BERT methods.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, the domain-specific CoT prompts #2–#5 did not significantly outperform the original CoT prompt #1.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that most of the questions answered incorrectly triggered generating CoTs that contained reasoning errors (pattern D, 86%) and that exhibited a lack of knowledge (pattern E, 74%).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The best performances on the USMLE and MedMCQA datasets were obtained by combining retrieval-augmented prompts and setting a maximum of 53.1% accuracy on the USMLE dataset and 48.8% validation accuracy on the MedMCQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, inspecting the CoTs leading to incorrect answers more closely, we found that 47% of those were inconclusive: the model could not narrow down the prediction to a single answer.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.”
§ Can large language models reason about medical questions? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-aligned largest GPT-3 text-davinci-001 scored 40.2%, whereas the largest code pre-trained Codex scored 52.9%, and the code-pre-trained and human-aligned InstructGPT scored 47.1%.”
§ Can large language models reason about medical questions? > Experiments > Datasets and models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 20
- The quote only partly states this claim. 2
- Probably not a substantial claim. 10
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct prompting led to overestimating labels C and D while underestimating label A.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performances were lower for each of the considered CoT cues, except in the case of the USMLE dataset, for which half of the CoT prompts resulted in small improvements over the direct prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
§ Can large language models reason about medical questions? > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining the predictions of multiple prompts outperformed the single-prompt predictions, except in the case of the PubMedQA dataset, for which the direct prompt performed exceptionally well.”
§ Can large language models reason about medical questions? > Experiments > Zero-shot benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both prompting schemes led to biased predictive frequencies.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge.”
- Also stated in: § Contributions · plain
“A medical expert annotated a subset of CoTs generated by zero-shot InstructGPT and supports that InstructGPT, in many cases, can reason and exploit memorized expert knowledge.”
§ Can large language models reason about medical questions? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
- Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
- Also stated in: § Contributions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting led to underestimating B and C while overestimating label D.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Best performances are obtained using Codex, outperforming the human-aligned InstructGPT, which is a fine-tuned version of Codex.”
§ Can large language models reason about medical questions? > Discussion > Scale, code pre-training, human-alignment, and few-shot learning
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 100 samples, Codex 5-shot CoT delivered unprecedented performances on the three datasets, bridging the gap with human-level performances and virtually passing the USMLE by 0.2% points.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
boosteddirectn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
- Rests on an assumption: The human baseline performances reported are valid lower bounds for human performance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although Codex performed exceptionally well with 5 shots, Codex yield feeble performances with zero-shot CoT; inspecting the generated CoTs revealed lesser-quality samples (supplemental information section I).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although such prompts, taken separately, did not outperform the original zero-shot CoT prompt (see Table S2 in supplemental information section A), more specific prompts appeared to trigger alternative strategies such as working by elimination or manipulating equations (see supplemental information sections B and I).”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Codex 5-shot CoT appears to be overall calibrated, although the calibration is worse for the PubMedQA dataset.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
hedgedcorrelationaln = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Conditioning the predictions on a knowledge base is an alternative research direction for improving language models.47,48,49”
§ Can large language models reason about medical questions? > Method > Retrieval augmentation
What the sources say
- Not checked yetcites [47] Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [48] Borgeaud, 2022, Improving language models by retrieving from trillions of tokens, PMLR
- Not checked yetcites [49] Lazaridou, 2022, Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Specialized datasets such as the MedQA-US Medical Licensing Examination (USMLE) enable assessing the capabilities of LLMs in realistic clinical scenarios requiring specialized medical knowledge, advanced reasoning capabilities, and human-level reading comprehension skills.32”
§ Can large language models reason about medical questions? > Introduction > LLMs and medical applications
What the sources say
- Not checked yetcites [32] Jin, 2021, What disease does this patient have? a large-scale open domain question answering dataset from medical exams, Appl. Sci. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“CoT prompting led to substantial improvements on many reasoning-intensive tasks,20,21,22,23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.24”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [20] Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst.
- Not checked yetcites [21] Zhou, 2022, Least-to-most Prompting Enables Complex Reasoning in Large Language Models, arXiv
- Not checked yetcites [22] Drozdov, 2022, Compositional Semantic Parsing with Large Language Models, arXiv
- Not checked yetcites [23] Nye, 2021, Show Your Work: Scratchpads for Intermediate Computation with Language Models, arXiv
- Not checked yetcites [24] Suzgun, 2022, Challenging BIG-bench Tasks and Whether Chain-of-thought Can Solve Them, arXiv
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We repeated the experiment using randomly permuted labels and observed similar patterns (see supplemental information section F).”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As an alternative to writing reference step-by-step solutions, zero-shot CoT allows for generating CoTs using a single and domain-agnostic cue: “Let’s think step by step”25”
§ Can large language models reason about medical questions? > Introduction > Chain-of-thought (CoT) prompting
What the sources say
- Not checked yetcites [25] Kojima, 2022, Large language models are zero-shot reasoners, Adv. Neural Inf. Process. Syst.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Combining multiple sampled CoTs (also known as self-consistency) has also been explored in past work, and showed improvements over single-sample CoT methods.42,43”
§ Can large language models reason about medical questions? > Method > Answer likelihood
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [43] Li, 2022, On the Advance of Making Language Models Better Reasoners, arXiv
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although benchmark contamination in training sets for both proprietary and open-source LLMs is a valid concern, these results indicate that both open- and closed-source LLMs hold great potential for assisting human decision-making in medicine and beyond.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Since the completion of v.2 of this work, both GPT-4 and MedPalm 2 have achieved performance on USMLE around 85%.35,63”
§ Can large language models reason about medical questions? > Discussion > Conclusion
What the sources say
- Not checked yetcites [35] Nori, 2023, Capabilities of GPT-4 on Medical Challenge Problems, arXiv
- Not checked yetcites [63] Singhal, 2023, Towards Expert-level Medical Question Answering with Large Language Models, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, CoTs are human readable and therefore interpretable.”
§ Can large language models reason about medical questions? > Discussion > Zero-shot CoT prompting often yields sound and interpretable step-by-step solutions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our expert evaluation suggests that LLMs can mostly comprehend complex medical questions, can often recall expert-domain knowledge, and can often perform non-trivial reasoning steps.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
- Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.
None of the sources cited for this claim have been checked yet.
In the paper
“CoTs can be combined and/or filtered using human or automated feedback.42,60”
§ Can large language models reason about medical questions? > Discussion > Generating and combining many CoTs bridges the gap with human-level performances
What the sources say
- Not checked yetcites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv
- Not checked yetcites [60] Cobbe, 2021, Training Verifiers to Solve Math Word Problems, arXiv
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CoT prompting proved to be a powerful tool leading to better performances and more interpretable predictions.”
§ Can large language models reason about medical questions? > Discussion > Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
It was unclear how large language models perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.
“Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.”
The paper relies on this without support.
At the time of the research, studies applying large language models to the medical domain were still needed.
“At the dawn of this research work, although LLMs had been tested on large benchmarks (MMLU,31 BIG-bench18), studies applied to the medical domain were still needed.”
The paper relies on this without support.
Large language models memorize part of the knowledge embedded into the training data.
“LLMs memorize part of the knowledge embedded into the training data; nonetheless, models might fail to reuse this knowledge effectively during prediction.”
The paper relies on this without support.
A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
“A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
The paper relies on this without support.
The generated CoTs reflect the model’s reasoning process.
“The CoTs that result from that prompt not only appear to expose valid reasoning but also translate into superior zero-shot performances (see example in Figure 1).”
The paper relies on this without support.
The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
“Zero-shot InstructGPT and Codex outperformed fine-tuned BERT models on three challenging question-answering datasets (see zero-shot benchmark and supplemental information section A).”
The paper relies on this without support.
The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
The paper relies on this without support.
The human baseline performances reported are valid lower bounds for human performance.
“We report the best fine-tuned models and the human baseline.”
The paper relies on this without support.
The expert evaluation of CoTs is a valid measure of soundness and interpretability.
“Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) [6,7] or general reasoning capabilities [2].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]
In the citing paper
“Finally, we classified the errors leading to incorrect recommendations by the LLM, leveraging a prior evaluation framework [29]. We classified each error as 1 of 4 error types: incorrect reading comprehension, incorrect recall of knowledge, incorrect reasoning step, and not enough information, as described in Table 1.”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
“the MedMCQA dataset (62.7% ≥ 50%)”
§ Can large language models reason about medical questions? > Method > Answer likelihood
“| Codex 5-shot CoTa | 2022 | 60.2 | 59.7 | 78.2 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Can large language models reason about medical questions? > Experiments > Expert evaluation of the generated CoTs
“Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”
§ Can large language models reason about medical questions? > Method > Answer likelihood
“| Human (passing score) | – | 60.0 | 50.0 | – | | Human (expert score) | – | 87.0 | 90.0 | 78.0 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“, 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer architecture (Vaswani et al., 2017) and its high compatibility with massively parallel computation hardware.”
BackgroundJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“…word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models are now ubiquitous in natural language processing, notably, thanks to the Transformer…”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Language representations have transformed the field of natural language processing, from simple word vectors (Mikolov et al., 2013; Pennington et al., 2014) to deep contextualized representations (Peters et al., 2018; Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2018), language models…”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“This so far translated sustain gains (Kaplan et al., 2020), enabled new ways to interact with language models and made many of the past benchmarks obsolete; which triggered a community-wide effort to design a sufficiently difficult benchmarks (e.”
BackgroundDense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · arXiv · doi:10.48550/arxiv.2004.04906 · Cited by 5,311 · cited 1×
“, 2022) or dense retrievers (Karpukhin et al., 2020), (ii) using a more complete knowledge base Borgeaud et al.”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“18 Pre-train, prompt, and predict 19 is an emerging paradigm for applying LLMs to new problems without fine-tuning THE BIGGER PICTURE Foundation models have changed the way machine learning is practiced.”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“18 Pre-train, prompt, and predict 19 is an emerging paradigm for applying LLMs to new problems without fine-tuning THE BIGGER PICTURE Foundation models have changed the way machine learning is practiced.”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“20 CoT prompting led to substantial improvements on many reasoning-intensive tasks, 20–23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“Based on detailed answers generated by LLMs, we show that LLMs can often answer challenging medical questions by mobilizing expert knowledge and advanced reasoning capabilities.”
BackgroundMethodKung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 2×
“33,37 Throughoutthe development of this research, significant progress happened in the open-source world (Llama 2 38 ), and recently, there has been an investigation on both generalist and fine-tuned open-source LLMs applied to medical benchmarks.”
BackgroundLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“33,37 Throughoutthe development of this research, significant progress happened in the open-source world (Llama 2 38 ), and recently, there has been an investigation on both generalist and fine-tuned open-source LLMs applied to medical benchmarks.”
BackgroundLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“Based on detailed answers generated by LLMs, we show that LLMs can often answer challenging medical questions by mobilizing expert knowledge and advanced reasoning capabilities.”
BackgroundMethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Conditioning the predictions on a knowledge base is an alternative research direction for improving language models (Lewis et al., 2020; Borgeaud et al., 2021; Lazaridou et al., 2022).”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“33,37 Throughoutthe development of this research, significant progress happened in the open-source world (Llama 2 38 ), and recently, there has been an investigation on both generalist and fine-tuned open-source LLMs applied to medical benchmarks.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · cited 1×
“35 ChatGPT sparked the interest of the public and the research community, which hastened to benchmark it against USMLE questions, 36,37 turning to self-curated data instead of the peer-reviewed MedQA benchmark.”
BackgroundLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · cited 2×
“In the zero-shot CoT setting, this corresponds to the setup described in Kojima et al., 25 and the direct setting corresponds to Brown et al. 7”
MethodLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2×
“In the zero-shot CoT setting, this corresponds to the setup described in Kojima et al., 25 and the direct setting corresponds to Brown et al. 7”
MethodJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 6×
“9 for Vicuna.”
BackgroundInfluentialToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“Performances were lower for each of the considered CoT cues, except in the case of the USMLE dataset, for which halfoftheCoT prompts resultedinsmallimprovementsover thedirect prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).”
MethodInfluentialRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · UCL Discovery (University College London) · Cited by 1,187
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · cited 1×
“…(Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021; Laurençon et al., 2023) to using hundreds of billions of parameters and to training on gigabytes of text.”
BackgroundTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 1×
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · cited 1×
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 7×
“Generating and combining many CoTs bridges the gap with human-level performances CoTs can be combined and/or filtered using human or automated feedback (Wang et al., 2022; Cobbe et al., 2021).”
BackgroundMethodInfluentialLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1×
“45 The three datasets are summarized in Table 2.”
BackgroundTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 1×
“Generating and combining many CoTs bridges the gap with human-level performances CoTs can be combined and/or filtered using human or automated feedback (Wang et al., 2022; Cobbe et al., 2021).”
BackgroundBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
LinkBERT: Pretraining Language Models with Document LinksPreprint
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.15827 · Cited by 543 · cited 1×
“Fine-tuned BERT, BioLinkBERT 58 ; DPR, dense passage retrieval.”
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 487 · cited 1×
“45 The three datasets are summarized in Table 2.”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q. G., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 480
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.551 · Cited by 474
(2021) · International Conference on Learning Representations · Cited by 435
Opt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382 · cited 2×
“39 ,”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · cited 1×
“CoT prompting led to substantial improvements on many reasoning-intensive tasks (Wei et al., 2022; Zhou et al., 2022; Drozdov et al., 2022; Nye et al., 2021), allowing to bridge the gap with human-level performances for most of the hard BIG-bench tasks (Suzgun et al.”
BackgroundTowards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 2×
“At the dawn of this research work, although LLMs had been tested on large benchmarks (MMLU, 31 BIG-bench 18 ), studies applied to the medical domain were still needed.”
MethodSmith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286 · cited 1×
“…Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021; Laurençon et al., 2023) to using hundreds of billions of parameters…”
BackgroundLIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275 · cited 1×
“31”
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275 · cited 1×
“31”
Korngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · cited 1×
“29,30 Nonetheless, LLMs are powerful tools and therefore have the potential to transform the field of machine intelligence.”
BackgroundChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187
On the Advance of Making Language Models Better ReasonersPreprint
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W.-Z. (2022) · arXiv.org · doi:10.48550/arxiv.2206.02336 · Cited by 165 · cited 2×
“…language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021; Laurençon et al., 2023) to using hundreds of billions of parameters and to training on…”
BackgroundMedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157 · cited 1×
“…question answering datasets: USMLE, whichincludesdifficult real-world medicalques-tions targeting medical professionals 32 ; the MedMCQA, which gathers questions from medical school entrance exams 44 ; and the PubMedQA, which includes reading comprehension questions about PubMed abstracts.”
BackgroundShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1×
“20 CoT prompting led to substantial improvements on many reasoning-intensive tasks, 20–23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.”
BackgroundSezgın, E., Sirrianni, J., & Linwood, S. L. (2022) · JMIR Medical Informatics · doi:10.2196/32875 · Cited by 129 · cited 1×
“Therefore, deploying LLMs into sensitive areas such as healthcare must be operated with great care (Korngiebel & Mooney, 2021; Sezgin et al., 2022).”
BackgroundMEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 1×
“39 ,”
Compositional Semantic Parsing with Large Language ModelsPreprint
Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., & Zhou, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15003 · Cited by 95 · cited 1×
“CoT prompting led to substantial improvements on many reasoning-intensive tasks (Wei et al., 2022; Zhou et al., 2022; Drozdov et al., 2022; Nye et al., 2021), allowing to bridge the gap with human-level performances for most of the hard BIG-bench tasks (Suzgun et al.”
BackgroundThe BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
Augmenting Black-box LLMs with Medical Textbooks for Biomedical Question Answering
Wang, Y., Ma, X., & Chen, W. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.95 · Cited by 93
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., & Li, B. (2021) · arXiv · doi:10.48550/arxiv.2111.02840 · Cited by 88 · cited 1×
“28 Therefore, deploying LLMs into sensitive areas such as healthcare must be operated with great care.”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 1×
“57 We usedgreedydecoding(temperature t = 0)with k = 1sampleunlessspecified otherwise (e.g., ensemble methods).”
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C. E., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02433-3 · Cited by 66 · cited 1×
“51 All our benchmark results are summarized in supplemental information section A and Table S2.”
BackgroundSelf-citationInternet-augmented language models through few-shot prompting for open-domain question answering
Lazaridou, A., Gribovskaya, E., Stokowiec, W., & Grigorev, N. (2022) · arXiv · doi:10.48550/arxiv.2203.05115 · Cited by 64 · cited 1×
“49 We investigated whether grounding the model with additional context could improve the answering accuracy.”
BackgroundChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 1×
“, 2021), allowing to bridge the gap with human-level performances for most of the hard BIG-bench tasks (Suzgun et al., 2022).”
BackgroundJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Variational Open-Domain Question Answering
Liévin, V., Motzfeldt, A. G., Jensen, I. R., & Winther, O. (2022) · Research at the University of Copenhagen (University of Copenhagen) · doi:10.48550/arxiv.2210.06345 · Cited by 4
Self-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 1×
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundDeep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.05365
vlievin/medical-reasoning: Official release
Liévin, V. (2023) · Zenodo · doi:10.5281/zenodo.10301874
Self-citation
Research works printed but not linked: 1
- Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper AI21 Labs, 1, 9
Cited by
Sorted by citations
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
“[50] used Instruct GPT-3, an instruction-tuned LLM [63], and applied chain-of-thought prompting [91] on top to improve the results on the MedQA, MedMCQA, and PubMedQA datasets.”
BackgroundMethodA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“In professional domain reasoning tasks, zeroshot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved (Liévin et al., 2022).”
BackgroundToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
“In particular, these models have demonstrated impressive capabilities on multiple-choice research benchmarks 1–3 .”
Metadata
- Authors
- Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O.
- Affiliations
- Technical University of Denmark (DK)
- Copenhagen University Hospital (DK)
- University of Copenhagen (DK)
- Corresponding author
- Liévin, Valentin
- Venue
- Patterns
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 5
- Issue
- 3
- Pages
- 100943
- ISSN
- 2666-3899 (ISSN-L)
- PMC
- PMC10935498
- OpenAlex
- W4392359953
- Semantic Scholar
- d697b440dd0e65a05fe027e4c0ea85f62dcba033
- Cited by
- 343
- References
- 69
- Citations by year
- 2026: 95 · 2025: 108 · 2024: 79
- FWCI
- 26.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Patterns (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Danmarks Grundforskningsfond
- Novo Nordisk Fonden (NNF20OC0062606)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, medical question answering, medical reasoning, chain-of-thought prompting, retrieval augmentation, few-shot prompting, prompt engineering, open-source language models, Llama 2, GPT-3.5, MedMCQA, PubMedQA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
