CoChat
Sign Up
Cited by 348Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Can large language models reason about medical questions?

Open access
Liévin, Valentin, Hother, Christoffer Egeberg, Motzfeldt, Andreas Geert, Winther, Ole·2024·Patterns
DOI 10.1016/j.patter.2024.100943PMC PMC10935498
Open this paper in CoChat — free accountOpen PDF

Abstract

Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.

Verified 2026-10-07 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

This paper evaluates whether GPT-3.5 and open-source LLMs can answer and reason about medical questions across MedQA-USMLE, MedMCQA, and PubMedQA using prompting, chain-of-thought (CoT), retrieval augmentation, and ensembling. It finds that LLMs can often reason and recall expert knowledge, and that Codex 5-shot CoT with many samples reaches passing scores on all three benchmarks.

Key findings

  • Zero-shot InstructGPT outperformed the best fine-tuned BERT baselines on all three datasets: +1.4% on USMLE, +1.0% on MedMCQA, and +1.1% on PubMedQA.
  • Expert annotation of 50 USMLE CoTs found that incorrect answers often contained reasoning errors (86%), insufficient/incorrect knowledge (74%), and reading-comprehension failures (50%); still, 59% of CoTs leading to incorrect answers contained at least one correct reasoning step and 65% showed correct knowledge recall.
  • Codex 5-shot CoT with an ensemble of up to k=100 samples reached passing scores: USMLE 60.2%, MedMCQA 62.7%, and PubMedQA 78.2%; Llama 2 70B also passed USMLE with 62.5% accuracy.
  • Codex 5-shot CoT and Llama 2 were close to well calibrated, though calibration was worse for PubMedQA; prompting also induced answer-position biases, with direct prompting overestimating C/D and underestimating A, and CoT prompting underestimating B/C while overestimating D.

Methods

  • The study used three medical multiple-choice QA datasets: MedQA-USMLE, MedMCQA, and PubMedQA, with questions and reference explanations used for few-shot CoT prompts.
  • Models included closed-source GPT-3.5 series (InstructGPT, text-davinci-002; Codex, code-davinci-002) and open-source models from 7B to 70B parameters, including Llama 2, Vicuna, Guanaco, Falcon, MPT, and GPT-NeoX.
  • Prompting conditions included zero-shot direct, zero-shot CoT with domain-specific cues, few-shot direct/CoT, BM25 retrieval augmentation over Wikipedia, and ensemble/self-consistency sampling up to k=100 completions; a medical expert annotated 50 generated CoTs using six success/failure patterns.

Limitations

  • The paper states that deploying LLMs in clinical scenarios requires more robust techniques, that it exposed only one form of bias (answer-option ordering), and that many more biases may exist, including hidden training-data biases such as gender and race.
  • The paper notes that benchmark contamination in training sets is a valid concern for both proprietary and open-source LLMs, and that a large gap remains between the models and human experts despite passing scores.
  • Evident from the text: the expert CoT evaluation is based on a subset of only 50 USMLE CoTs annotated by one medical expert, and the BM25 retrieval experiment is described as limited.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.patter.2024.100943

Cites

79 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    When using the direct prompt, BM25 grounding improved MedMCQA accuracy by +2.7%.

    “When using the direct prompt, we recorded gains on the USMLE (+1.3%) and on the MedMCQA (+2.7%) datasets, suggesting that retrieval augmentation might be beneficial.”
    Quote in the paperSubstantial claim
  • Findingown data

    The smaller model text-curie-002 delivered close to random performances on USMLE, with a maximum accuracy of 27.9%.

    “The smaller model text-curie-002 delivered close to random performances, with a maximum accuracy of 27.9%.”
    Quote in the paperSubstantial claim
  • Findingown data

    Without BM25 grounding, InstructGPT outperformed the best BERT methods on MedMCQA by +1.0%.

    “Without BM25 grounding, InstructGPT scored +1.4% on the USMLE questions, +1.0% on the MedMCQA exam questions, and +1.1% on PubMedQA over the best BERT methods.”
    Quote in the paperSubstantial claim
  • Findingown data

    Domain-specific CoT prompts #2-#5 did not significantly outperform the original CoT prompt #1.

    “Nonetheless, the domain-specific CoT prompts #2–#5 did not significantly outperform the original CoT prompt #1.”
    Quote in the paperSubstantial claim
  • Findingown data

    Among questions answered incorrectly, 86% of generated CoTs contained reasoning errors (pattern D).

    “We found that most of the questions answered incorrectly triggered generating CoTs that contained reasoning errors (pattern D, 86%) and that exhibited a lack of knowledge (pattern E, 74%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
    • Rests on an assumption: The generated CoTs reflect the model’s reasoning process.
  • May combine two claims. 20
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 10

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    GPT-3.5/Codex 5-shot CoT reaches passing scores on MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).

    “Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Direct prompting led to overestimating labels C and D while underestimating label A.

    “Direct prompting led to overestimating labels C and D while underestimating label A.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Performances were lower for each considered CoT cue except on USMLE, where half of the CoT prompts resulted in small improvements over the direct prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).

    “Performances were lower for each of the considered CoT cues, except in the case of the USMLE dataset, for which half of the CoT prompts resulted in small improvements over the direct prompt (+1.1% using CoT prompt #1 vs. using the direct prompt).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    GPT-3.5/Codex 5-shot CoT yields calibrated predictive distributions.

    “Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Combining the predictions of multiple prompts outperformed single-prompt predictions, except on PubMedQA, where the direct prompt performed exceptionally well.

    “Combining the predictions of multiple prompts outperformed the single-prompt predictions, except in the case of the PubMedQA dataset, for which the direct prompt performed exceptionally well.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both direct and CoT prompting schemes led to biased predictive frequencies on USMLE.

    “Both prompting schemes led to biased predictive frequencies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    InstructGPT can often read, reason, and recall/exploit expert knowledge.

    “Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge.”
    • Also stated in: § Contributions · plain
      “A medical expert annotated a subset of CoTs generated by zero-shot InstructGPT and supports that InstructGPT, in many cases, can reason and exploit memorized expert knowledge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.
    • Rests on an assumption: The generated CoTs reflect the model’s reasoning process.

  • May combine two claims.

    Findingown data

    CoT prompting led to underestimating labels B and C while overestimating label D.

    “CoT prompting led to underestimating B and C while overestimating label D.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    InstructGPT was often able to reason, recall medical knowledge, and comprehend the given problem.

    “Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.

  • May combine two claims.

    Findingown data

    Best performances are obtained using Codex, outperforming the human-aligned InstructGPT, which is a fine-tuned version of Codex.

    “Best performances are obtained using Codex, outperforming the human-aligned InstructGPT, which is a fine-tuned version of Codex.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With 100 samples, Codex 5-shot CoT delivered unprecedented performances on the three datasets, bridging the gap with human-level performances and virtually passing the USMLE by 0.2% points.

    “With 100 samples, Codex 5-shot CoT delivered unprecedented performances on the three datasets, bridging the gap with human-level performances and virtually passing the USMLE by 0.2% points.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
    • Rests on an assumption: The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.
    • Rests on an assumption: The human baseline performances reported are valid lower bounds for human performance.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Inspecting the generated CoTs revealed lesser-quality samples.

    “Although Codex performed exceptionally well with 5 shots, Codex yield feeble performances with zero-shot CoT; inspecting the generated CoTs revealed lesser-quality samples (supplemental information section I).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although such prompts, taken separately, did not outperform the original zero-shot CoT prompt, more specific prompts appeared to trigger alternative strategies such as working by elimination or manipulating equations.

    “Although such prompts, taken separately, did not outperform the original zero-shot CoT prompt (see Table S2 in supplemental information section A), more specific prompts appeared to trigger alternative strategies such as working by elimination or manipulating equations (see supplemental information sections B and I).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Codex 5-shot CoT appears to be overall calibrated, although the calibration is worse for the PubMedQA dataset.

    “Codex 5-shot CoT appears to be overall calibrated, although the calibration is worse for the PubMedQA dataset.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Conditioning predictions on a knowledge base is an alternative research direction for improving language models.

    “Conditioning the predictions on a knowledge base is an alternative research direction for improving language models.47,48,49”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [47] Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Adv. Neural Inf. Process. Syst.

      Not checked in the free run.

    • cites [48] Borgeaud, 2022, Improving language models by retrieving from trillions of tokens, PMLR

      Not checked in the free run.

    • cites [49] Lazaridou, 2022, Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    MedQA-USMLE enables assessing LLM capabilities in realistic clinical scenarios requiring specialized medical knowledge, advanced reasoning, and human-level reading comprehension.

    “Specialized datasets such as the MedQA-US Medical Licensing Examination (USMLE) enable assessing the capabilities of LLMs in realistic clinical scenarios requiring specialized medical knowledge, advanced reasoning capabilities, and human-level reading comprehension skills.32”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] Jin, 2021, What disease does this patient have? a large-scale open domain question answering dataset from medical exams, Appl. Sci. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    CoT prompting led to substantial improvements on many reasoning-intensive tasks, bridging the gap with human-level performances for most hard BIG-bench tasks.

    “CoT prompting led to substantial improvements on many reasoning-intensive tasks,20,21,22,23 allowing us to bridge the gap with human-level performances for most of the hard BIG-bench tasks.24”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [20] Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst.

      Not checked in the free run.

    • cites [21] Zhou, 2022, Least-to-most Prompting Enables Complex Reasoning in Large Language Models, arXiv

      Not checked in the free run.

    • cites [22] Drozdov, 2022, Compositional Semantic Parsing with Large Language Models, arXiv

      Not checked in the free run.

    • cites [23] Nye, 2021, Show Your Work: Scratchpads for Intermediate Computation with Language Models, arXiv

      Not checked in the free run.

    • cites [24] Suzgun, 2022, Challenging BIG-bench Tasks and Whether Chain-of-thought Can Solve Them, arXiv

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The experiment was repeated using randomly permuted labels, and similar patterns were observed.

    “We repeated the experiment using randomly permuted labels and observed similar patterns (see supplemental information section F).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Zero-shot CoT allows generating CoTs using a single domain-agnostic cue, "Let's think step by step".

    “As an alternative to writing reference step-by-step solutions, zero-shot CoT allows for generating CoTs using a single and domain-agnostic cue: “Let’s think step by step”25”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] Kojima, 2022, Large language models are zero-shot reasoners, Adv. Neural Inf. Process. Syst.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Combining multiple sampled CoTs (self-consistency) showed improvements over single-sample CoT methods.

    “Combining multiple sampled CoTs (also known as self-consistency) has also been explored in past work, and showed improvements over single-sample CoT methods.42,43”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv

      Not checked in the free run.

    • cites [43] Li, 2022, On the Advance of Making Language Models Better Reasoners, arXiv

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert using six categories.

    “A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Both open- and closed-source LLMs hold great potential for assisting human decision-making in medicine and beyond.

    “Although benchmark contamination in training sets for both proprietary and open-source LLMs is a valid concern, these results indicate that both open- and closed-source LLMs hold great potential for assisting human decision-making in medicine and beyond.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Both GPT-4 and MedPalm 2 have achieved performance on USMLE around 85%.

    “Since the completion of v.2 of this work, both GPT-4 and MedPalm 2 have achieved performance on USMLE around 85%.35,63”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35] Nori, 2023, Capabilities of GPT-4 on Medical Challenge Problems, arXiv

      Not checked in the free run.

    • cites [63] Singhal, 2023, Towards Expert-level Medical Question Answering with Large Language Models, arXiv

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    CoTs are human readable and therefore interpretable.

    “Nevertheless, CoTs are human readable and therefore interpretable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    LLMs can mostly comprehend complex medical questions, can often recall expert-domain knowledge, and can often perform non-trivial reasoning steps.

    “Our expert evaluation suggests that LLMs can mostly comprehend complex medical questions, can often recall expert-domain knowledge, and can often perform non-trivial reasoning steps.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.
    • Rests on an assumption: The expert evaluation of CoTs is a valid measure of soundness and interpretability.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    CoTs can be combined and/or filtered using human or automated feedback.

    “CoTs can be combined and/or filtered using human or automated feedback.42,60”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [42] Wang, 2022, Self-consistency Improves Chain of Thought Reasoning in Language Models, arXiv

      Not checked in the free run.

    • cites [60] Cobbe, 2021, Training Verifiers to Solve Math Word Problems, arXiv

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    CoT prompting proved to be a powerful tool leading to better performances and more interpretable predictions.

    “CoT prompting proved to be a powerful tool leading to better performances and more interpretable predictions.”
    Quote in the paperSubstantial claim
27 of 27

  • The paper relies on this without support.

    It was unclear how large language models perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.

    “Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.”

  • The paper relies on this without support.

    At the time of the research, studies applying large language models to the medical domain were still needed.

    “At the dawn of this research work, although LLMs had been tested on large benchmarks (MMLU,31 BIG-bench18), studies applied to the medical domain were still needed.”

  • The paper relies on this without support.

    Large language models memorize part of the knowledge embedded into the training data.

    “LLMs memorize part of the knowledge embedded into the training data; nonetheless, models might fail to reuse this knowledge effectively during prediction.”

  • The paper relies on this without support.

    A single medical expert’s annotation is sufficient to evaluate the quality of generated CoTs.

    “A subset of 50 CoTs generated based on USMLE questions were annotated by a medical expert (C.E.H.) using the six categories.”

  • The paper relies on this without support.

    The generated CoTs reflect the model’s reasoning process.

    “The CoTs that result from that prompt not only appear to expose valid reasoning but also translate into superior zero-shot performances (see example in Figure 1).”

  • The paper relies on this without support.

    The USMLE, MedMCQA, and PubMedQA datasets are valid benchmarks for medical reasoning ability.

    “Zero-shot InstructGPT and Codex outperformed fine-tuned BERT models on three challenging question-answering datasets (see zero-shot benchmark and supplemental information section A).”

  • The paper relies on this without support.

    The chosen passing-score thresholds (60% for USMLE, 50% for MedMCQA, 78% for PubMedQA) are appropriate for defining passing.

    “Using an ensemble of the k samples, Codex 5-shot CoT reaches the passing score on the three tasks (see Table 1): the USMLE dataset (60.2% ≥ 60%), the MedMCQA dataset (62.7% ≥ 50%), and the PubMedQA dataset (78.2% ≥ 78%).”

  • The paper relies on this without support.

    The human baseline performances reported are valid lower bounds for human performance.

    “We report the best fine-tuned models and the human baseline.”

  • The paper relies on this without support.

    The expert evaluation of CoTs is a valid measure of soundness and interpretability.

    “Our expert evaluation (see expert evaluation of the generated CoTs) revealed that CoTs are often sound: even though InstructGPT still does make mistakes, it was often able to reason, recall medical knowledge, and comprehend the given problem.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs.

    “Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”

    From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]

    The Experiments section reports that this study used 2,000 NVIDIA A100 GPU hours to benchmark open-source models and used the Codex beta program for other experiments. It does not establish that most published medical AI benchmarks use enterprise GPUs or cloud APIs, or that they omit hardware specifications and inference costs. In fact, it names its own GPU hardware. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Extensive prior work has benchmarked LLMs against medical board exams and sophisticated research-based questions.

    “Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”

    From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [14]

  • LLMs have surpassed the average human score on the USMLE, rising from 38.1% in September 2021 to 90.2% in November 2023, where the human passing score is 60% and human expert score is 87%.

    “LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”

    From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [4]

  • Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) or general reasoning capabilities.

    “Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) [6,7] or general reasoning capabilities [2].”

    From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [7]

  • Errors leading to incorrect recommendations by the large language model were classified using a prior evaluation framework into 4 error types: incorrect reading comprehension, incorrect recall of knowledge, incorrect reasoning step, and not enough information.

    “Finally, we classified the errors leading to incorrect recommendations by the LLM, leveraging a prior evaluation framework [29]. We classified each error as 1 of 4 error types: incorrect reading comprehension, incorrect recall of knowledge, incorrect reasoning step, and not enough information, as described in Table 1.”

    From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [29]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting Codex MedMCQA accuracy

    Major

    The abstract and results text report 62.7% MedMCQA accuracy for Codex 5-shot CoT, but Table 1 reports 59.7% for the same model and dataset.

  • PubMedQA passing score claimed despite none being listed

    Minor

    The text says Codex reached the passing score on PubMedQA by exceeding 78%, but Table 1 lists no PubMedQA passing score. It identifies 78% as the human expert score instead.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper AI21 Labs, 1, 9

Cited by

Sorted by citations

Related

Authors
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O.
Affiliations
  • Technical University of Denmark (DK)
  • Copenhagen University Hospital (DK)
  • University of Copenhagen (DK)
Corresponding author
Liévin, Valentin
ORCID
Venue
Patterns
Publisher
Elsevier BV
Published
2024-03
Volume
5
Issue
3
Pages
100943
ISSN
2666-3899 (ISSN-L)
DOI
10.1016/j.patter.2024.100943
PMC
PMC10935498
OpenAlex
W4392359953
Semantic Scholar
d697b440dd0e65a05fe027e4c0ea85f62dcba033
Cited by
343
References
69
Citations by year
2026: 95 · 2025: 108 · 2024: 79 · 2023: 60 · 2022: 1
FWCI
26.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Patterns (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Danmarks Grundforskningsfond
  • Novo Nordisk Fonden (NNF20OC0062606)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, medical question answering, medical reasoning, chain-of-thought prompting, retrieval augmentation, few-shot prompting, prompt engineering, open-source language models, Llama 2, GPT-3.5, MedMCQA, PubMedQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review