Cited by 67Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Reasoning with large language models for medical question answering

Open access
Lucas, Mary M, Yang, Justin, Pomeroy, Jon K, Yang, Christopher C·2024·Journal of the American Medical Informatics Association

Abstract

Objectives: To investigate approaches of reasoning with large language models (LLMs) and to propose a new prompting approach, ensemble reasoning, to improve medical question answering performance with refined reasoning and reduced inconsistency. Materials and Methods: We used multiple choice questions from the USMLE Sample Exam question files on 2 closed-source commercial and 1 open-source clinical LLM to evaluate our proposed approach ensemble reasoning. Results: On GPT-3.5 turbo and Med42-70B, our proposed ensemble reasoning approach outperformed zero-shot chain-of-thought with self-consistency on Steps 1, 2, and 3 questions (+3.44%, +4.00%, and +2.54%) and (2.3%, 5.00%, and 4.15%), respectively. With GPT-4 turbo, there were mixed results with ensemble reasoning again outperforming zero-shot chain-of-thought with self-consistency on Step 1 questions (+1.15%). In all cases, the results demonstrated improved consistency of responses with our approach. A qualitative analysis of the reasoning from the model demonstrated that the ensemble reasoning approach produces correct and helpful reasoning. Conclusion: The proposed iterative ensemble reasoning has the potential to improve the performance of LLMs in medical question answering tasks, particularly with the less powerful LLMs like GPT-3.5 turbo and Med42-70B, which may suggest that this is a promising approach for LLMs with lower capabilities. Additionally, the findings show that our approach helps to refine the reasoning generated by the LLM and thereby improve consistency even with the more powerful GPT-4 turbo. We also identify the potential and need for human-artificial intelligence teaming to improve the reasoning beyond the limits of the model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocae131

Cites

27 of 28 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Extensive prior work has benchmarked LLMs against medical board exams and sophisticated research-based questions.

    “Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”

    From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [15]

    The 'LLMs in healthcare' section describes multiple prior studies benchmarking LLMs on medical board-exam questions, including CIBE, AKT, NBME and USMLE questions. The passages do not establish that prior work also benchmarked LLMs on sophisticated research-based questions. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Frank MC. Baby steps in evaluating the capacities of large lan- guage models. Nat Rev Psychol. 2023;2(8):451-452. https://www. nature.com/articles/s44159-023-00211-x

Cited by

Sorted by citations

Authors
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C.
Affiliations
  • Drexel University (US)
  • University of Maryland, College Park (US)
  • University of Pennsylvania Health System (US)
Corresponding author
Yang, Christopher C
Venue
Journal of the American Medical Informatics Association
Published
2024-07-03
Volume
31
Issue
9
Pages
1964-1975
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4400307907
Cited by
66
References
21
Citations by year
2026: 32 · 2025: 31 · 2024: 3
FWCI
6.18
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Science Foundation (IIS-1741306, IIS-2235548)
  • Department of Defense (DoD W91XWH-05-1-023)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, medical question answering, prompting, chain-of-thought prompting, self-consistency, response consistency, human-AI teaming
MeSH terms
Educational Measurement, Humans, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review