CoChat

Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework

Open access
Aljohani, Bushra, Alsanoosy, Tawfeeq · 2026 · Information

The DOI matches its Crossref record; 73 of the 90 references it cites were found in a registry or an index, and 10 are not checked yet; 36 of the 47 sources its claims cite were checked for support.

Abstract

Evaluation was conducted on PubMedQA, MedMCQA, and MedQA-US using hybrid retrieval combining BM25 and MedCPT with GPT-4o and BioGPT; the hybrid retriever achieved 92.14% recall, 74.36% precision, and 82.30% F1, and GPT-4o with hybrid retrieval reached 89.4% faithfulness on PubMedQA.

Verified 2026-11-19 · 10 references not checked yet

Overview

CoChat Summary

## TL;DR

This paper proposes a modular retrieval-augmented generation (RAG) framework for medical question answering that combines BM25 and MedCPT retrieval with GPT-4o and BioGPT generation. On three healthcare QA datasets, the hybrid retriever outperformed single retrievers, and GPT-4o generally produced more faithful and better-balanced answers than BioGPT.

## Key findings

  • The proposed hybrid retriever achieved 92.14% context recall, 74.36% context precision, and 82.30% F1-score, outperforming BM25 (82.76% recall, 67.77% precision, 74.52% F1) and MedCPT (73.31% recall, 48.75% precision, 58.56% F1).
  • On PubMedQA, GPT-4o with hybrid retrieval reached 89.4% faithfulness, 82.7% answer relevancy, 93.1% context precision, 80.6% context recall, and 88.0% BERTScore F1.
  • On MedMCQA, GPT-4o achieved 78.1% answer relevancy and 77.9% context recall; on MedQA-US, it reached 34.7% faithfulness and 77.3% context recall.
  • BioGPT had very high context precision on PubMedQA (98.4%) but much lower faithfulness (32.3%), suggesting strong lexical overlap without consistently reliable grounding.

## Methods

  • The study used a modular RAG pipeline with four phases: data preparation and indexing, hybrid retrieval, response generation, and evaluation.
  • It evaluated three healthcare QA datasets: PubMedQA, MedMCQA, and MedQA-US; the text also says retrieval evaluation used a 15,000-question corpus, and generation evaluation was done on 3,000 questions sampled evenly from the three datasets.
  • Retrieval compared BM25, MedCPT, and a hybrid late-fusion approach that merged both retrievers; generation compared GPT-4o (few-shot prompting, temperature = 0, max_tokens = 512) and BioGPT (beam search).
  • Evaluation used context precision, context recall, F1, Average Precision, plus RAGAS and BERTScore for generation quality.

## Limitations

  • The paper explicitly states that it does not perform disease-specific or organ-level stratified analysis, so performance differences across subdomains such as cardiology, nephrology, and oncology were not separately examined.
  • It also states that future work could extend to subdomain-aware evaluation; this makes the current evaluation broad rather than specialty-specific.
  • Evident from the text: the framework was tested on only three datasets, and the reported generation evaluation was sampled from 3,000 questions, so the results may not cover all medical QA settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/info17020133

Cites

73 of 90 references verified0 citation errors · 7 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overview of the research methodology to enhance healthcare QA.
Figure 1 Overview of the research methodology to enhance healthcare QA.CC-BY · From the PDF
Figure 2 RAGAS evaluation results for generation across GPT-4o and BioGPT.
Figure 2 RAGAS evaluation results for generation across GPT-4o and BioGPT.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 3 BERTScore metric results.
Figure 3 BERTScore metric results.CC-BY · From the PDF

Verification

Claims the paper makes

19 Valid · 7 Not valid · 10 Not checked · 96 Own results and interpretations

67 more claims · 7 with their cited sources checked

Create a free account to see all 132 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 38
  • Probably not a substantial claim. 40
  • Borderline: one of the tests for a substantial claim is unclear. 1

Create a free account to see these claims.

Citations

Cites

Show all 90 references

Cited by

Metadata

Authors
Aljohani, B., & Alsanoosy, T.
Affiliations
Taibah University (SA)
Corresponding author
Alsanoosy, Tawfeeq
Venue
Information
Publisher
MDPI AG
Published
2026-02-01
Volume
17
Issue
2
Pages
133
ISSN
2078-2489 (ISSN-L)
OpenAlex
W7126397257
Cited by
3
References
62
Citations by year
2026: 3
FWCI
24.78
Type
journal article
Language
en
Open access
Locations
  • Information (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
medical question answering, retrieval-augmented generation, hybrid retrieval, sparse retrieval, dense retrieval, BM25, hallucination mitigation, GPT-4o, BioGPT, faithfulness, medical knowledge base, PubMedQA, MedMCQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.