CoChat
Sign Up
Cited by 2Artificial Intelligence in Healthcare and EducationBiomedical Text Mining and OntologiesTopic Modeling

VeReaFine: Iterative Verification Reasoning Refinement RAG for Hallucination-Resistant on Open-Ended Clinical QA

Open access
Phasook, Pakawat, Pitijaroonpong, Rapepong, Kinchagawat, Jiramet, Chinkamol, Amrest, Saengja, Tossaporn, Udomlapsakul, Kiartnarin, Sawatphol, Jitkapat, Ittichaiwong, Piyalitt·2025·Proceedings of the 24th Workshop on Biomedical Language Processing (Shared Tasks)
DOI 10.18653/v1/2025.bionlp-share.34
Open this paper in CoChat — free accountOpen PDF

Abstract

Pakawat Phasook, Rapepong Pitijaroonpong, Jiramet Kinchagawat, Amrest Chinkamol, Tossaporn Saengja, Kiartnarin Udomlapsakul, Jitkapat Sawatphol, Piyalitt Ittichaiwong. Proceedings of the 24th Workshop on Biomedical Language Processing (Shared Tasks). 2025.

AI snapshot

Written from the paper’s full text.

TL;DR

VeReaFine is a verifier-augmented RAG pipeline that iteratively retrieves, verifies, generates, and refines answers for open-ended clinical QA. On the BioNLP 2025 ClinIQLink shared task, its 7B generator with an 8B verifier rivals or surpasses a 32B medical model on several open-ended metrics, including P75 recall and multi-hop inverse step identification.

Key findings

  • Short-answer P75 recall: VeReaFine reaches 0.269, a +60% relative gain over Qwen2.5-7B without verification (0.168); M1-32B reaches 0.338 (+101% over Qwen2.5-7B).
  • Short-inverse P75 recall: VeReaFine attains 0.559, surpassing Qwen2.5-7B (0.358, +56%) and M1-32B (0.539, +4%).
  • Multi-hop and inverse P75 recall: VeReaFine hits 0.394 and 0.475; Qwen2.5-7B scores 0.236 and 0.387; M1-32B scores 0.396 and 0.387. The paper reports a +22% boost on inverse steps.
  • Multi-hop inverse step-identification: VeReaFine achieves 0.751 (+48% over Qwen2.5-7B’s 0.508), while M1-32B reaches 0.826. The abstract also states VeReaFine “reduces multi-hop inverse step-identification errors by 26%” and recovers over 90% of the 32B baseline’s step-identification fidelity.

Methods

  • Design: A tri-loop Verifier-RAG pipeline. Initial retrieval uses BM-Retriever-410M with FAISS to fetch top 10 passages, reranked by BM-Retriever-2B. MedReason-8B verifies relevance, builds a ground-truth context pool, and issues feedback queries for up to three retrieval iterations. Qwen2.5-7B-Instruct generates an answer from the ground-truth context; MedReason-8B labels claims as supported/unsupported, and the generator refines unsupported claims, capped at 2 iterations. Generator and verifier temperatures are set to 0.7.
  • Data: Hidden BioNLP 2025 ClinIQLink test set of 500 expert-curated medical QA pairs across four open-ended formats: short answer, short-inverse, multi-hop, and multi-hop-inverse. Retrieval indexes a curated 8 GB biomedical corpus (PubMed abstracts, clinical guidelines) via FAISS.
  • Analysis: Compares Sub1 VeReaFine, Sub2 M1-32B, and Sub3 Qwen2.5-7B without verification. Metrics include P25/P75 quantile recall over semantic partial matches, multi-hop inverse step-identification rate, and standard BLEU/METEOR/ROUGE for completeness.

Limitations

  • Stated: Shared-task constraints limited the team to three submissions, preventing direct comparison against RAG without a verifier and precluding evaluation on larger backbones (e.g., 30–70B models). A controlled RAG-only ablation remains future work. The verifier relies on a single 8B reasoning model; an ensemble of specialized verifiers could improve robustness.
  • Evident from the text: Sub3 is described as a “2.5B-parameter Qwen instruct model” in the experimental setup but elsewhere as Qwen2.5-7B, an inconsistency. The abstract’s “reduces multi-hop inverse step-identification errors by 26%” is not directly reconciled with the body’s reported step-identification rates (0.751 vs. 0.508 and 0.826).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.bionlp-share.34

Cites

0 citation errors · 2 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 20 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    VeReaFine recovers over 90% of the step-identification fidelity of a 32B baseline (M1-32B) in multi-hop inverse questions.

    “Our experiments show that, despite using a moderate-size 7B generator, VeReaFine achieves recall gains of +60-100% at the 75th percentile (P75) over the same model without verification, and recovers over 90% of the step-identification fidelity of a 32B baseline (Sub2: M1-32B (UCSC-VLAA, 2024; Huang et al., 2025; UCSC-VLAA, 2024)) in multi-hop inverse questions.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites UCSC-VLAA, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Huang et al., 2025 source
  • Premisecitation

    A key driver of hallucinations in medical LLMs is a shortage of domain knowledge, causing models to fabricate plausible-sounding but wrong information.

    “One key driver of hallucinations in medical LLMs is simply a shortage of domain knowledge: if the model's internal parameters don't”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites UCSC-VLAA, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Huang et al., 2025 source

      The Introduction (p. 3) attributes a medical-reasoning bottleneck to deficiencies in essential medical knowledge, and the Results (p. 7) show that missing or faulty knowledge can lead to incorrect reasoning or answers. The paper does not specifically identify knowledge shortage as a driver of hallucinations or describe models fabricating plausible-sounding information. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    RAG and CoT systems do not guarantee that each generated assertion is cross-checked against evidence, allowing unsupported claims to slip through.

    “Lack of explicit verification: RAG and CoT systems do not guarantee that each generated assertion is cross-checked against evidence, allowing unsupported claims to slip through (Maynez et al., 2020).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Maynez et al., 2020 source

      Section 5.2 reports that generated summaries contained spans unsupported by their source articles, supporting the general concern that unsupported claims can slip through. The paper studies abstractive summarization, not RAG or CoT systems, and does not establish whether those systems cross-check each assertion against evidence. Page numbers were not available in our copy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    RAG methods do not enforce that every generated claim is actually supported by the retrieval, allowing hallucinations to persist when models misinterpret or ignore evidence.

    “Despite these enhancements, RAG methods do not enforce that every generated claim is actually supported by the retrieval, allowing hallucinations to persist when models misinterpret or ignore evidence (Maynez et al., 2020).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Maynez et al., 2020 source

      Section 2 says summarization models can generate text inconsistent with their input, and Section 2.1 describes hallucinations that ignore source material. That supports the general concern about unsupported generation, but the paper examines abstractive summarization, not RAG; it does not establish what RAG methods enforce about retrieved evidence. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Systems typically assume retrieved passages suffice, failing to detect and handle cases where key evidence is missing.

    “No mechanism for insufficient-context detection: Systems typically assume retrieved passages suffice, failing to detect and handle cases where key evidence is missing (Song et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Song et al., 2024 source

      The Introduction (page unavailable) says hallucinations persist in RAG systems and identifies a lack of work integrating hallucination detection and mitigation. The detection discussion and prompts address whether generated statements are supported by the provided input, but do not say that systems typically assume retrieved passages are sufficient or fail to detect missing key evidence. Searches for insufficient context, missing evidence, and inadequate retrieved information found no passage establishing that claim. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
  • May combine two claims. 16
  • Classified as probably not a substantial claim. 6
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Chain-of-Thought prompting improves multi-step problem-solving but remains vulnerable when the model's internal knowledge is incomplete or outdated.

    “Chain-of-Thought (CoT) prompting results in intermediate reasoning text and improves multi-step problem-solving (Zhang et al., 2022), but remains vulnerable when its internal knowledge is incomplete or outdated (Madaan et al., 2023).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Zhang et al., 2022 source
    • cites Madaan et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Large language models often hallucinate unsupported or incorrect clinical assertions, risking patient safety and trust.

    “Large language models (LLMs) can generate medical responses, but they often”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Maynez et al., 2020 source

      The BERTS2S conclusions report frequent hallucinations in single-sentence summaries and say that over 90% of extrinsic hallucinations were erroneous. The study concerns abstractive document summarization, not clinical responses; it does not establish the claimed risks to patient safety or trust. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Chain-of-Thought prompting elicits explicit reasoning steps from LLMs, improving performance on multi-step tasks.

    “Chain-of-Thought (CoT) prompting elicits explicit reasoning steps from LLMs, improving performance on multi-step tasks (Wei et al., 2022).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wei et al., 2022 source

  • May combine two claims.

    Findingreasoning

    Sub2 reaches 0.826 on multi-hop inverse step-identification, owing to its stronger internal reasoning.

    “Sub2 reaches 0.826, owing to its stronger internal reasoning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In short-inverse, VeReaFine attains P75=0.559, surpassing Qwen2.5 (0.358, +56%) and M1-32B (0.539, +4%).

    “Here, VeReaFine attains P75=0.559,surpassing both Qwen2.5 (0.358, +56%) and even M1-32B (0.539, +4%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Sub3 (Qwen2.5) achieves P75=0.168 on short answer, covering only the top 17% of gold variants.

    “Short Answer Sub3 (Qwen2.5)achieves P75=0.168,indicating it covers only the top 17% of gold variants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    M1-32B reaches multi-hop P75=0.396 and inverse P75=0.387, while VeReaFine hits multi-hop P75=0.394 and inverse P75=0.475, a +22% boost on inverse steps.

    “M1-32B reaches (0.396, 0.387), while VeReaFine hits (0.394, 0.475)-a +22% boost on inverse steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    VeReaFine achieves short answer P75=0.269, a +60% relative increase over Sub3, despite using the same base 7B generator.

    “VeReaFine achieves P75 to 0.269-increasing by +60% relative to Sub3-despite using the same base 7B generator, demonstrating that the verification loop recovers critical answer fragments otherwise hallucinated or omitted.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    On the BioNLP 2025 ClinIQLink shared task, VeReaFine's 7B-parameter generator with its verifier nearly or surpasses M1-32B in some category on key open-ended metrics (P75 recall, step-identification rate).

    “On the BioNLP 2025 ClinIQLink”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Closed-ended questions do not have much improvement in this setting.

    “We focus our analysis on the open-ended QA because close-end questions do not have much improvement, and our pipeline is not designed to focus on solving the problems with closed-ended QA metrics most sensitive to hallucination:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    VeReaFine achieves 0.751 on multi-hop inverse step-identification (+48% over Sub3), recovering most of the gap by explicitly verifying each step against retrieved evidence.

    “VeReaFine achieves 0.751 (+48% over Sub3), recovering most of the gap by explicitly verifying each step against retrieved evidence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Retrieval-Augmented Generation mitigates some hallucination risks by providing relevant documents, but cannot ensure that the LLM correctly incorporates all retrieved facts or refrains from using incorrect contextual information.

    “Retrieval-Augmented Generation (RAG) mitigates some of these risks by providing relevant documents to the model, yet it cannot ensure that the LLM correctly incorporates all retrieved facts or refrains from utilizing incorrect contextual information (Lewis et al., 2020b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lewis et al., 2020b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Shared-task constraints limited the authors to only three submissions, preventing direct comparison against a RAG without verifier and precluding evaluation of VeReaFine on larger backbones (e.g., 30-70B models).

    “However, our shared-task constraints limited us to only three submissions, preventing direct comparison against a RAG without verifier and precluding evaluation of VeReaFine on larger backbones (e.g., 30-70B models).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    VeReaFine meets the desiderata of integrating explicit verification, adaptive retrieval expansion, and monotonic convergence guarantees by embedding a dedicated verifier into every retrieval and generation step, issuing targeted feedback queries when context is insufficient, and framing the end-to-end process as constraint-satisfaction with provable non-decreasing factuality.

    “VeReaFine meets these desiderata by embedding a dedicated verifier into every retrieval and generation step, issuing targeted feedback queries when context is insufficient, and framing the end-to-end process as a constraint-satisfaction with provable non-decreasing factuality.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The verifier has nonnegative correction fidelity.
    • Rests on an assumption: Iterative refinement loops lack formal assurances that factuality monotonically improves over successive passes.

  • May combine two claims.

    Premisecitation

    The BioNLP 2025 ClinIQLink test set is explicitly designed to surface subtle hallucinations in LLM outputs, provides ground truth, and requires evidence-grounded answers.

    “This testbed is explicitly designed to surface subtle hallucinations in LLM outputs, as it provides ground truth and requires evidence-grounded answers.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites BioNLP Shared Task Organizers, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationown data

    The verifier loop is especially potent at catching subtle logical missteps in inverse explanations, forcing the generator to ground its critique in actual evidence.

    “The verifier loop is especially potent at catching subtle logical missteps in inverse explanations, forcing the generator to ground its critique in actual evidence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Self-verification methods enhance self-consistency but remain limited by reliance on parametric knowledge rather than dynamic evidence acquisition.

    “These approaches enhance self-consistency but remain limited by reliance on parametric knowledge rather than dynamic evidence acquisition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Prior work has tried three main remedies-pure RAG grounding, chain-of-thought prompting, and self-verification loops-but none simultaneously guarantees that the generator receives just enough high-precision medical context and that each claim is checked against external evidence before being emitted.

    “Prior work has tried three main remedies-pure RAG grounding, chain-of-thought prompting, and self-verification loops-but none simultaneously guarantees that (a) the generator truly receives”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    The verifier effectively enforces evidence sufficiency, closing the factuality gap between moderate and large-scale models.

    “This highlights the power of LLM-as-judge paradigms in high-stakes domains: the verifier effectively enforces evidence sufficiency, closing the factuality gap between moderate and large-scale models (Madaan et al., 2023;Dhuliawala et al., 2023).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Madaan et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Dhuliawala et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

19 of 19

  • The paper relies on this without support.

    A naive RAG approach that overloads the generator with large bundles of retrieved text often backfires, confusing the LLM and leading it to latch onto irrelevant or outdated facts.

    “A naive RAG approach attempts to compensate by overloading the generator with large bundles of retrieved text, but this often backfires: too much loosely related information can confuse the LLM, leading it to latch onto irrelevant or outdated facts.”

  • The authors state this as an assumption.

    The verifier has nonnegative correction fidelity.

    “assuming a verifier with nonnegative correction fidelity”

  • The paper relies on this without support.

    Iterative refinement loops lack formal assurances that factuality monotonically improves over successive passes.

    “Absence of convergence guarantees: Iterative refinement loops lack formal assurances that factuality monotonically improves over successive passes.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Step-identification rate is lower than the 32B baseline

    Major

    The discussion claims a step-identification gain over the 32B model, but VeReaFine's reported rate is 0.751, below the 32B model's 0.826.

  • Claimed P75 gains conflict with multi-hop inverse results

    Minor

    The paper claims P75 recall gains of 60-100% over the unverified model, but its multi-hop inverse figures show a gain of about 23% (0.475 versus 0.387).

  • Sub3 is described as both a 7B and a 2.5B model

    Minor

    The same Sub3 baseline is labeled Qwen2.5-7B and described as having 2.5B parameters.

  • Refinement cap permits unsupported claims in the final answer

    Minor

    The pipeline claims to ensure every assertion is evidence-backed, but its return rule outputs the current draft when the refinement cap is reached, even if unsupported claims remain.

4 of 4

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 8

  1. Sahil Dhuliawala, Raghav Gupta, and Shashi Narayan. 2023. Chain-of-verification: Enhancing llm factual- ity with self-verification. In NAACL.
  2. Xiao Jiang, Rohan Patel, and Arjun Singh. 2024. Tc- rag: Turing-complete retrieval-augmented generation. In ACL.
  3. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Vivek Kulkarni, Minjoon Pasquale, Sebastian Riedel, Douwe Kiela, and 1 others. 2020a. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474.
  4. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hannaneh Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS.
  5. Arjun Madaan, Omar Khattab, Abhyudaya Jagannatha, and Ron Cohen. 2023. Self-refine: Iterative self- feedback for large language models. In Proceedings of ICLR 2023.
  6. Prakhar Manakul, Jesse Wu, and Venkatesh Jampani. 2023. Selfcheckgpt: Zero-resource hallucination de- tection in large language models. Transactions of the ACL.
  7. Jacob Menick, Ari Jacobson, and Rafael Pikeler. 2022. Gophercite: Training llms to search for evidence and abstain when uncertain. In ICML.
  8. Vikram Trivedi, Feng Zhao, and Eduard Hovy. 2023. Ircot: Interleaving retrieval with chain-of-thought for improved multi-hop question answering. In Proceed- ings of AAAI 2023, pages 4567-4574.

Cited by

Sorted by citations

Related

Authors
Phasook, P., Pitijaroonpong, R., Kinchagawat, J., Chinkamol, A., Saengja, T., Udomlapsakul, K., Sawatphol, J., & Ittichaiwong, P.
Affiliations
King Mongkut's University of Technology Thonburi (TH)
ORCID
Venue
Proceedings of the 24th Workshop on Biomedical Language Processing (Shared Tasks)
Publisher
Association for Computational Linguistics
Published
2025
Pages
281-288
DOI
10.18653/v1/2025.bionlp-share.34
OpenAlex
W4412889100
Semantic Scholar
d61a1d019bf6fc745280ffb9e1790c1e9059810a
Cited by
0
References
0
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Workshop on Biomedical Language Processing (BioNLP) (publishedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
hallucination resistance, retrieval-augmented generation, medical question answering, hallucination detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review