MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification

Probably not a substantial claim: may not pass S3 (load-bearing).

There is work demonstrating bias in using LLMs as evaluators.

“there is also work demonstrating bias in using LLMs as evaluators (Wang et al., 2024;Zheng et al., 2023).”

• cites Wang et al., 2024 sourceSupported
• cites Zheng et al., 2023 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

These factors are especially important given the potential harm of incorrect medical information and that LLMs have been known to add irrelevant or sometimes dangerous text.

“These factors are especially important given the potential harm of incorrect medical information and that LLMs have been known to add irrelevant or sometimes dangerous text (Giorgi et al., 2024;Mei et al., 2023).”

• cites Giorgi et al., 2024 sourcePartly supported

  The Results (Fact checking; page unavailable) document nonfactual and dangerous LLM responses, and the Conclusions warn that an inaccurate answer could have serious health consequences. The cited passages do not establish that the models added irrelevant text, so that part of the sentence is not supported by this source. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Mei et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Incorrect medical information could seriously harm the patient.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLMs have been shown to excel at following instructions and comprehending retrieved passages.

“While LLMs have been shown to excel at these tasks (Wang et al., 2022;Zhong et al., 2023)”

• cites Wang et al., 2022 sourcePartly supported

  Section 6.1 (page unavailable) reports that instruction-tuned models outperform untuned models and learn to follow instructions on unseen tasks. The retrieved-passage comprehension part is not established: the paper lists question-answering and related tasks, but the passages found do not assess comprehension of retrieved passages specifically. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Zhong et al., 2023 sourcePartly supported

  Section 4.5.2 (Strengths; page unavailable) says the models excel at comprehending context and questions, and Section 4.3 (Main Results; page unavailable) links strong zero-shot performance to instruction tuning and understanding tasks. The paper does not establish that the models excel specifically at following instructions or comprehending retrieved passages; its comprehension results concern exam questions and their context. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can effectively answer patient medical questions.

“Recent work has demonstrated that LLMs can effectively answer patient medical questions (Ayers et al., 2023b).”

• cites Ayers et al., 2023b sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

VeriScore fine-tuned a model for the decomposition step, largely solving the hallucinated, incorrectly structured, and redundant issues.

“Also, they fine-tuned a model for the decomposition step, largely solving the hallucinated, incorrectly structured, and redundant issues as well.”

• cites Song et al., 2024 sourcePartly supported

  Section 4.1 and Appendix F report fine-tuning an open-weight model for claim extraction, VeriScore’s decomposition step. But the Manual quality comparison says the fine-tuned model still sometimes combines information that GPT-4 separates and extracts claims outside the marked sentence. The paper does not establish that fine-tuning largely solved hallucinated, incorrectly structured, and redundant outputs. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The decompose-then-verify system popularized by FActScore has been iterated on by other systems.

“The decompose-then-verify system that was popularized by FActScore (Min et al., 2023) has been iterated on by other systems (Song et al., 2024;Jiang et al., 2025;Huang and Chen, 2024;Wei et al., 2024).”

• cites Min et al., 2023 sourceSupported
• cites Song et al., 2024 sourceSupported
• cites Jiang et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Specialized models like UniEval and Prometheus 2, and LLMs like GPT-4 can be used to judge the factuality of a passage.

“Specialized models like UniEval (Zhong et al., 2022) and Prometheus 2 (Kim et al., 2024), and LLMs like GPT-4 (OpenAI et al., 2024b) can be used to judge the”

• cites Zhong et al., 2022 sourceSupported
• cites Kim et al., 2024 sourceSupported
• cites OpenAI et al., 2024b sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs have demonstrated medical ability through tasks such as medical licensing exams, answering clinical questions from patient records, and reasoning about clinical case studies.

“LLMs have demonstrated medical ability through tasks such as medical licensing exams (Nori et al., 2023), answering clinical questions from patient records (Pampari et al., 2018;Patrick and Li, 2012;Bai et al., 2024), and reasoning about clinical case studies (Eriksen et al., 2024;Chen et al., 2025).”

• cites Nori et al., 2023 sourceSupported
• cites Pampari et al., 2018 sourcePartly supported

  Section 6.2 (p. 8) describes answering questions from clinical notes in electronic medical records, and §6.2.2 (p. 8) reports results for that task. But the evaluated systems include an RNN-based DrQA reader and logistic regression, not LLMs, so this paper does not establish the attributed claim about LLM medical ability. (LLM, conf=0.89)

  Experimental — accuracy not yet measured
• cites Eriksen et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

VeriScore expands the generation domains to include fictional stories, Wikipedia, biographies, and Reddit.

“VeriScore (Song et al., 2024) expands the generation domains to include fictional stories, Wikipedia, biographies, and an online forum, Reddit.”

• cites Song et al., 2024 sourcePartly supported

  Table 1 (section 3.2; page unavailable) includes WritingPrompts stories, biographies, Reddit-sourced datasets, and Wikipedia articles. But it marks the Wikipedia dataset, wikitext-103, for human evaluation of claim extraction only, not as a generation domain; Appendix J explicitly discusses fictional WritingPrompts generations. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

VeriScore introduced a context-inclusive prompt for the decomposition step, which addressed context-dependent and incomplete claims.

“Another iteration, VeriScore, introduced a context-inclusive prompt for the decomposition step, which addressed context-dependent and incomplete claims (Song et al., 2024).”

• cites Song et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Song et al. (2024) provide general, non-formalized reasons for why a claim can be unverifiable.

“Song et al. (2024) provide reasons for why a claim can be”

• cites Song et al. (2024) sourceSupported

May combine two claims.

The factuality score substantially varies by decomposition method, verification corpus, and used backbone LLM.

“The resulting factuality score substantially varies by decomposition method, verification corpus, and used backbone LLM”

May combine two claims.

MedScore extracts up to three times as many valid facts as existing methods, reducing hallucination and vague references, and retaining condition-dependency in facts.

“Our method extracts up to three times as many valid facts as existing methods, reducing hallucination and vague references, and retaining condition-dependency in facts.”

May combine two claims.

OpenBioLLM cannot distinguish valid and invalid claims well, overly assigning True to most invalid claims and resulting in inflated scores.

“OpenBioLLM cannot distinguish valid and invalid claims well, thus overly assigning True to most invalid claims and resulting in inflated scores.”

May combine two claims.

FActScore overly decomposes sentences into invalid claims, with an overall valid claim rate of 17%.

“From Table 2, we find that FActScore generates the most number of claims among the three decomposition methods, but it overly decomposes sentences into invalid claims, whose overall valid claim rate is only 17% in Table 3.”

May combine two claims.

Mistral Small 3, the backbone verifier LLM, has strong capabilities to understand medical passages, associate the claim with the passages, and reason about the truthfulness of the claim.

“In the external corpus verification, we find that Mistral Small 3, the backbone verifier LLM, has strong capabilities to understand medical passages, associate the claim with the passages, and reason about the truthfulness of the claim.”

• AssumptionRests on an assumption: The LLM verifiers' outputs are reliable enough to draw conclusions about factuality, despite potential bias.
• AssumptionRests on an assumption: The retrieved documents from MedCorp and Wikipedia are sufficient to verify claims.

May combine two claims.

FActScore overly decomposes sentences into invalid claims, and VeriScore overly omits claims that should be generated and verified.

“From Tables 2 and 4, we find the same pattern: FActScore overly decomposes sentences into invalid claims, and VeriScore overly omits claims that should be generated and verified.”

May combine two claims.

Although FActScore's verifiable rate is high, 60.09%, its claims are still invalid by containing lots of errors in the MedScoreTaxonomy.

“Although FActScore's verifiable rate is high, 60.09%, its claims are still invalid by containing lots of errors in the MedScoreTaxonomy.”

• AssumptionRests on an assumption: The manual annotations in PUMA (text spans) are accurate and can serve as ground truth for evaluating claim verifiability.
• AssumptionRests on an assumption: The MedScoreTaxonomy correctly defines which claims are verifiable.

May combine two claims.

MedScore has fewer invalid claims than FActScore and higher coverage than VeriScore.

“MedScore, a new taxonomy-based factuality evaluation pipeline for open-ended long-form medical answers with fewer invalid claims than FActScore and higher coverage than VeriScore, which is generalizable across domains with minor change;”

May combine two claims.

The main issue with applying VeriScore to medical responses is the high level of omitted claims; VeriScore decomposes a response into drastically fewer claims than other systems.

“The main issue with applying VeriScore to medical responses is the high level of omitted claims, as VeriScore decomposes a response into drastically fewer claims than other systems, shown in Section 4.”

May combine two claims.

Most of the generated claims by DnDScore on AskDocsAI are invalid and not sufficiently decontextualized.

“We reproduced DnDScore (Wanner et al., 2025) on AskDocsAI and found most of the generated claims are invalid and not sufficiently decontextualized, as shown in Table 6.”

• cites Wanner et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

MedScore achieves the highest factuality scores among most verification corpora on all datasets, showing its robustness to the style of writing in free-form answers by generating more valid claims than other methods.

“Table 4 shows that MedScore achieves the highest factuality scores among most verification corpora on all datasets, showing its robustness to the style of writing in free-form answers by generating more valid claims than other methods.”

May combine two claims.

DnDScore's first step generates similar quality claims as FActScore, and its second step focuses on only one error in the MedScoreTaxonomy, making it fail to create valid claims on medical domain text.

“Its first step generates similar quality claims as FActScore, and the second step focuses on only one error in the MedScoreTaxonomy, making it fail to create valid claims on medical domain text.”

May combine two claims.

Despite VeriScore being fine-tuned on Reddit ELI5 data to optimize performance, MedScore generates twice as many claims without any tuning.

“Despite VeriScore being fine-tuned on Reddit ELI5 data to optimize performance, MedScore generates twice as many claims without any tuning.”

May combine two claims.

MedScore is more robust to colloquial language and has a more reasonable 0-claim rate of 9.3% since some answers are too short and contain no verifiable information.

“MedScore is more robust to colloquial language and has a more reasonable 0-claim rate of 9.3% since some answers are too short (e.g.,”

• AssumptionRests on an assumption: The manual annotations in PUMA (text spans) are accurate and can serve as ground truth for evaluating claim verifiability.
• AssumptionRests on an assumption: The MedScoreTaxonomy correctly defines which claims are verifiable.

May combine two claims.

VeriScore and Core over-filter claims.

“VeriScore and Core over-filter claims.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CaLMQA doesn't separate questions from context, making it hard to calculate how many questions have additional context information.

“CaLMQA doesn't separate questions from context, so it is hard to calculate how many questions have additional context information, and we mark it as N/A.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AskDocsAI has doctor responses and chatbot responses, while PUMA and CaLMQA only have user responses.

“AskDocsAI has doctor responses and chatbot responses, while PUMA and CaLMQA only have user responses, so there are two N/A in their columns' doctor response rows.”

Classified as probably not a substantial claim. May combine two claims.

Unlike AskDocsAI, PUMA answers are neither written by doctors nor rewritten by an LLM, and are generally shorter and of lower quality.

“Unlike AskDocsAI, the answers are neither written by doctors nor rewritten by an LLM, and are generally shorter and of lower quality.”

May combine two claims.

LLMs cannot learn valid claim patterns only from 8 ICL examples, producing similar quality claims to FActScore.

“Results in Table 9 show that LLMs cannot learn valid claim patterns only from 8 ICL examples, producing similar quality claims to FActScore.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Doctor responses are not themselves complete and might have information omissions due to subjective writing preferences.

“In addition, doctor responses are not themselves complete and might have information omissions due to subjective writing preferences.”

Classified as probably not a substantial claim.

From an initial collection of 343 question-physician pairs, we randomly selected 300 samples for inclusion in the final AskDocsAI dataset.

“From this initial collection of 343 questionphysician pairs, we randomly selected 300 samples for inclusion in the final AskDocsAIdataset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

False positives happen when a too-general claim (typically incomplete or context-dependent) is highly relevant to many corpus documents, and is assigned True by the verifier, but should be False for that specific case.

“Another issue is false positives, which happen when a too-general claim (typically incomplete or contextdependent) is highly relevant to many corpus documents, and is assigned True by the verifier, but should be False for that specific case.”

• AssumptionRests on an assumption: The retrieved documents from MedCorp and Wikipedia are sufficient to verify claims.

May combine two claims.

Core cannot distinguish valid and invalid claims and equally filters them out by 33% on AskDocsAI, 49% on PUMA, and 44% on CaLMQA.

“Additionally, the post-filtering system, Core, cannot distinguish valid and invalid claims and equally filters them out by 33% on AskDocsAI and 49% on PUMA, and 44% on CaLMQA.”

May combine two claims.

FActScore underestimates and VeriScore overestimates the factuality score.

“FActScore underestimates and VeriScore overestimates the factuality score.”

May combine two claims.

MedScore includes the most verifiable information by the highest adjusted verifiable rate, 63.75%, with minimal inclusion of invalid claims.

“Tables 1 and 5 show that MedScore includes the most verifiable information by the highest adjusted verifiable rate, 63.75%, with minimal inclusion of invalid claims.”

• AssumptionRests on an assumption: The manual annotations in PUMA (text spans) are accurate and can serve as ground truth for evaluating claim verifiability.
• AssumptionRests on an assumption: The MedScoreTaxonomy correctly defines which claims are verifiable.

May combine two claims.

MedScore measures accuracy, but not completeness.

“MedScore measures accuracy, but not completeness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the AskDocsAI dataset, chatbot responses usually have more and longer sentences than doctor responses.

“In the AskDocsAI dataset, chatbot responses usually have more and longer sentences than doctor responses.”

May combine two claims.

VeriScore generates 0 claims for 44 responses and its 0-claim rate is 14.67% for 300 long responses in AskDocsAI.

“Table 2 shows that VeriScore generates 0 claims for 44 responses and its 0-claim rate is 14.67% for 300 long responses in AskDocsAI.”

May combine two claims.

Some MedScore claims are verifiable and valid, but there are no corresponding annotated verifiable spans in the PUMA dataset, which results in an underestimate of its verifiable rate.

“Some MedScore claims are verifiable and valid, but there are no corresponding annotated verifiable spans in the PUMA dataset, which results in an underestimate of its verifiable rate, as shown in Table 13.”

• AssumptionRests on an assumption: The manual annotations in PUMA (text spans) are accurate and can serve as ground truth for evaluating claim verifiability.
• AssumptionRests on an assumption: The MedScoreTaxonomy correctly defines which claims are verifiable.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the corpus ablation study, the Textbooks-only corpus yields a factuality score of 68.89%.

“| Textbooks-only | 125.8k | 68.89% | 25.75 min |”

The quote only partly states this claim.

MedScore retains more information in the claims from the input, including colloquial text.

“MedScore retains more information in colloquial text.”

May combine two claims.

Taxonomy-based Instruction most effectively guides LLM to avoid generating invalid claims, but often leads to plausible yet inference-based outputs not strictly grounded in the sentence.

“Taxonomy-based Instruction most effectively guides LLM to avoid generating invalid claims, but often leads to plausible yet inferencebased outputs not strictly grounded in the sentence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the corpus ablation study, using all 3 medical corpora yields a factuality score of 66.88%.

“| All 3 medical corpus | 24.3M | 66.88% | 54.35 min |”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the ablation study on AskDocsAI with MedScore decomposition, the factuality score for Mistral Small 3 is 99.35%.

“| Mistral Small 3 | 99.35% |”

May combine two claims.

As the relevancy decreases, the benefit of including more snippets decreases, and the running time increases much more.

“However, as the relevancy decreases, the benefit of including more snippets decreases, and the running time increases much more.”

May combine two claims.

In the total 17920 retrieved snippets for all claims, 16375 (91.4%) snippets are from PubMed, with 1297 (7.2%) from StatPearls and 248 (1.4%) from Textbooks.

“In the total 17920 retrieved snippets for all claims, 16375 (91.4%) snippets are from PubMed, with 1297 (7.2%) from StatPearls and 248 (1.4%) from Textbooks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The false negative NLI examples of PUMA claims by MedScore are valid and verifiable, but labeled as False because the original answers are not correctly labeled with verifiable spans.

“These claims are valid and verifiable, but labeled as False because the original answers are not correctly labeled with verifiable spans.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the ablation study on AskDocsAI with MedScore decomposition, the factuality score for Mistral-7B-Instruct-v0.3 is 98.35%.

“| Mistral-7B-Instruct-v0.3 | 98.35% |”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the ablation study on AskDocsAI with MedScore decomposition, the factuality score for Llama-3.1-8B-Instruct is 97.53%.

“| Llama-3.1-8B-Instruct | 97.53% |”

May combine two claims.

StatPearls and Textbooks are much smaller, thus quicker, and have 2%-3% higher scores than PubMed.

“StatPearls and Textbooks are much smaller, thus quicker, and have 2%-3% higher scores than PubMed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

FActScore has the most claims while VeriScoreQA has the fewest claims in Table 2.

“Similar to the chatbot response decomposition results, FActScore has the most claims while VeriScoreQA has the fewest claims in Table 2.”

Probably not a substantial claim: may not pass S2 (checkable).

Customizing each step is important for reliable factuality evaluation.

“highlighting the importance of customizing each step for reliable factuality evaluation by using our generalizable and modularized pipeline for domain adaptation.”

Classified as probably not a substantial claim.

The ability to correctly decompose a generation into atomic facts directly influences the resulting factuality score.

“Subsequent work has shown the importance of the decomposition step, as the ability to correctly decompose a generation into atomic facts directly influences the resulting factuality score (Wanner et al., 2024).”

• cites Wanner et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Medical answers are condition-dependent, conversational, hypothetical, sentence-structure diverse, and subjective, unlike biographical texts, making decomposition into valid facts challenging.

“This differs from condition-dependent, conversational, hypothetical, sentence-structure diverse, and subjective medical answers, making decomposition into valid facts challenging.”

• Also stated in: § 1 Introduction · hedged

  “Patient-facing medical answers usually contain subjective-sounding suggestions (”

Classified as probably not a substantial claim.

Evaluating open-ended medical tasks requires a more sophisticated approach because there is no gold answer or gold reference document for metrics calculation.

“However, evaluating open-ended medical tasks requires a more sophisticated approach because there is no gold answer or gold reference document that can be used for metrics calculation.”

• AssumptionRests on an assumption: There is no gold answer or gold reference document for open-ended medical tasks that can be used for metrics calculation.

Classified as probably not a substantial claim.

PUMA answers have text spans labeled with 5 categories: Cause, Suggestion, Experience, Question, and Information.

“PUMA answers have text spans labeled with 5 categories: Cause, Suggestion, Experience, Question, and Information.”

• cites Section 3.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The annotation was performed by two annotators with an overall interannotator agreement of 0.73 in Cohen's kappa.

“The annotation was performed by two annotators with an overall interannotator agreement of 0.73 in Cohen's kappa.”

May combine two claims.

CaLMQA is a cultural free-form QA dataset, whose 96 English QA pairs are collected from r/explainlikeimfive (ELIF) subreddit.

“CaLMQA is a cultural free-form QA datasetfoot_3 , whose 96 English QA pairs are collected from r/explainlikeimfivefoot_4 (ELIF) subreddit.”

May combine two claims.

Verification directly relies on the quality of the corpus retriever and on whether any supporting information is present in the corpus.

“However, verification also directly relies on the quality of the corpus retriever and on whether any supporting information is present in the corpus.”

• AssumptionRests on an assumption: The retrieved documents from MedCorp and Wikipedia are sufficient to verify claims.

Classified as probably not a substantial claim. May combine two claims.

Unverifiable claims include event narratives and patient-empathy statements, which are specific to the patient interaction and cannot be verified through an external objective knowledge corpus.

“An unverifiable claim includes event narratives (”

May combine two claims.

The tuning-free MedScore decomposition generalizes well to out-of-domain data and can be easily adapted to other fields and customized verifiability definitions via the released modular pipeline.

“The tuning-free MedScore decomposition generalizes well to out-of-domain data and can be easily adapted to other fields and customized verifiability definitions via our released modular pipeline.”

Classified as probably not a substantial claim. May combine two claims.

The '0-claim rate' is an indicator of omitted claims, and a rough estimation of claim recall and coverage quality.

“Therefore, we consider the”

• AssumptionThe authors state this as an assumption: For AskDocsAI, every response has at least one valid claim because the chatbot responses are high-quality, rewritten doctor responses.

May combine two claims.

VeriScore sacrifices the number of claims for claim quality, omitting a lot of informative facts that need to be verified.

“Table 3 shows that it generates the fewest number of invalid claims, which indicates that VeriScore sacrifices the number of claims for claim quality, omitting a lot of informative facts that need to be verified.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Existing popular systems, FActScore, VeriScore, and DnDScore, should not be used off-the-shelf for reliable factuality estimation.

“Existing popular systems, FActScore, VeriScore, and DnDScore, should not be used off-the-shelf for reliable factuality estimation.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Biographical text generations contain entity-centric, context-free, formulaic sentence structures, while medical answers are more context-dependent with complex sentence structures.

“Biographical text generations contain entity-centric, context-free, formulaic sentence structures (e.g.,”

Probably not a substantial claim: may not pass S3 (load-bearing).

We deliberately avoided using post score (upvotes) as a filtering criterion, as this can introduce bias toward popular but not necessarily more informative content.

“Importantly, we deliberately avoided using post score (upvotes) as a filtering criterion, as this can introduce bias toward popular but not necessarily more informative content.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In real-world free-form QA, there are no gold standard answers that can be seen as fully complete.

“In terms of real-world free-form QA, there are no gold standard answers that can be seen as fully complete.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The taxonomy identifies issues present in FActScore, some of which have been fixed in subsequent work.

“The taxonomy above identifies issues present in FActScore, some of which have been fixed in subsequent work.”

Classified as probably not a substantial claim.

If a valid claim is nested as a sub-clause, the wrong prefix will disturb verification judgment for the valid information.

“Also, if a valid claim is nested as a sub-clause (e.g.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Core introduced a filtering step between decomposition and verification that removes duplicate and unfaithful claims through measuring entailment.

“Core introduced a filtering step between the decomposition and verification steps, which removes duplicate and unfaithful claims through measuring entailment.”

• cites Jiang et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

PUMA contains 3,195 user questions with 10,067 answers (roughly 3 answers per question), and manually annotated answers' text spans with 5 labels: Cause, Suggestion, Experience, Question, and Information.

“PUMA contains 3,195 user questions with 10,067 answers (roughly 3 answers per question), and manually annotated answers' text spans with 5 labels: Cause, Suggestion, Experience, Question, and Information.”

Classified as probably not a substantial claim. May combine two claims.

When a decomposition method returns no claim for a response, this indicates that there is no verifiable claim in the response, which contradicts the fact that every response usually has at least one piece of verifiable medical information.

“When a decomposition method returns no claim for a response, this indicates that there is no verifiable claim in the response, which contradicts the fact that every response usually has at least one piece of verifiable medical information.”

• AssumptionThe authors state this as an assumption: For AskDocsAI, every response has at least one valid claim because the chatbot responses are high-quality, rewritten doctor responses.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We manually checked, replaced wrong claims, and added new examples to make sure all ICL examples include at least one example for each invalid category we defined in Section 2.

“Then we manually checked, replaced wrong claims, and added new examples to make sure all ICL examples include at least one example for each invalid category we defined in Section 2.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Our workflow offers a practical template for researchers to adapt the pipeline with minimal effort.

“Our workflow offers a practical template for researchers to adapt the pipeline with minimal effort.”

Classified as probably not a substantial claim.

Repeated claims can artificially raise or lower the factuality score of a passage, since the same claim would be evaluated multiple times.

“Repeated claims can artificially raise or lower the factuality score of a passage, since the same claim would be evaluated multiple times (Wanner et al., 2024).”

• cites Wanner et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The medical domain has context-dependent, subjective, and highly complex structured sentences, unlike domains for which FActScore and similar systems were developed.

“In contrast to domains for which FActScore and similar systems have been developed, the medical domain has context-dependent, subjective, and highly complex structured sentences.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Often, a redundant claim is also incomplete.

“Often, a redundant claim is also incomplete.”

May combine two claims.

MedScore is a customizable decompose-then-verify pipeline that extracts condition-aware valid claims from medical answers while preserving most of the informative content, and verifies them against a domain-adapted corpus.

“We proposed MedScore, a customizable decompose-then-verify pipeline that extracts condition-aware valid claims from medical answers while preserving most of the informative content, and verifies them against a domainadapted corpus.”

May combine two claims.

FActScore decomposes sentences into atomic facts while MedScore and VeriScore generate molecular facts by including more information from the given context.

“These claim lengths show that FActScore decomposes sentences into atomic facts while MedScore and VeriScore generate molecular facts by including more information from the given context.”

Classified as probably not a substantial claim.

A claim not in a declarative format, such as an imperative, is hard to verify because it does not match the format of sentences in the external authoritative corpus.

“A claim not in a declarative format, such as an imperative (e.g.”

May combine two claims.

The machine augmented answers are high quality because they are grounded in the doctor response with minimal hallucinations.

“indicating that the machine augmented answers are high quality because they are grounded in the doctor response with minimal hallucinations.”

• AssumptionRests on an assumption: The doctor responses in AskDocsAI are a reliable ground truth for verifying the factuality of machine augmented answers.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

A holistic quality evaluation should also ensure that all important information from a passage is covered, not only precision of generated claims.

“The above taxonomy only categorizes generated claims (i.e., precision), while a holistic quality evaluation should also ensure that all”

• AssumptionRests on an assumption: The definition of important changes based on the applied domain.

Probably not a substantial claim: may not pass S3 (load-bearing).

MedCorp consists of four corpora: PubMed, Stat-Pearls, Textbooks, and Wikipedia.

“MedCorp consists of four corpora: PubMed, Stat-Pearls, Textbooks, and Wikipediafoot_6 , and the ablation study for each corpora's impact is in Appendix A.8.”

Classified as probably not a substantial claim. May combine two claims.

DnDScore first generates subclaims for each sentence using the full response paragraph as context, then decontextualizes each subclaim by adding additional information.

“DnDScore first generates subclaims for each sentence, using the full response paragraph as the context, and then decontextualizes each subclaim into a decontextualized claim by adding additional information.”

• cites Wanner et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Claims with vague references to nouns with under-specified or overly-specified pronouns, entities, locations, and temporal information are difficult or impossible to verify.

“This includes claims with vague references to nouns with under-specified or overly-specified pronouns, entities, locations, and temporal information that are difficult or impossible to verify.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Allen et al. identified personalization, perceived empathy, and factuality as factors for evaluating patient-facing systems.

“To evaluate patient-facing systems, Allen et al. (2024) identified several factors, including personalization, perceived empathy (Ayers et al., 2023a), and factuality.”

• cites Allen et al. (2024) sourceNot checked — no full text of the source
• cites Ayers et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

MedScoreTaxonomy is generalizable to other decomposition tasks and non-medical domains.

“MedScoreTaxonomy, a taxonomy for claim decomposition for patient-facing medical text evaluation that highlights existing challenges, and is generalizable to other decomposition tasks and non-medical domains;”

Classified as probably not a substantial claim. May combine two claims.

AskDocsAI is a high-quality open-ended long-form medical QA dataset with real user questions and LLM-augmented doctor answers from Reddit's r/AskDocs.

“AskDocsAI, a high-quality open-ended longform medical QA dataset with real user questions and LLM-augmented doctor answers from Reddit's r/AskDocs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We applied several reproducible filtering steps by coded rules without human intervention or bias.

“To ensure quality and relevance, we applied several reproducible filtering steps by coded rules without human intervention or bias.”

Classified as probably not a substantial claim. May combine two claims.

Experience and Question spans describe users' first-hand experiences or questions, which are usually subjectively unverifiable and don't need to be verified.

“Experience and Question spans describe users' first-hand experiences or questions, which are usually subjectively unverifiable and don't need to be verified.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

While iterations introduce improvements to the decomposition stage, not all changes are best suited for the medical domain.

“While the iterations introduce improvements to the decomposition stage, not all changes are best suited for the medical domain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The selection process prioritized clarity, comprehensiveness, and diversity of medical topics.

“The selection process prioritized clarity, comprehensiveness, and diversity of medical topics.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The best verifier model should be able to understand this doctor's response, and only use this response's information to determine whether the claim is True.

“Therefore, the best verifier model should be able to understand this doctor's response, and only use this response's information to determine whether the claim is True.”

Probably not a substantial claim: may not pass S4 (based).

Theoretically, any claim decomposed from the doctor's response should be factual when compared to itself, i.e., always True.

“Theoretically, any claim decomposed from the doctor's response should be factual when compared to itself, i.e., always True.”