AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment

Classified as probably not a substantial claim.

In schema-based systems, treating source and target facts as atomic facts with constant named entities and predicates makes computing the F1 score straightforward.

“In schemabased systems (Goodrich et al., 2019), it is easy to treat F s and F t as atomic facts with constant named entities and predicates, thus computing the F1 score as described above is straightforward.”

• cites Goodrich et al., 2019 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and instruction_ep5 is 8.728525e-01.

“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section describes MIMIC-IV-BHC, not MIMIC-IV-Ext-BHC. The Statistical analysis section describes Wilcoxon signed-rank tests, but our copy provides no Friedman posthoc comparison between instruction_ep2 and instruction_ep5 or the stated p-value (searched for the model names, test, dataset, and value). Neither section has page numbers in our copy. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep5 and lm_ep2 is 1.712586e-07.

“| instruction_ep5 | 0.872852 | 1.0 | 1.712586e-07 | 0.000039 |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section describes MIMIC-IV-BHC, while Statistical analysis describes Wilcoxon signed-rank tests for clinician preferences; neither reports the claimed Friedman posthoc comparison. Searches for “1.712586e-07,” “instruction_ep5,” “lm_ep2,” and “MIMIC-IV-Ext-BHC” found no matching result in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and lm_ep2 is 9.882236e-06.

“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section describes MIMIC-IV-BHC, but does not report the claimed model comparison or p-value. The Statistical analysis section describes Wilcoxon signed-rank tests; searches for the exact p-value, model labels, and Friedman posthoc test found no passage reporting this result. Page numbers were unavailable in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

On MIMIC-IV-Ext-BHC, lm_ep5 has first rank 12.6%.

“| lm_ep5 | 12.6% | | | |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset and Model performance analysis sections describe MIMIC-IV-BHC and benchmarked models, but neither reports a first-rank result for lm_ep5. Searches for “lm_ep5,” “12.6%,” and first-rank results found no passage supporting the stated percentage. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

On MIMIC-IV-Ext-BHC, instruction_ep2 has first rank 23.7%.

“| instruction_ep2 | 23.7% | | | |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section describes MIMIC-IV-BHC, and the Model performance analysis section reports results for adaptation strategies, but neither identifies MIMIC-IV-Ext-BHC or reports a first-rank percentage for instruction_ep2. Searches for “instruction_ep2” and “23.7% first rank” found no passage supporting the attributed result. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between lm_ep2 and lm_ep5 is 0.723773.

“| lm_ep2 | 0.000010 | 1.712586e-07 | 1.0 | 0.723773 |”

• cites Aali et al., 2024 sourceCould not verify

  The Statistical analysis section describes Wilcoxon signed-rank tests, not a Friedman posthoc comparison. Searches for 0.723773, lm_ep2, lm_ep5, and Friedman found no passage reporting the claimed p-value; the paper’s Dataset section describes MIMIC-IV-BHC, but does not establish this comparison. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep5 and lm_ep5 is 0.000039.

“| instruction_ep5 | 0.872852 | 1.0 | 1.712586e-07 | 0.000039 |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset and Adaptation strategies sections describe MIMIC-IV-BHC and the models’ adaptation methods, but do not report a Friedman posthoc comparison. Searches for “instruction_ep5,” “lm_ep5,” “0.000039,” and “MIMIC-IV-Ext-BHC” found no passage establishing the claimed p-value. Page numbers were not available in our copy. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and lm_ep5 is 0.001026.

“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section identifies MIMIC-IV-BHC, while the Statistical analysis section describes Wilcoxon signed-rank tests, not the stated Friedman posthoc comparison. Searches for 0.001026, instruction_ep2, lm_ep5, and MIMIC-IV-Ext-BHC found no passage reporting the claimed p-value. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

On MIMIC-IV-Ext-BHC, instruction_ep5 has first rank 26.3%.

“| instruction_ep5 | 26.3% | | | |”

• cites Aali et al., 2024 sourceCould not verify

  The Dataset section describes MIMIC-IV-BHC, and the Model performance analysis section reports other performance comparisons, but neither gives a first-rank percentage for instruction_ep5 on MIMIC-IV-Ext-BHC. Searches for “instruction_ep5,” “26.3,” and the dataset and ranking terms found no passage addressing that result. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between PegasusDynamic and T5 is 0.0000.

“| PegasusDynamic 0.0000 | 0.0000 | 1.0000 | 0.0000 | |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between Pegasus and PegasusDynamic is 0.0000.

“| Pegasus | 1.110223e-16 | 1.0000 | 0.0000 | 1.566292e-11 |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Friedman test yielded p-values of 0.000 for both datasets, indicating that the models are statistically significantly different.

“The p-values p = 0.000 estimated by the Friedman test for both datasets demonstrate that the models are statistically significantly different.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Post-hoc tests reported in Table 1 show which models are statistically significantly different.

“After the Friedman test, a follow-up with post-hoc tests reported in Table 1 shows which models are statistically significantly different as expected by their number of parameters and training protocols.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between BART and Pegasus is 1.110223e-16.

“| BART | 1.0000 | 1.110223e-16 | 0.0000 | 0.0000 |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between BART and T5 is 0.0000.

“| BART | 1.0000 | 1.110223e-16 | 0.0000 | 0.0000 |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between Pegasus and T5 is 1.566292e-11.

“| Pegasus | 1.110223e-16 | 1.0000 | 0.0000 | 1.566292e-11 |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Friedman posthoc test on AgreeFact, the p-value for the comparison between BART and PegasusDynamic is 0.0000.

“| BART | 1.0000 | 1.110223e-16 | 0.0000 | 0.0000 |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

On AgreeFact, PegasusDynamic has first rank 2.3%.

“| PegasusDynamic | 2.3% | | | |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

On AgreeFact, T5 has first rank 0.4%.

“| T5 | 0.4% | | | |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

On AgreeFact, Pegasus has first rank 3.6%.

“| Pegasus | 3.6% | | | |”

• cites Tang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

The absence of granular insight makes it difficult to diagnose and correct inaccuracies effectively.

“This absence of granular insight makes it difficult to diagnose and correct inaccuracies effectively.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The TF-IDF-based weighting mechanism may not fully capture the nuanced importance of domain-specific entity types, especially where subtle distinctions carry significant implications.

“Second, although the TF-IDF-based weighting mechanism allows for some granularity, it may not fully capture the nuanced importance of domain-specific entity types, especially in contexts where subtle distinctions carry significant implications (e.g., between similar medical conditions or treatments).”

May combine two claims.

False positives and false negatives both negatively impact factual consistency.

“FP and FN both negatively impact the factual consistency.”

May combine two claims.

An increase in the number of false negatives reduces recall and signals factual inconsistency.

“While an increase in the number of FNs reduces Recall(R) and signals factual inconsistency, an increase in the number of FPs reduces Precision(P) and might signal hallucination.”

May combine two claims.

BLEU and ROUGE focus on surface-level information such as precision and recall over n-gram overlap, and thus fail to capture deeper semantic fluency and coherence in generated text.

“Popular metrics such as BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2002) and ROUGE (Recall Oriented Understudy for Gisting Evaluation) (Lin, 2004) focus on surface-level information, such as precision and recall over ngram overlap, thus fail to capture deeper semantic fluency and coherence in the generated text.”

• cites Papineni et al., 2002 sourceNot checked — no full text of the source
• cites Lin, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The approach has not yet been integrated into end-to-end generative model training, so its impact on actual model outputs and downstream task performance remains to be empirically demonstrated.

“Finally, our approach has not yet been integrated into end-to-end generative model training, and thus its impact on actual model outputs and downstream task performance remains to be empirically demonstrated.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The statistically significant differences between models are as expected by their number of parameters and training protocols.

“After the Friedman test, a follow-up with post-hoc tests reported in Table 1 shows which models are statistically significantly different as expected by their number of parameters and training protocols.”

Classified as probably not a substantial claim. May combine two claims.

An increase in the number of false positives reduces precision and might signal hallucination.

“While an increase in the number of FNs reduces Recall(R) and signals factual inconsistency, an increase in the number of FPs reduces Precision(P) and might signal hallucination.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The decomposition of text into atomic facts can introduce subjectivity or error depending on the quality of the fact extraction process.

“Third, the decomposition of text into atomic facts, while useful for consistency measurement, can introduce subjectivity or error depending on the quality of the fact extraction process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation primarily focused on controlled datasets, so the robustness of the scoring algorithm in highly noisy, real-world clinical or open-domain scenarios remains to be validated.

“First, our evaluation primarily focused on controlled datasets, and the robustness of the scoring algorithm in highly noisy, real-world clinical or opendomain scenarios remains to be validated.”

Classified as probably not a substantial claim.

The constituents of the AlignCheck score, FP and FN, provide interpretable insights into where a model went wrong by fabricating new facts (FP) or ignoring necessary facts (FN).

“In addition to the score, which shows the semantic overall based on tangible facts, the constituents of the score, including FP and FN, pro-vide interpretable insights of where the model went wrong by fabricating new facts enumerated in FP or ignoring necessary facts enumerated in FN.”