CoChat
Sign Up
Cited by 2Natural Language Processing TechniquesDiscourse Analysis in Language StudiesSentiment Analysis and Opinion Mining

STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond

Open access
Dycke, Nils, Zečević, Matej, Kuznetsov, Ilia, Suess, Beatrix, Kersting, Kristian, Gurevych, Iryna·2025·Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2025.acl-long.1107
Open this paper in CoChat — free accountOpen PDF

Abstract

Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, Iryna Gurevych. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

AI snapshot

Written from the paper’s full text.

TL;DR

STRICTA is a specification framework that models critical text assessment as an explicit, causally structured reasoning graph, instantiated for biomedical paper assessment with expert-generated data and LLM assistance experiments. It provides a dataset of over 4,000 reasoning steps and shows that LLMs can align with human reasoning under oversight but are prone to error propagation and decision-making differences.

Key findings

  • The biomedical paper assessment workflow contains 45 interdependent steps, and the collected dataset includes 4,371 step answers, 93 workflow executions, 22 papers, 240,110 tokens, and an average of 60.04 tokens per answer; Krippendorff’s α on boolean nodes is 0.424.
  • Human answer variability is moderate: semantic similarity averages 0.55, with INFER-KNOWLEDGE steps deviating most and EXTRACT steps most similar; substantial disagreement propagates to subsequent questions with probability 0.72.
  • Causal analysis finds step33 (alignment of conclusions and research questions) has the highest positive average causal effect on the final verdict at 0.37, while writing clarity (step48) also has ACE 0.20; in counterfactual analysis of negative verdicts, 60% could have been changed by intervening on a single figure-related step (step12).
  • In LLM experiments, GPT-4o with human oversight reaches F1 0.876 on boolean decisions versus a human baseline of 0.801, but LLMs as a program perform far lower (e.g., Llama3 Prg F1 0.170, GPT4o Prg F1 0.720); error propagation is identified as a factor, and human oversight mitigates it.

Methods

  • The paper formalizes STRICTA using structural causal models (SCMs), where a workflow is a directed acyclic graph with inputs, reasoning components, and final verdicts; it defines a three-step process: design the SCM structure, populate it with data, and analyze/automate.
  • For the biomedical case study, the workflow was elicited from two experienced biomedical researchers, validated with three PhD students, then used in an annotation study: JUNIOR stage with 36 postgraduate students and SENIOR stage with 8 doctoral/postdoctoral researchers plus 2 postgraduate students, covering 22 bioRxiv papers.
  • Analysis used a boolean decision SCM with Gaussian process classifiers, average causal effect estimation, and counterfactual queries; LLM experiments tested Llama 3 8B, Mixtral, GPT-3.5 turbo, and GPT-4o in an abductive setting with automatic metrics (BERT-F1, SummaC, TRUE, F1) and a human evaluation subset.

Limitations

  • Stated: the framework assumes verdicts and reasoning steps arise from consuming the text and do not exist independently, a perspective subject to philosophical discussion; the case study findings are limited to the biomedical domain, and the sample may retain topical and annotator-pool bias.
  • Stated: the LLM evaluation relies on automatic metrics that may not cover all aspects of suitable answers and seem to favor more extensive answers; the human baseline is a conservative lower-bound estimate, and more large-scale real-world user studies are needed.
  • Evident from the text: the transfer study to NLP paper assessment was formative and not quality-assured, so its output dataset is not suitable for evaluation or analysis; the dataset is not designed to compare behavioral patterns between groups with different experience levels.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.acl-long.1107

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 34 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Most studies in NLP for peer review focus exclusively on machine learning and NLP domains.

    “Second, most studies in NLP for peer review focus exclusively on machine learning and NLP domains (Staudinger et al., 2024); we contribute to bridging the gap towards other lessrepresented scientific domains (Kuznetsov et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Staudinger et al., 2024 source
  • Premisecitation

    Prior work largely treats critical text assessment as a black box, leaving unanswered how experts arrive at their judgement.

    “Yet, existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration.”
    • Also stated in: § 1 Introduction · plain
      “Yet, prior work largely treats critical text assessment as a black box, leaving a crucial question unanswered: How do the experts arrive at their judgement?”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Shen et al., 2019 source

      The Introduction says prior document-quality studies mainly focus on textual features, and Related Work describes features and models used to predict quality. Neither section says that critical text assessment is treated as a black box or that this limits interpretability or human–AI collaboration; searches for those concepts found no supporting passage in our copy. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Maillette de Buy Wenniger et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites D' Arcy et al., 2024 source

      Section 2.1 (page unavailable) describes limitations of earlier review-generation methods, including template-based comments, limited input size, and use of a single prompt. It does not say that prior work treats critical text assessment as a black box, leaves expert judgment unexplained, or limits interpretability and human–AI collaboration. Searches for those concepts found no passage addressing that claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Premisecitation

    Expert-elicited SCMs are commonly held as the gold standard.

    “Hereby, we ensure close alignment to the true human reasoning process and ensure the quality of the workflow since expert-elicited SCMs are commonly held as the gold standard (Zanga et al., 2022;Zhang et al., 2023).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Zanga et al., 2022 source

      Section 7.2 describes one study that used a causal graph retrieved from literature as a gold standard. Section 2.7 discusses expert-elicited causal knowledge, but the survey does not say that expert-elicited SCMs are commonly regarded as the gold standard. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Zhang et al., 2023 source

      The Introduction (page unavailable) says causal-model auditing typically requires a qualified human domain expert, but it does not say that expert-elicited SCMs are commonly regarded as the gold standard. Searches for expert-elicited models and the gold-standard claim found no passage addressing that point. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Premisereasoning

    Automatic review generation methods neglect the fine-grained decision-making factors in humans and LLMs that shape complex assessment.

    “While these methods offer valuable insights into downstream performance in peer review, they neglect the fine-grained decision-making factors in humans and LLMs that shape the complex assessment.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites D'Arcy et al., 2024 source

      Section 2.1 (page unavailable) says earlier review-generation methods are limited in scope or rely on templates rather than nuanced free-form comments; Section 1 describes the reasoning involved in assessing a paper. These passages support the concern that prior methods miss aspects of complex assessment, but they do not say those methods neglect fine-grained decision-making factors in both humans and LLMs. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Yuan et al., 2022 source
    • cites Du et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation

    LLMs integrate naturally with the causal graph to solve STRICTA problems reconstructing human reasoning from partial information by turning the causal graph into an LLM program.

    “In Section 5 we demonstrate that LLMs integrate naturally with the causal graph to solve STRICTA problems reconstructing human reasoning from partial information by turning the causal graph into an LLM program (Dohan et al., 2022).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dohan et al., 2022 source
  • Classified as probably not a substantial claim. 88
  • May combine two claims. 59
  • The quote only partly states this claim. 5

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Causality-aware text representation is a hard research problem without a general solution.

    “Causality-aware text representation is a hard research problem without a general solution(Veitch et al., 2020;Feder et al., 2022).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Veitch et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Feder et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    The JUNIOR stage involved 36 postgraduate students in the biomedical field; the SENIOR stage involved 8 doctoral and postdoctoral researchers and 2 postgraduate students.

    “In the JUNIOR stage, the annotation setup was validated within a university course, where 36 postgraduate students in the biomedical field participated voluntarily.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The sample of 200 papers was filtered manually by an expert annotator and subselected by annotator preference, resulting in 22 papers.

    “The resulting sample of 200 papers is filtered manually by an expert annotator to ensure high expertise and is subject to subselection by annotator preference, resulting in 22 papers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data

    Based on expert interviews, there are three key activities during assessment: READ, EXTRACT, and INFER.

    “Based on the expert interviews, there are three key activities during assessment:”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The resulting workflow has 45 interdependent paper quality assessment steps terminating in a final verdict on paper quality.

    “The resulting workflow has 45 interdependent paper quality assessment steps terminating in a final verdict on the paper quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Low lexical and syntactical similarity suggests that annotators employ their individual style and register and do not use template language.

    “Notably, the low lexical and syntactical similarity suggests that annotators employ their individual style and register and do not use template language, as has been previously observed for human-generated explanations (Camburu et al., 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Camburu et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The resulting dataset consists of a JUNIOR subset with 4 annotated papers and a SENIOR subset covering 18 papers.

    “The resulting dataset consists of a JUNIOR subset with 4 annotated papers and a SENIOR subset covering 18 papers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At least three annotators annotated each paper and workflow step; 11 papers have annotations by five different annotators.

    “At least three annotators have annotated each paper and workflow step; 11 papers have annotations by five different annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The impact of background variables is reasonably low, suggesting that no important factors were left unmodelled in the SCM.

    “The impact of background variables is reasonably low, suggesting that no important factors were left unmodelled in the SCM.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The average semantic similarity between annotators is 0.55, similar to open-ended text production tasks such as story generation.

    “The semantic similarity, on average, lies at 0.55, similar to open-ended text production tasks, such as story generation (Giulianelli et al., 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Giulianelli et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • The quote only partly states this claim.

    Findingown data

    Step33 (Alignment of Conclusions and Research Questions) has the highest positive impact on the final verdict, with ACE 0.37.

    “step33 (Alignment of Conclusions and Research Questions) has the highest positive impact on the final verdict.”
    • Also stated in: § Step Prompt ACE · plain
      “Considering the correct conclusions and the paper's research questions, do the conclusions fully answer the research questions? (step33) 0.37”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is significant overlap between human responses, but they raise different points.

    “Overall, there is a significant overlap between human responses, but they do raise different points.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Answers to steps based on background knowledge deviate the most, whereas EXTRACT steps are most similar.

    “Answers to the steps based on background knowledge deviate the most, whereas EXTRACT steps are most similar.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The dataset was split into 20 test papers (4089 answers) and 2 papers (282 answers) for prompt development.

    “We split the dataset into 20 test papers (4089 answers) and reserve 2 papers (282 answers) for prompt development.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There are two general types of papers in biomedicine relevant to the experts' work: natural science papers, which investigate natural phenomena, and method papers, which propose a new approach to conducting experiments.

    “First, there are two general types of papers in biomedicine relevant to their work: natural science papers, which investigate natural phenomena, and method papers, which propose a new approach to conducting experiments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Several heuristics exist to detect bad experiment design, such as the lack of controls or suspiciously high standard deviations on measurements.

    “Finally, several heuristics exist to detect bad experiment design, such as the lack of controls or suspiciously high standard deviations on measurements.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Nearly all annotators speak English as a second language.

    “Nearly all annotators speak English as a second language.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingown data

    Annotators need training to get an overview of the workflow graph to avoid redundantly stating similar facts across multiple steps.

    “Third, annotators need training to get an overview of the workflow graph to avoid redundantly stating similar facts across multiple steps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The deduplicated list of preprints resulted in roughly 100k preprint URLs.

    “We deduplicated the list, resulting in roughly 100k in preprint URLs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The dataset captures roughly 4000 reasoning steps, with three to five expert responses per paper, contributed by roughly 40 experts in over seven hundred hours of work.

    “The dataset captures roughly 4000 reasoning steps, with three to five expert responses per paper, contributed by roughly 40 experts in over seven hundred hours of work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Several of the JUNIOR annotators do not regularly read biomedical papers, but several consume them occasionally.

    “Several of the JUNIOR annotators do not regularly read biomedical papers, but several consume them occasionally.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average degree per node in the graph lies at 3.57.

    “The average degree per node lies at 3.57;”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Biomedical papers follow a relatively rigid surface structure with an introduction including background information, the methods section as a detailed experimental log, the results section reporting the exact measurements, and the discussion section followed by an overall conclusion; the results and discussion section can be intermingled.

    “Second, biomedical papers follow a relatively rigid surface structure with an introduction including background information, the methods section as a detailed experimental log, the results section reporting the exact measurements, and the discussion section followed by an overall conclusion.The results and discussion section can be intermingled.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingnone given

    Figures and tables play a major role in the understanding and assessment of a paper.

    “Third, figures and tables play a major role in the understanding and assessment of a paper.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The final dataset of 22 annotated papers shows a light skew towards accepted papers, but all world regions, versions, and publication statuses are present in similar proportions as in the full sample.

    “While observing a light skew towards accepted papers, all world regions, versions, and publication statuses are present in similar proportions as in the full sample.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Nearly all annotators are from the age group 20-29.

    “Nearly all annotators are from the age group 20 -29.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    All annotators self-report being proficient in English; half of them are native speakers.

    “All annotators self-report being proficient in English; half of them are native speakers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingnone given

    Methodological soundness and consistency of results are key indicators during text assessment.

    “Fourth, methodological soundness and consistency of results are key indicators during text assessment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    All papers in the dataset are written in English.

    “All papers are written in English which is the lingua franca across science.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    While some subordinate steps deviate and the researchers put different emphasis on each aspect, the overall assessment approach is very similar between the two manuals.

    “While some of the subordinate steps deviate and the researchers put a different emphasize on each of these aspects, the overall assessment approach is very similar.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Results show moderate agreement among humans and a moderate but significant correlation between automatic metrics and human judgment (BERT-F1 ρ = 0.37 to 0.54; SummaC ρ = -0.22 to 0.55).

    “Results show moderate agreement among humans -typical for challenging tasks -and a moderate but significant correlation between metrics and human judgment (between ρ = 0.37 and ρ = 0.54 for BERT-F1, and ρ = -0.22 and ρ = 0.55 for Sum-maC, depending on the step type) at similar levels as related automatic evaluation metrics (Fu et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There are 4 terminal nodes of which one is the final verdict, whereas the others are the assessment of clarity and the high-level summary based on the skim.

    “there are 4 terminal nodes of which one is the final verdict, whereas the others are the assessment of clarity and the high-level summary based on the skim.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Running GPT-4o independently for each step, using the full paper as input and applying the same self-refinement loop, results in improved performance compared to GPT-4o operating as a program, though still below its performance with human oversight.

    “Results, shown in the last line of Table 3, reveal improved performance compared to GPT-4o operating as a program, though still below its performance with human oversight.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LLMs tend to restate inputs, split up facts into small components, and make harsher decisions.

    “However, LLMs tend to restate inputs, split up facts into small components, and make harsher decisions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There are 3 roots in the graph (the READ nodes).

    “there are 3 roots in the graph (the READ) nodes;”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The consolidation of the sequential manuals results in a workflow of 106 reasoning steps covering all sections of the paper.

    “This results in a workflow of 106 reasoning steps covering all sections of the paper.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The result of the interviews is two different step-wise manuals on how to iterate through a biomedical paper and assess its methodological soundness.

    “The result of these interviews is two different step-wise manuals on how to iterate through a biomedical paper and assess its methodological soundness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The SENIOR annotators all consume papers regularly.

    “The SENIOR annotators all consume papers regularly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Upon qualitative inspection, the two major sections (results and discussion) form a dense network of descendants, while there are only a few connections between the nodes belonging to the different sections.

    “Upon qualitative inspection of the graph, it becomes clear that the two major sections (results and discussion) form a dense network of descendants, while there are only a few connections between the nodes belonging to the different sections.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The findings from our simulation study on human-AI collaboration support the principle that structured reasoning offers built-in interpretability and facilitates human-AI collaboration.

    “The findings from our simulation study on human-AI collaboration support this principle.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All LLMs demonstrate a notable performance increase in the io-condition, effectively adapting to the annotator's perspective based on their step-by-step inputs and producing more aligned responses.

    “All LLMs demonstrate a notable performance increase, effectively adapting to the annotator's perspective based on their step-by-step inputs and producing more aligned responses.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The human oversees and corrects the LLM at each step, providing the input as human-generated answers from parent steps or the paper for EXTRACT steps.

  • May combine two claims.

    Findingown data

    Humans reserve negative decisions for extreme cases and focus their answers on a minimal set of facts relevant to the individual assessment.

    “showing that humans reserve negative decisions for extreme cases and focus their answers on a minimal set of facts relevant to the individual assessment.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    In a small-scale user study with nine postgraduate biomedical students, the structured workflow received an average rating of 5.67 for usefulness in paper assessment.

    “On a scale from 1 (fully disagree) to 7 (fully agree), the structured workflow received an average rating of 5.67 for usefulness in paper assessment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The mean number of parents per node lies at 1.78.

    “the mean number of parents lies at 1.78;”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The distribution of genders among annotators is nearly balanced between male and female; diverse was never self-reported.

    “The distribution of genders is nearly balanced between male and female; diverse was never selfreported.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A stratified sample of 200 papers was manually inspected and filtered by one expert annotator, resulting in 57 eligible papers, of which 22 were annotated.

    “For this purpose, we take a stratified sample of 200, which is then manually inspected and filtered by one expert annotator, resulting in 57 eligible papers, of which 22 were annotated.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    GPT-4o's performance is better on boolean decision steps than Mixtral but worse during extraction of facts.

    “GPT-4o's performance is better on boolean decision steps than Mixtral but worse during extraction of facts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The graph structure of the SCM aligns well with the data, as there are no other permutations of the DAG inside its Markov equivalence class.

    “The graph structure of the SCM aligns well with the data, as there are no other permutations of the DAG inside its Markov equivalence class.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The non-root nodes of the SCM are sampled with additive noise.
    • The authors state this as an assumption: The SCM is invertible.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average amount of aligned explanations is 0.54 for BERT-F1.

    “BERT-F1 0.37 | 0.54”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Roughly 21% of all answers show substantial disagreements.

    “Under this assumption, roughly 21% of all answers show substantial disagreements.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Jaccard metric on facts for matching items is 0.37 for BERT-F1.

    “BERT-F1 0.37 | 0.54”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The annotation study results in 40 annotation items of which 14 items across the 8 workflow steps have at least two annotations.

    “The annotation study results in 40 annotation items of which 14 items across the 8 workflow steps have at least two annotations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The results are consistent across multiple reasoning settings.

    “The results are consistent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average amount of aligned explanations is 0.20 for TRUE.

    “TRUE 0.30 | 0.20”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Llama-3 with 70B parameters had insufficient context size to fit the relevant inputs.

    “We also tested other models including Llama-3 with 70B parameters but the context size was insufficient to fit the relevant inputs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average KL divergence between the distribution implied by the learned structural equations lies at 0.18 which shows an overall close fit to the data.

    “The average KL divergence between the distribution implied by the learned structural equations lies at 0.18 which shows an overall close fit to the data.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The non-root nodes of the SCM are sampled with additive noise.
    • The authors state this as an assumption: The SCM is invertible.

  • May combine two claims.

    Findingown data

    SummaC appears inappropriate for matching evaluation (weak negative correlation), but well-suited for alignment.

    “SummaC, on the other hand, appears inappropriate for matching evaluation (weak negative correlation), but well-suited for alignment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    7 annotators participated in the study; one annotator was filtered out based on screening.

    “7 annotators participated in the study; we filter out one annotator based on the described screening.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Humans mostly lean towards positive decisions (67% 'yes' answers), whereas most LLMs are more restrictive (44%-59% 'yes' answers).

    “Humans mostly lean towards positive decisions (67% 'yes' answers), whereas most LLMs are more restrictive (44%-59% 'yes' answers).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Jaccard metric on facts for matching items is -0.22 for SummaC.

    “SummaC -0.22 | 0.55”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    All annotators indicate that they read biomedical papers regularly.

    “All annotators indicate that they read biomedical papers regularly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average amount of aligned explanations is 0.55 for SummaC.

    “SummaC -0.22 | 0.55”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The results of the human evaluation closely resemble the findings based on the automatic metrics, with Mixtral and GPT-4o performing nearly on-par and the human conservative baseline lagging slightly behind.

    “The results closely resemble the findings based on the automatic metrics with Mixtral and GPT-4o performing nearly on-par and the human conservative baseline lagging slightly behind.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Substantial disagreement propagates to the subsequent questions with a 72% chance as compared to random disagreement with only 21%.

    “As expected, substantial disagreement propagates to the subsequent questions with a 72% chance as compared to random disagreement with only 21%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Jaccard metric on facts for matching items is 0.30 for TRUE.

    “TRUE 0.30 | 0.20”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Automatically generated answers to EXTRACT steps tend to split up facts exhaustively, whereas humans make broader and summarizing statements.

    “Upon manual inspection, we find that the automatically generated answers to EXTRACT steps tend to split up facts exhaustively, whereas humans make broader and summarizing statements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The agreement level is in line with agreement levels reported for peer review ratings and typical for subjective annotation tasks.

    “This is in line with agreement levels reported for peer review ratings (Bornmann, 2011;Dycke et al., 2021) and typical for subjective annotation tasks.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Bornmann, 2011 source
    • cites Dycke et al., 2021 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Working on Boolean variables makes standard causal estimation tractable.

    “Working on Boolean variables makes standard causal estimation tractable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Both JUNIOR and SENIOR stages used the same guidelines, setups, and annotation interfaces.

    “Both stages used the same guidelines, setups, and annotation interfaces.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The dataset size is sufficient for the purposes of the study, validated using statistical analysis.

    “Additionally, we validate that the dataset size is sufficient for the purposes of our study using statistical analysis and report the details in Appendix A.4.1.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Causal modeling is indispensable for studying decision factors during text assessment.

    “Thus, causal modeling is indispensable for studying decision factors during text assessment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The style and writing of the paper directly influence decision making and not only lead to higher variability due to different interpretations.

    “This result also shows that the style and writing of the paper directly influence decision making and not only lead to higher variability due to different interpretations of potentially unclear or ambiguous paper contents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    A fixed reasoning structure, where each step has a well-defined, context-independent meaning, enables quantitative comparison of human reasoning factors and fine-grained evaluation of automatic methods across instances.

    “A fixed reasoning structure, where each step has a well-defined, context-independent meaning, enables quantitative comparison of human reasoning factors and fine-grained evaluation of automatic methods across instances.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The workflow for biomedical paper assessment was derived from expert interviews and validated with three PhD students in a formative study.

    “We derive the workflow for biomedical text assessment based on expert interviews, followed by an independent validation step, opting against the automatic causal discovery of the workflow from data such as peer review reports.”
    • Also stated in: § … · plain
      “Finally, we validate the graph with three PhD students from the same field in a formative study where we ask them to assess a biomedical article using the reasoning graph.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    AI support during paper assessment on background knowledge shows great potential.

    “Considering variability as a proxy for step difficulty, AI support during paper assessment on background knowledge shows great potential.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Paper assessment is a critical step in scholarly peer review and can substantially benefit from AI assistance.

    “Paper assessment is a critical step in scholarly peer review and can substantially benefit from AI assistance (Kuznetsov et al., 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kuznetsov et al., 2024 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Ignoring training error, the resulting SCM from Step 2 perfectly models the human reasoning process for the given problem.

    “Ignoring training error, the resulting SCM from Step 2 perfectly models the human reasoning process for the given problem.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    By enforcing step dependencies during annotation, the study ensures the assumed causal constraints.

    “By enforcing the step dependencies during annotation, we ensure the assumed causal constraints.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Prior studies on explanations in fact-checking focus on short factual claims and often ignore ambiguity.

    “Prior studies have generated structured explanations for final verdicts as rule chains (Gad-Elrab et al., 2019), argumentation graphs (Si et al., 2024), multi-hop graphs (Jiang et al., 2020), or verification programs (Pan et al., 2023), focusing on short factual claims and often ignoring ambiguity (Glockner et al., 2024).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Gad-Elrab et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Si et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Jiang et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Automatic essay scoring can enhance teacher consistency and enable large-scale student self-teaching.

    “Automatic essay scoring (AES) can enhance teacher consistency and enable large-scale student self-teaching (Taghipour and Ng, 2016).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Taghipour and Ng, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The annotators involved in the human evaluation received 10.40£ per hour.

    “The annotators involved in the human evaluation received 10.40£ per hour.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The terminal nodes only exist to ease the annotation and have no meaning for the overall SCM modeling of the DAR problem.

    “These only exist to ease the annotation and have no meaning for the overall SCM modeling of the DAR problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    All human data were collected based on explicit consent.

    “All annotators could choose freely to reject participation in the study, and all human data were collected based on explicit consent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Our study paves the path for more explainable, controlled, and grounded AI assistance for assessment tasks.

    “Bridging the research in causality, human-AI collaboration, and LLMs, our study paves the path for more explainable, controlled, and grounded AI assistance for assessment tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All senior annotators and experts participating in the study received adequate compensation within their regular employment in the research project.

    “All senior annotators and experts participating in the study received adequate compensation within their regular employment in the research project.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisereasoning

    Paper assessment can be seen as a diagnostic task.

    “By the above general definition, paper assessment can be seen as a diagnostic task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given

    The task of multi-rubric essay scoring aligns well with STRICTA's scope.

    “This task aligns well with STRICTA's scope, and we propose a streamlined protocol based on our study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The subjective nature of the task and the complex interaction of quality criteria make it impossible to derive the verdict from formal rules.

    “The subjective nature of the task and the complex interaction of quality criteria make this impossible.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The annotation study employed guidelines and an annotation interface to ensure maximal consistency among annotators and to encourage step-wise causal thinking.

    “We employed the following guidelines and annotation interface to ensure maximal consistency among annotators and to encourage step-wise causal thinking.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The annotators were required to have at least a post-graduate degree in biomedical or biochemical sciences to ensure relevant expertise.

    “We require the annotators to have at least a post-graduate degree in biomedical or biochemical sciences to ensure relevant expertise.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    As a key quality measure, the study repeatedly engaged with individual annotators and the whole annotator group to calibrate and provide feedback on their annotations.

    “As another key quality measure, we repeatedly engaged with individual annotators and the whole annotator group to calibrate and provide feedback on their annotations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Our experiments demonstrated the versatility of our framework for analysis and LLM assistance.

    “Our experiments demonstrated the versatility of our framework for analysis and LLM assistance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We ensured high redundancy during annotation to avoid biasing the dataset to the idiosyncrasies of individual annotators.

    “We ensured high redundancy during annotation to avoid biasing the dataset to the idiosyncrasies of individual annotators.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The reasoning process of paper assessment is probabilistic, i.e., reviewers choose the best of all possible explanations for their verdict.

    “First, unlike deduction, the reasoning process of paper assessment is probabilistic, i.e., reviewers choose the best of all possible explanations for their verdict.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Publishing subjective assessments of the papers by study participants could potentially harm the reputation of the publication authors when they are assessed unfairly.

    “However, as we publish subjective assessments of the papers by our study participants, this could potentially harm the reputation of the publication authors when they are assessed unfairly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Structured reasoning offers built-in interpretability and facilitates human-AI collaboration by breaking down problems into smaller steps.

    “Structured reasoning offers built-in interpretability and facilitates human-AI collaboration by breaking down problems into smaller steps (Luo et al., 2024;Ferdaus et al., 2024).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Luo et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ferdaus et al., 2024 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The given sample of papers and the annotator pool might still introduce a certain level of bias to the dataset.

    “Yet, the given sample of papers and the annotator pool might still introduce a certain level of bias to the dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The underlying paper dataset was crawled from bioRxiv and ensures high sample diversity along relevant dimensions.

    “We crawl the underlying papers from bioRxiv and ensure a high sample diversity along relevant dimensions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Only SCMs allow for studying cause and effect relationships, making them strictly more powerful than the other approaches.

    “Only SCMs allow for studying cause and effect relationships (Pearl, 2000), making them strictly more powerful than the other approaches.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pearl, 2000

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The collected dataset is intended for research on reasoning during peer review across human subjects.

    “We highlight that the collected dataset is intended for research on reasoning during peer review across human subjects; it is not suitable for comparing behavioral patterns between groups, e.g. by varying levels of experience, which would require a different experimental design where annotators of different groups annotate the exact same papers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We conduct a human evaluation on a subset of the data to validate the automatic metrics.

    “We also conduct a human evaluation on a subset of the data to validate the automatic metrics.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    STRICTA is designed as a general domain-independent framework for critical text assessment.

    “We design STRICTA as a general domainindependent framework for critical text assessment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The probabilistic elements of SCMs can directly model the statistical variance induced by the ambiguity and subjectivity of most document assessment tasks.

    “On one hand, the probabilistic elements of this formalism can directly model the statistical variance induced by the ambiguity and subjectivity of most document assessment tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    In the given task, the goal is to find the unobserved intermediate steps that lead to the observed verdict.

    “Second, in the given task, framing the goal is to find the unobserved intermediate steps that lead to the observed verdict, which aligns precisely with the definition of abduction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Structured reasoning enables systematic analysis of human-LLM collaboration potential.

    “Structured reasoning also enables systematic analysis of human-LLM collaboration potential (see Section 5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The STRICTA framework allows for defining various reasoning tasks on the underlying workflow depending on the provided data.

    “The STRICTA framework allows for defining various reasoning tasks on the underlying workflow depending on the provided data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    Underlying our dataset is a set of publicly available papers with open licenses (CC-BY, CC0, ...) that allow the redistribution and extension of the contents.

    “Underlying our dataset is a set of publicly available papers with open licenses (CC-BY, CC0, ...) that allow the redistribution and extension of the contents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The ability to pose causal and counterfactual queries on SCMs enables a rich methodology to study the reasoning process of humans and potentially AI models in terms of explanations.

    “On the other hand, the ability to pose causal and counterfactual queries on SCMs enables a rich methodology to study the reasoning process of humans and potentially AI models in terms of explanations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Errors on earlier steps in the workflow deprive later steps of information or provide incorrect information for prediction.

    “This suggests that errors on earlier steps in the workflow deprive later steps of information or provide incorrect information for prediction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Participation in the annotation study did not impact the students' grading in the underlying course.

    “The participation in the annotation study did not impact the students' grading in the underlying course.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Since the reasoning structure for the explanation is fixed, the model output becomes easier to interpret.

    “Since the reasoning structure for the explanation is fixed, the model output becomes easier to interpret.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    By design, structured explanations allow verification of model outputs, promoting trustworthiness.

    “By design, they allow verification of model outputs, promoting trustworthiness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All annotators could choose freely to reject participation in the study.

    “All annotators could choose freely to reject participation in the study, and all human data were collected based on explicit consent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    These observations support the framing of paper assessment as a structured reasoning process and a strong alignment with the causal framework proposed in this paper.

    “These observations support the framing of paper assessment as a structured reasoning process and a strong alignment with the causal framework proposed in this paper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Framing paper assessment as a STRICTA problem opens up safe and interpretable human-AI collaboration.

    “Framing paper assessment as a STRICTA problem opens up safe and interpretable human-AI collaboration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study has been approved by the university's ethical board under proposal number EK 41/2024.

    “This study has been approved by the university's ethical board under the proposal number EK 41/2024.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    STRICTA supports effective and interpretable human-AI collaboration by enabling explainability and identifying opportunities for human-LLM collaboration.

    “Conceptually, STRICTA supports effective and interpretable human-AI collaboration in two ways: by enabling explainability and identifying opportunities for human-LLM collaboration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Our framework for STRICTA problems is designed to be general and applicable to any text assessment task.

    “Our framework for STRICTA problems is designed to be general and applicable to any text assessment task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductreasoning

    Some topical bias is inevitable to ensure relevance to the annotators' expertise.

    “However, some topical bias is inevitable to ensure relevance to the annotators' expertise.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The human evaluation selected 8 out of the 45 workflow steps, focusing on early-stage extract steps and critical boolean decision steps identified by their average causal effect on the final outcome.

    “To streamline this process, we select 8 out of the 45 workflow steps, focusing on early-stage steps (extract steps) and critical boolean decision steps identified by their average causal effect on the final outcome (see Section 4.3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The used automatic metrics capture human judgement to some extent and can serve as a solid estimate for performance.

    “In conclusion, the used automatic metrics capture human judgement to some extend and can serve as a solid estimate for performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The dataset size is sufficient for the objectives of this study since the results are not sensitive to adding or removing a paper.

    “Thus, the dataset size is sufficient for the objectives of this study since the results are not sensitive to adding or removing a paper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The human baselines are lower bound estimates of human performance because all answers are compared despite different verdicts.

    “The human baselines are lower bound estimates of human performance because all answers are compared despite different verdicts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    An accurate estimate would require humans to re-annotate steps using inputs from their fellow annotators.

    “An accurate estimate would require humans to re-annotate steps using inputs from their fellow annotators.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Both factors might contribute to the high factual alignment by automatic metrics because LLMs appear more specific and cover many facts, whereas humans tend to select a minimal set of facts relevant to their individual assessment process.

    “Both factors might contribute to the high factual alignment by automatic metrics because LLMs appear more specific and cover many facts, whereas humans tend to select a minimal set of facts relevant to their individual assessment process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    This moderate level of agreement is similar to other tasks aiming to identify links between two texts and points to the difficulty of the task similar to annotation scenarios in argumentation mining.

    “This moderate level of agreement is similar to other tasks aiming to identify links between two texts (e.g.implicit linking in Kuznetsov et al. (2022)) and point to the difficulty of the task similar to annotation scenarios in argumentation mining (Lawrence and Reed, 2020;Iskender et al., 2021).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Kuznetsov et al. (2022) source
    • cites Lawrence and Reed, 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Iskender et al., 2021 source
126 of 126

  • The paper relies on this without support.

    The lack of formal models and datasets that capture expert reasoning during critical text assessment prevents progress in this area.

    “Yet, the lack of formal models and datasets that capture expert reasoning during critical text assessment prevents progress in this area.”

  • The paper relies on this without support.

    Many text assessment tasks are partially subjective and allow for multiple valid interpretations.

    “Since many text assessment tasks are partially subjective and allow for multiple valid interpretations, we explore the use of structural causal models (SCMs) (Pearl, 2000) to explicitly account for this variance during reasoning.”

  • The authors state this as an assumption.

    The STRICTA framework assumes an expert extracts key information from the document as input for a series of interconnected reasoning steps.

    “This framework assumes an expert extracts key information from the document as input for a series of interconnected reasoning steps.”

  • The authors state this as an assumption.

    All problem instances in a STRICTA problem class share the same reasoning structure, with variation arising only in the graph's variables.

    “Notably, we assume that all problem instances in a STRICTA problem class share the same reasoning structure, with variation arising only in the graph's variables.”

  • The paper relies on this without support.

    Varying experiences and biases act as background factors.

    “Varying experiences and biases act as background factors.”

  • The paper relies on this without support.

    If an assessment task is subjective, background factors induce high variability in the answers.

    “If an assessment task is subjective, background factors induce high variability in the answers.”

  • The authors state this as an assumption.

    The human oversees and corrects the LLM at each step, providing the input as human-generated answers from parent steps or the paper for EXTRACT steps.

    “Here, we assume the human oversees and corrects the LLM at each step, providing the input as human-generated answers from parent steps or the paper for EX-TRACT steps.”

  • The authors state this as an assumption.

    The final verdicts and reasoning steps arise when consuming the input text and do not exist on their own.

    “Here, we assume that the final verdicts and reasoning steps arise when consuming the input text and do not exist on their own.”

  • The authors state this as an assumption.

    A text is consumed, causing the verdict, and the reasoning steps in between give rise to this verdict.

    “In practice, this means we assume that a text is consumed, causing the verdict (e.g.,”

  • The paper relies on this without support.

    For any reasonable text assessment process, the final verdict should depend on the input text of the problem.

    “Additionally, for any reasonable text assessment process, the final verdict should depend on the input text of the problem.”

  • The authors state this as an assumption.

    The read nodes of different sections are independent except for the clarity of the paper acting as a confounder.

    “We make the assumption that the read nodes of different sections are independent except for the clarity of the paper acting as a confounder.”

  • The paper relies on this without support.

    The automatic metrics used in this work are widely adopted in the community, but they remain only imperfect approximators of the real performance perceived by the users.

    “While the automatic metrics used in this work are widely adopted in the community, they still remain only imperfect approximators of the real performance perceived by the users.”

  • The authors state this as an assumption.

    The non-root nodes of the SCM are sampled with additive noise.

    “the other nodes are assumed to be sampled with additive noise.”

  • The authors state this as an assumption.

    The SCM is invertible.

    “The model is represented as an invertible structural causal model”
    “The invertibility assumption of the nodes”
14 of 14

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Factual-alignment scores are not generally above the human baseline

    Major

    The text says the GPT models and Mixtral score above the human baseline on SummaC and TRUE. In Table 3, GPT-4o is below the human baseline on both metrics, while Mixtral is below it on TRUE.

  • Correlation method is called both Pearson and Spearman

    Minor

    The evaluation description says it computes Pearson correlation, but the caption of the corresponding Table 16 calls the reported coefficients Spearman's ρ.

  • Average tokens per answer does not match the reported totals

    Minor

    Dividing 240110 tokens by 4371 step answers gives 54.93 tokens per answer, not 60.04. The table does not identify a smaller denominator for this average.

    Reported 60.04 · recomputed 54.93251

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 42

  1. Minnie Mary Alice Ashdown and Giorgos Petrou. 2023. Development of a directed acyclic graph through ex- pert elicitation for causal inference in built environ- ment research. In Proceedings of Building Simula- tion 2023: 18th Conference of IBPSA, volume 18 of Building Simulation, pages 1184-1191, Shanghai, China. IBPSA. · DOI
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  3. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4157- 4165, Online. Association for Computational Lin- guistics. · DOI
  4. Eric Chamoun, Michael Schlichtkrull, and Andreas Vla- chos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the As- sociation for Computational Linguistics: ACL 2024, pages 9742-9763, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  5. Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entail- ment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7358-7370, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI
  6. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Sri- nath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, and Wenpeng Yin. 2024. LLMs assist NLP researchers: Critique paper (meta-)reviewing. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 5081-5099, Miami, Florida, USA. Association for Computational Linguistics. · DOI · Paper page
  7. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049- 5073, Toronto, Canada. Association for Computa- tional Linguistics. · DOI · Paper page
  8. James Fiacco, Shiyan Jiang, David Adamson, and Car- olyn Rosé. 2022. Toward automatic discourse pars- ing of student writing motivated by neural interpreta- tion. In Proceedings of the 17th Workshop on Inno- vative Use of NLP for Building Educational Applica- tions (BEA 2022), pages 204-215, Seattle, Washing- ton. Association for Computational Linguistics. · DOI
  9. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556-6576, Mexico City, Mexico. Association for Computational Lin- guistics. · DOI
  10. Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Ur- bani, and Gerhard Weikum. 2019. Exfakt: A frame- work for explaining facts over knowledge graphs and text. In Proceedings of the Twelfth ACM Interna- tional Conference on Web Search and Data Mining, WSDM '19, page 87-95, New York, NY, USA. As- sociation for Computing Machinery. · DOI
  11. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? evaluating uncertainty in neural text generators against human production variability. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 14349-14371. Association for Computational Linguistics. · DOI
  12. Max Glockner, Ieva Staliūnaitė, James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2024. AmbiFC: Fact-checking ambiguous claims with evi- dence. Transactions of the Association for Computa- tional Linguistics, 12:1-18. · DOI
  13. Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Adam Rodman, and Jonathan H. Chen. 2024. Large language model influence on diagnostic rea- soning: A randomized clinical trial. JAMA Network Open, 7(10):e2440969-e2440969. · DOI · Paper page
  14. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206. · DOI · Paper page
  15. Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Hyunseung Lim, Yoonsu Kim, Tak Yeon Lee, Hwa- jung Hong, Juho Kim, So-Yeon Ahn, and Alice Oh. 2024. LLM-as-a-tutor in EFL writing education: Fo- cusing on evaluation of student-LLM interaction. In Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), pages 284-293, Miami, Florida, USA. Association for Computational Linguistics. · DOI
  16. Matthew Hoffman, Francis Bach, and David Blei. 2010. Online learning for latent dirichlet allocation. In Advances in Neural Information Processing Systems, volume 23.
  17. Curran Associates, Inc. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3905-3920, Seattle, United States. Association for Computational Lin- guistics. · DOI
  18. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441-3460, Online. Association for Computational Linguistics. · DOI
  19. Rahul Kumar, Sandeep Mathias, Sriparna Saha, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1485-1495, Seattle, United States. Association for Computational Linguistics. · DOI
  20. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:163-177. · DOI · Paper page
  21. Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2-17. · DOI · Paper page
  22. Stephan Lewandowsky, John Cook, Ullrich Ecker, Do- lores Albarracin, Panayiota Kendeou, Eryn J New- man, Gordon Pennycook, Ethan Porter, David G Rand, David N Rapp, Jason Reifler, Jon Roozenbeek, Philipp Schmid, Colleen M. Seifer, Gale M. Sina- tra, Briony Swire-Thompson, Sander van der Linden, Thomas J. Wood, and Maria S. Zaragoza. 2020. The debunking handbook 2020. · DOI
  23. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023. CoAnnotating: Uncertainty-guided work allocation between human and large language models for data annotation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 1487-1505, Singapore. Association for Computational Linguistics. · DOI
  24. Gideon Maillette de Buy Wenniger, Thomas van Don- gen, Eleri Aedmaa, Herbert Teun Kruitbosch, Ed- win A. Valentijn, and Lambert Schomaker. 2020. Structure-tags improve text classification for schol- arly document quality prediction. In Proceedings of the First Workshop on Scholarly Document Process- ing, pages 158-167, Online. Association for Compu- tational Linguistics. · DOI
  25. Haile Misgna, Byung-Won On, Ingyu Lee, and Gyu Sang Choi. 2025. A survey on deep learning- based automated essay scoring and feedback genera- tion. Artificial Intelligence Review, 58(2):1-40.
  26. Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. Fact-checking complex claims with program-guided reasoning. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6981-7004, Toronto, Canada. Association for Com- putational Linguistics. · DOI
  27. Judea Pearl. 2000. Causality: Models, reasoning and inference, 2 edition, volume 7. Cambridge University Press.
  28. Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press.
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics. · DOI · Paper page
  30. Raymond Reiter. 1987. A theory of diagnosis from first principles. Artificial Intelligence, 32(1):57-95. · DOI · Paper page
  31. Nihar B. Shah. 2022. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6):76-87. · DOI · Paper page
  32. Jiasheng Si, Yibo Zhao, Yingjie Zhu, Haiyang Zhu, Wenpeng Lu, and Deyu Zhou. 2024. CHECKWHY: Causal fact verification via argument structure. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15636-15659, Bangkok, Thai- land. Association for Computational Linguistics. · DOI
  33. Swapna Somasundaran, Michael Flor, Martin Chodorow, Hillary Molloy, Binod Gyawali, and Laura McCulla. 2018. Towards evaluating nar- rative quality in student writing. Transactions of the Association for Computational Linguistics, 6:91-106. · DOI · Paper page
  34. Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation metrics in the era of GPT-4: Reli- ably evaluating large language models on sequence to sequence tasks. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 8776-8788, Singapore. Associa- tion for Computational Linguistics. · DOI
  35. Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of the Thirty-Sixth Conference on Un- certainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124 of Proceed- ings of Machine Learning Research, pages 919-928.
  36. Thomas Verma and Judea Pearl. 1990. Equivalence and synthesis of causal models. In Proceedings of the Sixth Annual Conference on Uncertainty in Artificial Intelligence, UAI '90, page 255-270, USA. Elsevier Science Inc.
  37. Yupei Wang, Renfen Hu, and Zhe Zhao. 2024b. Be- yond agreement: Diagnosing the rationale align- ment of automated essay scoring methods based on linguistically-informed counterfactuals. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8906-8925, Miami, Florida, USA. Association for Computational Linguistics. · DOI
  38. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Bench- marks 2021, December 2021, virtual.
  39. Nathan Young, Qiming Bao, Joshua Bensemann, and Michael Witbrock. 2022. AbductionRules: Train- ing transformers to explain unexpected inputs. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 218-227, Dublin, Ireland. Association for Computational Linguistics. · DOI
  40. Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Ren- Jing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, and Xiang Li. 2024. Automated peer reviewing in paper SEA: Standardization, eval- uation, and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10164-10184, Miami, Florida, USA. Association for Computational Linguistics. · DOI
  41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
  42. Dennis Zyska, Nils Dycke, Jan Buchmann, Ilia Kuznetsov, and Iryna Gurevych. 2023. CARE: Col- laborative AI-assisted reading environment. In Pro- ceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 3: System Demonstrations), pages 291-303, Toronto, Canada. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Related

Authors
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I.
Affiliations
  • National Research Center for Applied Cybersecurity ATHENE (DE)
  • Technische Universität Darmstadt (DE)
ORCID
Venue
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
22687-22727
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2025.acl-long.1107
OpenAlex
W4412944753
Cited by
1
References
0
Citations by year
2026: 1
FWCI
0.96
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
  • European Commission
  • Bundesministerium für Bildung und Forschung (01IS19081, FKZ 01IS19081)
  • HORIZON EUROPE European Research Council (101054961)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
peer review, structured reasoning, text quality evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review