CoChat
Sign Up
PreprintCited by 0Logic, Reasoning, and KnowledgeTopic ModelingNatural Language Processing Techniques

Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming

Open access
Chen, Zirong, Ma, Meiyi·2026·arXiv
DOI 10.48550/arxiv.2607.23019arXiv 2607.23019
Open this paper in CoChat — free accountOpen PDF

Abstract

Chain-of-thought (CoT) prompting enables large language models (LLMs) to tackle multi-step reasoning tasks, yet the generated intermediate steps are not guaranteed to be logically sound. We present Reason Popper-ly, a neurosymbolic framework that uses inductive logic programming (ILP) to learn relation composition rules from reasoning traces and deploys them as an online verifier for step-level correction. Given an LLM-generated trace, the method checks each inferred step against the learned rule table, diagnoses the violation type, rewrites incorrect steps with symbolically derived repairs, and regenerates the remaining suffix so that the model can produce its final answer conditioned on a verified trace. We evaluate on CLUTRR, a multi-hop kinship reasoning benchmark, using five language models over reasoning chains of 2 to 10 hops. Across all models, Reason Popper-ly consistently improves terminal accuracy over standard CoT, with gains of up to 48 percentage points for small models and 15 points for frontier models on the longest chains. Compared with a fully exogenous symbolic pipeline, our method performs better on harder instances by preserving the model's successful grounding while correcting only verifiable reasoning failures. In addition, step-level ILP verification yields a fine-grained error taxonomy that provides diagnostic insight beyond final-answer accuracy.

AI snapshot

Written from the paper’s full text.

TL;DR

Reason Popper-ly is a neurosymbolic framework that uses inductive logic programming to learn relation-composition rules from LLM reasoning traces and applies them as an online step-level verifier and repair mechanism. On CLUTRR kinship reasoning, it consistently improves terminal accuracy over standard chain-of-thought, with especially large gains on longer chains.

Key findings

  • Across all five evaluated models and all reasoning depths, Reason Popper-ly outperforms standard single-shot CoT. At 8–10 hops, Qwen-3.5:4B improves from 19.09% to 61.93%, Gemma-4:E4B from 14.39% to 62.74%, LLaMA-3.2:3B from 18.72% to 42.48%, Claude Sonnet 4.6 from 56.92% to 71.38%, and GPT-5.4 from 59.91% to 75.09%.
  • The paper reports gains of up to 48 percentage points for small models and 15 points for frontier models on the longest chains.
  • Compared with a fully exogenous symbolic pipeline, Reason Popper-ly overtakes it as chain length grows; at 8–10 hops its advantage ranges from 4.59 points for Claude Sonnet to 20.52 points for Gemma.
  • Step-level ILP verification yields an error taxonomy: smaller local models show a balanced mix with wrong rule dominant (43–49%), while frontier models shift toward missing inv (75% of GPT-5.4 errors and 78% of Sonnet-4.6 errors); hallucinated errors are nearly absent (0–2%).

Methods

  • Evaluation uses CLUTRR, with 3,073 randomly sampled instances stratified across reasoning chains of 2 to 10 hops; 200 held-out 2-hop instances are reserved for ILP rule learning and validation, while longer chains are used only at evaluation time.
  • Five language models are tested: Qwen-3.5:4B, Gemma-4:E4B, LLaMA-3.2:3B, Claude Sonnet 4.6, and GPT-5.4. Baselines are endogenous single-shot CoT and an exogenous symbolic composition pipeline.
  • Offline, Popper induces a composition table over `compose/3` from extracted positive and negative triples; online, each inferred step is checked against the table, violations are diagnosed into four types, invalid steps are patched, and the remaining suffix is regenerated conditioned on the corrected prefix. The primary metric is terminal accuracy.

Limitations

  • The paper states that evaluation is limited to kinship reasoning, where relation composition is well defined and grounding is relatively tractable through templated parsing; domains with more ambiguous predicate structures would be substantially more complex.
  • The paper states that the method assumes the explicit CoT trace remains causally relevant to the final answer, but prior work shows CoT explanations can be post-hoc rationalizations, so patching intermediate steps may have limited influence if the model has already committed to an answer.
  • Evident from the text: the empirical evaluation is confined to CLUTRR and the listed models, with ILP rules learned from 200 held-out 2-hop instances and longer chains used only for evaluation, so generality beyond this benchmark and rule-learning setup is not demonstrated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.23019

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

19 of 30 cited sources checked · 12 supported · 2 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 2

    Retry-based refinement methods often discard the entire context and regenerate from scratch, losing correct intermediate work.

    “Retry-based refinement methods Madaan et al. [2023], Gou et al. [2024] often discard the entire context and regenerate from scratch, losing correct intermediate work.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Madaan et al. [2023] source

      The SELF-REFINE algorithm section says the method retains previous feedback and outputs in the prompt to inform each refinement. That is contrary to the claim that this method discards the entire context and regenerates from scratch. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Gou et al. [2024]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation
    § 1 Introduction · p. 1

    Existing neurosymbolic approaches typically operate at the level of the full problem or the full reasoning trace.

    “but they typically operate at the level of the full problem or the full reasoning trace.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Pan et al. [2023] source

      The Problem Goal Answer passage and §3.2 describe Logic-LM as a neuro-symbolic method that formulates the problem and goal for a symbolic solver, supporting the full-problem part of the claim. Section 2 discusses prior neuro-symbolic work but does not say that such approaches typically operate at the level of the full problem or the full reasoning trace. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Olausson et al. [2023] source

      The Introduction and §2 describe LINC as processing the premises and conclusion together through semantic parsing and a theorem prover, which supports the full-problem characterization for LINC. The related-work discussion describes other neurosymbolic approaches, including one that decides how an inference rule applies at each stage, but does not establish that such approaches typically operate on the full problem or full reasoning trace. Page numbers are unavailable in our copy. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
    • cites Xu et al. [2024]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation
    § 1 Introduction · p. 2

    The grounding bottleneck has historically limited symbolic reasoning pipelines to relatively narrow and well-formalized domains.

    “More broadly, this bottleneck has historically limited symbolic reasoning pipelines to relatively narrow and well-formalized domains Olausson et al. [2023], Lyu et al. [2023].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Olausson et al. [2023] source

      The Limitations section says LINC uses short statements to make formalization tractable and has more semantic-parsing failures on naturalistic text. This supports the grounding bottleneck, but the paper does not establish the broader historical claim that it limited symbolic reasoning pipelines generally to narrow, well-formalized domains. (LLM, conf=0.81)

      Experimental — accuracy not yet measured
    • cites Lyu et al. [2023] source

      The paper’s Related Work discussion (page unavailable) says two concurrent symbolic-language approaches tackled only math and simple symbolic reasoning tasks; Appendix B.3 repeats that narrower scope. It does not establish that a grounding bottleneck historically limited symbolic reasoning pipelines generally to well-formalized domains. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • Rests on an assumption: Symbolic verification operates over structured predicates, whereas CoT traces are written in natural language.
  • Premisecitation
    § Abstract · p. 1

    The intermediate steps generated by CoT prompting are not guaranteed to be logically sound.

    “yet the generated intermediate steps are not guaranteed to be logically sound”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Turpin et al. [2023] source
    • cites Lanham et al. [2023] source

      The Introduction and §2.3 discuss whether generated chain-of-thought steps faithfully reflect the model’s reasoning, not whether those steps are logically sound. Section 2.4 introduces mistakes experimentally; it does not establish that unmodified CoT steps contain logical errors. I searched for discussion of incorrect or logically invalid original reasoning steps but found no passage addressing that claim. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1 Introduction · p. 1

    CoT explanations can be plausible while remaining unfaithful to the model's actual decision process.

    “Prior work shows that CoT explanations can be plausible while remaining unfaithful Turpin et al. [2023], Lanham et al. [2023]”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Turpin et al. [2023] source
    • cites Lanham et al. [2023] source
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 10
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives, or rely on hand-crafted formal grammars specified per domain.

    “Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives de Souza et al. [2025], or rely on hand-crafted formal grammars specified per domain Pan et al. [2023], Chen et al. [2022, 2023].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites de Souza et al. [2025] source

      The citation attaches to the retraining branch of the sentence, but the cited paper describes a different approach. Figure 1 (page unavailable in our copy) says an LLM generates theories and feedback from a logic interpreter is used to refine the prompts iteratively—not to retrain the model with logic-informed objectives. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Pan et al. [2023] source
    • cites Chen et al. [2022, 2023]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    Logic-LM partially addresses the grounding bottleneck by using LLMs as translators from natural language to formal logic, but the translation stage itself introduces an additional failure mode.

    “Logic-LM Pan et al. [2023] partially addresses this issue by using LLMs as translators from natural language to formal logic, but the translation stage itself introduces an additional failure mode.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Pan et al. [2023] source
    • Rests on an assumption: Symbolic verification operates over structured predicates, whereas CoT traces are written in natural language.

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    Recent neurosymbolic approaches either translate natural-language problems into formal representations and delegate reasoning to external solvers, or inject symbolic structure into the reasoning process to improve faithfulness and controllability.

    “Recent neurosymbolic approaches either translate natural-language problems into formal representations and delegate reasoning to external solvers Pan et al. [2023], Olausson et al. [2023], or inject symbolic structure into the reasoning process to improve faithfulness and controllability Xu et al. [2024], Lyu et al. [2023].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Pan et al. [2023] source
    • cites Olausson et al. [2023] source
    • cites Xu et al. [2024]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 4 Evaluation · p. 6

    Solving a CLUTRR instance requires composing a sequence of intermediate kinship relations.

    “Solving an instance requires composing a sequence of intermediate kinship relations, for example composing sister and father to derive aunt .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sinha et al. [2019] source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 4 Evaluation · p. 6

    CLUTRR is a benchmark for multi-hop kinship reasoning.

    “We evaluate on CLUTRR Sinha et al. [2019], a benchmark for multi-hop kinship reasoning.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sinha et al. [2019] source

  • May combine two claims.

    Findingown data
    § Error Type Distribution (% among all errors) · p. 8

    Smaller local models exhibit a more balanced mix of wrong rule, missing inv, and direction rev, with wrong rule being the dominant category (43-49%).

    “Smaller local models exhibit a more balanced mix of wrong rule , missing inv , and direction rev , with wrong rule being the dominant category (43-49%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 4 Evaluation · p. 8

    Claude Sonnet 4.6 and GPT-5.4 start from much stronger endogenous CoT baselines than the smaller local models, but both still exhibit substantial degradation as reasoning depth increases.

    “Claude Sonnet 4.6 and GPT-5.4 start from much stronger endogenous CoT baselines than the smaller local models, but both still exhibit substantial degradation as reasoning depth increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 Evaluation · p. 7

    As chain length increases, Reason Popper-ly overtakes the exogenous baseline and the gap widens.

    “As chain length increases, however, Reason Popper -ly overtakes the exogenous baseline and the gap widens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 1 Introduction · p. 2

    Reason Popper-ly preserves the model's own reasoning whenever it is already sound, while introducing symbolic correction only where it is needed.

    “This design preserves the model's own reasoning whenever it is already sound, while introducing symbolic correction only where it is needed.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The extraction and normalization procedure accurately parses inferred steps into composition triples.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Abstract · p. 1

    Chain-of-thought (CoT) prompting enables large language models (LLMs) to tackle multistep reasoning tasks.

    “Chain-of-thought (CoT) prompting enables large language models (LLMs) to tackle multistep reasoning tasks”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al. [2022] source

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    SymbCoT integrates symbolic expressions into chain-of-thought with an LLM-based verification stage, but does not localize failures to specific reasoning steps or systematically classify their types.

    “SymbCoT Xu et al. [2024] integrates symbolic expressions into chain-of-thought with an LLMbased verification stage, but does not localize failures to specific reasoning steps or systematically classify their types.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Xu et al. [2024]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Current neurosymbolic verifiers often reduce verification to a binary accept-or-reject decision over the full trace or full solution.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 1 Introduction · p. 2

    Replacing a single step is more targeted, but it also changes the context for all subsequent steps.

    “Replacing a single step is more targeted, but it also changes the context for all subsequent steps.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4 Evaluation · p. 7

    The pattern suggests a shift in the dominant failure mode: for short chains, local reasoning mistakes dominate; for longer chains, extraction and graph-construction errors accumulate, which increasingly harms fully exogenous symbolic pipelines.

    “This pattern suggests a shift in the dominant failure mode. For short chains, local reasoning mistakes dominate. For longer chains, extraction and graph-construction errors accumulate, which increasingly harms fully exogenous symbolic pipelines.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given
    § 6 Discussion, Limitation, and Future Work · p. 9

    Our evaluation focuses on kinship reasoning, where relation composition is well defined and grounding from natural language to structured predicates is relatively tractable through templated parsing.

    “Our evaluation focuses on kinship reasoning, where relation composition is well defined and grounding from natural language to structured predicates is relatively tractable through templated parsing.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationreasoning
    § Abstract · p. 1

    Reason Popper-ly performs better on harder instances by preserving the model's successful grounding while correcting only verifiable reasoning failures.

    “by preserving the model's successful grounding while correcting only verifiable reasoning failures”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The extraction and normalization procedure accurately parses inferred steps into composition triples.
    • Rests on an assumption: The LLM-generated reasoning traces can be reliably partitioned into stated and inferred steps with correct supporting indices.
    • Rests on an assumption: The gender-aware inverse map correctly normalizes relations into chain-consistent form.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 4 Evaluation · p. 7

    Endogenous reasoning (single-shot CoT) isolates the model's native compositional reasoning ability.

    “This is the standard CoT setup Wei et al. [2022] and isolates the model's native compositional reasoning ability.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al. [2022] source

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 3.2 Online: Verify, Diagnose, Patch · p. 6

    The procedure acts as a local repair operator over the reasoning trace: it preserves the model's original reasoning wherever possible, intervenes only at rule-violating steps, and restores downstream coherence by regenerating only the affected suffix.

    “Conceptually, this procedure acts as a local repair operator over the reasoning trace: it preserves the model's original reasoning wherever possible, intervenes only at rule-violating steps, and restores downstream coherence by regenerating only the affected suffix.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The extraction and normalization procedure accurately parses inferred steps into composition triples.
    • Rests on an assumption: The LLM-generated reasoning traces can be reliably partitioned into stated and inferred steps with correct supporting indices.

  • May combine two claims.

    Interpretationreasoning
    § 4 Evaluation · p. 8

    Compositional reasoning failures are not restricted to weaker models; they remain present, detectable, and actionable even in strong frontier systems.

    “compositional reasoning failures are not restricted to weaker models. Instead, they remain present, detectable, and actionable even in strong frontier systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Error Type Distribution (% among all errors) · p. 8

    Step-level symbolic diagnosis not only enables targeted repair, but also exposes qualitatively different reasoning weaknesses across model scales.

    “These findings support the value of step-level symbolic diagnosis: it not only enables targeted repair, but also exposes qualitatively different reasoning weaknesses across model scales.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 6 Discussion, Limitation, and Future Work · p. 9

    In domains with more ambiguous predicate structures, such as legal reasoning or scientific argumentation, both the grounding process and the induced rule space would be substantially more complex.

    “In domains with more ambiguous predicate structures, such as legal reasoning or scientific argumentation, both the grounding process and the induced rule space would be substantially more complex.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 6 Discussion, Limitation, and Future Work · p. 9

    CLUTRR is a useful testbed for studying whether symbolic verification can improve step-level reasoning.

    “This makes CLUTRR a useful testbed for studying whether symbolic verification can improve step-level reasoning”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § 1 Introduction · p. 2

    The taxonomy is enabled by the structured nature of ILP-based verification and supports localized attribution of reasoning failures.

    “The taxonomy is enabled by the structured nature of ILP-based verification and supports localized attribution of reasoning failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Error Type Distribution (% among all errors) · p. 8

    Scaling reduces core composition mistakes but does not eliminate structured reasoning failures; instead, the dominant failure mode shifts toward relation-direction normalization errors.

    “This suggests that scaling reduces core composition mistakes but does not eliminate structured reasoning failures; instead, the dominant failure mode shifts toward relation-direction normalization errors .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 6 Discussion, Limitation, and Future Work · p. 10

    The assumption that explicit CoT trace is causally relevant is at least partially valid in the evaluated setting, since symbolic patching yields consistent gains in terminal accuracy.

    “Our empirical results suggest that the assumption is at least partially valid in the evaluated setting, since symbolic patching yields consistent gains in terminal accuracy.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The explicit CoT trace remains causally relevant to the model's final answer, so that editing an intermediate step can redirect the downstream conclusion.
24 of 24

  • The paper relies on this without support.

    The extraction and normalization procedure accurately parses inferred steps into composition triples.

    “we parse each inferred step into a composition triple ( r 1 , r 2 , r 3 ) using the same extraction and normalization procedure as in the offline phase.”
    “This extraction requires direction normalization : relations expressed in the story may not align with the direction needed by the reasoning chain.”

  • The paper relies on this without support.

    The LLM-generated reasoning traces can be reliably partitioned into stated and inferred steps with correct supporting indices.

    “each step explicitly indicates whether it is stated or inferred , together with the indices of the supporting prior steps.”

  • The paper relies on this without support.

    The gender-aware inverse map correctly normalizes relations into chain-consistent form.

    “We therefore map each relation into chain-consistent form using a gender-aware inverse map when necessary.”

  • The paper relies on this without support.

    Symbolic verification requires explicit inference rules.

    “Symbolic verification requires explicit inference rules.”

  • The paper relies on this without support.

    ILP has rarely been used as a runtime verifier for LLM-generated reasoning traces.

    “but it has rarely been used as a runtime verifier for LLM-generated reasoning traces.”

  • The paper relies on this without support.

    Symbolic verification operates over structured predicates, whereas CoT traces are written in natural language.

    “Symbolic verification operates over structured predicates, whereas CoT traces are written in natural language.”

  • The paper relies on this without support.

    Current neurosymbolic verifiers often reduce verification to a binary accept-or-reject decision over the full trace or full solution.

    “Current neurosymbolic verifiers often reduce verification to a binary accept-or-reject decision over the full trace or full solution.”

  • The authors state this as an assumption.

    The explicit CoT trace remains causally relevant to the model's final answer, so that editing an intermediate step can redirect the downstream conclusion.

    “Our method assumes that the explicit CoT trace remains causally relevant to the model's final answer, so that editing an intermediate step can redirect the downstream conclusion.”
8 of 8

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Ryan Hsiang, William Adkisson, Robert Joseph George, and Anima Anandkumar. Leandojov2: A comprehensive library for AI-assisted theorem proving in lean. In The 5th Workshop on Mathematical Reasoning and AI at NeurIPS 2025 , 2025. URL https://openreview.net/ forum?id=tnx1VvrcAn .
  2. Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrievalaugmented language models. Advances in Neural Information Processing Systems , 36:2157321612, 2023.

Authors
Chen, Z., & Ma, M.
Affiliations
Vanderbilt University (US)
ORCID
Ma, Meiyi
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.23019
arXiv
2607.23019
OpenAlex
W7171530237
Semantic Scholar
c11bd1cc4e6ccb4fe315791f80b88d71819498aa
Cited by
0
References
30
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
chain-of-thought prompting, inductive logic programming, neuro-symbolic reasoning, large language models, multi-step reasoning, error taxonomy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review