CoChat
Sign Up
PreprintCited by 154Logic, Reasoning, and KnowledgeTopic ModelingNatural Language Processing Techniques

Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Open access
Pan, Liangming, Albalak, Alon, Wang, Xinyi, Wang, William Yang·2023·arXiv
DOI 10.48550/arxiv.2305.12295arXiv 2305.12295
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.

AI snapshot

Written from the paper’s full text.

TL;DR

LOGIC-LM combines LLMs with deterministic symbolic solvers by having the LLM translate natural-language logical problems into symbolic formulations, then offloading inference to solvers and interpreting results back into answers. Across five logical reasoning datasets, it improves over standard prompting by 39.2% on average and over chain-of-thought by 18.4% on average.

Key findings

  • LOGIC-LM outperforms standard LLMs and CoT across all five datasets; with GPT-3.5, average improvement is 39.2% over standard prompting and 18.4% over CoT.
  • With GPT-4, LOGIC-LM further improves over standard prompting by an average of 24.98% and over CoT by 10.44%.
  • Performance gains grow with reasoning depth on ProofWriter: LOGIC-LM outperforms CoT by 7.1%, 5.0%, 12.7%, 20.0%, and 39.4% on depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.
  • Self-refinement increases executable symbolic formulations by 5.01% on average, but accuracy tends to stagnate in later rounds even as executability continues to rise.

Methods

  • Design: three-stage framework — Problem Formulator (LLM translates NL into symbolic language), Symbolic Reasoner (deterministic solver performs inference), and Result Interpreter (maps solver output to natural-language answer). A self-refinement module revises symbolic formulations using solver error messages.
  • Data: five logical reasoning datasets — PrOntoQA, ProofWriter, FOLIO, LogicalDeduction, and AR-LSAT — covering deductive reasoning, first-order logic, constraint satisfaction, and analytical reasoning. Test sizes reported in the paper’s table are PrOntoQA 500, ProofWriter 600, FOLIO 204, LogicalDeduction 300, and AR-LSAT 230; the text separately says AR-LSAT has 231 multiple-choice questions.
  • Analysis: compares LOGIC-LM against standard in-context prompting and chain-of-thought prompting using ChatGPT (gpt-3.5-turbo), GPT-3.5 (text-davinci-003), and GPT-4 (gpt-4), with temperature 0 and accuracy as the metric. Symbolic solvers include Pyke, Prover9, python-constraint, and Z3.

Limitations

  • Stated: LOGIC-LM’s applicability is bounded by the expressiveness of the symbolic solver, since not all problems can be easily encoded in formalisms such as first-order logic.
  • Stated: it depends on in-context learning plus self-refinement to convert NL into symbolic representations, which may struggle with intricate grammar structures such as probabilistic soft logic due to limited demonstrations and context size.
  • Evident from the paper’s analysis: self-refinement mainly fixes invalid symbolic forms using solver error messages, but a valid symbolic representation is not necessarily a correct problem formulation; the paper notes accuracy can stagnate even as executable rate increases. Error cases also show GPT-4 struggles with predicate consistency, interpreting expressions like “below” vs. “above,” and FOL grammar.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.12295

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 13 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    LLMs still struggle with complex logical reasoning problems.

    “Despite the advances of LLMs, they still struggle with complex logical reasoning problems (Liu et al., 2023b).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al., 2023b source
  • Premisecitation

    LLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain.

    “Recent studies (Golovneva et al., 2023;Ribeiro et al., 2023b;Lyu et al., 2023) found that LLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Golovneva et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ribeiro et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lyu et al., 2023 source
  • Premisecitation

    LLMs have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs.

    “However, they have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs, such as mathematical equations (He-Yueya et al., 2023) or Python codes (Gao et al., 2023).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites He-Yueya et al., 2023 source
    • cites Gao et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    LOGIC-LM becomes increasingly effective as the required reasoning depth increases.

    “We also find that LOGIC-LM becomes increasingly effective as the required reasoning depth increases ( § 4.3).”
    • Also stated in: § 4.3 Robustness of Reasoning · hedged
      “The results, shown in Figure 3, indicate that LOGIC-LM becomes increasingly effective as the required reasoning depth increases.”
    Quote in the paperNot a substantial claim
  • Findingown data

    GPT-4 has difficulty fully grasping the rules of FOL grammar, evidenced by the invalid generated formula: Rating(subway, y) ∧ y > 9.

    “Example 4 exemplifies GPT-4's challenge with fully grasping the rules of FOL grammar, evidenced by the invalid generated formula: Rating(subway, y) ∧ y > 9.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 13

  • May combine two claims.

    Premisecitation

    Prior neuro-symbolic methods suffer from limitations such as hand-crafted or specialized module designs that are not easily generalizable, or brittleness due to the difficulty of optimization.

    “Although prior works (Mao et al., 2019;Gupta et al., 2020;Manhaeve et al., 2021;Cai et al., 2021;Tian et al., 2022;Pryor et al., 2023) also propose neuro-symbolic methods to combine neural networks with symbolic reasoning, these methods suffer from limitations such as hand-crafted or specialized module designs that are not easily generalizable, or brittleness due to the difficulty of optimization.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Mao et al., 2019 source

      The Training Paradigm (p. 6) says curriculum learning is essential to joint optimization, and Appendix E (p. 19) reports that training can fail to converge without it. But the paper describes NS-CL as robust and demonstrates several forms of generalization (p. 2); it does not support the unqualified claim that its module design is not easily generalizable or that the trained method is brittle. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Gupta et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Manhaeve et al., 2021 source

      The Related Work section describes limitations of some prior neuro-symbolic methods, including restricted expressivity, and §7.3 identifies an approach specialized for NLP tasks. It does not establish that prior methods generally rely on hand-crafted modules that are hard to generalize or are brittle because optimization is difficult. Page numbers were not available in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The main enhancement of GPT-4 lies in its ability to carry out complex reasoning.

    “This aligns with the assertion that the main enhancement of GPT-4 lies in its ability to carry out complex reasoning (OpenAI, 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites OpenAI, 2023 source

      The Observed Safety Challenges section (p. 44) says GPT-4 improves on earlier models in reasoning, but also names knowledge retention and coding. The report does not identify complex reasoning as GPT-4’s main enhancement. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Chain-of-thought prompting yields comparatively less substantial or even negative improvements on FOLIO, LogicalDeduction, and AR-LSAT, with improvements of 11.75%, 9.41%, and -3.2%, respectively.

    “While integrating CoT generally enhances LLM performance, we find its benefits comparatively less substantial or even negative on FOLIO, LogicalDeduction, and AR-LSAT, with a modest improvement of 11.75%, 9.41%, and -3.2%, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4's Exe_Rate is notably less promising on FOLIO and AR-LSAT, with scores of 79.9% and 32.6%, respectively.

    “When it comes to datasets comprising real-world, expertly crafted problems, such as FOLIO and AR-LSAT, GPT-4's performance is notably less promising, with Exe_Rate scores of 79.9% and 32.6% respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LOGIC-LM achieves an average improvement of 39.2% over standard prompting and 18.4% over chain-of-thought prompting.

    “On average, LOGIC-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-ofthought prompting.”
    • Also stated in: § Problem Goal Answer · boosted
      “Our evaluations show that the strategy of integrating LLMs with symbolic solvers performs significantly better than purely relying on LLMs for logical reasoning, with an average improvement of 39.2% over the standard prompting and 18.4% over the chain-of-thought prompting ( § 4.1).”
    • Also stated in: § 4.1 Main Results · plain
      “With GPT-3.5, our method outperforms standard LLM on all datasets, with an average improvement of 39.2%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LOGIC-LM significantly outperforms standard LLMs and chain-of-thought across all datasets.

    “Logic-LM significantly outperforms standard LLMs and CoT across all datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    As rounds of self-refinement increase, the percentage of executable formulations consistently increases, leading to an enhancement in final performance.

    “We find that as the rounds of self-refinement increase, the percentage of executable formulations consistently increases, leading to an enhancement in the final performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Findingnone given

    There remain certain difficulties in accurately understanding the semantics of the problem.

    “Nonetheless, there remain certain difficulties in accurately understanding the semantics of the problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    LOGIC-LM outperforms CoT by 7.1%, 5.0%, 12.7%, 20.0%, and 39.4% on depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.

    “LOGIC-LM outperforms CoT by 7.1%, 5.0%, 12.7%, 20.0%, and 39.4% on depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LOGIC-LM improves GPT-4 by an average of 24.98% for standard prompting and 10.44% for chain-of-thought prompting.

    “By delegating the reasoning to symbolic solvers, our method can further improve GPT-4 by an average of 24.98% and 10.44% for standard prompting and CoT prompting, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Chain-of-thought prompting yields improvements of 51.59% on ProntoQA and 33.82% on ProofWriter.

    “On the contrary, the benefits of CoT on ProntoQA and ProofWriter are 51.59% and 33.82%, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Accuracy tends to stagnate in subsequent rounds of self-refinement, even though Exe_Rate continues to increase.

    “However, the accuracy tends to stagnate in subsequent rounds, even though the Exe_Rate continues to increase.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Compared to CoT, integrating symbolic solvers is better suited to nonlinear reasoning styles, yielding more marked improvements on FOLIO (+21.85%), LogicalDeduction (+45.67%), and AR-LSAT (+24.14%).

    “Compared to CoT, the integration of symbolic solvers is better suited to these reasoning styles, hence yielding a more marked improvement on FO-LIO (+21.85%),LogicalDeduction (+45.67%), and AR-LSAT (+24.14%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    In LOGIC-LM, multi-step logical reasoning is delegated to external symbolic solvers, transitioning the challenge of LLM from problem-solving to problem representation.

    “In LOGIC-LM, multi-step logical reasoning is delegated to external symbolic solvers, thereby transitioning the challenge of LLM from problemsolving to problem representation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    For reproducible results, the temperature was set to 0 and the response with the highest probability was selected from LLMs.

    “For reproducible results, we set the temperature to 0 and select the response with the highest probability from LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The robustness analysis randomly selected 300 examples from each ProofWriter subset, ensuring a balanced label distribution.

    “We randomly selected 300 examples from each subset of ProofWriter, ensuring a balanced label distribution.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    LOGIC-LM's applicability is inherently bounded by the expressiveness of the symbolic solver; not all problems can be easily encoded in first-order logic.

    “As a consequence, the model's applicability is inherently bounded by the expressiveness of the symbolic solver, for example, not all problems can be easily encoded in first-order logic.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    For un-executable cases where the symbolic solver does not return an answer due to grammar errors, the method falls back on using chain-of-thought to predict the answer.

    “For these un-executable cases, we fall back on using chain-of-thought to predict the answer.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    A plausible explanation for CoT's performance pattern is that CoT emulates human forward-chain reasoning: beginning with known facts and sequentially deriving new conclusions until the goal is met.

    “that CoT emulates human forward-chain reasoning: beginning with known facts and sequentially deriving new conclusions until the goal is met.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    By incorporating LLMs with symbolic solvers, one can exploit the robust natural language understanding capabilities of LLMs to precisely represent the problem using symbolic representations, while also taking advantage of the logical faithfulness and transparency offered by symbolic solvers.

    “By incorporating LLMs with symbolic solvers, we can exploit the robust natural language understanding capabilities of LLMs to precisely represent the problem using symbolic representations, while also taking advantage of the logical faithfulness and transparency offered by symbolic solvers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    By defining additional problem-specific formulations, the LOGIC-LM framework retains the flexibility to accommodate a wider range of reasoning tasks.

    “By defining additional problem-specific formulations, our framework retains the flexibility to accommodate a wider range of reasoning tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Model performance was evaluated based on the accuracy of selecting the correct answer.

    “Since all examples are formed as multiple-choice questions, we evaluate model performance based on the accuracy of selecting the correct answer.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AR-LSAT is particularly challenging, with state-of-the-art models only achieving performance slightly better than random guessing.

    “AR-LSAT is particularly challenging, with state-of-the-art models only achieving performance slightly better than random guessing (Liang et al., 2022;Ribeiro et al., 2023a).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Liang et al., 2022 source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites Ribeiro et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    This inconsistency is a recurring issue in GPT-4's symbolic form generation, illustrating that the model sometimes struggles to maintain an overarching understanding of the problem when forming logical symbols.

    “This inconsistency is a recurring issue in GPT-4's symbolic form generation, illustrating that the model sometimes struggles to maintain an overarching understanding of the problem when forming logical symbols.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To ensure fair comparisons, the same in-context examples were used for all models.

    “To ensure fair comparisons, we use the same in-context examples for all models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The flexible design of LOGIC-LM facilitates integration of a more diverse set of symbolic solvers.

    “The flexible design of LOGIC-LM facilitates this integration.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The difficulty with intricate grammar structures arises from the difficulty in conveying complex grammatical rules to the language model through a limited number of demonstrations within a constrained context size.

    “This arises from the difficulty in conveying complex grammatical rules to the language model through a limited number of demonstrations within a constrained context size.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The wide range of reasoning tasks that we can instantiate our LOGIC-LM framework on shows its general applicability.

    “The wide range of reasoning tasks that we can instantiate our LOGIC-LM framework on shows its general applicability.”
    Quote in the paperNot a substantial claim
28 of 28

  • The paper relies on this without support.

    The main obstacle in using symbolic solvers for logical reasoning is accurately translating natural language problems into symbolic representations.

    “The main obstacle is how to accurately translate a problem into symbolic representations, considering the inherent ambiguity and flexibility of natural language.”

  • The paper relies on this without support.

    Language models have inherent limitations such as the inability to access up-to-date information, take actions, or perform precise mathematical reasoning.

    “Language models have inherent limitations such as the inability to access up-to-date information, take actions, or perform precise mathematical reasoning.”

  • The paper relies on this without support.

    Many analytical reasoning problems can be formulated as SAT problems.

    “Many analytical reasoning problems can be formulated as SAT problems.”

  • The paper relies on this without support.

    A lot of real-world analytical reasoning problems can be represented as problems of solving a system of equations.

    “A lot of real-world analytical reasoning problems can be represented as problems of solving a system of equations.”

  • The paper relies on this without support.

    For complex problems, generating the correct logical form may become challenging for LLMs.

    “For complex problems, generating the correct logical form may become challenging for LLMs.”

  • The paper relies on this without support.

    While logic programming efficiently represents common deductive reasoning problems, it may fail to represent more complex first-order logic problems.

    “While the logic programming language efficiently represents common deductive reasoning problems, it may fail to represent more complex first-order logic (FOL) problems.”

  • The authors state this as an assumption.

    The complexity of formally representing a problem statement in logical form should remain relatively constant, regardless of whether the questions require simple or complex reasoning.

    “Ideally, the complexity of formally representing a problem statement in logical form should remain relatively constant, regardless of whether the questions require simple or complex reasoning.”

  • The paper relies on this without support.

    The problems in ProntoQA, ProofWriter, and LogicalDeduction are mostly synthetically generated, limiting language variability.

    “given that their problems are mostly synthetically generated, limiting language variability.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing neurosymbolic approaches typically operate at the level of the full problem or the full reasoning trace.

    “but they typically operate at the level of the full problem or the full reasoning trace.”

    From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]

    The Problem Goal Answer passage and §3.2 describe Logic-LM as a neuro-symbolic method that formulates the problem and goal for a symbolic solver, supporting the full-problem part of the claim. Section 2 discusses prior neuro-symbolic work but does not say that such approaches typically operate at the level of the full problem or the full reasoning trace. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • LOGIC-LM creates a deterministic symbolic solver that expresses reasoning in a specific symbolic format to obtain actual results.

    “LOGIC-LM [15] creates a deterministic symbolic solver that expresses reasoning in a specific symbolic format to obtain actual results.”

    From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [15]

    The Problem Goal Answer passage describes LOGIC-LM using an LLM to convert a problem into a symbolic formulation, then a deterministic solver to perform inference. Section 3.2 says the framework calls a task-dependent external solver; it does not say LOGIC-LM creates that solver or that the solver itself expresses the problem in symbolic form. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The improvement aligns with recent research integrating LLMs with symbolic reasoners for various reasoning tasks.

    “This aligns with recent research integrating LLMs with symbolic reasoners for various reasoning tasks (Olausson et al., 2023;Pan et al., 2023).”

    From TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025) · cites this paper as Pan et al., 2023

  • Logic-LM partially addresses the grounding bottleneck by using LLMs as translators from natural language to formal logic, but the translation stage itself introduces an additional failure mode.

    “Logic-LM Pan et al. [2023] partially addresses this issue by using LLMs as translators from natural language to formal logic, but the translation stage itself introduces an additional failure mode.”

    From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]

  • Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives, or rely on hand-crafted formal grammars specified per domain.

    “Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives de Souza et al. [2025], or rely on hand-crafted formal grammars specified per domain Pan et al. [2023], Chen et al. [2022, 2023].”

    From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • LOGIC-LM does not outperform CoT on every dataset

    Major

    The across-all-datasets claim conflicts with Table 2: on PrOntoQA, ChatGPT with LOGIC-LM scores 61.00 versus 67.80 with CoT, and GPT-4 with LOGIC-LM scores 83.20 versus 98.79 with CoT.

  • AR-LSAT test-set size differs between text and table

    Minor

    The dataset description states that the test set has 231 questions, while the dataset-statistics table lists 230.

  • Reasoning-depth discussion refers to a nonexistent depth-4 group

    Minor

    The reported ProofWriter depth groups are 0, 1, 2, 3, and 5; the discussion instead calls the fourth group depth 4.

  • Oldest-vehicle answer has the wrong option letter

    Minor

    The convertible is option B in the stated choices, but the answer labels it option A.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Stephen Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2017. Hinge-loss markov random fields and probabilistic soft logic. Journal of Machine Learning Research (JMLR), 18(1):1-67.
  2. Kostas S. Metaxiotis, Dimitris Askounis, and John E. Psarras. 2002. Expert systems in production planning and scheduling: A state-of-the-art survey. Journal of Intelligent Manufacturing, 13(4):253-260. · DOI
  3. Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Henry Zhu, Rui Dong, Deguang Kong, Juliette Burger, An- jelica Ramos, William Yang Wang, Zhiheng Huang, George Karypis, Bing Xiang, and Dan Roth. 2023b. STREET: A multi-task structured reasoning and ex- planation benchmark. In Proceedings of the 11th International Conference on Learning Representa- tions (ICLR). · DOI

Cited by

Sorted by citations

Related

Authors
Pan, L., Albalak, A., Wang, X., & Wang, W. Y.
Affiliations
University of California, Santa Barbara (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
3806-3824
DOI
10.48550/arxiv.2305.12295
arXiv
2305.12295
OpenAlex
W4389519818
Semantic Scholar
9e9e4df2996bac794c4f04cb887df3e553bae4fd
Cited by
144
References
56
Citations by year
2026: 59 · 2025: 54 · 2024: 25 · 2023: 6
FWCI
25.82
Type
conference paper
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2048122)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, symbolic solvers, logical reasoning, self-refinement, chain-of-thought prompting, proof writing, first-order logic, logical deduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review