The DOI matches its DataCite record; 62 of the 62 references it cites were found in a registry or an index; 7 of the 13 sources its claims cite were checked for support.
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Open accessAbstract
Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
LOGIC-LM combines LLMs with deterministic symbolic solvers by having the LLM translate natural-language logical problems into symbolic formulations, then offloading inference to solvers and interpreting results back into answers. Across five logical reasoning datasets, it improves over standard prompting by 39.2% on average and over chain-of-thought by 18.4% on average.
Key findings
- LOGIC-LM outperforms standard LLMs and CoT across all five datasets; with GPT-3.5, average improvement is 39.2% over standard prompting and 18.4% over CoT.
- With GPT-4, LOGIC-LM further improves over standard prompting by an average of 24.98% and over CoT by 10.44%.
- Performance gains grow with reasoning depth on ProofWriter: LOGIC-LM outperforms CoT by 7.1%, 5.0%, 12.7%, 20.0%, and 39.4% on depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.
- Self-refinement increases executable symbolic formulations by 5.01% on average, but accuracy tends to stagnate in later rounds even as executability continues to rise.
Methods
- Design: three-stage framework — Problem Formulator (LLM translates NL into symbolic language), Symbolic Reasoner (deterministic solver performs inference), and Result Interpreter (maps solver output to natural-language answer). A self-refinement module revises symbolic formulations using solver error messages.
- Data: five logical reasoning datasets — PrOntoQA, ProofWriter, FOLIO, LogicalDeduction, and AR-LSAT — covering deductive reasoning, first-order logic, constraint satisfaction, and analytical reasoning. Test sizes reported in the paper’s table are PrOntoQA 500, ProofWriter 600, FOLIO 204, LogicalDeduction 300, and AR-LSAT 230; the text separately says AR-LSAT has 231 multiple-choice questions.
- Analysis: compares LOGIC-LM against standard in-context prompting and chain-of-thought prompting using ChatGPT (gpt-3.5-turbo), GPT-3.5 (text-davinci-003), and GPT-4 (gpt-4), with temperature 0 and accuracy as the metric. Symbolic solvers include Pyke, Prover9, python-constraint, and Z3.
Limitations
- Stated: LOGIC-LM’s applicability is bounded by the expressiveness of the symbolic solver, since not all problems can be easily encoded in formalisms such as first-order logic.
- Stated: it depends on in-context learning plus self-refinement to convert NL into symbolic representations, which may struggle with intricate grammar structures such as probabilistic soft logic due to limited demonstrations and context size.
- Evident from the paper’s analysis: self-refinement mainly fixes invalid symbolic forms using solver error messages, but a valid symbolic representation is not necessarily a correct problem formulation; the paper notes accuracy can stagnate even as executable rate increases. Error cases also show GPT-4 struggles with predicate consistency, interpreting expressions like “below” vs. “above,” and FOL grammar.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
24 Own interpretation · 20 Own result · 5 Partly supported · 5 Own method · 3 No basis given · 1 Supported · 1 Sources not found · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“Despite the advances of LLMs, they still struggle with complex logical reasoning problems (Liu et al., 2023b).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Problem Goal Answer
What the sources say
- Supportedcites Liu et al., 2023b source
The Conclusion reports that ChatGPT and GPT-4 face challenges on new and out-of-distribution logical-reasoning data and do not perform especially well on natural-language inference requiring logical reasoning. This supports the citing sentence’s broad claim that advanced language models still struggle with difficult logical-reasoning problems. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent studies (Golovneva et al., 2023;Ribeiro et al., 2023b;Lyu et al., 2023) found that LLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Problem Goal Answer
What the sources say
- Not matchedcites Golovneva et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Ribeiro et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Lyu et al., 2023 source
The Input section states that in standard chain-of-thought prompting, the final answer does not necessarily follow from the generated reasoning chain. Its Figure 1 example shows an answer that does not follow from the chain, matching the cited claim. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, they have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs, such as mathematical equations (He-Yueya et al., 2023) or Python codes (Gao et al., 2023).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 3.1 Problem Formulator
What the sources say
- Supportedcites He-Yueya et al., 2023 source
Section 3.1 says the LLM formalizes math word problems as variables and equations. The Conclusion reports that this approach was effective, supporting the claim as applied to mathematical equations; the separate Python-code example is attributed to another source. (LLM, conf=0.93)
- Not matchedcites Gao et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also find that LOGIC-LM becomes increasingly effective as the required reasoning depth increases ( § 4.3).”
- Also stated in: § 4.3 Robustness of Reasoning · hedged
“The results, shown in Figure 3, indicate that LOGIC-LM becomes increasingly effective as the required reasoning depth increases.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Problem Goal Answer
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.3 Robustness of Reasoning · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Example 4 exemplifies GPT-4's challenge with fully grasping the rules of FOL grammar, evidenced by the invalid generated formula: Rating(subway, y) ∧ y > 9.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.5 Case Study
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 3, we find that self-refinement is effective in fixing the in-executable symbolic formulations, increasing the Exe_Rate by 5.01 on average.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.4 Impact of Self-Refinement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, by analyzing the impact of selfrefinement, we highlight the effectiveness of incrementally revising symbolic formalizations when interacting with the symbolic solver ( § 4.4).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Problem Goal Answer
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 outperforms GPT-3.5 by a large margin of 48.46% on average for the standard prompting.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance of Standard and CoT declines precipitously with the escalation of problem complexity.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, LLM demonstrates high proficiency in transcribing problems into symbolic formats, evidenced by its near 100% Exe_Rate on ProntoQA, ProofWriter, and Logi-calDeduction.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.2 Effectiveness of Problem Formulator
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 13
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although prior works (Mao et al., 2019;Gupta et al., 2020;Manhaeve et al., 2021;Cai et al., 2021;Tian et al., 2022;Pryor et al., 2023) also propose neuro-symbolic methods to combine neural networks with symbolic reasoning, these methods suffer from limitations such as hand-crafted or specialized module designs that are not easily generalizable, or brittleness due to the difficulty of optimization.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 2 Related Work
What the sources say
- Partly supportedcites Mao et al., 2019 sourceExperimental — accuracy not yet measured
The Training Paradigm (p. 6) says curriculum learning is essential to joint optimization, and Appendix E (p. 19) reports that training can fail to converge without it. But the paper describes NS-CL as robust and demonstrates several forms of generalization (p. 2); it does not support the unqualified claim that its module design is not easily generalizable or that the trained method is brittle. (LLM, conf=0.86)
- Not matchedcites Gupta et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Manhaeve et al., 2021 sourceExperimental — accuracy not yet measured
The Related Work section describes limitations of some prior neuro-symbolic methods, including restricted expressivity, and §7.3 identifies an approach specialized for NLP tasks. It does not establish that prior methods generally rely on hand-crafted modules that are hard to generalize or are brittle because optimization is difficult. Page numbers were not available in our copy. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This aligns with the assertion that the main enhancement of GPT-4 lies in its ability to carry out complex reasoning (OpenAI, 2023).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
What the sources say
- Partly supportedcites OpenAI, 2023 sourceExperimental — accuracy not yet measured
The Observed Safety Challenges section (p. 44) says GPT-4 improves on earlier models in reasoning, but also names knowledge retention and coding. The report does not identify complex reasoning as GPT-4’s main enhancement. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“AR-LSAT is particularly challenging, with state-of-the-art models only achieving performance slightly better than random guessing (Liang et al., 2022;Ribeiro et al., 2023a).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4 Experiments
What the sources say
- Partly supportedcites Liang et al., 2022 sourceExperimental — accuracy not yet measured
The Reasoning section (page unavailable) says most evaluated models scored around the 20% chance level on LSAT questions, supporting the claim that the task is challenging. It does not establish that the best-performing models scored slightly above chance; it also reports 0% accuracy for the code models evaluated on LSAT. (LLM, conf=0.84)
- Not matchedcites Ribeiro et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While integrating CoT generally enhances LLM performance, we find its benefits comparatively less substantial or even negative on FOLIO, LogicalDeduction, and AR-LSAT, with a modest improvement of 11.75%, 9.41%, and -3.2%, respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When it comes to datasets comprising real-world, expertly crafted problems, such as FOLIO and AR-LSAT, GPT-4's performance is notably less promising, with Exe_Rate scores of 79.9% and 32.6% respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.2 Effectiveness of Problem Formulator
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, LOGIC-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-ofthought prompting.”
- Also stated in: § Problem Goal Answer · boosted
“Our evaluations show that the strategy of integrating LLMs with symbolic solvers performs significantly better than purely relying on LLMs for logical reasoning, with an average improvement of 39.2% over the standard prompting and 18.4% over the chain-of-thought prompting ( § 4.1).”
- Also stated in: § 4.1 Main Results · plain
“With GPT-3.5, our method outperforms standard LLM on all datasets, with an average improvement of 39.2%.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Problem Goal Answer · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Logic-LM significantly outperforms standard LLMs and CoT across all datasets.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that as the rounds of self-refinement increase, the percentage of executable formulations consistently increases, leading to an enhancement in the final performance.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.4 Impact of Self-Refinement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nonetheless, there remain certain difficulties in accurately understanding the semantics of the problem.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.5 Case Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LOGIC-LM outperforms CoT by 7.1%, 5.0%, 12.7%, 20.0%, and 39.4% on depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By delegating the reasoning to symbolic solvers, our method can further improve GPT-4 by an average of 24.98% and 10.44% for standard prompting and CoT prompting, respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the contrary, the benefits of CoT on ProntoQA and ProofWriter are 51.59% and 33.82%, respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the accuracy tends to stagnate in subsequent rounds, even though the Exe_Rate continues to increase.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.4 Impact of Self-Refinement
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to CoT, the integration of symbolic solvers is better suited to these reasoning styles, hence yielding a more marked improvement on FO-LIO (+21.85%),LogicalDeduction (+45.67%), and AR-LSAT (+24.14%).”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In LOGIC-LM, multi-step logical reasoning is delegated to external symbolic solvers, thereby transitioning the challenge of LLM from problemsolving to problem representation.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For reproducible results, we set the temperature to 0 and select the response with the highest probability from LLMs.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We randomly selected 300 examples from each subset of ProofWriter, ensuring a balanced label distribution.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a consequence, the model's applicability is inherently bounded by the expressiveness of the symbolic solver, for example, not all problems can be easily encoded in first-order logic.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For these un-executable cases, we fall back on using chain-of-thought to predict the answer.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“that CoT emulates human forward-chain reasoning: beginning with known facts and sequentially deriving new conclusions until the goal is met.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By incorporating LLMs with symbolic solvers, we can exploit the robust natural language understanding capabilities of LLMs to precisely represent the problem using symbolic representations, while also taking advantage of the logical faithfulness and transparency offered by symbolic solvers.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Problem Goal Answer
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By defining additional problem-specific formulations, our framework retains the flexibility to accommodate a wider range of reasoning tasks.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 3.1 Problem Formulator
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Since all examples are formed as multiple-choice questions, we evaluate model performance based on the accuracy of selecting the correct answer.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This inconsistency is a recurring issue in GPT-4's symbolic form generation, illustrating that the model sometimes struggles to maintain an overarching understanding of the problem when forming logical symbols.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.5 Case Study
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure fair comparisons, we use the same in-context examples for all models.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The flexible design of LOGIC-LM facilitates this integration.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This arises from the difficulty in conveying complex grammatical rules to the language model through a limited number of demonstrations within a constrained context size.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The wide range of reasoning tasks that we can instantiate our LOGIC-LM framework on shows its general applicability.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
The main obstacle in using symbolic solvers for logical reasoning is accurately translating natural language problems into symbolic representations.
“The main obstacle is how to accurately translate a problem into symbolic representations, considering the inherent ambiguity and flexibility of natural language.”
The paper relies on this without support.
Language models have inherent limitations such as the inability to access up-to-date information, take actions, or perform precise mathematical reasoning.
“Language models have inherent limitations such as the inability to access up-to-date information, take actions, or perform precise mathematical reasoning.”
The paper relies on this without support.
Many analytical reasoning problems can be formulated as SAT problems.
“Many analytical reasoning problems can be formulated as SAT problems.”
The paper relies on this without support.
A lot of real-world analytical reasoning problems can be represented as problems of solving a system of equations.
“A lot of real-world analytical reasoning problems can be represented as problems of solving a system of equations.”
The paper relies on this without support.
For complex problems, generating the correct logical form may become challenging for LLMs.
“For complex problems, generating the correct logical form may become challenging for LLMs.”
The paper relies on this without support.
While logic programming efficiently represents common deductive reasoning problems, it may fail to represent more complex first-order logic problems.
“While the logic programming language efficiently represents common deductive reasoning problems, it may fail to represent more complex first-order logic (FOL) problems.”
The authors state this as an assumption.
The complexity of formally representing a problem statement in logical form should remain relatively constant, regardless of whether the questions require simple or complex reasoning.
“Ideally, the complexity of formally representing a problem statement in logical form should remain relatively constant, regardless of whether the questions require simple or complex reasoning.”
The paper relies on this without support.
The problems in ProntoQA, ProofWriter, and LogicalDeduction are mostly synthetically generated, limiting language variability.
“given that their problems are mostly synthetically generated, limiting language variability.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“but they typically operate at the level of the full problem or the full reasoning trace.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LOGIC-LM [15] creates a deterministic symbolic solver that expresses reasoning in a specific symbolic format to obtain actual results.”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This aligns with recent research integrating LLMs with symbolic reasoners for various reasoning tasks (Olausson et al., 2023;Pan et al., 2023).”
From TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025) · cites this paper as Pan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Logic-LM Pan et al. [2023] partially addresses this issue by using LLMs as translators from natural language to formal logic, but the translation stage itself introduces an additional failure mode.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives de Souza et al. [2025], or rely on hand-crafted formal grammars specified per domain Pan et al. [2023], Chen et al. [2022, 2023].”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Recent neurosymbolic approaches either translate natural-language problems into formal representations and delegate reasoning to external solvers Pan et al. [2023], Olausson et al. [2023], or inject symbolic structure into the reasoning process to improve faithfulness and controllability Xu et al. [2024], Lyu et al. [2023].”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“These approaches show that symbolic constraints can improve reasoning reliability”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Pan et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Neither [11] nor [13] pose a viable approach in this current work, however, due to the complexity of questions expected for a KBQA system, and our subsequent interest in leveraging the generative power of an LLM to answer inputted queries.”
From ProSLM: A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering (2024) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“While LLMs can generate logic programs, their outputs in formal domains remain prone to errors [15].”
From ADVENT: LLM-Driven Automatic Predicate Invention for ILP (2026) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“External tools used for self- correction include code interpreters for code generation tasks (Chen et al., 2024e;Gou et al., 2024) and symbolic reasoners for logical reasoning tasks (Pan et al., 2023).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Pan et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.1 Main Results
“1. Logic-LM significantly outperforms standard LLMs and CoT across all datasets.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Query:
“| PrOntoQA | 47.40 | 67.80 | 61.00 | 51.80 | 83.00 | 85.00 | 77.40 | 98.79 | 83.20 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4 Experiments
“We use the test set which has 231 multiplechoice questions.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Query:
“| AR-LSAT | AR | 230 | 5 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
“depth-0, depth-1, depth-2, depth-4, and depth-5 problems, respectively.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > 4.3 Robustness of Reasoning
“The problems in these subsets require 0, <=1, <=2, <=3, and <=5 hops of reasoning, respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Answer
“A) The convertible is the oldest.”
§ LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning > Which of the following is true?
“A) The tractor is the oldest.B) The convertible is the oldest.C) The minivan is the oldest.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
de Moura, L., & Bjørner, N. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-78800-3_24 · Cited by 6,595 · cited 2×
“For solving SAT problems, we use the Z3 theorem prover (de Moura and Bjørner, 2008), a satisfiability modulo theories (SMT) solver developed by Microsoft 4 .”
BackgroundMethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 7×
“”, these models are able to answer questions with explicit reasoning steps (Wei et al., 2022b; Kojima et al., 2022).”
BackgroundMethodInfluentialSelf-citationTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Machine-Oriented Logic Based on the Resolution Principle
Robinson, J. A. (1965) · Journal of the ACM · doi:10.1145/321250.321253 · Cited by 4,011 · cited 2×
“It initially converts FOL statements to conjunctive normal form (CNF) and then performs resolution (Robinson, 1965) on the CNF to deduce whether a conclusion is true, false, or unknown.”
BackgroundMethodEnergy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Richardson, M., & Domingos, P. (2006) · Machine Learning · doi:10.1007/s10994-006-5833-1 · Cited by 2,676 · cited 2×
“For example, statistical relational learning (SRL) systems such as Markov logic networks (Richardson and Domingos, 2006) and probabilistic soft logic (Bach et al., 2017) have demonstrated great promise in reasoning under uncertainty and integration with our framework would enable even more adaptive…”
BackgroundA Mathematical Introduction to Logic
(2001) · Elsevier eBooks · doi:10.1016/c2009-0-22107-6 · Cited by 2,398 · cited 1×
“To address this, we also include the FOL grammar (Enderton, 2001) in Appendix A.”
MethodSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Research works printed but not linked: 3
- Stephen Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2017. Hinge-loss markov random fields and probabilistic soft logic. Journal of Machine Learning Research (JMLR), 18(1):1-67.
- Kostas S. Metaxiotis, Dimitris Askounis, and John E. Psarras. 2002. Expert systems in production planning and scheduling: A state-of-the-art survey. Journal of Intelligent Manufacturing, 13(4):253-260. · DOI
- Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Henry Zhu, Rui Dong, Deguang Kong, Juliette Burger, An- jelica Ramos, William Yang Wang, Zhiheng Huang, George Karypis, Bing Xiang, and Dan Roth. 2023b. STREET: A multi-task structured reasoning and ex- planation benchmark. In Proceedings of the 11th International Conference on Learning Representa- tions (ICLR). · DOI
Cited by
Sorted by citations
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
Ma, Y., Cao, Y., Hong, Y. S., & Sun, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.710 · Cited by 193
“…Chowdhery et al. 2022; Touvron et al. 2023) have shown remarkable abilities on various NLP applications such as factual question answering (Yu et al., 2023; Sun et al., 2023), arithmetic reasoning (Chen et al., 2022a; Qian et al., 2023) and logical reasoning (Jung et al., 2022; Pan et al., 2023).”
BackgroundEnhancing emergency decision-making with knowledge graphs and large language models
Chen, M., Tao, Z., Tang, W., Qin, T., Yang, R. C., & Zhu, C. (2024) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2024.104804 · Cited by 112
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86
Self-citationWhen Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57
“Second, Logic-LM employs a self-refinement strategy, which has shown promise across code generation and NLP tasks (Zhang et al., 2023b; Chen et al., 2023a; Peng et al., 2023; Madaan et al., 2023; Olausson et al., 2023) but which we do not consider here.”
MethodLLM-Based Agents for Tool Learning: A Survey
Xu, W., Huang, C., Gao, S., & Shang, S. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00296-9 · Cited by 87
“Similarly, Logic-lm [94] transforms described problems into first-order formal logic (FOL [7]).”
Pallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Lior Horesh, C., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 76
“…and planning involves transforming natural language instructions into structured planning languages (Wong et al. 2023; Kelly et al. 2023; Yang 2023; Pan et al. 2023; Xie et al. 2023; Yang, Ishay, and Lee 2023; Lin et al. 2023c; Sakib and Sun 2023; Yang et al. 2023b; Parakh et al. 2023; Yang et al.…”
Large Language Models Can Learn Temporal Reasoning
Xiong, S., Payani, A., Kompella, R., & Fekri, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.563 · Cited by 47
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 66
“Hybrid models that combine LLMs with symbolic solvers (Pan et al., 2023), constraint programming, or external simulation tools (Gao et al., 2024a) are emerging.”
A survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 45
ArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemes
Hong, S., Xiao, L., Zhang, X., & Chen, J. (2024) · 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822109 · Cited by 21
Machine Learning in FCAPS: Toward Enhanced Beyond 5G Network Management
Mekrache, A., Ksentini, A., & Verikoukis, C. V. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3395414 · Cited by 21
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationBioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 51
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24
Self-citationLogic Augmented Multi-Decision Fusion Framework for Stance Detection on Social Media
Zhang, B., Jun, M., Fu, X., & Dai, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103214 · Cited by 13
A Survey of Task Planning with Large Language Models
Zhai, W., Liao, J., Chen, Z., Su, B., & Zhao, X. (2025) · Intelligent Computing · doi:10.34133/icomputing.0124 · Cited by 21
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.61 · Cited by 12
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
“Several efforts (e.g. Liu et al. (2023); Pan et al. (2023); Xie et al. (2023)) advocate using LLMs only as glorified translators–for converting reasoning problems embedded in textual format to symbolic representations, which are then pawed off to external classical symbolic solvers (with all their…”
BackgroundORFS-agent: Tool-Using Agents for Chip Design Optimization
Ghose, A., Kahng, A. B., Kundu, S., & Wang, Z. (2025) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad65511.2025.11189204 · Cited by 20
“Blending BO with symbolic reasoning [42] [52] yields targeted, adaptive flows.”
Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models
Liu, J., Huang, Z., Liu, Q., Ma, Z., Zhai, C., & Chen, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3556367 · Cited by 13
“Nevertheless, behind these successes, numerous studies have indicated that the current LLMs are still prone to exhibit logical errors when solving problems [10], [11], [12], [13].”
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8
“Despite facing challenges in multi-step logical tasks involving contemporary information or complex logical sequences [9], emerging approaches like self-consistency [10] show promise in enhancing performance, particularly in areas such as arithmetic and common sense reasoning.”
BackgroundLoRP: LLM-based Logical Reasoning via Prolog
Di, Z., Zhang, C., Lv, H., Cui, L., & Liu, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114140 · Cited by 11
DSPy-based neural-symbolic pipeline to enhance spatial reasoning in LLMs
Wang, R., & Sun, K. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108022 · Cited by 8
Design Patterns for Large Language Model Based Neuro-Symbolic Systems
de Boer, M., Smit, Q. T. S., van Bekkum, M. A., Meyer-Vitali, A., & Schmid, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251377499 · Cited by 13
“Secondly, we test validity and usefulness of the Box-ology and our extensions in this field on example architectures and applications, such as ChatGPT, KnowGL, GENOME and Logic-LM.”
Tapia-Mandiola, S., & Araya, R. (2024) · AI · doi:10.3390/ai5040093 · Cited by 11
Advancing Few-Shot Named Entity Recognition with Large Language Model
Xiao, Y., Zou, J., & Yang, Q. (2025) · Applied Sciences · doi:10.3390/app15073838 · Cited by 10
An LLM-Enabled Multi-Agent Autonomous Mechatronics Design Framework
Wang, Z., Lo, F. P.-W., Chen, Q., Zhang, Y., Chen, L., Chen, X., Yu, Z., Thompson, A. J., Yeatman, E. M., & Lo, B. P. L. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00404 · Cited by 7
Vakharia, P., Kufeldt, A., Meyers, M., Lane, I., & Gilpin, L. H. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-71170-1_23 · Cited by 5
“Neither [11] nor [13] pose a viable approach in this current work, however, due to the complexity of questions expected for a KBQA system, and our subsequent interest in leveraging the generative power of an LLM to answer inputted queries.”
BackgroundMethodCausal Reasoning in Large Language Models using Causal Graph Retrieval Augmented Generation
Samarajeewa, C., Silva, D. D., Osipov, E., Alahakoon, D., & Manic, M. (2024) · Conference on Human System Interactions · doi:10.1109/hsi61632.2024.10613566 · Cited by 10
“In [18], Pan et al. propose an approach to improve logical reasoning capabilities by combining large language models with symbolic solvers.”
BackgroundGraph of Logic: Enhancing LLM Reasoning with Graphs and Symbolic Logic
Alotaibi, F., Kulkarni, A., & Zhou, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825450 · Cited by 5
Agile Logical Semantics for Natural Languages
Manca, V. (2024) · Information · doi:10.3390/info15010064 · Cited by 6
ToM-LM: Delegating Theory of Mind Reasoning to External Symbolic Executors in Large Language Models
Tang, W.-Z., & Belle, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_20 · Cited by 11
A dataset for evaluating clinical research claims in large language models
Zhang, B., Bornet, A., Yazdani, A., Khlebnikov, P., Milutinovic, M., Rouhizadeh, H., Amini, P., & Teodoro, D. (2025) · Scientific Data · doi:10.1038/s41597-025-04417-x · Cited by 11
Li, Y., Fu, X., Verma, G., Buitelaar, P., & Liu, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24476 · Cited by 4
“Contemporary research has increasingly explored the integration of symbolic reasoning [104, 145] with LLMs to improve the logical consistency and reliability of model reasoning processes.”
Schmidt, D. M., & Cimiano, P. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1406857 · Cited by 7
“…tasks requiring a structured prediction, e.g., event extraction (Lu et al., 2021), syntactic and semantic parsing (Roy et al., 2024), symbolic expression generation like SMT formulas (Pan et al., 2023; Sun et al., 2023) or SQL query generation from text (Scholak et al., 2021; Lin et al., 2020).”
InfluentialLLM-Driven Knowledge Extraction in Temporal and Description Logics
Duranti, D., Giorgini, P., Mazzullo, A., Robol, M., & Roveri, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_12 · Cited by 7
Safety and Trust in Artificial Intelligence with Abstract Interpretation
Singh, G., Laurel, J., Misailovíc, S., Banerjee, D., Singh, A., Xu, C., Ugare, S., & Zhang, H. (2025) · Foundations and Trends® in Programming Languages · doi:10.1561/2500000062 · Cited by 4
Dynamic knowledge correction via abductive for domain question answering
Zhou, Y., Huang, R., Lin, C., Liu, L., & Qin, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104306 · Cited by 4
Final: Combining First-Order Logic With Natural Logic for Question Answering
Shi, J., Ding, X., Hui, S. C., Yan, Y., Zhao, H., Liu, T., & Qin, B. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3551231 · Cited by 3
“Furthermore, some research efforts [63], [64], [65] leverage language models to convert natural language problems into symbolic formulations, which are then solved using deterministic symbolic solvers.”
Su, P., Xu, R., Wen-bin, W., & Chen, D. (2025) · Applied System Innovation · doi:10.3390/asi8060160 · Cited by 3
Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language
Faghihi, H. R., Nafar, A., Uszok, A., Karimian, H. R., & Kordjamshidi, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_25 · Cited by 5
Can Large Language Models Put 2 and 2 Together? Probing for Entailed Arithmetical Relationships
Panas, D., Seth, S., & Belle, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_21 · Cited by 4
Towards Interpretable and Consistent Multi-Step Mathematical Reasoning in Large Language Models
Huang, X., Wang, Z., Liu, X., et al. (2025) · International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM) · doi:10.1109/aiim67611.2025.11232978 · Cited by 3
“Wang et al. [1] combined SymPy with LLMs for algebraic manipulation and reasoning, while Yu et al. [2] fused chain-of-thought prompting with rule-based deduction.”
Giabbanelli, P. J., Beverley, J., Dávid, I., & Tolk, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5365069 · Cited by 3
SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2410.20285 · Cited by 3
“Search methods Various search approaches have been applied to Large Language Models (LLMs) to facilitate System 2 (Kahneman, 2011; Saha et al., 2024; Pan et al., 2023; Bounsi et al., 2024) thinking in non-linear reasoning structures.”
Self-citationEvidence from formal logical reasoning reveals that the language of thought is not natural language
Kean, H., Fung, A., Jaggers, P., Chen, J., Rule, J. S., Benn, Y., Tenenbaum, J. B., Piantadosi, S. T., Varley, R. A., & Fedorenko, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520095123 · Cited by 2
Neuro-symbolic systems: a perspective of uncertainty management
Li, Z., Yao, Y., Ma, X., & lu, J. (2024) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0163 · Cited by 2
Semantic-aware query answering with Large Language Models
Atzeni, P., Baldazzi, T., Bellomarini, L., Laurenza, E., & Sallinger, E. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102494 · Cited by 4
Kabir, I., Reza, M. A., & Billah, S. M. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128818 · Cited by 2
“LOGIC-LM [9] demonstrated the integration of large language models with symbolic solvers for enhanced logical reasoning. a) Components of FOL: FOL consists of predicates , functions , constants , variables , and quantifiers .”
Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-agent Systems
Kostka, A., & Chudziak, J. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_2 · Cited by 4
“…applications, like learned models of other agents’ goals and behaviors [17, 35, 19], more advanced integration of LLMs with symbolic reasoning beyond what is provided by ASP-based verification [23, 36, 28], and hybrid approaches to reasoning that leverage both neural and symbolic strengths [31].”
Fine-Tuned Large Language Models for Logical Translation: Reducing Hallucinations with Lang2Logic
Pan, M., Kodakandla, D., & Farooque, M. (2025) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc66965.2025.11250432 · Cited by 2
“Recent works such as LogicLLaMA [6] and LOGIC-LM [7] have pioneered advancements in logical reasoning by fine-tuning LLMs for specialized tasks.”
Bridging Large Language Models and Logic Programming
Crocetti, V., & Dragoni, A. F. (2025) · 2025 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE) · doi:10.1109/metroxraine66377.2025.11340308
“A highly relevant framework, Logic-LM, was recently proposed by Pan et al. (2023) [9].”
DCoL-A: Agentic dual chain of thinking helps LLMs pretend logic solvers
Chen, M., Wu, L.-I., Liu, R., Chang, X., Xue, J., & Li, G. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103780 · Cited by 2
OrLog: Resolving Complex Queries with LLMs and Probabilistic Reasoning
Hoveyda, M., Piepenbrock, J., de Vries, A. P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_7 · Cited by 2
Modeling Patterns for Neural-Symbolic Reasoning Using Energy-based Models
Dickens, C., Pryor, C., & Getoor, L. C. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31187 · Cited by 2
“We compare our predictive performance with the baseline and models presented in Pan et al. (2023) on gpt-3.”
MethodResultLarge Language Models for Logical Fallacy Detection
Teo, N., Huang, D., Cambria, E., & Wang, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8197-6_29 · Cited by 3
CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Causal Significance and Consistency
Wang, K., Zhang, X., Lyu, J., Hu, T., & Ma, H. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209798 · Cited by 2
“Large language models (LLMs) have shown promising potential in solving structured logical reasoning problems [1]–[3], unstructured mathematical reasoning problems [4]– [6] and downstream tasks such as autonomous driving [7], [8].”
Benchmarking Graph Construction by Large Language Models for Coherence-Driven Inference
Huntsman, S., & Thomas, J. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00116 · Cited by 2
“However, the computation of a transformer with L layers on a prompt of length n can be performed using O ( L log n ) bits of memory under common assumptions.”
Can Language Models Pretend Solvers? Logic Code Simulation with LLMs
Chen, M., Li, G., Wu, L.-I., Liu, R., Su, Y., Chang, X., & Xue, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0602-3_6 · Cited by 6
Case-Based Reasoning in Generative Agents: Review and Prospect
Zhan, H., Wu, Y., Ding, J. L., Xu, B., Yang, L., & Lin, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96559-3_12 · Cited by 1
On the Relevance of Logic for Artificial Intelligence, and the Promise of Neurosymbolic Learning
Belle, V. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251339951 · Cited by 1
“2 More generally, recent approaches seek to incorporate logical solvers as oracles (Persia & Ozaki, 2022) that can validate or disprove the predictions of neural architectures, including LLMs (Miceli-Barone et al., 2023; Pan et al., 2023; Panas et al., 2024; Zhang et al., 2023).”
He, R., Zhang, Y., Zhang, M., et al. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00040 · Cited by 1
“Recent work has shown that large language models can generate Boolean logic, propositional logic, and even more expressive systems like predicate logic (first-order logic) [8], [9].”
Trustworthy Machine Learning in the Era of Foundation Models
Han, B. (2025) · IEEE Intelligent Systems · doi:10.1109/mis.2025.3621938 · Cited by 1
From Legal Text to Tech Specs: Generative AI’s Interpretation of Consent in Privacy Law
Kesari, A., Breaux, T. D., Santos, S., Norton, T. B., & Singhal, A. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769241 · Cited by 1
“Formal syntax constrains problem spaces, yielding precise and computationally tractable solutions compared to natural language reasoning [21, 23].”
He, Y., Liu, J., Wen, W., Guo, Y., Chen, H., Ma, Y., Ke, W., & Wang, P. (2026) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2026.3654446 · Cited by 1
“Meanwhile,inspiredbypreviousworks[46],[47],we constructacandidatedemonstrationpoolof1,000machine-translatedexamplesperdataset,followingaschema-guidedFOL translationframeworkthatensuressyntacticvalidityandlogi-calconsistency.”
MAPPING RESEARCH ON GENERATIVE AI AND CRITICAL THINKING IN EDUCATION
Idrus, N. M., Ghani, N. A. A., & Rahman, H. A. (2025) · International Journal of Education Psychology and Counseling · doi:10.35631/ijepc.1061067 · Cited by 1
DL-ReasonSuite: A Benchmark for Evaluating Description Logic Reasoning in Large Language Models
Olucoglu, M., & Bursa, O. (2026) · Applied Sciences · doi:10.3390/app16041821 · Cited by 1
Hersche, M., Camposampiero, G., Wattenhofer, R. P., Sebastian, A., & Rahimi, A. (2026) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732261419316 · Cited by 1
Algebraic Semantics for Interpretable Reasoning in Large Language Models
Liu, P., & Li, Y. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123383 · Cited by 1
Neuro-symbolic NLP: taxonomy, assessment, and directions
Chatzikyriakidis, S., & Lappin, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1797587 · Cited by 2
“Logic-LM (Pan et al., 2023) maps natural language to FOL or SAT constraints for solver refinement, boosting performance by 39.2% over standard prompting.”
InfluentialVeriTrans: Fine-Tuned LLM-Assisted NL→PL Translation via a Deterministic Neuro-symbolic Pipeline
Liu, X. M., Kodakandla, D., Srivastva, K., & Farooque, M. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814906 · Cited by 3
neucsestu@CC-MMD 2026: SC-MACA via Multi-Agent Cultural Reasoning for Misogyny Detection
Zhang, S., & Li, Z. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3832462 · Cited by 1
“Inspired by logical prompting and multi-model refinement [2, 7, 9, 15], the pipeline keeps its intermediate rationales structured enough for downstream learning.”
Trustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2
Rule-Based Moral Principles for Explaining Uncertainty in Natural Language Generation
Atf, Z., & Lewis, P. R. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00018 · Cited by 1
Giabbanelli, P. J., Beverley, J. R., Dávid, I., & Tolk, A. (2025) · Winter Simulation Conference · doi:10.1109/wsc68292.2025.11338852 · Cited by 1
“However, as shown by Pan et al. (2023) through their experiments with 1 700 code samples in C, C++, Go, Java, and Python, LLMs are not yet reliable enough for code translation, with the reported proportion of correct translations ranging between 2.1% and 47.3%.”
InfluentialLimitations of and Lessons from the Learning of Large Language ModelsPreprint
Oldenburg, R. (2023) · Qeios · doi:10.32388/9fh6ad · Cited by 1
Xu, F., Zhou, T., Nguyen, T., Bao, H., Lin, C., & Du, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805664 · Cited by 1
Not Explainable but Verifiable
Burgess, P., & Shareghi, E. (2025) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0093 · Cited by 1
Ge, Y., Romeo, S., Cai, J., Shu, R., Sunkara, M., Benajiba, Y., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.01630 · Cited by 3
“These results align with recent research that combines LLMs with symbolic reasoners for various reasoning tasks (Olausson et al., 2023; Pan et al., 2023).”
InfluentialReason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming
Chen, Z., & Ma, M. (2026) · arXiv · doi:10.48550/arxiv.2607.23019
“Logic-LM Pan et al. [2023] translates natural-language problems into formal logic and delegates inference to an external solver.”
InfluentialAlviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
“…fine-tuning of specialized LLMs to support diverse logic tasks (Coppolillo et al. 2024); and (iii) solver-in-the-loop methodologies that use feedback from the solver either to refine the extracted ASP code on the fly (Pan et al. 2023) or to guide the LLM fine-tuning stage (Schrader et al. 2026).”
ADVENT: LLM-Driven Automatic Predicate Invention for ILP
Yu, T., Huang, P.-C., Hsu, C., Ku, C.-T., & Kang, Y. (2026) · 2026 IEEE International Conference on Information Reuse and Integration for Data Science (IRI) · doi:10.1109/iri69576.2026.00025
Legally-Guided Automated Decision-Making System Using Language Model Agents for Autonomous Driving
Wang, Y., Barta, D., Hesse, J., Buchwald, P., & Paschke, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72407-7_17
A dataset for evaluating clinical research claims in large language modelsPreprint
Zhang, B., Yazdani, A., Bornet, A., Khlebnikov, P., Milutinovic, M., Rouhizadeh, H., Amini, P., & Teodoro, D. (2024) · medRxiv · doi:10.1101/2024.10.08.24315103
Identifying Logical Patterns in Text for Reasoning
Armary, P., Vaigh, C. B. E., Spicher, A., Labbani, O., & Nicolle, C. (2024) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai62512.2024.00122
“The two best models are Logic-LM [13], which is based on GPT-4 (1.7T parameters) and Mistral (7B parameters).”
InfluentialKinder, L., & Käfer, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78955-7_21
Beyond Training Data: Zero-Shot Mathematical Problem Solving with Large Language Models
Parmar, S. H., & Goriya, M. (2024) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei64305.2024.10912380
Zhang, H., Zhu, X., Wang, X., Song, Y., Gu, H., & Ma, L. (2025) · IEEE International Conference on Mobile Data Management (MDM) · doi:10.1109/mdm65600.2025.00024 · Cited by 1
Self-citationNARS-GPT: An Integrated Reasoning System for Natural Language Interactions
Isaev, P., & Hammer, P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-99965-9_25 · Cited by 1
A Little Help Goes a Long Way: Tutoring LLMs in Solving Competitive Programming Through Hints
Zhang, Y., Dong, W., Liu, J., Wang, S., Wang, D., Ma, T., Li, Y., & Yang, K. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3614204
“While prompt-based generation using powerful LLMs such as GPT-4o can produce plausible results, prior work [17], [18] shows that such models often require reliable external guidance to avoid compounding early-stage errors.”
Formal Reasoning for Intelligent QA Systems: A Case Study in the Educational Domain
Tuan, B., Nguyen, A. D., Thai, P., Hua, M., Ngan Pham, L. N., Ngan Pham, T. B., Le, D. N. H., Nguyen, L. S. T., Tran, T.-T., Bui, T., et al. (2025) · Proceedings of the 2nd ACM Workshop in AI-powered Question & Answering Systems · doi:10.1145/3746274.3760392 · Cited by 2
Le, T. D., Thai, P. K., Nguyen, S., Hua, M. T., Pham, N., Bui, T., Quan, T., & Bui, T. V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_14 · Cited by 1
Zhang, X., Zhang, P., Liu, Q., & Zhao, F. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4990-0_9
Weng, T.-T., Sun, J.-T., Fu, Z.-X., & Lee, C.-H. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-09945-7_14
Minimal reduct for propositional circumscription
Xie, Z.-T., Wang, Y., Yang, L., & Feng, R. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1614894
“Within this trend, classical logic techniques are being revisited as essential tools to provide strong guarantees and structured reasoning capabilities that purely statistical methods often lack (Ryu et al., 2025; Ye et al., 2023; Pan et al., 2023; Olausson et al., 2023; Callewaert et al., 2025).”
Three Tiers of Computation in Transformers and in Brain Architectures
Graham, E., & Granger, R. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844612
FoVer: First-Order Logic Verification for Natural Language Reasoning
Pei, Y., Du, Y., & Jin, X. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.41
Zheng, X., Li, N., Luan, X., Wang, K., Shi, L., Sun, M., & Wang, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787820 · Cited by 6
“18) If something likes the cat then it visits the dog.”
Metadata
- Authors
- Pan, L., Albalak, A., Wang, X., & Wang, W. Y.
- Affiliations
- University of California, Santa Barbara (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 3806-3824
- arXiv
- 2305.12295
- OpenAlex
- W4389519818
- Semantic Scholar
- 9e9e4df2996bac794c4f04cb887df3e553bae4fd
- Cited by
- 144
- References
- 56
- Citations by year
- 2026: 59 · 2025: 54 · 2024: 25 · 2023: 6
- FWCI
- 25.82
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2048122)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, symbolic solvers, logical reasoning, self-refinement, chain-of-thought prompting, proof writing, first-order logic, logical deduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-19, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-19 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
