Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Open accessAbstract
Xiaocheng Yang, Bingsen Chen, Yik-Cheung Tam. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2024.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes generating Prolog programs with LLMs for arithmetic reasoning, so the model extracts predicates and symbolic formulas while an external Prolog interpreter performs calculation. It also introduces PROPER, a predicate-permutation data augmentation method, and reports that Prolog generation and PROPER outperform Chain-of-Thought on GSM8K and GSM-HARD across three 7B LLMs.
Key findings
- Prolog generation consistently beats CoT across Llama-2, CodeLlama, and Mistral on GSM8K, with a 10.9% average margin; Table 1 reports GSM8K accuracy of 41.5% vs. 33.8% for Llama-2, 55.0% vs. 37.5% for CodeLlama, and 66.3% vs. 58.9% for Mistral.
- The gap expands to 22.6% on GSM-HARD on average; Table 1 reports 32.4% vs. 12.0% for Llama-2, 41.6% vs. 13.9% for CodeLlama, and 50.6% vs. 30.8% for Mistral.
- PROPER improves accuracy with a proper permutation ratio: adding two permuted samples per original increased accuracy by 9.5% for Llama-2 and 4.0% for CodeLlama; Table 2 reports 1:2 accuracies of 51.0% for Llama-2, 59.0% for CodeLlama, and 68.8% for Mistral, with Mistral lower than its 1:1 result of 70.2%.
- Validation loss did not reliably track accuracy: choosing checkpoints by validation accuracy improved Prolog from 41.5% to 47.9% and PROPER 1:2 from 51.0% to 51.3% in Table 3; Python generation with Llama2 reached 55.12% on GSM8K, better than both Prolog and PROPER.
Methods
- The authors curated GSM8K-Prolog semi-automatically by prompting GPT-4 to generate Prolog programs for GSM8K questions, manually correcting malfunctioning samples, and obtaining 100% accuracy in code results; the corpus has 8792 samples, with 7473 training, 1319 test, and 100 validation samples.
- They finetuned 7B versions of Llama2, CodeLlama, and Mistral using 8-bit quantization and LoRA, adopting (r, α) = (32, 64), which trained 0.248% of parameters for Llama2 and CodeLlama and 0.188% for Mistral; inference used beam search with beam size 4 and PySwip as the Prolog interpreter.
- PROPER permutes facts/rules and goals in Prolog programs, samples up to 10 goal permutations and up to 10 fact/rule permutations, and mixes permuted samples into training at ratios 1:0, 1:1, and 1:2; evaluation used accuracy on GSM8K and GSM-HARD, restricted to integer-answer samples because PySwip could not handle decimal answers.
Limitations
- Stated in the paper: full-parameter finetuning was unsatisfying due to limited corpus size, so they could not compare with methods like ToRA or RFT; they did not scale base models beyond 7B; and PySwip restricted solvable questions to integer answers.
- Evident from the text: initial PROPER validation involved soft data leakage because permutations of validation samples were included in the training set, and validation loss did not align with accuracy, requiring checkpoint selection by validation accuracy.
- Evident from the text: the evaluation is limited to 7B models and integer-answer GSM8K/GSM-HARD samples, and the reported Python-generation baseline of 55.12% with Llama2 outperformed their Prolog and PROPER results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 2 cited sources checked · 0 supported · 0 flagged
- Findingown data
7% of the 70 sampled problematic Prolog codes generated by finetuned Mistral using the 1:1 permutation setting had syntax errors (Swi-Prolog yields syntax errors on those codes).
“7% of them had syntax errors, meaning the Swi-Prolog yields syntax errors on those codes.”
Quote in the paperNot a substantial claim - Findingreasoning
Increasing the permutation ratio results in significantly lowered validation loss.
“As is shown in Figure 3, increasing the permutation ratio results in significantly lowered validation loss.”
Quote in the paperSubstantial claim - Findingown data
The permutation ratio of 1:2 yielded a weakened performance on Mistral compared with its case of one permutation per sample.
“On the other hand, the lowered accuracy of Mistral, compared with its case of one permutation per sample, suggests that PROPER might be limited for models already with high Prolog generation capacity.”
Quote in the paperSubstantial claim - Findingown data
100% of the 70 sampled problematic Prolog codes generated by finetuned Mistral using the 1:1 permutation setting had semantic errors (their logic of solution was wrong).
“100% of them had semantic errors, meaning their logic of solution was wrong.”
Quote in the paperNot a substantial claim - Findingown data
Lowered validation loss from PROPER does not lead to higher accuracy.
“Lowered validation loss from PROPER does not lead to higher accuracy.”
Quote in the paperSubstantial claim
- Findingown data
Prolog-based arithmetic problem-solving outperforms CoT generation on GSM8K across three distinct LLMs, with an average margin of 10.9% over the CoT baseline.
“Experimental results show that our Prolog-based arithmetic problem-solving outperforms CoT generation in the GSM8K benchmark across three distinct LLMs.”
- Also stated in: § 4.2 Results · plain
“Prolog generation performs consistently better than CoT across three models.”
- Also stated in: § 4.2 Results · plain
“According to Table 1, generating Prolog to solve mathematical questions yields significantly more accurate results with a 10.9% margin over the CoT baseline on average across all models on GSM8K.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 Results · plain
- Findingown data
Increased validation loss from PROPER does not lead to decreased validation accuracy.
“Increased validation loss from PROPER does not lead to decreased validation accuracy.”
Quote in the paperSubstantial claim - Findingown data
The 1:1 permutation setting yielded the highest accuracy among the permutation settings tested.
“using the 1:1 permutation setting, which had yielded the highest accuracy”
Quote in the paperNot a substantial claim - Findingown data
Prolog generation outperforms CoT on GSM-HARD by 22.6% on average across models.
“This gap further expands to 22.6% on GSM-HARD, indicating exceptional superiority over CoT when large number calculations are involved.”
Quote in the paperSubstantial claim - Findingreasoning
Using the same instruction for both original ground truth codes and permuted samples degraded the performance of the model.
“Notably, when we used PROPER to augment the dataset, we used slightly altered input prompts for permuted samples because we found that using the same instruction for both the original ground truth codes and the permuted ones degraded the performance of the model.”
Quote in the paperSubstantial claim - Findingown data
Choosing checkpoints based on validation accuracy instead of validation loss improves performance across all methods.
“As is shown in Table 3, by choosing checkpoints based on validation accuracy instead of validation loss, the performance can be improved across all methods.”
Quote in the paperSubstantial claim - Interpretationown data
PROPER might be limited for models already with high Prolog generation capacity.
“On the other hand, the lowered accuracy of Mistral, compared with its case of one permutation per sample, suggests that PROPER might be limited for models already with high Prolog generation capacity.”
Quote in the paperSubstantial claim - Interpretationown data
Prolog generation works well regardless of the model's inherent arithmetic reasoning capability.
“This observation indicates that Prolog generation works well regardless of the model's inherent arithmetic reasoning capability.”
Quote in the paperSubstantial claim - Interpretationown data
There is a larger divergence between the cross-entropy loss objective and the ultimate accuracy of Prolog generation.
“suggesting a larger divergence between the objective of cross entropy loss and the ultimate accuracy of Prolog generation.”
Quote in the paperSubstantial claim - Study design and conductreasoning
The GSM8K-Prolog dataset has 100% accuracy in terms of the code results.
“In this manner, we obtained a high-quality corpus with 100% accuracy in terms of the code results.”
Quote in the paperSubstantial claim - Interpretationown data
Learning the non-sequential structure of Prolog predicates is helpful for LLMs to generate correct Prolog programs to solve arithmetic problems.
“This improvement indicates that learning the non-sequential structure of Prolog predicates is helpful for LLMs to generate correct Prolog programs to solve arithmetic problems.”
Quote in the paperSubstantial claim
- May combine two claims. 7
- The quote only partly states this claim. 2
- Classified as probably not a substantial claim. 6
May combine two claims.
Findingown dataLoRA rank r = 32 and alpha = 64 yielded significantly better results than other settings and was adopted for all experiments.
“With more trainable parameters, r = 32, α = 64 yielded significantly better results, which we thereby adopted as the configuration for all the experiments.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataPROPER Prolog generation achieves higher accuracy than regular Prolog generation and the CoT baseline on GSM8K and GSM-HARD across Llama-2, CodeLlama, and Mistral.
“| CoT | 33.8% | 12.0% | 37.5% | 13.9% | 58.9% | 30.8% |”
- Also stated in: § A.4 Error Analysis · plain
“| Prolog | 41.5% | 32.4% | 55.0% | 41.6% | 66.3% | 50.6% |”
- Also stated in: § A.4 Error Analysis · plain
“| PROPER | 51.0% | 37.4% | 59.0% | 45.9% | 70.2% | 54.4% |”
Quote in the paperSubstantial claim- Also stated in: § A.4 Error Analysis · plain
May combine two claims.
FindingreasoningPython generation gives an accuracy of 55.12% on GSM8K using Llama2 as the base model, better than both Prolog and PROPER.
“It gives an accuracy of 55.12% on GSM8K using Llama2 as the base model, better than both Prolog and PROPER.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingnone givenThe PySwip library cannot handle decimal answers.
“Notably, since we noticed that the PySwip library cannot handle decimal answers, we only considered the samples with an integer answer.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataPROPER with a proper permutation ratio further enhances LLM's arithmetic reasoning with Prolog generation, improving accuracy on GSM8K-Prolog and mitigating early convergence during training.
“With a proper permutation ratio, PROPER further enhances LLM's arithmetic reasoning with Prolog generation.”
- Also stated in: § 6 Conclusion · plain
“PROPER further improves the model's accuracy on GSM8K-Prolog and mitigates early convergence during training.”
Quote in the paperSubstantial claim- Also stated in: § 6 Conclusion · plain
May combine two claims.
Findingown dataAdding two permuted samples per original sample increased accuracy by 9.5% for Llama-2 and 4.0% for CodeLlama on the test set.
“As shown in Table 2, by adding two permuted samples for each original sample, we observed an increased accuracy of 9.5% and 4.0% of Llama-2 and CodeLlama respectively on the test set.”
Quote in the paperSubstantial claimMay combine two claims.
FindingreasoningThe permutation ratio of 1:2 yielded a weakened performance on Mistral although the validation loss was the lowest.
“Therefore, according to Table 2, the permutation ratio of 1:2 yielded a weakened performance on Mistral although the validation loss was the lowest.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationXLNet performs a tokenwise permutation via attention masking, which inspired the use of the permutation technique in PROPER.
“inspired by XLNet (Yang et al., 2020) that performs a tokenwise permutation via attention masking, we decided to also use the permutation technique.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yang et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
PremisereasoningProlog codes are not sequential; the order of facts and rules does not alter the result of the program.
“Moreover, Prolog codes are not sequential like Python, meaning that the order of facts and rules does not alter the result of the program.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe error types of the sampled Prolog codes were manually checked using the Swi-Prolog interpreter.
“manually checked the type of errors using the Swi-Prolog interpreter”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenProlog solves arithmetic reasoning tasks by defining an unordered set of predicates and running queries over them.
“Prolog solves arithmetic reasoning tasks by defining an unordered set of predicates and running queries over them.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationXLNet via permutation can attend to tokens on both sides during training and thus can partially obtain the property of autoencoding while maintaining the property of autoregressive modeling.
“The XLNet, via the permutation, can attend to tokens on both sides during training and thus can partially obtain the property of autoencoding while maintaining the property of autoregressive modeling.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yang et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningCodeLlama's larger performance gain when switching from CoT to Prolog generation is potentially attributed to its pretraining on the code-related corpus.
“which is potentially attributed to its pretraining on the code-related corpus.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
If the facts and rules are accurately captured, a Prolog interpreter can precisely solve for a correct answer in a deterministic way.
“If the facts and rules are accurately captured, a Prolog interpreter can precisely solve for a correct answer in a deterministic way.”
The paper relies on this without support.
There has not been a dataset for solving mathematical questions with Prolog.
“To our knowledge, there has not been a dataset for solving mathematical questions with Prolog.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mapping math word problems to Prolog predicates yields higher accuracy than Python PoT for problems requiring constraint satisfaction or symbolic manipulation.
“GSM8K-Prolog, presented by Yang et al. (2024), demonstrated that mapping math word problems to Prolog predicates yields higher accuracy than Python PoT for problems requiring constraint satisfaction or symbolic manipulation.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Yang et al. (2024)
Section 4.2 (page unavailable) reports that Python-code generation achieved 55.12% accuracy on GSM8K with Llama2, better than both Prolog and PROPER. The paper reports Prolog outperforming a chain-of-thought baseline, not Python, and does not report a separate accuracy comparison for constraint-satisfaction or symbolic-manipulation problems. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe 7B model's one-shot pass@4 on GSM8K (0.893) outperforms PROPER (0.72).
“thus outperforming both PROPER, introduced by Yang et al. (2024), a variant of Mistral-7B, presented by Jiang et al. (2023), with specialized supervised fine-tuning (0.72)”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Yang et al. (2024)
The Conclusion identifies PROPER as the authors’ data-augmentation method, and the Setup says they fine-tuned a 7B Mistral model. But Table 1 reports 70.2% GSM8K accuracy for Mistral with PROPER, not 0.72; the paper also calls PROPER a method rather than a model variant. The cited paper does not establish the citing model’s 0.893 pass@4 result. (LLM, conf=0.88)
Experimental — accuracy not yet measuredOptimized Qwen2.5-Coder-7B outperforms PROPER (0.72) on Prolog GSM8K.
“remarkably outperforming PROPER (0.72), a Mistral 7B variant that underwent specialized supervised fine-tuning for Prolog code generation in GSM8K.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Yang et al. (2024)
The Setup (§4.1) describes fine-tuning a Mistral 7B model, and Table 1 reports PROPER accuracy on GSM8K. But Table 1 gives 70.2% (0.702), not 0.72; the paper does not evaluate Qwen2.5-Coder-7B or establish that it outperforms PROPER. (LLM, conf=0.90)
Experimental — accuracy not yet measured[15] fine-tune LLMs to generate Prolog for arithmetic word problems.
“[15] fine-tune LLMs to generate Prolog for arithmetic word problems, while [16] use a Prolog engine to produce verified reasoning trajectories that an LLM imitates as structured chain-of-thought.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [15]
A growing line of work has explored combinations of LLMs and Prolog.
“A growing line of work has explored combinations of LLMs and Prolog.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [15]
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48. · DOI
- Ivan Bratko. 2012. Prolog programming for Artificial Intelligence. Addison-Wesley.
- Michael A. Covington. 2002. Natural language pro- cessing for Prolog programmers. Prentice Hall.
- PMLR.
- Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yu- jiu Yang. 2023. Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. · DOI
Cited by
Sorted by citations
Evaluating and enhancing spatial cognition abilities of large language models
Yang, A.-R., Fu, C., Jia, Q.-R., Dong, W., Ma, M.-Y., Chen, H., Yang, F., & Wu, H. (2025) · International Journal of Geographical Information Science · doi:10.1080/13658816.2025.2490701 · Cited by 25 · see citing works
Avila, C., Ilbay, D., & Rivera, D. (2025) · Buildings · doi:10.3390/buildings15173190 · Cited by 16 · see citing works
Large Language Models for Computer-Aided Design: A Survey
Zhang, L., Le, B., Akhtar, N., Lam, S.-K., & Ngo, D. (2026) · ACM Computing Surveys · doi:10.1145/3787499 · Cited by 23 · see citing works
LoRP: LLM-based Logical Reasoning via Prolog
Di, Z., Zhang, C., Lv, H., Cui, L., & Liu, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114140 · Cited by 11 · see citing works
An LLM-based Neuro-symbolic Approach for Designing Fuzzy Logic Controllers
Rashid, F., & Long, Y. (2025) · International Conference on Frontiers of Information Technology · doi:10.1109/fit67061.2025.11333630 · Cited by 1 · see citing works
Xie-Li, D., Agüero-Elizondo, N., & Fallas-Moya, F. (2025) · International Conference of the Chilean Computer Science Society · doi:10.1109/sccc67219.2025.11420610 · Cited by 1 · see citing works
Fine-Tuning Large Language Models for Solving Math Word Problems: A Case Study with LLaMA-3
Nie, H. (2024) · 2024 4th International Signal Processing, Communications and Engineering Management Conference (ISPCEM) · doi:10.1109/ispcem64498.2024.00024 · Cited by 2 · see citing works
From Reasoning to Code: GRPO Optimization for Underrepresented Languages
PENNINO, F., RAIMONDI, B., RONDELLI, M., GURIOLI, A., & GABBRIELLI, M. (2026) · Theory and Practice of Logic Programming · doi:10.1017/s1471068426100489 · Cited by 1 · see citing works
PrologMCP: A Standardized Prolog Tool Interface for LLM Agents
Mensfelt, A., Prabhakaran, A., Haret, A., Trencsenyi, V., & Stathis, K. (2026) · arXiv · doi:10.48550/arxiv.2606.14935 · Cited by 1 · see citing works
Performance Analysis of Chinese Large Language Models in Solving Math Word Problems
Pan, S., Liu, K., Chen, W., & He, B. (2024) · 2024 International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir62538.2024.10960109 · Cited by 1 · see citing works
Peshave, A., Hossain, K., Kubricht, J., Narayanan, R., Burpee, Z., & Agarwal, A. (2025) · 2025 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw69685.2025.00072
Large Language Model-Empowered Debate Outcome Prediction via Argument Quality Assessment
Chen, X., Li, Y., Lee, O.-K. D., & Chin, A. G. (2026) · Group Decision and Negotiation · doi:10.1007/s10726-026-09986-9 · Cited by 2 · see citing works
Autonomous Business System via Neuro-symbolic AI
Pang, C., & Sayama, H. (2026) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon66367.2026.11503621
A DSL-Based Agent Framework for Reliable LLM Arithmetic using Model Context Protocol
C, R., & Jishnu, K. (2026) · 2026 International Conference on Data Science and Business Systems (ICDSBS) · doi:10.1109/icdsbs69077.2026.11634997
Alviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
- Authors
- Yang, X., Chen, B., & Tam, Y.-C.
- Affiliations
- New York University Shanghai (CN)
- Venue
- Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
- Published
- 2024
- Pages
- 699-710
- OpenAlex
- W4401042728
- Cited by
- 12
- References
- 0
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 2
- FWCI
- 3.81
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- arithmetic reasoning, large language models, permutation generation, code generation, mathematical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
