CoChat
Sign Up
PreprintCited by 2Logic, Reasoning, and KnowledgeTopic ModelingNatural Language Processing Techniques

Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation

Open access
Bao, Qiming, Peng, Alex Yuxuan, Hartill, Tim, Tan, Neset, Deng, Zhenyun, Witbrock, Michael, Liu, Jiamou·2022·arXiv
DOI 10.48550/arxiv.2207.14000arXiv 2207.14000
Open this paper in CoChat — free accountOpen PDF

Abstract

Combining deep learning with symbolic logic reasoning aims to capitalize on the success of both fields and is drawing increasing attention. Inspired by DeepLogic, an end-to-end model trained to perform inference on logic programs, we introduce IMA-GloVe-GA, an iterative neural inference network for multi-step reasoning expressed in natural language. In our model, reasoning is performed using an iterative memory neural network based on RNN with a gated attention mechanism. We evaluate IMA-GloVe-GA on three datasets: PARARULES, CONCEPTRULES V1 and CONCEPTRULES V2. Experimental results show DeepLogic with gated attention can achieve higher test accuracy than DeepLogic and other RNN baseline models. Our model achieves better out-of-distribution generalisation than RoBERTa-Large when the rules have been shuffled. Furthermore, to address the issue of unbalanced distribution of reasoning depths in the current multi-step reasoning datasets, we develop PARARULE-Plus, a large dataset with more examples that require deeper reasoning steps. Experimental results show that the addition of PARARULE-Plus can increase the model's performance on examples requiring deeper reasoning depths. The source code and data are available at https://github.com/Strong-AI-Lab/Multi-Step-Deductive-Reasoning-Over-Natural-Language.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.14000

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The PARARULE-Plus dataset is a large-scale synthetic dataset for multi-step deductive reasoning over natural language, extending the original PARARULE with deeper inference chains.

    “The PARARULE-Plus dataset [24] is a large-scale synthetic dataset for multi-step deductive reasoning over natural language, extending the original PARARULE [32] with deeper inference chains.”

    From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [24]

    The Datasets (p. 9) describes PARARULE-Plus as a large natural-language multi-step reasoning dataset with nearly 400,000 samples, including many at depths 2–5. But Related Work (p. 5) says the original PARARULES already reached depth 5. The source supports adding more examples requiring deep reasoning, not extending the maximum inference-chain depth. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The paper provides an empirical evaluation of the formalizer agent on PARARULE-Plus, comparing it against standard LLM and reasoning-model inference.

    “III. An empirical evaluation of the formalizer agent on PARARULE-Plus [24], comparing it against standard LLM and reasoning-model inference (Section 4).”

    From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [24]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 35

  1. [1][1] N. Cingillioglu, A. Russo, Deeplogic: Towards end-to-end differentiable logical reasoning, in: AAAI-MAKE, 2019. -
  2. [2][2] S. Russell, P. Norvig, E. Davis, Artificial intelligence: A modern approach (3rd (global) ed.), 2016. -
  3. [3][3] P. Clark, O. Tafjord, K. Richardson, Transformers as soft reasoners over language, in: IJCAI, 2020, pp. 3882-3890. -
  4. [4][4] J. Weston, A. Bordes, S. Chopra, T. Mikolov, Towards ai-complete question answering: A set of prerequisite toy tasks, in: ICLR, 2016. -
  5. [5][5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). - · arXiv
  6. [9][9] C. Liang, J. Berant, Q. Le, K. D. Forbus, N. Lao, Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision, in: ACL, 2017, pp. 23-33. -
  7. [10][10] S. Reed, N. De Freitas, Neural programmer-interpreters, arXiv preprint arXiv:1511.06279 (2015). - · arXiv
  8. [11][11] Z. Jiang, S. Luo, Neural logic reinforcement learning, in: ICML, PMLR, 2019, pp. 3110-3119. -
  9. [12][12] T. Rocktäschel, S. Riedel, End-to-end differentiable proving, in: NIPS, 2017, pp. 3788-3800. -
  10. [13][13] K. Richardson, H. Hu, L. S. Moss, A. Sabharwal, Probing natural language inference models through semantic fragments., in: AAAI, 2020, pp. 8713-8721. -
  11. [14][14] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, C. D. Manning, Hotpotqa: A dataset for diverse, explainable multi-hop question answering, arXiv preprint arXiv:1809.09600 (2018). - · arXiv
  12. [15][15] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: NAACL-HLT, 2019. -
  13. [16][16] N. Young, Q. Bao, J. Bensemann, M. Witbrock, AbductionRules: Training transformers to explain unexpected inputs, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Findings of the Association for Computational Linguistics: ACL 2022, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 218-227. URL: https://aclanthology.org/2022.findings-acl.19/. doi: 10.18653/v1/2022.findings-acl.19 . - · DOI
  14. [17][17] N. Tan, T. Nguyen, J. Bensemann, A. Peng, Q. Bao, Y. Chen, M. Gahegan, M. Witbrock, Multi2Claim: Generating scientific claims from multi-choice questions for scientific factchecking, in: A. Vlachos, I. Augenstein (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 2652-2664. URL: https:// aclanthology.org/2023.eacl-main.194/. doi: 10.18653/v1/2023.eacl-main.194 . - · DOI
  15. [18][18] Z. Wang, J. Liu, Q. Bao, H. Rong, J. Zhang, Chatlogic: Integrating logic programming with large language models for multi-step reasoning, in: 2024 International Joint Conference on Neural Networks (IJCNN), IEEE, 2024, pp. 1-8. -
  16. [19][19] Q. Bao, M. Witbrock, J. Liu, Natural language processing and reasoning, IEEE Vehicular Technology Society Invited Talk (2022). -
  17. [20][20] J. Bensemann, Q. Bao, G. Gendron, T. Hartill, M. Witbrock, Relating blindsight and ai: A review, Journal of Artificial Intelligence and Consciousness 09 (2022) 111-125. URL: https://doi.org/10.1142/S2705078521500156. doi: 10.1142/S2705078521500156 . arXiv:https://doi.org/10.1142/S2705078521500156 . - · DOI
  18. [21][21] G. Gendron, Q. Bao, M. Witbrock, G. Dobbie, Large language models are not strong abstract reasoners, arXiv preprint arXiv:2305.19555 (2023). - · arXiv
  19. [22][22] X. Xiao, S. Shen, Q. Bao, H. Rong, K. Liu, Z. Wang, J. Liu, Cora: Optimizing low-rank adaptation with common subspace of large language models, arXiv preprint arXiv:2409.02119 (2024). - · arXiv
  20. [23][23] N. Ö. Tan, A. Y. Peng, J. Bensemann, Q. Bao, T. Hartill, M. Gahegan, M. Witbrock, Inputlength-shortening and text generation via attention values, arXiv preprint arXiv:2303.07585 (2023). - · arXiv
  21. [24][24] Q. Bao, G. Gendron, A. Y. Peng, W. Zhong, N. Tan, Y. Chen, M. Witbrock, J. Liu, Assessing and enhancing the robustness of large language models with task structure variations for logical reasoning, arXiv preprint arXiv:2310.09430 (2023). - · arXiv
  22. [25][25] Q. Bao, M. Witbrock, J. Liu, From symbolic logic reasoning to soft reason-ing: A neuralsymbolic paradigm, The 1st New Zealand Workshop on Artificial Intelligence Research (NZAIR) (2021). -
  23. [26][26] Q. Bao, A. Peng, Z. Deng, W. Zhong, G. Gendron, T. Pistotti, N. Tan, N. Young, Y. Chen, Y. Zhu, P. Denny, M. Witbrock, J. Liu, Abstract Meaning Representation-based logicdriven data augmentation for logical reasoning, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 5914-5934. URL: https:// aclanthology.org/2024.findings-acl.353/. doi: 10.18653/v1/2024.findings-acl.353 . - · DOI
  24. [27][27] Q. Bao, J. Leinonen, A. Y. Peng, W. Zhong, G. Gendron, T. Pistotti, A. Huang, P. Denny, M. Witbrock, J. Liu, Exploring iterative enhancement for improving learnersourced multiple-choice question explanations with large language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025, pp. 28955-28963. -
  25. [28][28] Q. Bao, Developing And Assessing Language Models For Logical Reasoning Over Natural Language, Ph.D. thesis, University of Auckland, 2025. -
  26. [29][29] Q. Qi, Q. Bao, A. Y. Peng, J. Liu, M. Witbrock, A dynamic prompt-tuning method for data augmentation with associated knowledge, 2023. URL: https://openreview.net/forum?id= hli7A0ioiS\_. -
  27. [30][30] Q. Bao, L. Ni, J. Liu, Hhh: An online medical chatbot system based on knowledge graph and hierarchical bi-directional attention, in: Proceedings of the Australasian Computer Science Week Multiconference, ACSW '20, Association for Computing Machinery, New York, NY, USA, 2020. URL: https://doi.org/10.1145/3373017.3373049. doi: 10.1145/3373017. 3373049 . - · DOI
  28. [31][31] L. Ni, Q. Bao, X. Li, Q. Qi, P. Denny, J. Warren, M. Witbrock, J. Liu, Deepqr: Neural-based quality ratings for learnersourced multiple-choice questions, Proceedings of the AAAI Conference on Artificial Intelligence 36 (2022) 12826-12834. URL: https://ojs.aaai.org/index. php/AAAI/article/view/21562. doi: 10.1609/aaai.v36i11.21562 . - · DOI
  29. [32][32] J. Pennington, R. Socher, C. Manning, GloVe: Global vectors for word representation, in: EMNLP, Doha, Qatar, 2014, pp. 1532-1543. -
  30. [33][33] K. Cho, B. van Merriënboer, D. Bahdanau, Y. Bengio, On the properties of neural machine translation: Encoder-decoder approaches, in: SSST-8, 2014, pp. 103-111. -
  31. [34][34] C. Xiong, S. Merity, R. Socher, Dynamic memory networks for visual and textual question answering, in: ICML, PMLR, 2016, pp. 2397-2406. -
  32. [37][37] A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, V. Zhong, R. Paulus, R. Socher, Ask me anything: Dynamic memory networks for natural language processing, in: ICML, volume 48, 2016, pp. 1378-1387. -
  33. [38][38] D. A. Hudson, C. D. Manning, Compositional attention networks for machine reasoning, in: ICLR, 2018. -
  34. [39][39] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: ICLR, 2015. -
  35. [40][40] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, M. Auli, fairseq: A fast, extensible toolkit for sequence modeling, in: NAACL-HLT, 2019.

Cited by

Sorted by citations

Authors
Bao, Q., Peng, A. Y., Hartill, T., Tan, N., Deng, Z., Witbrock, M., & Liu, J.
Affiliations
  • University of Auckland (NZ)
  • University of Cambridge (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2207.14000
arXiv
2207.14000
OpenAlex
W4288804529
Cited by
2
References
0
Citations by year
2025: 1 · 2023: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • ResearchSpace (University of Auckland) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
natural language reasoning, out-of-distribution generalization, logic programs, gated attention, recurrent neural networks, rule-based reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review