ProofWriter: Generating implications, proofs, and abductive statements over natural language
Open accessAbstract
Transformers have been shown to emulate logical deduction over natural language theories (logical rules expressed in natural language), reliably assigning true/false labels to candidate implications.However, their ability to generate implications of a theory has not yet been demonstrated, and methods for reconstructing proofs of answers are imperfect.In this work we show that a generative model, called ProofWriter, can reliably generate both implications of a theory and the natural language proofs that support them.In particular, iterating a 1-step implication generator results in proofs that are highly reliable, and represent actual model decisions (rather than post-hoc rationalizations).On the RuleTaker dataset, the accuracy of ProofWriter's proofs exceed previous methods by +9% absolute, and in a way that generalizes to proof depths unseen in training and on out-of-domain problems.We also show that generative techniques can perform a type of abduction with high precision: Given a theory and an unprovable conclusion, identify a missing fact that allows the conclusion to be proved, along with a proof.These results significantly improve the viability of neural methods for systematically reasoning over natural language.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ProofWriter is a synthetic dataset created to assess deductive reasoning abilities of LLMs.
“ProofWriter (Tafjord et al., 2021) is a synthetic dataset created to assess deductive reasoning abilities of LLMs.”
From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Tafjord et al., 2021
The Abstract calls ProofWriter a generative model, not a dataset. Section 4, “Datasets,” says the authors evaluate ProofWriter using the RuleTaker datasets and two new variants. The paper therefore distinguishes ProofWriter from the datasets used to assess its reasoning. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredProofWriter is a synthetically generated dataset for logical reasoning over natural language.
“ProofWriter, meanwhile, is a synthetically generated dataset for logical reasoning over natural language.”
From LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023) · cites this paper as Tafjord et al., 2021
The Introduction calls ProofWriter “our system,” not a dataset. Section 4 says the system is evaluated using RuleTaker datasets and two new variants; those datasets contain natural-language reasoning questions. The claim confuses the system with the data used to evaluate it. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPrevious deductive reasoning benchmarks such as RuleTaker, ProofWriter, and ProntoQA studied proof depths up to five steps or hops.
“Early work on deductive reasoning benchmarks such as RuleTaker (Clark et al., 2020) and ProofWriter (Tafjord et al., 2021) established the basic setup of query answering over sets of rules and facts, with ProofWriter reaching a maximum proof depth of five steps.ProntoQA (Saparov and He, 2023) introduces proof-structure controls and formal error attribution, also evaluated up to five hops.”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Tafjord et al., 2021
These previous benchmarks revealed that models at the time struggle as proof depth increases, but the depths studied remain shallow.
“These benchmarks revealed that models at the time struggle as proof depth increases, but the depths studied remain shallow.”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Tafjord et al., 2021
A model that has learned a robust implementation of the proof-tree tracing algorithm should, in principle, generalize beyond the shallow depths seen in previous benchmarks.
“This process is algorithmic; a model that has learned a robust implementation of this algorithm should, in principle, generalize beyond the shallow depths seen in previous benchmarks (Clark et al., 2020;Tafjord et al., 2021;Saparov and He, 2023).”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Tafjord et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hobbs, J. R., Stickel, M. E., Appelt, D. E., & Martin, P. (1988) · Artificial Intelligence · doi:10.1016/0004-3702(93)90015-4 · Cited by 1,222 · see citing works
Applications of circumscription to formalizing common-sense knowledge
McCarthy, J. (1986) · Artificial Intelligence · doi:10.1016/0004-3702(86)90032-9 · Cited by 1,114 · see citing works
McCarthy, J. (1960) · Cited by 799 · see citing works
What you always wanted to know about Datalog (and never dared to ask)
Ceri, S., Gottlob, G., & Tanca, L. (1989) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.43410 · Cited by 704 · see citing works
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · see citing works
e-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 2× · see citing works
“Similarly, work on explanations has sought to gen-Proof erate human-style justifications, which again are typically supporting evidence rather than a fully-formed line of reasoning, and without explicit reasoning rules (Camburu et al., 2018; Jhamtani and Clark, 2020; Inoue et al., 2020).”
BackgroundLanguage Models as Knowledge Bases?Preprint
Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A., & Riedel, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01066 · Cited by 364 · see citing works
Transformers as Soft Reasoners over Language
Clark, P., Tafjord, O., & Richardson, K. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/537 · Cited by 176 · see citing works
Self-citationWT5?! Training Text-to-Text Models to Explain their PredictionsPreprint
Narang, S., Raffel, C., Lee, K., Roberts, A., Fiedel, N., & Malkan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14546 · Cited by 141 · see citing works
A Survey on Semantic ParsingPreprint
Kamath, A., & Das, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00978 · Cited by 83 · see citing works
Generative Language Modeling for Automated Theorem Proving
Polu, S., & Sutskever, I. (2020) · arXiv · doi:10.48550/arxiv.2009.03393 · Cited by 82 · see citing works
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language
Weber, L., Minervini, P., Münchmeyer, J., Leser, U., & Rocktäschel, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1618 · Cited by 81 · see citing works
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit KnowledgePreprint
Talmor, A., Tafjord, O., Clark, P. E., Goldberg, Y., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06609 · Cited by 65 · see citing works
Self-citationPRover: Proof Generation for Interpretable Reasoning over Rules
Saha, S., Ghosh, S., Srivastava, S., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.9 · Cited by 50 · see citing works
R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
Inoue, N., Stenetorp, P., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.602 · Cited by 43 · see citing works
Jhamtani, H., & Clark, P. E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.10 · Cited by 43 · see citing works
Self-citationMeasuring Systematic Generalization in Neural Proof Generation with TransformersPreprint
Gontier, N., Sinha, K., Reddy, S., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.14786 · Cited by 37 · see citing works
Obtaining Faithful Interpretations from Compositional Neural Networks
Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.495 · Cited by 36 · see citing works
Abductive theories in artificial intelligence
Konolige, K. (1997) · Center for the Study of Language and Information eBooks · Cited by 20 · see citing works
Learning to Prove Theorems by Learning to Generate TheoremsPreprint
Wang, M., & Deng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07019 · Cited by 17 · see citing works
Musen, M. A., & van der Lei, J. (1988) · Machine intelligence and pattern recognition · doi:10.1016/b978-0-444-87137-4.50029-1 · Cited by 13 · see citing works
Flagged first, then by citations
4 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, S. Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In ICLR'20.
- Tim Rocktäschel and S. Riedel. 2017. End-to-end dif- ferentiable proving. In NeurIPS.
- Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. 2019. Learning a SAT solver from single-bit supervision. In ICLR.
- Alon Talmor, Oyvind Tafjord, P. Clark, Y. Goldberg, and Jonathan Berant. 2020. LeapOfThought: Teach- ing pre-trained models to systematically reason over implicit knowledge. In NeurIPS.
Cited by
Sorted by citations
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“We randomly selected 300 examples from each subset of ProofWriter, ensuring a balanced label distribution.”
MethodInfluentialReasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 143 · see citing works
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 124 · see citing works
“Lastly, leveraging LLMs to generate each reasoning step iteratively via a selection and inference mechanism [16, 15, 54].”
BackgroundMethodFaith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023) · arXiv · Cited by 124 · see citing works
The formal rationality of artificial intelligence-based algorithms and the problem of bias
Nishant, R., Schneckenberg, D., & Ravishankar, M. N. (2023) · Journal of Information Technology · doi:10.1177/02683962231176842 · Cited by 109 · see citing works
Explaining Answers with Entailment Trees
Dalvi, B. B., Jansen, P. A., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.585 · Cited by 96 · see citing works
Self-citationPan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 76 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
A Survey on LLM-Based Code Generation for Low-Resource and Domain-Specific Programming Languages
Joel, S., Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770084 · Cited by 93 · see citing works
AI Reasoning in Deep Learning Era: From Symbolic AI to Neural–Symbolic AI
Liang, B., Wang, Y., & Tong, C. (2025) · Mathematics · doi:10.3390/math13111707 · Cited by 79 · see citing works
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57 · see citing works
“Datasets: Our experiments use tasks from two existing datasets: FOLIO (Han et al., 2022) and ProofWriter (Tafjord et al., 2021), both of which have been shown to be challenging for off-the-shelf LLMs (Han et al., 2022; Creswell et al., 2023).”
MethodUniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., & Liang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.218 · Cited by 51 · see citing works
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 50 · see citing works
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Kazemi, M., Kim, N., Bhatia, D. S., Xu, X., & Ramachandran, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.361 · Cited by 34 · see citing works
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
Hessel, J., Hwang, J. D., Park, J. S., Zellers, R., Bhagavatula, C., Rohrbach, A., Saenko, K., & Choi, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_32 · Cited by 29 · see citing works
Generating Natural Language Proofs with Verifier-Guided Search
Yang, K., Deng, J., & Chen, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.7 · Cited by 29 · see citing works
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Tafjord, O., Mishra, B. D., & Clark, P. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.134 · Cited by 31 · see citing works
Self-citationSay What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge
Chen, J., Shi, W., Fu, Z., Cheng, S., Li, L., & Xiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.550 · Cited by 30 · see citing works
Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
Sclar, M., Kumar, S., West, P., Suhr, A., Choi, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.780 · Cited by 27 · see citing works
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner
Ribeiro, D. N., Wang, S., Ma, X., Dong, R., Wei, X., Zhu, H., Chen, X., Xu, P., Huang, Z., Arnold, A., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.35 · Cited by 22 · see citing works
Demystifying Chains, Trees, and Graphs of Thoughts
Besta, M., Memedi, F., Zhang, Z., Gerstenberger, R., Piao, G., Blach, N., Nyczyk, P., Copik, M., Kwaśniewski, G., Müller, J., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3598182 · Cited by 34 · see citing works
Flexible Generation of Natural Language Deductions
Bostrom, K., Zhao, X., Chaudhuri, S., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.506 · Cited by 21 · see citing works
METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation
Hong, R., Zhang, H., Yu, X., & Zhang, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.145 · Cited by 21 · see citing works
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“Additionally, we release the predictions of all models used in our study to ensure that all re-ported results can be reproduced and verified: https://github.com/booydar/babilong/tree/ predictions_06_2024 .”
MethodFaiRR: Faithful and Robust Deductive Reasoning over Natural Language
Sanyal, S., Singh, H., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.77 · Cited by 20 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models
Saeed, M., Ahmadi, N., Nakov, P., & Papotti, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.110 · Cited by 16 · see citing works
Generative Multi-hop Retrieval
Lee, H., Yang, S., Oh, H., & Seo, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.92 · Cited by 20 · see citing works
Natural Language Deduction through Search over Statement Compositions
Bostrom, K., Sprague, Z., Chaudhuri, S., & Durrett, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.358 · Cited by 20 · see citing works
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
NaturalProofs: Mathematical Theorem Proving in Natural LanguagePreprint
Welleck, S., Liu, J., Bras, R. J. L., Hajishirzi, H., Choi, Y., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01112 · Cited by 25 · see citing works
AbductionRules: Training Transformers to Explain Unexpected Inputs
Young, N., Bao, Q., Bensemann, J., & Witbrock, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.19 · Cited by 13 · see citing works
Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability
Richardson, K., & Sabharwal, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21371 · Cited by 12 · see citing works
Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning
Mirzaee, R., & Kordjamshidi, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.413 · Cited by 11 · see citing works
Neural Unification for Logic Reasoning over Natural Language
Picco, G., Hoang, T. L., Sbodio, M. L., & López, V. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.331 · Cited by 10 · see citing works
RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees
Liu, T., Guo, Q., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.483 · Cited by 11 · see citing works
IAG: Induction-Augmented Generation Framework for Answering Reasoning Questions
Zhang, Z., Zhang, X., Ren, Y., Shi, S., Han, M., Wu, Y., Lai, R., & Cao, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1 · Cited by 10 · see citing works
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8 · see citing works
LoRP: LLM-based Logical Reasoning via Prolog
Di, Z., Zhang, C., Lv, H., Cui, L., & Liu, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114140 · Cited by 11 · see citing works
MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation
Saha, S., Yu, X., Bansal, M., Pasunuru, R., & Çelikyılmaz, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.704 · Cited by 8 · see citing works
Explicit Planning Helps Language Models in Logical Reasoning
Zhao, H., Wang, K., Yu, M., & Mei, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.688 · Cited by 11 · see citing works
Learning Symbolic Rules for Reasoning in Quasi-Natural LanguagePreprint
Yang, K., & Deng, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12038 · Cited by 8 · see citing works
Lee, J., & Lee, G. G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113091 · Cited by 8 · see citing works
Yao, B., Jindal, I., Popa, L. C., Katsis, Y., Ghosh, S., He, L., Lu, Y., Srivastava, S., Li, Y., Hendler, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.778 · Cited by 7 · see citing works
Aghahadi, Z., & Talebpour, A. (2022) · Journal of Applied Non-Classical Logics · doi:10.1080/11663081.2022.2041352 · Cited by 7 · see citing works
Interpretable Proof Generation via Iterative Backward Reasoning
Qu, H., Cao, Y. H., Gao, J., Ding, L., & Xu, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.216 · Cited by 7 · see citing works
Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations
Zhao, W., Chiu, J., Cardie, C., & Rush, A. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.831 · Cited by 8 · see citing works
RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners
Sanyal, S., Liao, Z., & Ren, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.653 · Cited by 11 · see citing works
Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs
Richardson, K., Tamari, R., Sultan, O., Shahaf, D., Tsarfaty, R., & Sabharwal, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.658 · Cited by 6 · see citing works
Saha, S., Yadav, P., & Bansal, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.85 · Cited by 7 · see citing works
AlKhamissi, B., Verma, S. S., Yu, P., Jin, Z., Çelikyılmaz, A., & Diab, M. (2023) · doi:10.18653/v1/2023.nlrse-1.10 · Cited by 6 · see citing works
Betz, G., & Richardson, K. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.2 · Cited by 6 · see citing works
Can Pretrained Language Models (Yet) Reason Deductively?
Yuan, Z., Hu, S., Vulić, I., Korhonen, A., & Meng, Z. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.106 · Cited by 7 · see citing works
PAT-Agent: Autoformalization for Model Checking
Zuo, X., Zhang, Y., Wang, H., Cai, Y., Hóu, Z., Sun, J., & Dong, J. S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00176 · Cited by 5 · see citing works
Graph of Logic: Enhancing LLM Reasoning with Graphs and Symbolic Logic
Alotaibi, F., Kulkarni, A., & Zhou, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825450 · Cited by 5 · see citing works
ALERT: Adapt Language Models to Reasoning Tasks
Yu, P., Wang, T., Golovneva, O., AlKhamissi, B., Verma, S. S., Jin, Z., Ghosh, G., Diab, M., & Çelikyılmaz, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.60 · Cited by 14 · see citing works
MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure
Huang, Y., Zhang, H., Hong, R., Liang, X., Zhang, C., & Yu, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.310 · Cited by 5 · see citing works
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
Sanyal, S., Yi‐chong, X., Wang, S., Yang, Z., Pryzant, R., Yu, W., Zhu, C., & Ren, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.347 · Cited by 6 · see citing works
Li, Y., Fu, X., Verma, G., Buitelaar, P., & Liu, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24476 · Cited by 4 · see citing works
Haji, S., Suekane, K., Sano, H., & Takagi, T. (2023) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc56153.2023.00036 · Cited by 4 · see citing works
World Models for Math Story Problems
Opedal, A., Stoehr, N., Saparov, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.579 · Cited by 8 · see citing works
Reliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3 · see citing works
“This method has shown significant performance gains on the FOLIO [16] and ProofWriter [34]”
REASON: Accelerating Probabilistic Logical Reasoning for Scalable Neuro-Symbolic Intelligence
Wan, Z.-S., Liu, C.-K., Qian, J., Yang, H.-C., Raychowdhury, A., & Krishna, T. (2026) · International Symposium on High-Performance Computer Architecture · doi:10.1109/hpca68181.2026.11408458 · Cited by 2 · see citing works
Betz, G., & Richardson, K. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281372 · Cited by 4 · see citing works
Experiments with LLMs for Converting Language to Logic
Tammet, T., Järv, P., Verrev, M., & Draheim, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_24 · Cited by 3 · see citing works
Towards General Natural Language Understanding with Probabilistic Worldbuilding
Saparov, A., & Mitchell, T. M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00463 · Cited by 5 · see citing works
Porada, I., Sordoni, A., & Cheung, J. C. K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.337 · Cited by 3 · see citing works
Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inference
Thayaparan, M., Valentino, M., Ferreira, D. M., Rozanova, J., & Freitas, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00508 · Cited by 3 · see citing works
DCoL-A: Agentic dual chain of thinking helps LLMs pretend logic solvers
Chen, M., Wu, L.-I., Liu, R., Chang, X., Xue, J., & Li, G. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103780 · Cited by 2 · see citing works
Betz, G., & Richardson, K. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.900943 · Cited by 2 · see citing works
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Hou, Y., Li, J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.299 · Cited by 10 · see citing works
Identifying the limits of transformers when performing model-checking with natural language
Madusanka, T., Batista-Navarro, R., & Pratt-Hartmann, I. E. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.257 · Cited by 2 · see citing works
Madusanka, T., Zahid, I., Li, H., Pratt-Hartmann, I. E., & Batista-Navarro, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.536 · Cited by 2 · see citing works
A Generative Symbolic Model for More General Natural Language Understanding and ReasoningPreprint
Saparov, A., & Mitchell, T. M. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Large-Scale Commonsense Knowledge for Default Logic Reasoning
Järv, P., Tammet, T., Verrev, M., & Draheim, D. (2023) · SN Computer Science · doi:10.1007/s42979-023-01963-2 · Cited by 1 · see citing works
Can Language Models Pretend Solvers? Logic Code Simulation with LLMs
Chen, M., Li, G., Wu, L.-I., Liu, R., Su, Y., Chang, X., & Xue, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0602-3_6 · Cited by 3 · see citing works
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1 · see citing works
Think straight or think again? Continual joint learning of deduction, abduction and induction
Xiong, K., Ding, X., Cao, Y., Zhao, Y., Liu, T., & Qin, B. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107768 · Cited by 1 · see citing works
Question Answering with LLMs and Learning from Answer Sets
Borroto, M., GALLAGHER, K., Ielo, A., KAREEM, I., Ricca, F., & Russo, A. (2025) · Theory and Practice of Logic Programming · doi:10.1017/s1471068425100343 · Cited by 1 · see citing works
EMRDCM : An experience-enhanced multi-role debate method based on compromise mechanisms
Zhang, Y., Feng, J., Yiming, X., Wang, Z., Yang, T., & Wang, X. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01022-1 · Cited by 1 · see citing works
DL-ReasonSuite: A Benchmark for Evaluating Description Logic Reasoning in Large Language Models
Olucoglu, M., & Bursa, O. (2026) · Applied Sciences · doi:10.3390/app16041821 · Cited by 1 · see citing works
LogiCue: Targeted Prompting for Improved Modal and Conditional Reasoning in Large Language Models
Shahrokhshahi, F., Mohammadi, F., & Sönmez, F. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.01.057 · Cited by 1 · see citing works
Large Language Model for OWL Proofs
Yang, H., Chen, J., & Sattler, U. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792395 · Cited by 1 · see citing works
Neuro-symbolic NLP: taxonomy, assessment, and directions
Chatzikyriakidis, S., & Lappin, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1797587 · Cited by 1 · see citing works
Investigating Transformer-Guided Chaining for Interpretable Natural Logic Reasoning
Kanagasabai, R., Rajamanickam, S., & Shi, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.588 · Cited by 1 · see citing works
MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning
Li, Y., Tian, J., Fan, C., Chen, W., He, H., & Jin, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.640 · Cited by 1 · see citing works
Transformers in the Service of Description Logic-Based Contexts
Poulis, A., Tsalapati, E., & Koubarakis, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_20 · Cited by 1 · see citing works
Advances in reasoning by prompting large language models: A survey
Hong, R., Pang, X., & Zhang, C. (2026) · Cybernetics and Intelligence · doi:10.26599/cai.2024.9390004 · Cited by 1 · see citing works
Language-Based Syllogistic Reasoning Using Deep Neural Networks
Aghahadi, Z., & Talebpour, A. (2022) · Cognitive Semantics · doi:10.1163/23526416-bja10026 · Cited by 1 · see citing works
Empirical Investigation of Neural Symbolic Reasoning Strategies
Aoki, Y., Kudo, K., Kuribayashi, T., Brassard, A., Yoshikawa, M., Sakaguchi, K., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.86 · Cited by 1 · see citing works
Evaluating and Improving the Reasoning Abilities of Language Models
Helwe, C. (2024) · theses.fr (ABES) · doi:10.70675/7fc70e59zdd27z4b0fza38dz03a2c67cda50 · Cited by 1 · see citing works
Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog
Al-Negheimish, H., Ilieva, J., & Kim, Y. (2026) · arXiv · doi:10.48550/arxiv.2610.11592
“This process is algorithmic; a model that has learned a robust implementation of this algorithm should, in principle, generalize beyond the shallow depths seen in previous benchmarks (Clark et al., 2020; Tafjord et al., 2021; Saparov and He, 2023).”
Neural Unification for Logic Reasoning over Natural LanguagePreprint
Picco, G., Lam, H. T., Sbodio, M. L., & Garcia, V. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08460 · Cited by 1 · see citing works
ProofInfer: Generating Proof via Iterative Hierarchical Inference
Fei, Z., Zhang, Q., Zhou, X., Gui, T., & Huang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.747
Natural Language Annotations for Reasoning about Program Semantics
Zocca, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.601
Zhang, Z., Storks, S., Hu, F., Sohn, S., Lee, M., Lee, H., & Chai, J. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.456 · Cited by 1 · see citing works
- Authors
- Tafjord, O., Dalvi, B., & Clark, P.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Clark, Peter
- Venue
- Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
- Published
- 2021
- Pages
- 3621-3634
- OpenAlex
- W3115201256
- Cited by
- 122
- References
- 25
- Citations by year
- 2026: 22 · 2025: 18 · 2024: 14 · 2023: 32 · 2022: 27 · 2021: 9
- FWCI
- 9.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- proof generation, abductive reasoning, large language models, generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
