The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Transformers as Soft Reasoners over Language
Open accessAbstract
Beginning with McCarthy's Advice Taker (1959), AI has pursued the goal of providing a system with explicit, general knowledge and having the system reason over that knowledge. However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research. This paper investigates a modern approach to this problem where the facts and rules are provided as natural language sentences, thus bypassing a formal representation. We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call RuleTakers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language. These findings are significant as it suggests a new role for transformers, namely as limited "soft theorem provers" operating over explicit theories in language. This in turn suggests new possibilities for explainability, correctability, and counterfactual reasoning in question-answering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This process is algorithmic; a model that has learned a robust implementation of this algorithm should, in principle, generalize beyond the shallow depths seen in previous benchmarks (Clark et al., 2020;Tafjord et al., 2021;Saparov and He, 2023).”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The PARARULE-Plus dataset [24] is a large-scale synthetic dataset for multi-step deductive reasoning over natural language, extending the original PARARULE [32] with deeper inference chains.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [32]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Unlike synthetically templated logic datasets (Han et al., 2022;Clark et al., 2020) in which all premises are stated as short clauses, NLR problems have explicit statements along with implicit information that must be inferred from common sense and context.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Clark et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Navigate, like many other benchmark datasets such as RuleTaker (Clark et al., 2020), is systematically generated.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Clark et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“These benchmarks revealed that models at the time struggle as proof depth increases, but the depths studied remain shallow.”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Clark et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Early work on deductive reasoning benchmarks such as RuleTaker (Clark et al., 2020) and ProofWriter (Tafjord et al., 2021) established the basic setup of query answering over sets of rules and facts, with ProofWriter reaching a maximum proof depth of five steps.ProntoQA (Saparov and He, 2023) introduces proof-structure controls and formal error attribution, also evaluated up to five hops.”
From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
ASH, R. B. (1966) · Cited by 3,990
NIPS 2016 Tutorial: Generative Adversarial NetworksPreprint
Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.00160 · Cited by 2,048
A Decomposable Attention Model for Natural Language Inference
Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1244 · Cited by 1,351
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
Towards a Theory of Declarative Knowledge
Apt, K. R., Blair, H. A., & Walker, A. (1988) · Foundations of Deductive Databases and Logic Programming · doi:10.1016/b978-0-934613-40-8.50006-3 · Cited by 1,151
Applications of circumscription to formalizing common-sense knowledge
McCarthy, J. (1986) · Artificial Intelligence · doi:10.1016/0004-3702(86)90032-9 · Cited by 1,114
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
The logic theory machine-A complex information processing system
Newell, A., & Simon, H. (1956) · IEEE Transactions on Information Theory · doi:10.1109/tit.1956.1056797 · Cited by 846
Research works printed but not linked: 5
- [Abdelaziz et al. , 2020] Ibrahim Abdelaziz, Veronika Thost, Maxwell Crouse, and Achille Fokoue. An experimental study of formula embeddings for automated theorem proving in first-order logic. arXiv , 2002.00423, 2020. · arXiv
- [Richardson and Sabharwal, 2019] Kyle Richardson and Ashish Sabharwal. What does my qa model know? devising controlled probes using expert knowledge. ArXiv , abs/1912.13337, 2019. · arXiv
- [Saxton et al. , 2019] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In ICLR , 2019.
- [Selsam et al. , 2019] Daniel Selsam, Matthew Lamm, Benedikt B¨ unz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning a SAT solver from single-bit supervision. In ICLR , 2019.
- [Talmor et al. , 2019] A. Talmor, Y. Elazar, Y. Goldberg, and J. Berant. oLMpics - on what language model pre-training captures. ArXiv , abs/1912.13283, 2019. · arXiv
Cited by
Sorted by citations
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
“Another line of work aims to enable language models to perform logical or formal reasoning, often by verablizing the rules in natural language formal rules using language (Clark et al., 2020; Saeed et al., 2021; Liang et al., 2021).”
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationA comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 590
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248
Interactive computer-aided diagnosis on medical image using large language models
Wang, S., Zhao, Z., Ouyang, X., Liu, T., Wang, Q., & Shen, D. G. (2024) · Communications Engineering · doi:10.1038/s44172-024-00271-8 · Cited by 157
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
“Recent works in adapting large language models for logical reasoning tasks can be broadly categorized into two groups: 1) fine-tuning approaches that optimize LLMs’ reasoning ability through fine-tuning or training specialized modules (Clark et al., 2020; Tafjord et al., 2022; Yang et al., 2022), and 2) incontext lear…”
BackgroundReasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 143
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134
“There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
BackgroundFaith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 335
“[14] Peter Clark, Oyvind Tafjord, and Kyle Richardson.”
BackgroundFaith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023) · arXiv · Cited by 124
ProofWriter: Generating implications, proofs, and abductive statements over natural language
Tafjord, O., Dalvi, B., & Clark, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.317 · Cited by 122
Self-citationLevy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119
HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
Jiang, Y., Bordia, S., Zhong, Z., Dognin, C., Singh, M., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.309 · Cited by 173
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectPreprint
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01958 · Cited by 92
Self-citationGenerating Role-Playing Game Quests With GPT Language Models
Värtinen, S., Hämäläinen, P., & Guckelsberger, C. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3228480 · Cited by 98
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 164
Explanations for CommonsenseQA: New Dataset and Models
Aggarwal, S., Mandowara, D., Agrawal, V., Khandelwal, D., Singla, P., & Garg, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.238 · Cited by 85
On the Paradox of Learning to Reason from Data
Zhang, H., Li, L. H., Meng, T., Chang, K.-W., & Broeck, G. V. D. (2022) · arXiv · doi:10.48550/arxiv.2205.11502 · Cited by 58
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Świątczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 59
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
PRover: Proof Generation for Interpretable Reasoning over Rules
Saha, S., Ghosh, S., Srivastava, S., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.9 · Cited by 53
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 56
On the Paradox of Learning to Reason from Data
Zhang, H., Li, L. H., Meng, T., Chang, K.-W., & Van den Broeck, G. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/375 · Cited by 58
Exploring Length Generalization in Large Language Models
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2207.04901 · Cited by 44
“Tasks that have a reasoning component are especially well represented in this category — multi-hop reasoning [7], program execution [8], deductive reasoning [9] and theorem proving [10], to name a few.”
BackgroundKnowledge-Aware Language Model PretrainingPreprint
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 84
Are Pretrained Language Models Symbolic Reasoners over Knowledge?
Kassner, N., Krojer, B., & Schütze, H. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.45 · Cited by 57
Yoran, O., Talmor, A., & Berant, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.416 · Cited by 40
REFINER: Reasoning Feedback on Intermediate RepresentationsPreprint
Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosselut, A., West, R., & Faltings, B. V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.01904 · Cited by 35
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 40
Self-citationRelational World Knowledge Representation in Contextual Language Models: A Review
Safavi, T., & Koutra, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.81 · Cited by 52
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts
Liu, H., Cui, L., Liu, J., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17580 · Cited by 41
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language InstructionsPreprint
Mishra, S. R., Khashabi, D., Baral, C. R., & Hajishirzi, H. (2021) · arXiv (Cornell University) · Cited by 32
MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning
Jiao, F., Guo, Y., Song, X., & Nie, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.276 · Cited by 35
Generating Natural Language Proofs with Verifier-Guided Search
Yang, K., Deng, J., & Chen, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.7 · Cited by 55
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
Zhou, P., Khanna, R., Lee, S., Lin, B. Y., Ho, D., Pujara, J., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.598 · Cited by 41
From F to A on the New York Regents Science Exams — An Overview of the Aristo Project
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2020) · AI Magazine · doi:10.1609/aimag.v41i4.5304 · Cited by 143
Self-citationEntailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Tafjord, O., Mishra, B. D., & Clark, P. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.134 · Cited by 35
Self-citationBeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
Kassner, N., Tafjord, O., Schütze, H., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.697 · Cited by 47
Self-citationSay What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge
Chen, J., Shi, W., Fu, Z., Cheng, S., Li, L., & Xiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.550 · Cited by 35
Logiqa 2.0—An improved dataset for logical reasoning in natural language understanding
Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., & Zhang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3293046 · Cited by 26
Gómez-Pérez, J. M., & Ortega, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.441 · Cited by 27
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI
Tian, J., Li, Y., Chen, W., Xiao, L., He, H., & Jin, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.303 · Cited by 41
Chinese legal judgment prediction via knowledgeable prompt learning
Sun, J., Huang, S., & Wei, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122177 · Cited by 25
Flexible Generation of Natural Language Deductions
Bostrom, K., Zhao, X., Chaudhuri, S., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.506 · Cited by 25
Robust and explainable identification of logical fallacies in natural language arguments
Sourati, Z., Venkatesh, V. P. P., Deshpande, D. G., Rawlani, H., Ilievski, F., Sandlin, H.-Â., & Mermoud, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110418 · Cited by 24
METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation
Hong, R., Zhang, H., Yu, X., & Zhang, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.145 · Cited by 21
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19
“Additionally, we release the predictions of all models used in our study to ensure that all re-ported results can be reproduced and verified: https://github.com/booydar/babilong/tree/ predictions_06_2024 .”
MethodLiu, Q., Han, S., Cambria, E., Li, Y., & Kwok, K. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10345-6 · Cited by 21
Ortloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 26
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
Saha, S., Yadav, P., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.287 · Cited by 19
FaiRR: Faithful and Robust Deductive Reasoning over Natural Language
Sanyal, S., Singh, H., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.77 · Cited by 26
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
Zhang, H., Huang, J., Li, Z., Naik, M., & Xing, E. P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.191 · Cited by 31
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models
Saeed, M., Ahmadi, N., Nakov, P., & Papotti, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.110 · Cited by 16
Generative Multi-hop Retrieval
Lee, H., Yang, S., Oh, H., & Seo, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.92 · Cited by 26
LLMs for legal reasoning: A unified framework and future perspectives
Nguyen, H.-T., Fungwacharakorn, W., Zin, M. M., Goebel, R. G., Toni, F., Stathis, K., & Satoh, K. (2025) · Computer law & security review · doi:10.1016/j.clsr.2025.106165 · Cited by 26
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., & Chang, K.-W. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.364 · Cited by 25
GNN is a Counter? Revisiting GNN for Question AnsweringPreprint
Wang, K., Zhang, Y., Yang, D., Song, L., & Qin, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03192 · Cited by 16
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 32
Self-citationNaturalProofs: Mathematical Theorem Proving in Natural LanguagePreprint
Welleck, S., Liu, J., Bras, R. J. L., Hajishirzi, H., Choi, Y., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01112 · Cited by 60
Exploring Transitivity in Neural NLI Models through Veridicality
Yanaka, H., Mineshima, K., & Inui, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.78 · Cited by 13
AbductionRules: Training Transformers to Explain Unexpected Inputs
Young, N., Bao, Q., Bensemann, J., & Witbrock, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.19 · Cited by 19
A BERT-based deontic logic learner
Sun, J., Huang, S., & Wei, C. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103374 · Cited by 15
Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning
Wang, C., Zheng, B., Niu, Y., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_61 · Cited by 15
Critical Thinking for Language ModelsPreprint
Betz, G., Voigt, C., & Richardson, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07185 · Cited by 18
Self-citationPushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability
Richardson, K., & Sabharwal, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21371 · Cited by 18
Self-citationMachine Reasoning: Technology, Dilemma and Future
Duan, N., Tang, D., & Zhou, M. (2020) · doi:10.18653/v1/2020.emnlp-tutorials.1 · Cited by 14
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge.Preprint
Talmor, A., Tafjord, O., Clark, P. E., Goldberg, Y., & Berant, J. (2020) · arXiv (Cornell University) · Cited by 13
Self-citationAnalyzing Commonsense Emergence in Few-shot Knowledge ModelsPreprint
Da, J., Bras, R. J. L., Lu, X., Choi, Y., & Bosselut, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00297 · Cited by 30
Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning
Mirzaee, R., & Kordjamshidi, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.413 · Cited by 26
Case-Based Reasoning with Language Models for Classification of Logical Fallacies
Sourati, Z., Ilievski, F., Sandlin, H.-Â., & Mermoud, A. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/576 · Cited by 25
FAQ-Based Question Answering via Knowledge Anchors
Xie, R., Lu, Y., Lin, F., & Lin, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_1 · Cited by 11
Neural Unification for Logic Reasoning over Natural Language
Picco, G., Hoang, T. L., Sbodio, M. L., & López, V. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.331 · Cited by 11
Yu, S. (2023) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3616961.3616965 · Cited by 16
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Sileo, D., & Lernould, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.303 · Cited by 22
What About the Precedent: An Information-Theoretic Analysis of Common Law
Valvoda, J., Pimentel, T., Stoehr, N., Cotterell, R., & Teufel, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.181 · Cited by 11
Inferring Implicit Relations in Complex Questions with Language Models
Katz, U., Geva, M., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.188 · Cited by 13
Natural Language Deduction with Incomplete Information
Sprague, Z., Bostrom, K., Chaudhuri, S., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.564 · Cited by 15
Exploring formal defeasible reasoning of large language models: A Chain-of-Thought approach
Li, Z., Chen, C., Li, M., & Liao, B. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113564 · Cited by 9
End-to-end Case-Based Reasoning for Commonsense Knowledge Base Completion
Yang, Z., Du, X., Cambria, E., & Cardie, C. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.255 · Cited by 9
Explainable Multi-hop Verbal Reasoning Through Internal Monologue
Liang, Z., Bethard, S. J., & Surdeanu, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.97 · Cited by 12
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8
LoRP: LLM-based Logical Reasoning via Prolog
Di, Z., Zhang, C., Lv, H., Cui, L., & Liu, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114140 · Cited by 14
Explicit Planning Helps Language Models in Logical Reasoning
Zhao, H., Wang, K., Yu, M., & Mei, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.688 · Cited by 15
Finlayson, M., Richardson, K., Sabharwal, A., & Clark, P. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.27 · Cited by 10
Self-citationLearning Symbolic Rules for Reasoning in Quasi-Natural LanguagePreprint
Yang, K., & Deng, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12038 · Cited by 8
Biomedical text normalization through generative modeling
Berkowitz, J. S., Srinivasan, A., Cortina, J. M. A., Fatapour, Y., & Tatonetti, N. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104850 · Cited by 10
Coursey, K. (2020) · Springer eBooks · doi:10.1007/978-3-658-29864-7_3 · Cited by 7
Aghahadi, Z., & Talebpour, A. (2022) · Journal of Applied Non-Classical Logics · doi:10.1080/11663081.2022.2041352 · Cited by 9
Interpretable Proof Generation via Iterative Backward Reasoning
Qu, H., Cao, Y. H., Gao, J., Ding, L., & Xu, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.216 · Cited by 7
RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners
Sanyal, S., Liao, Z., & Ren, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.653 · Cited by 18
Learning Paradigms for Hybrid Decision-Making Systems
Punzi, C., Pellungrini, R., Setzu, M., Giannotti, F., & Pedreschi, D. (2026) · ACM Computing Surveys · doi:10.1145/3802522 · Cited by 11
AlKhamissi, B., Verma, S. S., Yu, P., Jin, Z., Çelikyılmaz, A., & Diab, M. (2023) · doi:10.18653/v1/2023.nlrse-1.10 · Cited by 8
Betz, G., & Richardson, K. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.2 · Cited by 7
Self-citationCan Pretrained Language Models (Yet) Reason Deductively?
Yuan, Z., Hu, S., Vulić, I., Korhonen, A., & Meng, Z. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.106 · Cited by 8
Conversational Neuro-Symbolic Commonsense Reasoning
Arabshahi, F., Lee, J., Gawarecki, M., Mazaitis, K., Azaria, A., & Mitchell, T. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16623 · Cited by 5
Discourse-Aware Graph Networks for Textual Logical Reasoning
Huang, Y., Liu, L., Xu, K., Fang, M., Lin, L., & Liang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3280178 · Cited by 7
Metadata
- Authors
- Clark, P., Tafjord, O., & Richardson, K.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
- Published
- 2020-07
- Pages
- 3882-3890
- OpenAlex
- W3006188107
- Cited by
- 176
- References
- 41
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 15 · 2023: 41 · 2022: 37 · 2021: 37 · 2020: 18
- FWCI
- 12.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language reasoning, transformer models, synthetic training data, rule-based reasoning, knowledge representation, theorem proving, question answering, explainability, counterfactual reasoning, knowledge distillation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
