Cited by 1Logic, Reasoning, and KnowledgeNatural Language Processing TechniquesAI-based Problem Solving and Planning
SLR: Automated Synthesis for Scalable Logical Reasoning
Open accessHelff, Lukas, Omar, Ahmad, Friedrich, Felix, Wüst, Antonia, Shindo, Hikaru, Mitchell, Rupert, Woydt, Tim, Schramowski, Patrick +2 more·2026·Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Abstract
Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Crossref · 10.18653/v1/2026.acl-long.16
Cites
0 citation errors · 0 not checkable
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complexity groupings follow the binning used in the original SLR-Bench paper.
“Complexity groupings follow the binning used in the original SLR-Bench paper [Helff et al., 2025].”
From ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis (2026) · cites this paper as Helff et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localiza- tion, text reading, and beyond. arXiv preprint arXiv:2308.12966. · arXiv
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. · arXiv
- Quentin Delfosse, Jannis Blüml, Fabian Tatai, Théo Vincent, Bjarne Gregori, Elisabeth Dillies, Jan Pe- ters, Constantin Rothkopf, and Kristian Kersting. 2025. Deep reinforcement learning agents are not even close to human intelligence. Preprint, arXiv:2505.21731. · arXiv
- Long Phan et.al. 2025. Humanity's last exam. Preprint, arXiv:2501.14249. · arXiv
- Siqi Fan, Peng Han, Shuo Shang, Yequan Wang, and Aixin Sun. 2025. Cothink: Token-efficient reason- ing via instruct models guiding reasoning models. Preprint, arXiv:2505.22017. · arXiv
- Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Eka- terina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Mal- colm Sailor, Ansong Ni, Linyong Nan, Jungo Ka- sai, Tao Yu, and 7 others. 2022. Folio: Natural lan- guage reasoning with first-order logic. arXiv preprint arXiv:2209.00840. · arXiv
- Lukas Helff, Wolfgang Stammer, Hikaru Shindo, De- vendra Singh Dhami, and Kristian Kersting. 2025. V-lol: A diagnostic dataset for visual logical learning. Journal of Data-centric Machine Learning Research.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
- Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaib- hav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, and Orhan Firat. 2025. Big-bench extra hard. Preprint, arXiv:2502.19187. · arXiv
- Jiin Kim, Byeongjun Shin, Jinha Chung, and Minsoo Rhu. 2025. The cost of dynamic reasoning: Demysti- fying ai agents and test-time scaling from an ai infras- tructure perspective. Preprint, arXiv:2506.04301. · arXiv
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
- Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Py- atkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with chal- lenging tasks from real users in the wild. Preprint, arXiv:2406.04770. · arXiv
- Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, and Junxian He. 2025. Syn- logic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. Preprint, arXiv:2505.19641. · arXiv
- John W Lloyd. 2012. Foundations of logic program- ming. Springer Berlin, Heidelberg.
- Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, and Chitta Baral. 2024. Towards logiglue: A brief survey and a benchmark for analyzing logical rea- soning capabilities of language models. Preprint, arXiv:2310.00836. · arXiv
- Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2024. Kor-bench: Benchmarking lan- guage models on knowledge-orthogonal reasoning tasks. Preprint, arXiv:2410.06526. · arXiv
- Tom Michael Mitchell. 1997. Machine learning, inter- national edition. In McGraw-Hill Series in Computer Science.
- Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. 2024. Enhancing reasoning capabilities of llms via principled synthetic logic cor- pus. In Annual Conference on Neural Information Processing Systems.
- Stephen Muggleton and Luc de Raedt. 1994. Induc- tive logic programming: Theory and methods. The Journal of Logic Programming, 19-20:629-679.
- Bhrij Patel, Souradip Chakraborty, Wesley A. Sut- tle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha. 2024a. Aime: Ai system optimization via multiple llm evaluators. Preprint, arXiv:2410.03131. · arXiv
- Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. 2024b. Multi-logieval: Towards eval- uating multi-step logical reasoning ability of large language models. Preprint, arXiv:2406.17169. · arXiv
- Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, and 1 others. 2025. Codeelo: Benchmarking competition-level code gen- eration of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257. · arXiv
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
- Abulhair Saparov and He He. 2023. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations.
- DeepSeek-AI Team. 2025. Deepseek-r1: Incentiviz- ing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. · arXiv
- Meta Team. 2024a. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv: Artificial Intelligence.
- Antonia Wüst, Tim Tobiasch, Lukas Helff, Inga Ibs, Wolfgang Stammer, Devendra S Dhami, Con- stantin A Rothkopf, and Kristian Kersting. 2025. Bongard in wonderland: Visual puzzles that still make ai go mad? In Proceedings of the 42nd Inter- national Conference on Machine Learning (ICML).
- Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2024. On memorization of large language models in logical reasoning. Preprint, arXiv:2410.23123. · arXiv
- Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhi- rong Wu, and Chong Luo. 2025. Logic-rl: Un- leashing llm reasoning with rule-based reinforcement learning. Preprint, arXiv:2502.14768. · arXiv
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
- Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics.
- Yujun Zhou, Jiayi Ye, Zipeng Ling, Yufei Han, Yue Huang, Haomin Zhuang, Zhenwen Liang, Kehan Guo, Taicheng Guo, Xiangqi Wang, and Xiangliang Zhang. 2025. Dissecting logical reasoning in llms: A fine-grained evaluation and supervision study. Preprint, arXiv:2506.04810. · arXiv
Cited by
Sorted by citations
ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
Naik, A., Mathur, Y., Prakam, Rose, C., & Mortensen, D. (2026) · arXiv · doi:10.48550/arxiv.2605.05485
“We evaluate R EA C OMP on recent, challenging benchmarks — PBEBench [Naik et al., 2025b] and SLR-Bench [Helff et al., 2025] — and show that induced symbolic solvers achieve strong standalone performance while significantly improving inference efficiency.”
- Authors
- Helff, L., Omar, A., Friedrich, F., Wüst, A., Shindo, H., Mitchell, R., Woydt, T., Schramowski, P., Stammer, W., & Kersting, K.
- Affiliations
- Technische Universität Darmstadt (DE)
- Venue
- Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2026
- Pages
- 402-426
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W7166856071
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- Bundesministerium für Bildung und Forschung (16IS22091, 01IS22091)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- automated synthesis, symbolic reasoning, knowledge representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
