The DOI matches its DataCite record; 26 of the 26 references it cites were found in a registry or an index.
Solving Quantitative Reasoning Problems with Language Models
Open accessAbstract
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Wei et al. [4], Nye et al. [15] and Lewkowycz et al. [5] showed that combining prompting (i.e. in-context learning) with scratchpad strategies present a powerful combination.”
From Exploring Length Generalization in Large Language Models (2022) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Alemi, A. A., Chollet, F., Een, N., Irving, G., Szegedy, C., and Urban, J. (2016). Deepmath - Deep Sequence Models for Premise Selection. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, pages 2243-2251, USA. Curran Associates Inc.
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.
- Grabowski, A., Kornilowicz, A., and Naumowicz, A. (2010). Mizar in a nutshell. J. Formalized Reasoning , 3(2):153-245.
- Han, J. M., Rute, J., Wu, Y., Ayers, E., and Polu, S. (2022). Proof artifact co-training for theorem proving with language models. In International Conference on Learning Representations .
- Kaliszyk, C., Urban, J., Michalewski, H., and Olšák, M. (2018). Reinforcement learning of theorem proving. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.
- Li, W., Yu, L., Wu, Y., and Paulson, L. C. (2021). Isarstep: a benchmark for high-level mathematical reasoning. In ICLR .
- Megill, N. D. and Wheeler, D. A. (2019). Metamath: A Computer Language for Pure Mathematics . Lulu Press, Morrisville, North Carolina. http://us.metamath.org/downloads/metamath.pdf .
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and Szegedy, C. (2021). LIME: learning inductive bias for primitives of mathematical reasoning. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11251-11262. PMLR.
Cited by
Sorted by citations
Items 101–200 of 386
Li, Y., Yuan, P., Feng, S., Pan, B., Sun, B., Wang, X. X., Wang, H., & Li, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29821 · Cited by 20
Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models
Liu, J., Huang, Z., Liu, Q., Ma, Z., Zhai, C., & Chen, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3556367 · Cited by 13
“For example, Minerva [49] is an improved version of Google’s PaLM by fine-tuning on mathematical formula data, while Lightman et al. [50] designed the Process-Supervised Model that is optimized from the reasoning steps, not just the final answer.”
Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 23
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10
“There has been significant interest recently in understanding and improving the reasoning abilities of Transformer-based models (Cobbe et al., 2021; Magister et al., 2023; Wei et al., 2022; Dziri et al., 2023; Suzgun et al., 2022; Lewkowycz et al., 2022a; Press et al., 2023).”
BackgroundOrchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI
Kandogan, E., Bhutani, N., Zhang, D., Chen, R. L., Gurajada, S., & Junior, E. R. H. (2025) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew67478.2025.00007 · Cited by 23
Reasoning in Large Language Models Through Symbolic Math Word Problems
Gaur, V., & Saunshi, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.364 · Cited by 25
“…et al., 2020) has led to many other LLMs (Rae et al., 2021; Smith et al., 2022; Chowdhery et al., 2022) which have, perhaps surprisingly, taken big strides in solving difficult tasks like common sense reasoning, math and science problems (Lewkowycz et al., 2022), and writing code (Li et al., 2022).”
BackgroundMethodLarge Language Models are Catalyzing Chemistry EducationPreprint
Du, Y., Duan, C., Bran, A. M., Sotnikova, A., Qu, Y., Kulik, H. J., Bosselut, A., Xu, J., & Schwaller, P. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-h722v · Cited by 10
Symbolic equation solving via reinforcement learning
Dabelow, L., & Ueda, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128732 · Cited by 11
Exploring formal defeasible reasoning of large language models: A Chain-of-Thought approach
Li, Z., Chen, C., Li, M., & Liao, B. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113564 · Cited by 9
Liu, J., Huang, Z., Zhai, C., & Liu, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25571 · Cited by 9
Gemini-the most powerful LLM: Myth or TruthPreprint
Ahmed, I., & Islam, R. (2024) · doi:10.36227/techrxiv.171177477.70151414/v1 · Cited by 9
Conversational Alignment With Artificial Intelligence in Context
Sterken, R., & Kirkpatrick, J. R. (2024) · Philosophical Perspectives · doi:10.1111/phpe.12205 · Cited by 13
Demonstrating Large Language Models on Robots
DeepMind, G. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.024 · Cited by 12
“Large language models exhibit impressive zero-shot reasoning capabilities: from planning [18] to writing math programs [13]; from solving science problems [26] to using trained verifiers [11] for math word problems.”
BackgroundCollaboration, collusion, and barter-cheating: an analysis of academic help-seeking behaviors
Amigud, A., & Hosseini, S. (2023) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2259631 · Cited by 10
Standardization in artificial general intelligence model for education
Xu, Q., Wu, Y., Zheng, H., Yan, H., Wu, H.-C., Qian, Y., Wu, Y., & Liu, B. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104006 · Cited by 11
Self-citationBMCook: A Task-agnostic Compression Toolkit for Big Models
Zhang, Z., Gong, B., Chen, Y., Han, X., Zeng, G., Zhao, W., Chen, Y., Liu, Z., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.40 · Cited by 8
“Our BMCook is the first compression toolkit to support all four compression techniques. models could not do, such as quantitative reasoning (Lewkowycz et al., 2022) and long-form question answering (Nakano et al., 2021).”
MethodMulti-Label Classification with Generative Large Language Models
Niraula, N. B., Ayhan, S., Chidambaram, B., & Whyatt, D. (2024) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc62030.2024.10748883 · Cited by 7
“The rapid rise of generative large language models (LLMs) enables new approaches to many Natural Language Processing (NLP) tasks such as content generation [1], summarization [2], machine translation [3], problem solving [4], software code generation [5] and chat [6].”
Towards Multimodal In-context Learning for Vision and Language Models
Doveh, S., Perek, S., Mirza, M. J., Lin, W. R., Alfassy, A., Arbelle, A., Ullman, S., & Karlinsky, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93806-1_19 · Cited by 45
DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
Zhang, Y., Hu, Y., Zhao, R., Lui, J. C. S., & Chen, H. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764810 · Cited by 35
Edson, A. J., Fabry, A., Kohar, A. W., Bondaryk, L. G., & Phillips, E. D. (2024) · Digital Experiences in Mathematics Education · doi:10.1007/s40751-024-00163-8 · Cited by 9
“…case, mathematics, the most successful of these systems rely on pattern matching of content with human supervisory training by either tagging the input data to define it for the machine or by generating examples that demonstrate a particular pattern (Drori et al., 2022; Lewkowycz et al., 2022).”
Should artificial intelligence be interpretable to humans?
Schwartz, M. D. (2022) · Nature Reviews Physics · doi:10.1038/s42254-022-00538-z · Cited by 7
Sundaram, S. S., Gurajada, S., Padmanabhan, D., Abraham, S. S., & Fisichella, M. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1534 · Cited by 31
“A mathematical LLM has been developed (Lewkowycz et al., 2022) using mathematical data.”
Formalizing chemical physics using the Lean theorem prover
Bobbin, M., Sharlin, S., Feyzishendi, P., Dang, A. H., Wraback, C. M., & Josephson, T. R. (2023) · Digital Discovery · doi:10.1039/d3dd00077j · Cited by 7
Key-Point-Driven Data Synthesis with Its Enhancement on Mathematical Reasoning
Huang, Y., Liu, X., Gong, Y., Gou, Z., Shen, Y., Duan, N., & Chen, W. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34593 · Cited by 59
A Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 13
Self-citationSystematic Human Learning and Generalization From a Brief Tutorial With Explanatory Feedback
Nam, A., & McClelland, J. L. (2024) · Open Mind · doi:10.1162/opmi_a_00123 · Cited by 12
“Large language models (LLMs) (Bommasani et al., 2021) such as GPT-3 (Brown et al., 2020) are showing increasing proficiency in solving mathematical (Cobbe et al., 2021; Hendrycks et al., 2021; Lewkowycz et al., 2022; Uesato et al., 2022) and analogical problems (Webb et al., 2023).”
BackgroundComplex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 32
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“…GPT-3 (Brown et al., 2020), and ChatGPT, used as approximate stand-ins for shifts in how the community uses and therefore evaluates LMs. Common practice for evaluating autoregressive language models today diverges from the method described in the paper for all listed tasks except MMLU and MATH.”
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data
Zhang, Z., Li, X., Gao, Y., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.132 · Cited by 10
“Although LLMs show powerful reasoning abilities on various tasks (Qin et al., 2023), they have limitations on various fields (Lewkowycz et al., 2022; Ziems et al., 2023”
Predictive design of crystallographic chiral separation
Elijošius, R., King‐Smith, E., Faber, F. A., Bernier, L., Berritt, S., Farrell, W. P., Hou, X., Klug-McLeod, J. L., Mustakis, J. G., Sach, N. W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62825-4 · Cited by 8
Insights into Moral Reasoning of AI: A Comparative Study Between Humans and Large Language Models
Bajpai, S., Sameer, A., & Fatima, R. (2025) · Journal of Media Ethics · doi:10.1080/23736992.2025.2553146 · Cited by 5
“LLM models illustrated impressive abilities in solving simple quantitative reasoning tasks (Lewkowycz et al., 2022) and, with decomposed prompting, LLM models can perform significantly better in symbolic reasoning tasks (Khot et al., 2023).”
MacBehaviour: An R package for behavioural experimentation on large language models
Duan, X., Li, S., & Cai, Z. G. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02524-y · Cited by 7
“Current benchmarks of LLMs focused on quantifiable tasks like translation and world knowledge (Hendrycks et al., 2021; Zellers et al., 2019), math (Lewkowycz et al., 2022; Zhou et al., 2023) and coding (Peng et al., 2024); while these tests offer a good understanding of model performance, they are…”
BackgroundSymbolic Math Reasoning with Language Models
Gaur, V., & Saunshi, N. (2022) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc56832.2022.10002218 · Cited by 7
“Prior work has taken the finetuned approach with high accuracy [5].”
MethodInterpretable Math Word Problem Solution Generation via Step-by-step Planning
Zhang, M., Wang, Z., Yang, Z., Feng, W., & Lan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.379 · Cited by 27
CoqPyt: Proof Navigation in Python in the Era of LLMs
Carrott, P., Saavedra, N., Thompson, K., Lerner, S., Ferreira, J. F., & First, E. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663814 · Cited by 5
“Pretrained LLMs have been shown to have quantitative reasoning capabilities [3, 18, 27], especially when they undergo continued training on math data [1, 14].”
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5
Involving LLMs in legal processes is risky
Ladkin, P. B. (2023) · Digital Evidence and Electronic Signature Law Review · doi:10.14296/deeslr.v20i.5610 · Cited by 5
LIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 4
“Third, we incorporate multi-disciplinary benchmarks including Minerva (Lewkowycz et al., 2022) (undergraduate-level STEM) and GPQA (Rein et al., 2023) to evaluate the transfer of mathematical reasoning skills to broader contexts beyond our training domain.”
SigChord: Sniffing Wide Non-sparse Multiband Signals for Terrestrial and Non-terrestrial Networks
Peng, J., Duan, J., Lin, Z., Yuan, H., Gao, Y., & Chen, Z. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729159 · Cited by 4
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
Wu, H., Hui, W., Chen, Y., Wu, W., Tu, K., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.427 · Cited by 8
Ginde, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00034 · Cited by 4
A chit-chat between Llama 2 and ChatGPT for the automated creation of exploits
Caturano, F., Ciotola, J., Romano, S. P., & Varlese, M. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111501 · Cited by 4
Few-shot Adaptation Works with UnpredicTable Data
Chan, J. S., Pieler, M., Jao, J., Scheurer, J., & Perez, E. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.102 · Cited by 7
World Models for Math Story Problems
Opedal, A., Stoehr, N., Saparov, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.579 · Cited by 12
Self-Influence Guided Data Reweighting for Language Model Pre-training
Thakkar, M., Bolukbasi, T., Ganapathy, S., Vashishth, S., Chandar, S., & Talukdar, P. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.125 · Cited by 19
Recursos linguísticos para o PLN específico de domínio: o Petrolês
Freitas, C., SOUZA, E. A. D., Castro, M. C., Cavalcanti, T., da Silva, P. F., & Cordeiro, F. C. (2023) · Linguamática · doi:10.21814/lm.15.2.412 · Cited by 4
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 804
“We compare the pass@ k curves of base and zero-RL models on benchmarks of varying difficulty: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), Olympiad (He et al., 2024), AIME24, and AMC23.”
How Much Do Prompting Methods Help LLMs on Quantitative Reasoning with Irrelevant Information?
Song, S. H., & Tavanapong, W. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679840 · Cited by 3
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
“Despite the success, performing multi-hop and compositional reasoning for complex tasks can still face challenges (Hendrycks et al., 2021; Lewkowycz et al., 2022), even when the required knowledge is limited to the scope of primary school (Cobbe et al., 2021).”
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
“Targeted fine-tuning (Minerva) improves maths proficiency but may not generalize to abstract IQ formats (Lewkowycz, Andreassen, et al., 2022)As per journal style, section headings are numbered.”
Meadows, J., & Freitas, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00594 · Cited by 30
“Logical puzzle solvers (Groza and Nitu, 2022) and informal reasoning generation models (Lewkowycz et al., 2022) exist far into the abstractive side, due to the step-wise and some-times symbolic reasoning required to address them.”
BackgroundInfluentialEnhanced matrix inference with Seq2seq models via diagonal sorting
Peng, W., Wang, Y., & Wu, M. (2024) · Scientific Reports · doi:10.1038/s41598-023-50919-2 · Cited by 3
“Lewkowycz et al. 17 trained a large language model called Minerva that is capable of solving middle school to college level math and science problems without relying on external tools, by continuing pretraining it on technical subject data.”
BackgroundReeping, D. P., & Shah, A. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--47047 · Cited by 3
Nehring, A., Buschhüter, D., Kubsch, M., Ludwig, T., Wulff, P., & Neumann, K. (2025) · Zeitschrift für Didaktik der Naturwissenschaften · doi:10.1007/s40573-025-00177-8 · Cited by 3
“…Großer Sprachmo-K delle (GSM) naturwissenschaftliche Fähigkeiten weit über den Trainingskontext hinaus diagnostizieren können (Frie-der et al. 2023; Lewkowycz et al. 2022) und dass Anwen-dungen generativer Sprachmodelle, wie z.B. ChatGPT, na-turwissenschaftsdidaktische Aufgaben (etwa das…”
de Mántaras, R. L., Gibert, K., Alier, M., García, C. U. C., Estampa, Hernández-Fernändez, A., Balas, D. F., Carreras, A., Calle, A., & Sora, C. (2023) · Iniciativa Digital Politècnica. Oficina de Publicacions Acadèmiques Digitals de la UPC eBooks · doi:10.5821/ebook-9788410008090 · Cited by 3
What Can Large Language Models Do for Theorem Proving and Formal Methods?
Johansson, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46002-9_25 · Cited by 3
Imerman, M. B., & Fabozzi, F. J. (2025) · Finance research letters · doi:10.1016/j.frl.2025.107491 · Cited by 2
Large Language Models for Safe Minimization
Yadavally, A., Rong, X., Nguyen, P., & Nguyen, T. N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00203 · Cited by 2
“Recently, large language models (LLMs) have shown emergent behaviors with the abilities to reason in various domains, e.g., arithmetic [29], formal logic [29], [30], source code [31], [32], etc .”
Li, Y., & Fowlie, M. (2025) · LDV-Forum/Journal for language technology and computational linguistics · doi:10.21248/jlcl.38.2025.285 · Cited by 2
Župan, M. (2025) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.70011 · Cited by 6
ImagineNav++: Prompting Vision-Language Models as Embodied Navigator Through Scene Imagination
Wang, T., Zhao, X., Cai, W., & Sun, C. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688692 · Cited by 6
Kashefi, A., & Mukerji, T. (2026) · International Journal for Numerical Methods in Fluids · doi:10.1002/fld.70085 · Cited by 2
“Additionally, we aim to explore the integration of KANs with neural operators [135–139] and large language models [140–144].”
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. D., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 19
The Ascent of Machine Cognition: A Systematic Review of Large Reasoning Models
Sewak, M. (2026) · doi:10.37819/jaike.25.0010 · Cited by 2
“Minerva (Lewkowycz et al., 2022), fine-tuned on extensive mathematical and scientific corpora, demonstrated significant improvements in solving complex quantitative problems by learning to utilize appropriate mathematical notation and concepts, even without relying on external execution.”
de Souza, E. A., & Freitas, C. (2023) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2023.233980 · Cited by 2
Media and the transformative potential of AI in the scientific field
Sudmann, A., & Schröter, J. (2023) · AI critique · doi:10.1515/9783839467664-013 · Cited by 2
ArithmeticGPT: empowering small-size large language models with advanced arithmetic skills
Liu, Z., Zheng, Y., Yin, Z., Chen, J., Liu, T., Mi, T., & Luo, W. (2025) · Machine Learning · doi:10.1007/s10994-024-06681-1 · Cited by 2
“…2023), PoT (Chen et al., 2022), PAL (Gao et al., 2023), MathCoder (Wang et al., 2023a), and ToRa (Gou et al., 2023), continual pretraining large-scale base models with a mixture of mathematical code, webpages and papers, such as L LemmA (Azerbayev et al., 2023) and Minerva (Lewkowycz et al., 2022).”
Improving ToM Capabilities of LLMs in Applied Domains
Galitsky, B. A. (2026) · Springer eBooks · doi:10.1007/978-3-032-16524-4_4 · Cited by 3
Xie, F., & Schwertfeger, S. (2024) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio64047.2024.10907658 · Cited by 8
“Research on LLMs in mathematics has shown their challenges with numbers[8][9].”
Exploring Human and Language Model Alignment in Perceived Design Similarity Using Ordinal Embeddings
Keeler, M., Fuge, M., Peng, A., & Miller, S. R. (2025) · Journal of Mechanical Design · doi:10.1115/1.4069129 · Cited by 3
Improving Mathematics Tutoring With A Code Scratchpad
Upadhyay, S., Ginsberg, E., & Callison-Burch, C. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.2 · Cited by 4
“…wide variety of tasks, the improvement in dialogue systems stemming from alignment as seen in models like ChatGPT and Sparrow (Glaese et al., 2022), and the success of LLMs (especially those that generate code) in the related domain of Math Word Problem Solving (Li et al., 2022; Gao et al., 2022).”
MethodFERMAT: An Alternative to Accuracy for Numerical Reasoning
Sivakumar, J., & Moosavi, N. S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.838 · Cited by 2
20Q: Overlap-Free World Knowledge Benchmark for Language Models
Bruyn, M. D., Lotfi, E., Buhmann, J., & Daelemans, W. M. P. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.46 · Cited by 2
“Much previous work focused on what these models can do: question-answering, mathematics, translation, or code generation (Wei et al., 2022; Chen et al., 2021; Cobbe et al., 2021; NLLB Team et al., 2022; Lewkowycz et al., 2022 benchmark.”
BackgroundLanguage Model-Based Player Goal Recognition in Open World Digital Games
Kim, Y. J., Goslen, A., Rowe, J., Mott, B., & Lester, J. C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v19i1.27503 · Cited by 2
Hernández, G. J. P., Bonilla, M. L., Uribe-Suarez, D. A., & Palacio, L. F. C. (2023) · Revista de Investigación Desarrollo e Innovación · doi:10.19053/20278306.v13.n2.2023.16838 · Cited by 2
“The analysis has focused on generating playful learning spaces that make it possible to share the improvement of this competence in students (Ozaltun-Celik, 2021; Lewkowycz et al., 2022).”
BackgroundThe Two Faces of AI in Green Mobile Computing: A Literature Review
Siemers, W., Sallou, J., & Cruz, L. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00053 · Cited by 2
“…in deep learning, such as shown by Mnih et al. in reinforcement learning [14], Goodfellow et al. in the development of the Generative Adversarial Network (GAN) [15], and the recent development of models like Minerva, which can solve undergraduate-level mathematical and engineering problems [16].”
MethodRazeghi, Y., Mekala, R. R., Logan, R. L., Gardner, M., & Singh, S. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.39 · Cited by 2
“By leveraging diverse data sources such as GitHub and arXiv, these models have demonstrated the ability to perform complicated tasks such as quantitative reasoning (Lewkowycz et al., 2022) and writing computer programs (Chen et al., 2021).”
MethodHigher Cognition: A Mechanical Perspective
Friedman, R. (2022) · Encyclopedia · doi:10.3390/encyclopedia2030102 · Cited by 2
“An ideal example is a mathematical symbol, such as an operator for the addition of numbers [55].”
BackgroundTowards provably efficient quantum algorithms for large-scale machine-learning modelsPreprint
Liu, J., Liu, M., Liu, J.-P., Ye, Z., Wang, Y., Alexeev, Y., Eisert, J., & Jiang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.03428 · Cited by 2
“It is widely believed that large-scale machine learning might be one of the most revolutionary technologies benefiting society [1], including already important breakthroughs in digital arts [2], conversation like GPT-3 [3, 4], and mathematical problem solving [5].”
BackgroundUnderstanding Logical Reasoning Ability of Large Language ModelsPreprint
Chan, E. (2024) · SSRN Electronic Journal · doi:10.20944/preprints202408.1712.v1 · Cited by 2
A Catalog of Transformer Models
D’Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r609337 · Cited by 3
Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
Wu, M., Zhang, Z., Dong, Q., Xi, Z., Zhao, J., Jin, S., Fan, X., Zhou, Y., Lv, H., Zhang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.10532 · Cited by 75
“Standard suites such as MATH-500 (Hendrycks et al. 2021), AIME (Li et al. 2024), AMC (Li et al. 2024), and Minerva Math (Lewkowycz et al. 2022) require models to parse natural-language problem statements, uncover the latent mathematical structure, and generate exact numeric answers.”
RewardBench 2: Advancing Reward Model Evaluation
Malik, S., Pyatkin, V., Land, S., Morrison, J., Smith, N. A., Hajishirzi, H., & Lambert, N. (2025) · arXiv · doi:10.48550/arxiv.2506.01937 · Cited by 1
“Using a pool of models strong at math, we sampled five completions per model at a temperature of 1.0 and used majority voting to select a gold answer, as is common practice [66, 67].”
InfluentialConceptual Attention in StyleGAN
Suekane, K., Haji, S., Sano, H., & Takagi, T. (2023) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc56153.2023.00027 · Cited by 1
“In particular, the multi-head attention proposed for Trans-former [1] has shown excellent results on various datasets including those for natural language processing [39], [40], computer vision [22], [27], and speech processing [42].”
BackgroundEvaluating Logical Reasoning Ability of Large Language Models
Chan, E., & Chan, A. C.-F. (2026) · doi:10.1109/icnlp69856.2026.11527900 · Cited by 1
History of Large Language Models
Luzi, F. D. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_2 · Cited by 1
Benchmarking and Advancing Large Language Models for Local Life Services
Lan, X., Feng, J., Lei, J., Shi, X., & Li, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737196 · Cited by 3
ZhongJingGPT: An Expert Knowledge-Guided Language Model for Traditional Chinese Medicine
Kang, Y., Chang, Y., Wu, S., Wu, X., Jiao, Y., Fu, J., Ma, Q., Fang, Y., Chen, Y., Zhao, X., et al. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010046 · Cited by 2
Chowdhury, S., Chowdhury, A. F., Banu, A., & Hossain, R. (2026) · IEEE International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain69676.2026.11545654 · Cited by 1
GradeSQL: Outcome reward models for intelligent Text-to-SQL generation from LLMs
Tritto, M., Farano, G., Palma, D. D., Rossiello, G., Subramanian, D., Narducci, F., & Noia, T. D. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01071-6 · Cited by 1
“We selected N = 32 following Li (2025), as it represents a well-balanced trade-off between performance and computational cost (Lewkowycz, 2022; Wang et al., 2023 ).”
From Seed to Scope: Reasoning to Identify Change Impact Sets
Yadavally, A., & Nguyen, T. N. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773265 · Cited by 1
Learning to solve arithmetic problems with a virtual abacus
Petruzzellis, F., Chen, L., & Testolin, A. (2023) · Proceedings of the Northern Lights Deep Learning Workshop · doi:10.7557/18.6805 · Cited by 1
“Indeed, even state-of-the-art language models fall short in mathematical tasks that require strong generalization capabilities [3] (though very recent work has shown that performance significantly improves following fine-tuning on large-scale mathematical datasets [4]).”
BackgroundInfluentialSimple algorithm for judging equivalence of differential-algebraic equation systems
Kato, S., Zhang, C., & Kano, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-38254-y · Cited by 1
AI direct tests: LNE and NIST evaluations
Messina, E. R., Avrin, G., & Ribeiro, S. (2023) · Educational research and innovation · doi:10.1787/982e32ab-en · Cited by 1
Assessing AI capabilities on occupational tests
Kalamova, M. (2023) · Educational research and innovation · doi:10.1787/4bd0d136-en · Cited by 1
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 5
Is word order considered by foundation models? A comparative task-oriented analysis
Zhao, Q., Li, J., Liu, J., Kang, Z., & Zhou, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122700 · Cited by 3
Uncovering the Interpretation of Large Language Models
Akter, M. S., Shahriar, H., Cuzzocrea, A., & Wu, F. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00143 · Cited by 1
“A large number of recent studies suggests various strategies for enhancing LLM performance on mathematical tasks, whether through specialized pre-training schemes [3–5] or through unique prompting methods [6].”
BackgroundTowards Analysis and Interpretation of Large Language Models for Arithmetic Reasoning
Akter, M. S., Shahriar, H., & Cuzzocrea, A. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00049 · Cited by 1
“A large number of recent studies suggests various strategies for enhancing LLM performance on mathematical tasks, whether through specialized pre-training schemes [3]–[5] or through unique prompting methods [6].”
BackgroundDas, S., Ghosh, S., Tiwari, A., Lyngkhoi, R. E. Z. M., Saha, S., Murad, Z., & Maurya, A. (2025) · IEEE Access · doi:10.1109/access.2025.3551124 · Cited by 1
“Although Large Language Models (LLMs) have excelled in arithmetic, commonsense, and symbolic reasoning [34], [35], [36], there’s a research gap in summarizing financial video advice.”
Metadata
- Authors
- Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., & Misra, V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 3843-3857
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2206.14858
- OpenAlex
- W7133211296
- Cited by
- 358
- References
- 0
- Citations by year
- 2026: 55 · 2025: 53 · 2024: 67 · 2023: 173 · 2022: 10
- FWCI
- 287.45
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Education › Mathematics Education and Pedagogy
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- quantitative reasoning, language models, mathematical problem solving, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
