The DOI matches its DataCite record; 26 of the 26 references it cites were found in a registry or an index.
Solving Quantitative Reasoning Problems with Language Models
Open accessAbstract
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Wei et al. [4], Nye et al. [15] and Lewkowycz et al. [5] showed that combining prompting (i.e. in-context learning) with scratchpad strategies present a powerful combination.”
From Exploring Length Generalization in Large Language Models (2022) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Alemi, A. A., Chollet, F., Een, N., Irving, G., Szegedy, C., and Urban, J. (2016). Deepmath - Deep Sequence Models for Premise Selection. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, pages 2243-2251, USA. Curran Associates Inc.
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.
- Grabowski, A., Kornilowicz, A., and Naumowicz, A. (2010). Mizar in a nutshell. J. Formalized Reasoning , 3(2):153-245.
- Han, J. M., Rute, J., Wu, Y., Ayers, E., and Polu, S. (2022). Proof artifact co-training for theorem proving with language models. In International Conference on Learning Representations .
- Kaliszyk, C., Urban, J., Michalewski, H., and Olšák, M. (2018). Reinforcement learning of theorem proving. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.
- Li, W., Yu, L., Wu, Y., and Paulson, L. C. (2021). Isarstep: a benchmark for high-level mathematical reasoning. In ICLR .
- Megill, N. D. and Wheeler, D. A. (2019). Metamath: A Computer Language for Pure Mathematics . Lulu Press, Morrisville, North Carolina. http://us.metamath.org/downloads/metamath.pdf .
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and Szegedy, C. (2021). LIME: learning inductive bias for primitives of mathematical reasoning. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11251-11262. PMLR.
Cited by
Sorted by citations
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
“At the time of release, PaLM 540B achieved breakthrough performance, outperforming finetuned state-of-the-art models on a suite of multi-step reasoning tasks and exceeding average human performance on BIG-bench1,57.”
BackgroundMethodInfluentialLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“We also add a new subcorpora for arXiv and math webpages from the dataset used in Lewkowycz et al. (2022) in the proportion of 5%.”
MethodSelf-citationCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767
“Large language models exhibit impressive zero-shot reasoning capabilities: from planning [14] to writing math programs [43]; from solving science problems [44] to using trained verifiers [45] for math word problems.”
BackgroundUnleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401
“for coding decisions, similar to those generated by Minerva for mathematics problems [22].”
MethodLarge Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 430
“The adaptability of LLMs to unfamiliar tasks and their apparent reasoning abilities (22) are captivating.”
BackgroundSolving olympiad geometry without human demonstrations
Trinh, T. H., Wu, Y., Le, Q. V., He, H., & Luong, T. (2024) · Nature · doi:10.1038/s41586-023-06747-5 · Cited by 311
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks 26–29 .”
BackgroundSelf-citationA Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, V. N. (2024) · Applied Sciences · doi:10.3390/app14052074 · Cited by 355
“…GPT NeoX 20B [48] 3.15 M tokens 2048 150 K steps -, AdamW with ZeRO few-shot OPT [49] 2 M tokens 2048 - ReLU, AdamW few-shot, zero-shot MINERVA [50] - 1024 399 K SwiGLU, Adafactor few-shot, chain-of-thought context AlexaTM 20 B [51] 2 million tokens - - Adam Finetuning, few-shot, one-shot,…”
BackgroundMethodInfluentialRT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
“…for a wide range of downstream tasks: large language models can enable not only fluent text generation [1, 2, 3] but emergent problem-solving [4, 5, 6] and creative generation of prose [7, 2] and code [8], while vision-language models enable open-vocabulary visual recognition [9, 10, 11] and…”
BackgroundSelf-citationLarge Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230
Self-citationAGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211
“For MATH, we use the same instances as in the appendices of Lewkowycz et al. (2022).”
Autonomous GIS: the next-generation AI-powered GIS
Li, Z., & Ning, H. (2023) · International Journal of Digital Earth · doi:10.1080/17538947.2023.2278895 · Cited by 261
“Various strategies have been proposed, including employing specific training datasets (Lewkowycz et al. 2022; Yuan et al. 2023), implementing causal analysis (Stolfo et al. 2023), developing a ‘chain of thought’ approach (Wei et al. 2023), and integrating plug-ins (e.g. Wolfram).”
MethodWang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 199
“Google Research introduced Minerva [30], an advanced model that builds upon the PaLM [31] while incorporating a science-and-math-focused dataset.”
MethodPAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155
“In fact, even when fine-tuning a PaLM-based model on 164B tokens of explicit mathematical content, its two most common failures are reportedly “incorrect reasoning” and “incorrect calculation” (Lewkowycz et al., 2022).”
BackgroundA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
Is GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149
Large Language Models Are Reasoning Teachers
Ho, N., Schmid, L., & Yun, S.-Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.830 · Cited by 403
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Shtedritski, A., Rupprecht, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01101 · Cited by 191
Large Language Models as Evolutionary Optimizers
Liu, S., Chen, C., Qu, X., Tang, K., & Ong, Y.-S. (2024) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec60901.2024.10611913 · Cited by 209
“Among them of particular interest are tasks involving reasoning and decision-making, such as planning [39]–[41] and solving mathematical problems [35]–[37].”
BackgroundTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 196
“2022b), Minerva (Lewkowycz et al. 2022), and DIVERSE (Li et al.”
BackgroundFaithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.20 · Cited by 201
Espejel, J. L., Ettifouri, E. H., Alassan, M. S. Y., Chouham, E. M., & Dahhane, W. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100032 · Cited by 148
Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency Department
Huang, J., Neill, L., Wittbrodt, M., Melnick, D., Klug, M., Thompson, M., Bailitz, J., Loftus, T., Malik, S., Phull, A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36100 · Cited by 113
ParameterNet: Parameters are All You Need for Large-Scale Visual Pretraining of Mobile Networks
Han, K., Wang, Y., Guo, J., & Wu, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01491 · Cited by 124
“Our training dataset is a mixture of several sources, including C4 [39], Wikipedia [54], and ArXiv [31].”
MethodWang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 108
“LLMs have an impressive ability to adapt to unfamiliar tasks and demonstrate remarkable reasoning skills [75,76].”
BackgroundA Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 160
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 1,202
“If the butter costs $4.00 a pound, how much will it cost him to purchase 6 pounds of butter? sider generating responses from multiple instances of model and performing majority voting across responses (Wang et al., 2022; Lewkowycz et al., 2022).”
InfluentialSurvey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 130
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94
“of scientific papers from the arXiv preprint server and web pages that include mathematical expressions [52].”
BackgroundMethodInfluentialTowards provably efficient quantum algorithms for large-scale machine-learning models
Liu, J., Liu, M., Liu, J.-P., Ye, Z., Wang, Y., Alexeev, Y., Eisert, J., & Jiang, L. (2024) · Nature Communications · doi:10.1038/s41467-023-43957-x · Cited by 121
“It is widely believed that large-scale machine learning might be one of the most revolutionary technologies benefiting society [1], including already important breakthroughs in digital arts [2], conversation like GPT-3 [3, 4], and mathematical problem solving [5].”
BackgroundTurpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
“In particular, their high performance on numerical tasks, such as addition and multiplication, suggests that models may have learned the ability to perform the underlying reasoning operations simply through a combination of pre-training and model size (Lewkowycz et al., 2022).”
BackgroundLi, X., Chan, S., Zhu, X., Pei, Y., Ma, Z., Liu, X., & Shah, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.39 · Cited by 158
“For example, Sing-hal et al. (2022) applied GPT-3 in the legal domain, and Lewkowycz et al. (2022) adapted T5 to the financial domain.”
A Scoping Survey of ChatGPT in Mathematics Education
Pepin, B., Buchholtz, N. F., & Salinas-Hernández, U. (2025) · Digital Experiences in Mathematics Education · doi:10.1007/s40751-025-00172-1 · Cited by 166
“Other studies also explored the capabilities of generative AI technologies in general in producing accurate mathematical solutions (Buchholtz et al., 2024; Hend-rycks et al., 2021; Lewkowycz et al., 2022).”
Harnessing LLMs for Temporal Data - A Study on Explainable Financial Time Series Forecasting
Yu, X., Chen, Z., & Lu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.69 · Cited by 89
“…models capable of learning the underlying complex relations among textual sequences (Yun et al., 2019; Rong et al., 2020; Zhang et al., 2020; Dwivedi and Bresson, 2020; Ying et al., 2021) and solving quantitative problems (Wei et al., 2022; Lewkowycz et al., 2022; Imani et al., 2023).”
BackgroundDistilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79
“…of the modern approaches for these problems are shifting towards using large language models, often relying on approaches involving prompting or in-context learning (Cobbe et al., 2021; Kojima et al., 2022; Wei et al., 2022b; Chowdhery et al., 2022; Lewkowycz et al., 2022; Srivastava et al., 2022).”
BackgroundFew-shot In-context Learning on Knowledge Base Question Answering
Li, T., Ma, X., Zhuang, A., Gu, Y., Su, Y., & Chen, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.385 · Cited by 126
Do Large Language Models Know What They Don’t Know?
Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.551 · Cited by 75
Integration of large language models and federated learning
Chen, C., Feng, X., Li, Y., Lyu, L., Zhou, J., Zheng, X., & Yin, J. (2024) · Patterns · doi:10.1016/j.patter.2024.101098 · Cited by 140
Large Language Models Meet Next-Generation Networking Technologies: A Review
Hang, C. N., Yu, P.-D., Morabito, R., & Tan, C. W. (2024) · Future Internet · doi:10.3390/fi16100365 · Cited by 75
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 145
AI Reasoning in Deep Learning Era: From Symbolic AI to Neural–Symbolic AI
Liang, B., Wang, Y., & Tong, C. (2025) · Mathematics · doi:10.3390/math13111707 · Cited by 100
Use of Artificial Intelligence Chatbots in Interpretation of Pathology Reports
Steimetz, E., Minkowitz, J. M., Gabutan, E. C., Ngichabe, J., Attia, H., Hershkop, M., Ozay, F., Hanna, M. G., & Gupta, R. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.12767 · Cited by 66
Gemini-the most powerful LLM: Myth or Truth
Islam, R., & Ahmed, I. (2024) · Information Communication Technologies Conference (ICTC) · doi:10.1109/ictc61510.2024.10602253 · Cited by 98
Is Education Losing the Race with Technology?
OECD (2023) · Educational research and innovation · doi:10.1787/73105f99-en · Cited by 86
Kolmogorov–Arnold PointNet: Deep learning for prediction of fluid fields on irregular geometries
Kashefi, A. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.117888 · Cited by 55
“Integrating KANs into Fourier neural operators [113–117] and large language models [118–122] could be potentially beneficial to the fields of computational physics and computational mathematics.”
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52
From Deep Learning to Rational Machines
Buckner, C. J. (2023) · doi:10.1093/oso/9780197653302.001.0001 · Cited by 57
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 56
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296
Schorcht, S., Buchholtz, N. F., & Baumanns, L. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1386075 · Cited by 89
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 56
“The year 2022 witnessed an explosion in the development of LLMs, with models like MT-NLG [76], InstructGPT [77], LaMDA [78], Chin-chilla [79], PaLM [39], OPT [80], BLOOM/BLOOMZ [81,82], Min-erva [83], Sparrow [84], Flan-PaLM [85], Galactica [86], AnthropicLM v 4 -s 3 [73], OPT-IML [87], etc.”
Exploring Length Generalization in Large Language Models
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2207.04901 · Cited by 44
Self-citationLearning Mathematics with Large Language Models
Matzakos, N., Doukakis, S., & Moundridou, M. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i20.42979 · Cited by 51
Gandolfi, A. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00403-3 · Cited by 59
“ChatGPT’s performance can be moderately improved for datasets simpler than Calculus through prompt engineering or verification training (Liu et al. (2023); Frieder et al. (2024); Lewkowycz et al. (2022)).”
InfluentialAutomated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Babovic, Z. B., Bajat, B., Djokić, V., Đorđević, F., Drašković, D., Filipović, N., Furht, B., Gačić, N., Ikodinović, I. B., Ilić, M., et al. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00731-6 · Cited by 36
“Authors also demonstrate the model’s ability to solve complex equation systems and logic puzzles requiring a higher level of reasoning [56].”
BackgroundLimitations of Language Models in Arithmetic and Symbolic Induction
Jing, Q., Wang, H., Li, Z., Li, S., & Yan, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.516 · Cited by 90
TelecomRAG: Taming Telecom Standards with Retrieval Augmented Generation and LLMs
Yilma, G. M., Ayala‐Romero, J. A., García‐Saavedra, A., & Costa‐Pérez, X. (2024) · ACM SIGCOMM Computer Communication Review · doi:10.1145/3711992.3711996 · Cited by 30
“In the domain of scientific research, LLMs specially pre-trained on scientific-related corpora have been proposed (e.g. Galactica [11], Minerva [9]).”
Symbol tuning improves in-context learning in language models
Wei, J., Hou, L., Lampinen, A. K., Chen, X., Huang, D. Y., Tay, Y., Chen, X., Lu, Y., Zhou, D., Ma, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.61 · Cited by 80
AECR: Automatic attack technique intelligence extraction based on fine-tuned large language model
Chen, M., Zhu, K., Lu, B., Li, D., Yuan, Q., & Zhu, Y. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104213 · Cited by 31
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30
Human I/O: Towards a Unified Approach to Detecting Situational Impairments
Liu, X., Li, J. N., Kim, D., Chen, X., & Du, R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642065 · Cited by 26
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 95
Silva, D. D., Kaynak, O., El-Ayoubi, M., Mills, N., Alahakoon, D., & Manic, M. (2024) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2024.3382962 · Cited by 31
Exploring Effective Factors for Improving Visual In-Context Learning
Sun, Y., Chen, Q., Wang, J., Wang, J., & Li, Z. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3554410 · Cited by 61
“This is done by using a majority voting strategy [17].”
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29
“…users to complete multiple tasks (e.g., question answering (Thop-pilan et al., 2022), code generation (Chen et al., 2021), math problem solving (Lewkowycz et al., 2022), and scientific literature generation (Taylor et al., 2022)) through a natural language interface (Liu et al., 2023) with both…”
BackgroundTestolin, A. (2024) · Applied Sciences · doi:10.3390/app14020744 · Cited by 24
“For example, Minerva achieves over 80% accuracy on 10-digit addition and over 20% accuracy on 18-digit addition, but calculation accuracy is never at ceiling even for easy (e.g., 5-digit) problems, which highlights that even such a sophisticated model still has a very limited understanding of elementary arithmetic.”
BackgroundInfluentialEXIF as Language: Learning Cross-Modal Associations between Images and Camera Metadata
Zheng, C., Shrivastava, A., & Owens, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00671 · Cited by 25
“This allows us to directly process camera metadata using models from natural language processing — an approach that has successfully been applied to processing various text-like inputs other than language, such as math [45] and code [11].”
MethodOpportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25
AI and the Future of Skills, Volume 2
OECD (2023) · Educational research and innovation · doi:10.1787/a9fe53cb-en · Cited by 43
Data Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 28
Tree-Based Representation and Generation of Natural and Mathematical Language
Scarlatos, A., & Lan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.205 · Cited by 29
REAL: Resilience and Adaptation using Large Language Models on Autonomous Aerial Robots
Tagliabue, A., Kondo, K., Zhao, T., Peterson, M. B., Tewari, C. T., & How, J. P. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10885890 · Cited by 29
“Recently, foundational models, and especially Large Language Models (LLMs) pre-trained on internet-scale datasets [16]–[18], have demonstrated impressive performance on a variety of reasoning problems, including natural language [19], [20] and mathematics [21].”
MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting
Inaba, T., Kiyomaru, H., Cheng, F., & Kurohashi, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.130 · Cited by 35
Using large language models to accelerate communication for eye gaze typing users with ALS
Cai, S., Venugopalan, S., Seaver, K., Xiao, X., Tomanek, K., Jalasutram, S., Morris, M. R., Kane, S. K., Narayanan, A., MacDonald, R. L., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-53873-3 · Cited by 41
“Second, we integrated the SpeakFaster UI with the eye-gaze keyboards of two users with ALS and demonstrated a signi fi cant enhancement in text-entry speed (29 – 60”
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 106
“…such as text generation and reasoning (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2022a; Mishra et al., 2022), mathematical reasoning (Lewkowycz et al., 2022; Gao et al., 2022; Arora et al., 2023), and code generation (Li et al., 2022; Madaan et al., 2023b), has underscored the…”
BackgroundIntegration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 20
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationCapturing advanced human cognitive abilities with deep neural networks
McClelland, J. L. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.018 · Cited by 18
“Indeed, several recent transformer-based neural network systems [9] have demonstrated a new level of success in mathematical problem solving after training them with a corpus of text-and symbolic-formula-based presentations of mathematical ideas and arguments found in web pages and scienti fi c…”
BackgroundCordeiro, F. C., da Silva, P. F., Tessarollo, A., Freitas, C., de Souza, E. A., Gomes, D. D. S. M., Souza, R. R., & Coelho, F. C. (2024) · Computers & Geosciences · doi:10.1016/j.cageo.2024.105714 · Cited by 16
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement
Xi, Z., Jin, S., Zhou, Y., Zheng, R., Gao, S., Liu, J., Gui, T., Zhang, Q., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.762 · Cited by 40
Salikutluk, V., Koert, D., & Jäkel, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230081 · Cited by 16
“…brainstorming tasks [2,19,20], writing [21,22,3], or mood board creation [2], we propose the use of an AI-aided brainstorming tool for the specific use-case Pre-trained Transformer 3 (GPT-3), which is an LLM [23] that was already used in several different application areas [1,24,25,26,27,28].”
MethodExplaining Data Patterns in Natural Language with Language Models
Singh, C., Morris, J. X., Aneja, J., Rush, A. M., & Gao, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.3 · Cited by 17
“Moreover, LLMs have demonstrated the capacity to excel at real-world problems, such as mathematics (Lewkowycz et al., 2022), scientific question answering (Sa-dat and Caragea, 2022), predicting brain responses (Schrimpf et al., 2021), and classifying proteins and chemical compounds (Taylor et al.,…”
BackgroundQuantum many-body physics calculations with large language models
Pan, H., Mudur, N., Taranto, W., Tikhanovskaya, M., Venugopalan, S., Bahri, Y., Brenner, M. P., & Kim, E.-A. (2025) · Communications Physics · doi:10.1038/s42005-025-01956-y · Cited by 15
“In the natural sciences, recent works that demonstrate the utility of LLMs [21] have focused on testing their knowledge in different domains [6, 22–27] or on their ability to generate code to help with automating experimentation processes that utilize software [12, 28].”
Fang, M., Wan, X., Lu, F., Xing, F., & Zou, K. (2025) · Scientific Data · doi:10.1038/s41597-025-05283-3 · Cited by 59
“…2021] 2021 8.5k middle-school level math word problems 1k MATH [Hendrycks et al., 2021a] 2021 12.5k high-school math competitions 5k OCWCourses [Lewkowycz et al., 2022] 2022 University-level, MIT’s OpenCourseWare 272 MiniF2F [Zheng et al., 2022] 2023 Olympiad-level 488 SAT [Azerbayev et al.,…”
Stolfo, A., Belinkov, Y., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.435 · Cited by 107
“…of works proposing methods to improve the performance of large LMs on math benchmark datasets through enhanced pre-training (Spokoyny et al., 2022; Lewkowycz et al., 2022; Liu and Low, 2023) or specific prompting techniques (Wei et al., 2022b; Kojima et al., 2022; Yang et al., 2023, inter alia ).”
MethodCoLE: A collaborative legal expert prompting framework for large language models in law
Li, B., Fan, S., Zhu, S., & Wen, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113052 · Cited by 15
Hou, R., Wang, X., Zhang, W., Song, Z., Wang, K., Chen, Y., Liu, J., & Ruan, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3529682 · Cited by 15
“When constructing examples within ICL to tackle a specific task, common approaches involve randomly selecting examples [26], [27], utilizing hand-constructed examples [13].”
Towards Specification-Driven LLM-Based Generation of Embedded Automotive Software
Patil, M. S., Ung, G., & Nyberg, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75434-0_9 · Cited by 20
Multimodal Fake News Detection Based on Chain-of-Thought Prompting Large Language Models
Xu, Y., Ge, J., Lyu, G., Li, G., & Li, H. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831252 · Cited by 14
“Early explorations include OPT [16], GPT-NEOX [17] and Flan-T5 [18], which served as the foundation for the future development of LLMs. Nowadays, LLMs have been used in various NLP tasks, such as language generation [19], [20], knowledge completion [21]–[23] and complex reasoning [24]–[26].”
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding
Süzgün, M., Melas-Kyriazi, L., & Jurafsky, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.262 · Cited by 27
A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them In Zero Shot
Bhattacharyya, A., Singla, Y., Krishnamurthy, B., Shah, R. R., & Chen, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.608 · Cited by 28
Generative AI and digital twin integrated intelligent process planning：A conceptual frameworkPreprint
Xu, Q., Zhou, G., Zhang, C., Chang, F., Cao, Y., & Zhao, D. (2023) · Research Square · doi:10.21203/rs.3.rs-3652246/v1 · Cited by 12
Metadata
- Authors
- Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., & Misra, V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 3843-3857
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2206.14858
- OpenAlex
- W7133211296
- Cited by
- 358
- References
- 0
- Citations by year
- 2026: 55 · 2025: 53 · 2024: 67 · 2023: 173 · 2022: 10
- FWCI
- 287.45
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Education › Mathematics Education and Pedagogy
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- quantitative reasoning, language models, mathematical problem solving, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
