A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
We demonstrate that a neural network pretrained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI’s Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a dataset of questions from Massachusetts Institute of Technology (MIT)’s largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University’s Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pretrained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8 to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work automatically solves university-level mathematics course questions at a human level and explains and generates university-level mathematics course questions at scale, a milestone for higher education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Specifically, GPT-2 and GPT-3 have performed better than BERT-based models in content generation and evaluation tasks, such as generating university math problems (Drori et al., 2022) and evaluating the quality of student-generated short answer questions (Moore et al., 2022).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Drori et al., 2022
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Natural science education: LLM-based agents can communicate with humans fluently, often being utilized to develop agent-based educational tools [115,117-119].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [118]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Educational technologies developed by fine-tuning Codex also demonstrated the capability of resolving 81% of the advanced mathematics problems (Drori et al., 2022).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Drori et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLMs) have demonstrated astounding capabilities in educational assessments (e.g., Drori et al. 2022, Terwiesch 2023).”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Drori et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Socratic guiding questions in synthetic arithmetic data: matched LoRA runs (revision v2)
O'Grady, J., Gürhan, A. I., Chee, F. T., & Ramlan, E. I. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23236017 · Cited by 462
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
A Goal-Driven Tree-Structured Neural Model for Math Word Problems
Xie, Z., & Sun, S. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/736 · Cited by 193
MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving
Liang, Z., Zhang, J., Wang, L., Qin, W., Lan, Y., Shao, J., & Zhang, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.74 · Cited by 153
Graph-to-Tree Learning for Solving Math Word Problems
Zhang, J., Wang, L., Lee, R. K.-W., Bin, Y., Wang, Y., Shao, J., & Lim, E.-P. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.362 · Cited by 125
Li, S., Wu, L., Feng, S., Xu, F., Xu, F., & Zhong, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.255 · Cited by 93
Generate & Rank: A Multi-task Framework for Math Word Problems
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., & Liu, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.195 · Cited by 88
A Knowledge-Aware Sequence-to-Tree Network for Math Word Problem Solving
Wu, Q., Zhang, Q., Fu, J., & Huang, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.579 · Cited by 80
Choi, C. Q. (2021) · IEEE Spectrum · doi:10.1109/mspec.2021.9563958 · Cited by 71
Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems
Qin, J., Lin, L., Liang, X., Zhang, R., & Lin, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.309 · Cited by 59
Solving Machine Learning ProblemsPreprint
Tran, S., Krishna, P., Pakuwal, I., Kafle, P., Singh, N., Lynch, J., & Drori, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01238 · Cited by 5
Self-citationMWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem SolvingPreprint
Liang, Z., Zhang, J., Wang, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13435 · Cited by 3
Research works printed but not linked: 3
- A. Vaswani, “Attention is all you need” in Proceedings of Advances in Neural Information Processing Systems (2017), eds. I Guyon, et al. (Curran Associates, Inc., Long Beach, CA), vol. 30.
- I. Drori et al., Github repository for A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level (2022) https://github.com/idrori/mathQ. PMC9371704 35917350
- I. Drori, A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. arXiv e-prints (2022) https://arxiv.org/abs/2112.15594 (accessed: May 30, 2022). PMC9371704 35917350 · arXiv
Cited by
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
“Natural Science Education: LLM-based agents can communicate with humans fluently, often being utilized to develop agent-based educational tools [9, 145, 34, 19].”
BackgroundMethodA Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
“Natural Science Education: LLM-based agents can communicate with humans fluently, often being utilized to develop agent-based educational tools [9, 145, 34, 19].”
BackgroundMethodPractical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 1,435
“Surveys have been used to evaluate students’ experience with LLMs-based innovations from multiple perspectives, such as the quality and difficulty of AI-generated questions (Drori et al., 2022; Li and Xing, 2021) and potential learning benefits of the systems (Jayaraman and Black, 2022).”
BackgroundMethodInfluentialCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767
“Large language models exhibit impressive zero-shot reasoning capabilities: from planning [14] to writing math programs [43]; from solving science problems [44] to using trained verifiers [45] for math word problems.”
BackgroundCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07753 · Cited by 767
“Large language models exhibit impressive zero-shot reasoning capabilities: from planning [14] to writing math programs [43]; from solving science problems [44] to using trained verifiers [45] for math word problems.”
Background
Metadata
- Authors
- Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G.
- Affiliations
- Massachusetts Institute of Technology (US)
- Harvard University (US)
- Columbia University (US)
- MIT Media Lab (US)
- University of Waterloo (CA)
- Corresponding author
- Drori, Iddo
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2022-08-02
- Volume
- 119
- Issue
- 32
- Pages
- e2123433119-e2123433119
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC9371704
- OpenAlex
- W4289523162
- Semantic Scholar
- 257aee73d83a87921fd2d56b524de394dcf6a264
- Cited by
- 187
- References
- 19
- Citations by year
- 2026: 17 · 2025: 27 · 2024: 35 · 2023: 104 · 2022: 4
- FWCI
- 13.99
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
- Keywords
- program synthesis, few-shot learning, codices, automated problem solving, mathematics education, question generation, MATH benchmark, university mathematics courses, code generation, mathematical reasoning
- MeSH terms
- Humans, Massachusetts, Mathematics, Problem Solving, Universities, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.