Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level

Open access

Abstract

We demonstrate that a neural network pretrained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI’s Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a dataset of questions from Massachusetts Institute of Technology (MIT)’s largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University’s Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pretrained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8 to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work automatically solves university-level mathematics course questions at a human level and explains and generates university-level mathematics course questions at scale, a milestone for higher education.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2123433119

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. We apply a neural network, OpenAI Codex, to solve, explain, and generate mathematics problems. We randomly sample the input math problems from MIT and Columbia University courses and the MATH dataset (Left). We use zero-shot and few-shot learning to automatically generate programs that solve 81% of the questions. We then use Codex to explain the generated programs. The generated programs can output diverse forms of answers, like printing a numerical answer or generating a plot (Right): for example, in Calculus, the volume generated by rotating the finite two-dimensional region bounded by two two-dimensional graphs about the plotted axis (Top Right); in Differential Equations, the Lorenz strange attractor (Middle Right); in Linear Algebra, the geometry of the SVD (Middle Right). An example of Codex’s ability to produce line-by-line explanations of synthesized programs is demonstrated for a problem from Introduction to Probability and Statistics (Bottom Right).
Fig. 1. We apply a neural network, OpenAI Codex, to solve, explain, and generate mathematics problems. We randomly sample the input math problems from MIT and Columbia University courses and the MATH dataset (Left). We use zero-shot and few-shot learning to automatically generate programs that solve 81% of the questions. We then use Codex to explain the generated programs. The generated programs can output diverse forms of answers, like printing a numerical answer or generating a plot (Right): for example, in Calculus, the volume generated by rotating the finite two-dimensional region bounded by two two-dimensional graphs about the plotted axis (Top Right); in Differential Equations, the Lorenz strange attractor (Middle Right); in Linear Algebra, the geometry of the SVD (Middle Right). An example of Codex’s ability to produce line-by-line explanations of synthesized programs is demonstrated for a problem from Introduction to Probability and Statistics (Bottom Right).CC BY · PubMed Central
Fig. 2. We select a random sample of questions from each course or topic that do not contain input images or require proofs. A language model pretrained on text (GPT-3 text-davinci-002) automatically solves only 18% (for courses) and 25.5% (for the MATH benchmark topics) of these questions. In contrast, using zero-shot learning with a network pretrained on text and fine-tuned on code (OpenAI Codex code-davinci-002), we synthesize programs that automatically solve 71% (for courses) and 72.2% (for the MATH benchmark topics) of the questions. Using the same network but using few-shot learning, we automatically solve 81% (for courses) and 81.1% (for the MATH benchmark topics) of the questions. We use the nearest embedded zero-shot questions and their synthesized code for few-shot learning. The remaining 19% of the course questions and 18.9% of MATH benchmark topic questions are manually prompted to solve the question.
Fig. 2. We select a random sample of questions from each course or topic that do not contain input images or require proofs. A language model pretrained on text (GPT-3 text-davinci-002) automatically solves only 18% (for courses) and 25.5% (for the MATH benchmark topics) of these questions. In contrast, using zero-shot learning with a network pretrained on text and fine-tuned on code (OpenAI Codex code-davinci-002), we synthesize programs that automatically solve 71% (for courses) and 72.2% (for the MATH benchmark topics) of the questions. Using the same network but using few-shot learning, we automatically solve 81% (for courses) and 81.1% (for the MATH benchmark topics) of the questions. We use the nearest embedded zero-shot questions and their synthesized code for few-shot learning. The remaining 19% of the course questions and 18.9% of MATH benchmark topic questions are manually prompted to solve the question.CC BY · PubMed Central
Fig. 3. Comparison of the automatic solve rates on (A) MIT math courses and a Columbia University course and on (B) MATH benchmark dataset. The latest OpenAI GPT-3 (text-davinci-002), a transformer pretrained on text, achieves on the MIT math courses (A) 18.8% with zero-shot and 30.8% with few-shot and CoT and on the MATH benchmark (B) 25.5% with zero-shot and 42.2% with few-shot and CoT. In contrast, program synthesis using the latest OpenAI Codex (code-davinci-002), a transformer pretrained on text and fine-tuned on code, achieves automatic solve rates on the MIT math courses of (A) 71.1% with zero-shot learning and 81.1% with few-shot learning and on the MATH benchmark (B) 72.2% with zero-shot learning and 81.1% with few-shot learning.
Fig. 3. Comparison of the automatic solve rates on (A) MIT math courses and a Columbia University course and on (B) MATH benchmark dataset. The latest OpenAI GPT-3 (text-davinci-002), a transformer pretrained on text, achieves on the MIT math courses (A) 18.8% with zero-shot and 30.8% with few-shot and CoT and on the MATH benchmark (B) 25.5% with zero-shot and 42.2% with few-shot and CoT. In contrast, program synthesis using the latest OpenAI Codex (code-davinci-002), a transformer pretrained on text and fine-tuned on code, achieves automatic solve rates on the MIT math courses of (A) 71.1% with zero-shot learning and 81.1% with few-shot learning and on the MATH benchmark (B) 72.2% with zero-shot learning and 81.1% with few-shot learning.CC BY · PubMed Central
Fig. 4. Example pipelines automatically solve questions from MIT mathematics courses and explain the solutions. In the 18.01 Single Variable Calculus Zero-Shot example, given a question and the automatically generated prefix “using SymPy,” Codex is prompted and outputs a program. Running the program results in equations that are the correct answer. The program is then fed to Codex again with an automatic prompt, resulting in a generated code explanation. In the 18.02 Multivariable Calculus Few-Shot example, given a question, the prefix “write a program using SymPy” is automatically generated. The question is embedded with the other zero-shot questions in the course. The nearest zero-shot question and its corresponding code are used as a few-shot example. The few-shot example pair and the input question are fed into Codex, which generates a program that solves the question. The question, program, and prompt for explanation are fed into Codex to generate the explanation. In the 18.03 Differential Equations Zero-Shot example, the answer is both a vector and a plot. In the 18.05 Introduction to Probability and Statistics Zero-Shot example, given the question, a probabilistic program is generated by adding “using simulation” to the prompt. In the 18.06 Linear Algebra Zero-Shot example, the output answer is the correct vector.
Fig. 4. Example pipelines automatically solve questions from MIT mathematics courses and explain the solutions. In the 18.01 Single Variable Calculus Zero-Shot example, given a question and the automatically generated prefix “using SymPy,” Codex is prompted and outputs a program. Running the program results in equations that are the correct answer. The program is then fed to Codex again with an automatic prompt, resulting in a generated code explanation. In the 18.02 Multivariable Calculus Few-Shot example, given a question, the prefix “write a program using SymPy” is automatically generated. The question is embedded with the other zero-shot questions in the course. The nearest zero-shot question and its corresponding code are used as a few-shot example. The few-shot example pair and the input question are fed into Codex, which generates a program that solves the question. The question, program, and prompt for explanation are fed into Codex to generate the explanation. In the 18.03 Differential Equations Zero-Shot example, the answer is both a vector and a plot. In the 18.05 Introduction to Probability and Statistics Zero-Shot example, given the question, a probabilistic program is generated by adding “using simulation” to the prompt. In the 18.06 Linear Algebra Zero-Shot example, the output answer is the correct vector.CC BY · PubMed Central
Fig. 5. Imported Python programming libraries by course: NumPy is used by nearly all courses. Matplotlib is used in courses with questions that involve plotting. Sympy is used by most of the courses and SciPy by half of the courses.
Fig. 5. Imported Python programming libraries by course: NumPy is used by nearly all courses. Matplotlib is used in courses with questions that involve plotting. Sympy is used by most of the courses and SciPy by half of the courses.CC BY · PubMed Central
Fig. 6. Student survey example question: For each of 60 questions, students are asked 1) if the question is human written or machine generated, 2) if the question is appropriate or inappropriate for the course, and 3) to rate the difficulty level of each question on a scale between 1 (easiest) and 5 (hardest).
Fig. 6. Student survey example question: For each of 60 questions, students are asked 1) if the question is human written or machine generated, 2) if the question is appropriate or inappropriate for the course, and 3) to rate the difficulty level of each question on a scale between 1 (easiest) and 5 (hardest).CC BY · PubMed Central
Fig. 7. Visualization of embeddings of course questions: We embed the course questions into a 2,048-dimensional space using OpenAI’s text-similarity-babbage-001 embedding engine, which captures semantic similarity between texts. We then use uniform manifold approximation and projection to reduce the dimensionality to two. This shows distinctive clusters based on topics. We see clusters of questions from MIT’s 18.06 Linear Algebra and Columbia’s COMS3251 Computational Linear Algebra at Top Right. At Left, we see a cluster of the questions from MIT’s 18.01, 18.02, and 18.03. At Bottom Right, we see a cluster of the questions from MIT’s 18.05 Introduction to Probability and Statistics and 6.042 Mathematics for Computer Science, covering probability and statistics.
Fig. 7. Visualization of embeddings of course questions: We embed the course questions into a 2,048-dimensional space using OpenAI’s text-similarity-babbage-001 embedding engine, which captures semantic similarity between texts. We then use uniform manifold approximation and projection to reduce the dimensionality to two. This shows distinctive clusters based on topics. We see clusters of questions from MIT’s 18.06 Linear Algebra and Columbia’s COMS3251 Computational Linear Algebra at Top Right. At Left, we see a cluster of the questions from MIT’s 18.01, 18.02, and 18.03. At Bottom Right, we see a cluster of the questions from MIT’s 18.05 Introduction to Probability and Statistics and 6.042 Mathematics for Computer Science, covering probability and statistics.CC BY · PubMed Central
Fig. 8. Student survey results. A compares the level of difficulty of human-written questions and questions generated by our approach for each course based on the student ratings. The plot shows the means of the difficulty ratings between 1 (easiest) and 5 (hardest) and their 95% CIs. B shows the percentage of human-written and machine-generated questions rated as appropriate and not appropriate for the course. C shows the percentage of human-written questions rated as human written or machine generated (Left) and the percentage of machine-generated questions rated as human written or machine generated (Right).
Fig. 8. Student survey results. A compares the level of difficulty of human-written questions and questions generated by our approach for each course based on the student ratings. The plot shows the means of the difficulty ratings between 1 (easiest) and 5 (hardest) and their 95% CIs. B shows the percentage of human-written and machine-generated questions rated as appropriate and not appropriate for the course. C shows the percentage of human-written questions rated as human written or machine generated (Left) and the percentage of machine-generated questions rated as human written or machine generated (Right).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 19 references

Research works printed but not linked: 2

  1. A. Vaswani “Attention is all you need” in Proceedings of Advances in Neural Information Processing Systems (2017) eds. I Guyon et al. (Curran Associates Inc. Long Beach CA) vol. 30.
  2. I. Drori A neural network solves explains and generates university math problems by program synthesis and few-shot learning at human level. arXiv e-prints (2022) https://arxiv.org/abs/2112.15594 (accessed: May 30 2022). · arXiv

Cited by

See all 234

Metadata

Authors
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Harvard University (US)
  • Columbia University (US)
  • MIT Media Lab (US)
  • University of Waterloo (CA)
Corresponding author
Drori, Iddo
Venue
Proceedings of the National Academy of Sciences
Published
2022-08-02
Volume
119
Issue
32
Pages
e2123433119-e2123433119
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4289523162
Cited by
187
References
19
Citations by year
2026: 17 · 2025: 27 · 2024: 35 · 2023: 104 · 2022: 4
FWCI
13.99
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
Keywords
program synthesis, few-shot learning, codices, automated problem solving, mathematics education, question generation, MATH benchmark, university mathematics courses, code generation, mathematical reasoning
MeSH terms
Humans, Massachusetts, Mathematics, Problem Solving, Universities, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. https://app.cochat.ai/papers/10.1073/pnas.2123433119
The paper
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G. (2022). A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2123433119

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.