Math Education with Large Language Models: Peril or Promise?
Open accessAbstract
The widespread availability of large language models (LLMs) has provoked both fear and excitement in education. On one hand, there is the concern that students will offload their coursework to LLMs, limiting what they themselves learn. On the other hand, there is the hope that LLMs might serve as scalable, personalized tutors. We conducted a set of large, pre-registered experiments to investigate how LLM-based explanations affect learning. Participants first practiced high school-level math problems where we varied the assistance they received. Then all participants were asked to answer new but similar test questions without any assistance. In the first experiment, we found that LLM-based explanations positively impacted learning relative to seeing only correct answers, and that benefits were largest for those who attempted problems on their own first before consulting LLM explanations. In the second experiment, results showed that despite the presence of arithmetic errors and incorrect final answers, learning gains from incorrect LLM-based explanations fell in between those from seeing only the correct answer and seeing correct LLM-based explanations. These results suggest that, when used appropriately, LLM explanations can foster learning gains and that certain types of errors may not be as detrimental as previously thought.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM explanations improve students' learning outcomes.
“Still, LLM explanations improve students' learning outcomes (Kumar et al. 2023, Pardos and Bhandari 2023).”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Kumar et al. 2023
Previous studies show mixed results on the effectiveness of learning, even with the most advanced AI models.
“Despite this recent excitement, previous studies show mixed results on the effectiveness of learning, even with the most advanced AI models2,3.”
From AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting (2025) · cites this paper as [3]
Access to personalized conversational AI tools could support knowledge acquisition and facilitate a growth mindset in learners.
“Early evidence suggests that access to these personalized conversational AI tools could not only support knowledge acquisition [28] but also facilitate a growth mindset in learners [20], making them a valuable asset in the educational toolkit.”
From Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms (2024) · cites this paper as [28]
Several studies have found that LLM-generated explanations improve learning in mathematics and physics.
“Several studies have found that LLM-generated explanations improve learning in mathematics (Pardos and Bhandari 2023, Prihar et al. 2023, Kumar et al. 2023) and physics (Kestin et al. 2024).”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Kumar et al. 2023
Cites
Assessment and Classroom Learning
Black, P., & Wiliam, D. (1998) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969595980050102 · Cited by 8,152 · cited 1× · see citing works
“Feedback plays a vital role in the learning process as it minimizes the gap between actual and desired knowledge [7].”
BackgroundDunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013) · Psychological Science in the Public Interest · doi:10.1177/1529100612453266 · Cited by 3,138 · see citing works
ChatGPT User Experience: Implications for Education
Zhai, X. (2023) · doi:10.2139/ssrn.4312418 · Cited by 935 · see citing works
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1× · see citing works
“[6] have further shown how human-AI team can achieve complementary performance , where it outperforms both the AI and human acting alone in a decision-making task.”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Conjectures and Refutations: The Growth of Scientific Knowledge
Popper, K. (2014) · doi:10.4324/9780203538074 · Cited by 327 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083 · see citing works
Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-97652-6 · Cited by 214 · see citing works
Conceptualizing generative AI as style engines: Application archetypes and implications
Riemer, K., & Peter, S. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102824 · Cited by 63 · see citing works
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714198 · Cited by 51 · see citing works
Self-citationKumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., et al. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662042 · Cited by 53 · see citing works
Self-citationAI Meets the Classroom: When Do Large Language Models Harm Learning?
Lehmann, M., Cornelius, P. B., & Sting, F. J. (2024) · arXiv · doi:10.48550/arxiv.2409.09047 · Cited by 11 · see citing works
- Authors
- Kumar, H., Rothschild, D. M., Goldstein, D. G., & Hofman, J.
- Affiliations
- University of Toronto (CA)
- Microsoft Research New York City (United States) (US)
- Corresponding author
- Hofman, Jake
- Venue
- Lecture notes in computer science
- Publisher
- Elsevier BV
- Published
- 2023
- Pages
- 60-75
- ISSN
- 0302-9743 (ISSN-L)
- OpenAlex
- W4412874171
- Cited by
- 54
- References
- 6
- Citations by year
- 2026: 19 · 2025: 25 · 2024: 8 · 2023: 2
- FWCI
- 42.09
- Type
- conference paper
- Language
- en
- Open access
- green
- doi.org
- Locations
- Lecture notes in computer science (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- large language models, mathematics learning, personalized tutoring, calculation errors, learning gains, self-directed training, educational technology, adult learners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
