AI Tutoring Outperforms Active Learning
Open accessPreprint — published as 10.1038/s41598-025-97652-6 · Paper page
Abstract
Advances in generative artificial intelligence (GAI) show great potential for improving education. Yet little is known about how this new technology should be used and how effective it can be. Here we report a randomized, controlled study measuring college students’ learning and their perceptions when content is presented through an AI-powered tutor compared with an active learning class. The AI tutor was developed with the same pedagogical best practices as the lectures. We find that students learn more than twice as much in less time when using an AI tutor, compared with the active learning class. They also feel more engaged and more motivated. These findings offer empirical evidence for the efficacy of a widely accessible AI-powered pedagogy in significantly enhancing learning outcomes, presenting a compelling case for its broad adoption in learning environments. * These authors contributed equally to this work. Additionally, please note that Gregory Kestin is the corresponding author.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous studies restricted LLMs to certain use cases, in particular by limiting substitutive use.
“Previous studies restricted LLMs to certain use cases, in particular by limiting substitutive use (Nie et al. 2024, Kestin et al. 2024).”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Kestin et al. 2024
The Implications section recommends using AI tutoring for introductory material and cautions that it should not replace in-person teaching wholesale. But the Study design and Results sections say the trial used an AI-supported lesson at home in place of an in-class lesson. The paper therefore supports limits on substitution, not a characterization that its study avoided substitutive use. (LLM, conf=0.85)
Experimental — accuracy not yet measuredNie et al. and Kestin et al. implement safeguards against students asking an LLM for solutions to practice problems.
“Nie et al. and Kestin et al. also implement safeguards against students asking an LLM for solutions to practice problems.”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Kestin et al. 2024
In “Designing successful student-AI interactions” (page unavailable), Kestin et al. describe a system prompt intended to promote active engagement and a platform that guides students through problems one part at a time. These are safeguards against bypassing the learning process, but our copy does not say the tutor prevents students from asking for, or receiving, solutions to practice problems. This verdict addresses Kestin et al. only; their paper cannot establish what Nie et al. implemented. (LLM, conf=0.78)
Experimental — accuracy not yet measuredKestin et al. (2024) find that students achieve higher learning outcomes when learning with an LLM-based personal tutor than when attending traditional active learning classes.
“Kestin et al. (2024) conduct a field experiment in university-level physics classes, finding that students achieve higher learning outcomes when learning with an LLM-based personal tutor than when attending traditional active learning classes.”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Kestin et al. 2024
Promising work by Kestin et al (2024) in an introductory college physics class demonstrated the potential of carefully designed genAI-based tutors in enhancing student engagement and learning outcomes while saving time.
“promising work by Kestin et al (2024) in an introductory college physics class demonstrated the potential of carefully designed genAI-based tutors in enhancing student engagement and learning outcomes while saving time.”
From Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education (2025) · cites this paper as Kestin et al (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 24
- Forero, M. G. & Herrera-Suárez, H. J. ChatGPT in the classroom: boon or bane for physics students' academic performance? Preprint at arXiv:2312.02422 (2023). · arXiv
- Kumar, H., Rothschild, D. M., Goldstein, D. G., & Hofman, J. M. Math education with large language models: peril or promise? SSRN https://doi.org/10.2139/ssrn.4641653 (2023). · DOI · Paper page
- Sweller, J. Cognitive load theory. In The Psychology of Learning and Motivation: Cognition in Education (eds Mestre, J. P. & Ross, B. H.) 37-76 (Elsevier Academic Press, 2011).
- Kortemeyer, G. Could an artificial-intelligence agent pass an introductory physics course?. Phys. Rev. Phys. Educ. Res. 19(1), 010132 (2023).
- Henderson, C. & Dancy, M. H. Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Phys. Rev. Spec. Top.-Phys. Educ. Res. 3(2), 020102 (2007).
- Stains, M. et al. Anatomy of STEM teaching in North American universities. Science 359(6383), 1468-1470 (2018).
- Freeman, S. et al. Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci.
- Fraser, J. M. et al. Teaching and physics education research: Bridging the gap. Rep. Prog. Phys. 77, 032401 (2014).
- Thornton, R. K., Kuhl, D., Cummings, K. & Marx, J. Comparing the force and motion conceptual evaluation and the force concept inventory. Phys. Rev. ST Phys. Educ. Res. 5, 010105 (2009).
- Bloom, B. S. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educ. Res. 13(6), 4-16 (1984).
- Yeager, D. S. & Dweck, C. S. What can be learned from growth mindset controversies?. Am. Psychol. 75(9), 1269 (2020).
- Hestenes, D., Wells, M. & Swackhamer, G. Force concept inventory. Phys. Teach. 30(3), 141-158 (1992).
- Fredricks, J. A., Blumenfeld, P. C. & Paris, A. H. School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59-109. https://doi.org/10.3102/00346543074001059 (2004). · DOI · Paper page
- Wood, D., Bruner, J. S., & Ross, G. The role of tutoring in problem-solving. J. Child Psychol. Psychiatry 17(2), 89-100 (1976).
- Shute, V. J. Focus on formative feedback. Rev. Educ. Res. 78(1), 153-189 (2008).
- Meyer, J. G. et al. ChatGPT and large language models in academia: Opportunities and challenges. BioData Min. 16(1), 20-20. https://doi.org/10.1186/s13040-023-00339-9 (2023). · DOI · Paper page
- Nye, M. et al. Show your work: Scratchpads for intermediate computation with language models. arXiv:2112.00114 (2021). · arXiv
- Krupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., & Karolus, J., Unreflected acceptance-investigating the negative consequences of ChatGPT-assisted problem solving in physics education. In HHAI 2024: Hybrid Human AI Systems for the Social Good, Frontiers in Artificial Intelligence and Applications, Vol. 386, 199-212. IOS Press (2024).
- Caballero, M. D. et al. Comparing large lecture mechanics curricula using the Force Concept Inventory: A five thousand student study. Am. J. Phys. 80(7), 638-644 (2012).
- Perkins, K. K., Adams, W. K., Pollock, S. J., Finkelstein, N. D., & Wieman, C. E. Correlating student beliefs with student learning using the Colorado Learning Attitudes about Science Survey. AIP Conf. Proc. 790, 61-64 (2005).
- McCarty, L. S. & Deslauriers, L. Transforming a large university physics course to student-centered learning, without sacrificing content: A case study. In The Routledge International Handbook of Student-Centered Learning and Teaching in Higher Education, 186-200 (Routledge, 2020).
- Miller, K., Callaghan, K., McCarty, L. S. & Deslauriers, L. Increasing the effectiveness of active learning using deliberate practice: A homework transformation. Phys. Rev. Phys. Educ. Res. 17(1), 010129 (2021).
- Freeman, S. et al. Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. 111(23), 8410-8415 (2014).
- Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K. & Kestin, G. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc. Natl. Acad. Sci. 116(39), 19251-19257 (2019).
Cited by
Sorted by citations
Wang, K. D., Wu, Z., Tufts, L., Wieman, C., Salehi, S., & Haber, N. (2025) · 2025 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016406 · Cited by 28 · see citing works
AI Meets the Classroom: When Do Large Language Models Harm Learning?
Lehmann, M., Cornelius, P. B., & Sting, F. J. (2024) · arXiv · doi:10.48550/arxiv.2409.09047 · Cited by 11 · see citing works
AI chatbots in higher education: Comparing expectations to evidence
Thoeni, A., & Fryer, L. K. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101061 · Cited by 4 · see citing works
- Authors
- Kestin*, G., Miller*, K., Klales, A., Milbourne, T., & Ponti, G.
- Affiliations
- Department of Physics, Harvard University, 17 Oxford Street, Cambridge, MA 02138
- School of Engineering and Applied Sciences, 29 Oxford Street, Harvard University, Cambridge, MA 02138
- Harvard University (US)
- Venue
- Scientific Reports
- Published
- 2024-05-14
- Volume
- 15
- Issue
- 1
- Pages
- 17458-17458
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC12179260
- OpenAlex
- W4411505840
- Cited by
- 214
- References
- 0
- Citations by year
- 2026: 162 · 2025: 51 · 2013: 1
- FWCI
- 84.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Scientific Reports (publishedVersion, cc-by-nc-nd)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Harvard University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- AI tutoring, generative AI in education, active learning, university students, learning outcomes, student engagement, student motivation, personalized learning
- MeSH terms
- Adult, Artificial Intelligence, Female, Humans, Learning, Male, Students, Universities, Problem-Based Learning, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
