CoChat
Sign Up
PreprintCited by 75Intelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in EducationArtificial Intelligence in Healthcare and Education

Learning gain differences between ChatGPT and human tutor generated algebra hints

Open access
Pardos, Zachary A., Bhandari, Shreya·2023·arXiv
DOI 10.48550/arxiv.2302.06871arXiv 2302.06871
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs), such as ChatGPT, are quickly advancing AI to the frontiers of practical consumer use and leading industries to re-evaluate how they allocate resources for content production. Authoring of open educational resources and hint content within adaptive tutoring systems is labor intensive. Should LLMs like ChatGPT produce educational content on par with human-authored content, the implications would be significant for further scaling of computer tutoring system approaches. In this paper, we conduct the first learning gain evaluation of ChatGPT by comparing the efficacy of its hints with hints authored by human tutors with 77 participants across two algebra topic areas, Elementary Algebra and Intermediate Algebra. We find that 70% of hints produced by ChatGPT passed our manual quality checks and that both human and ChatGPT conditions produced positive learning gains. However, gains were only statistically significant for human tutor created hints. Learning gains from human-created hints were substantially and statistically significantly higher than ChatGPT hints in both topic areas, though ChatGPT participants in the Intermediate Algebra experiment were near ceiling and not even with the control at pre-test. We discuss the limitations of our study and suggest several future directions for the field. Problem and hint content used in the experiment is provided for replicability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.06871

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLM explanations improve students' learning outcomes.

    “Still, LLM explanations improve students' learning outcomes (Kumar et al. 2023, Pardos and Bhandari 2023).”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Pardos and Bhandari 2023

    The Abstract reports positive learning gains in the ChatGPT-hint conditions, but says those gains were not statistically significant and were significantly lower than gains with human-authored hints. The Conclusions and Discussion identify participants as Mechanical Turk workers, not students, and note that immediate feedback may have contributed to the gains. The source therefore does not establish the broader claim that LLM explanations improve students’ learning outcomes. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Several studies have found that LLM-generated explanations improve learning in mathematics and physics.

    “Several studies have found that LLM-generated explanations improve learning in mathematics (Pardos and Bhandari 2023, Prihar et al. 2023, Kumar et al. 2023) and physics (Kestin et al. 2024).”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Pardos and Bhandari 2023

    The Abstract reports positive learning gains in the ChatGPT-hint algebra conditions, but says those gains were not statistically significant and were substantially lower than gains with human-authored hints. The study therefore supports observed gains in mathematics, not a firm finding that LLM-generated explanations improve learning. The physics claim is attached to a different citation. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Students prefer human explanations over LLM explanations.

    “Students perceive LLM tutors to be helpful (Liu et al. 2024b), but prefer human explanations over LLM explanations (Pardos and Bhandari 2023, Prihar et al. 2023).”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Pardos and Bhandari 2023

    The Introduction describes an experiment comparing learning gains from human-authored and ChatGPT algebra hints among Mechanical Turk participants. The Conclusions and Discussion report higher learning gains with human-authored hints, but no passage found reports whether participants preferred them. Learning gains do not establish a preference for human explanations. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 26

  1. Vincent Aleven, Bruce M McLaren, Jonathan Sewall, and Kenneth R Koedinger. 2006. The cognitive tutor authoring tools (CTAT): Preliminary evaluation of effi- ciency gains. In International Conference on Intelligent Tutoring Systems. Springer, 61-70.
  2. Tiffany Barnes and John Stamper. 2008. Toward automatic hint generation for logic proof tutoring using historical student data. In Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, Canada, June 23-27, 2008 Proceedings 9. Springer, 373-382.
  3. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). · arXiv
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
  5. Milo Buwalda, Johan Jeuring, and Nico Naus. 2018. Use Expert Knowledge Instead of Data: Generating Hints for Hour of Code Exercises. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S '18). Association for Computing Machinery, New York, NY, USA, Article 32, 4 pages. https://doi.org/10.1145/3231644.3231690 · DOI
  6. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017).
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
  8. Roberto Gozalo-Brizuela and Eduardo C Garrido-Merchan. 2023. ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv preprint arXiv:2301.04655 (2023). · arXiv
  9. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv preprint arXiv:2301.12597 (2023). · arXiv
  10. Stephen MacNeil, Andrew Tran, Juho Leinonen, Paul Denny, Joanne Kim, Arto Hellas, Seth Bernstein, and Sami Sarsa. 2022. Automatically Generating CS Learning Materials with Large Language Models. arXiv preprint arXiv:2212.05113 (2022). · arXiv
  11. Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the gpt-3 large language model. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2. 37-39.
  12. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 1906-1919. https://doi.org/10.18653/v1/ 2020.acl-main.173 · DOI · Paper page
  13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022). · arXiv
  14. Eleanor O'Rourke, Eric Butler, Armando Díaz Tolentino, and Zoran Popović. 2019. Automatic generation of problems and explanations for an intelligent algebra tutor. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20. Springer, 383-395.
  15. Chris Piech, Mehran Sahami, Jonathan Huang, and Leonidas Guibas. 2015. Au- tonomously generating hints by inferring problem solving policies. In Proceedings of the second (2015) acm conference on learning@ scale. 195-204.
  16. Thomas W Price, Yihuan Dong, Rui Zhi, Benjamin Paaßen, Nicholas Lytle, Veron- ica Cateté, and Tiffany Barnes. 2019. A comparison of the quality of data-driven programming hint generation algorithms. International Journal of Artificial Intelligence in Education 29 (2019), 368-395.
  17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  18. Leena Razzaq, Jozsef Patvarczki, Shane F Almeida, Manasi Vartak, Mingyu Feng, Neil T Heffernan, and Kenneth R Koedinger. 2009. The Assistment Builder: Supporting the life cycle of tutoring system content creation. IEEE Transactions on Learning Technologies 2, 2 (2009), 157-166.
  19. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). · arXiv
  20. Kelly Rivers and Kenneth R Koedinger. 2014. Automating hint generation with solution space path construction. In Intelligent Tutoring Systems: 12th International Conference, ITS 2014, Honolulu, HI, USA, June 5-9, 2014. Proceedings 12. Springer, 329-339.
  21. Rohan Roy Choudhury, Hezheng Yin, and Armando Fox. 2016. Scale-driven automatic hint generation for coding style. In Intelligent Tutoring Systems: 13th International Conference, ITS 2016, Zagreb, Croatia, June 7-10, 2016. Proceedings 13. Springer, 122-132.
  22. Jürgen Rudolph, Samson Tan, and Shannon Tan. 2023. ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching 6, 1 (2023).
  23. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567 (2021). · arXiv
  24. John Stamper, Michael Eagle, Tiffany Barnes, and Marvin Croy. 2013. Experi- mental evaluation of automatic hint generation for a logic tutor. International Journal of Artificial Intelligence in Education 22, 1-2 (2013), 3-17.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  26. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). · arXiv

Cited by

Sorted by citations

Related

Authors
Pardos, Z. A., & Bhandari, S.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2302.06871
arXiv
2302.06871
OpenAlex
W4321012208
Cited by
70
References
0
Citations by year
2025: 9 · 2024: 31 · 2023: 30
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, large language models, adaptive tutoring systems, learning gains, elementary algebra, intermediate algebra, open educational resources, AI-generated educational content, content authoring

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review